Raaaaaaaay86 commented on issue #30663:
URL: https://github.com/apache/beam/issues/30663#issuecomment-2561877988
Facing same problem. Does anyone has solution? I'm trying to run a basic
Kafka pipeline on my M1 MacBook.
This is the docker-compose which I'm running as an environment.
```
version: v3
services:
kafka-1:
container_name: my_kafka_cluster_node_1
image: 'bitnami/kafka:3.5.1'
deploy:
resources:
limits:
memory: 2G
networks:
my_network:
ipv4_address: 175.26.0.5
ports:
- 9097:9097
environment:
- KAFKA_KRAFT_CLUSTER_ID=CqhzvKYNSvmn-8NmZWgOHb
- KAFKA_ENABLE_KRAFT=true
- KAFKA_CFG_NODE_ID=1
- KAFKA_CFG_PROCESS_ROLES=broker,controller
- [email protected]:9093
-
KAFKA_CFG_LISTENERS=INTERNAL://175.26.0.5:9092,CONTROLLER://175.26.0.5:9093,EXTERNAL://:9097
-
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
-
KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL://175.26.0.5:9092,EXTERNAL://localhost:9097
- KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
- KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERNAL
flink-jobmanager:
image: flink:1.18
networks:
my_network:
ipv4_address: 175.26.0.23
ports:
- "8081:8081"
command: jobmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: flink-jobmanager
jobmanager.memory.process.size: 5gb
flink-taskmanager:
image: flink:1.18
networks:
my_network:
ipv4_address: 175.26.0.24
depends_on:
- flink-jobmanager
command: taskmanager
scale: 1
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: flink-jobmanager
taskmanager.numberOfTaskSlots: 20
taskmanager.memory.process.size: 5gb
beam-worker-pool:
image: apache/beam_python3.11_sdk:latest
networks:
my_network:
ipv4_address: 175.26.0.25
ports:
- "50000:50000"
command: --worker_pool
environment:
- BEAM_WORKER_POOL_IN_DOCKER_VM=1
networks:
my_network:
driver: bridge
ipam:
driver: default
config:
- subnet: 175.26.0.0/24
```
and I run a beam_flink1.18_job_server seperatly by docker command.
```
docker run -d --network my_network -p 8099:8099 -p 8098:8098 -p 8097:8097 -v
/tmp/f-conf:/flink-conf apache/beam_flink1.18_job_server:latest
--flink-conf-dir=/flink-conf --flink-master=cre-center-flink-jobmanager:8081
```
and this is my Beam pipeline
```
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.io.kafka import ReadFromKafka
if __name__ == "__main__":
options = PipelineOptions([
"--runner=PortableRunner",
"--job_endpoint=localhost:8099",
"--environment_type=EXTERNAL",
"--environment_config=175.26.0.25:50000"
])
with beam.Pipeline(options=options) as pipeline:
(
pipeline
| "Read Data" >> ReadFromKafka(
consumer_config={
"bootstrap.servers": "175.26.0.5:9092",
"auto.offset.reset": "earliest",
},
topics=["member_entered_store"])
)
```
finally, I'll execute the pipeline by running command on MacBook
```
python beam_playground/main.py
```
and I receive same error in beam_flink1.18_job_server container
```
Cannot run program "docker": error=2, No such file or directory
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
