I cloned your repo on my Linux machine, which is super useful to run. Not sure why you use Beam 2.41 but anyway, I tried this on my Linux machine:
python t.py \ --topic test --group test-group --bootstrap-server localhost:9092 \ --job_endpoint localhost:8099 \ --artifact_endpoint localhost:8098 \ --environment_type=EXTERNAL \ --environment_config=localhost:50000 Note I replaced host.docker.internal with localhost and it runs well. I then tried to use host.docker.internal and it also runs well, Maybe this is related to your Mac setting? On Sun, Mar 17, 2024 at 8:34 AM Lydian Lee <[email protected]> wrote: > > Hi, > > Just FYI, the similar things works on a different image with the one I > built using my company’s image as base image. I’ve only replaced the base > image with ubuntu. But given that the error log is completely not helpful, > it’s really hard for me to continue debugging on the issue though. > > The docker is not required on my base image as I’ve already add extra args > to ReadFromKafka with default environment to be Process. This is proof to > work with my company’s docker image. For the host.internal.docker which is > also supported by docker for mac. The only thing i need to do is to > configure /etc/hosts so that i can submit the job directly from the laptop > and not the flink master. > > > > > On Sun, Mar 17, 2024 at 2:40 AM Jaehyeon Kim <[email protected]> wrote: > >> Hello, >> >> The pipeline runs in host while host.docker.internal would only be >> resolved on the containers that run with the host network mode. I guess the >> pipeline wouldn't be accessible to host.docker.internal and fails to run. >> >> If everything before ReadFromKafka works successfully, a docker container >> will be launched with the host network mode so that >> host.docker.internal:9092 can be resolved inside the container. As far as >> I've checked, however, it fails when I start a flink cluster on docker and >> I had to rely on a local flink cluster. If you'd like to try to use docker, >> you should have docker installed on your custom docker image and >> volume-map /var/run/docker.sock to the flink task manager. Otherwise, it >> won't be able to launch a Docker container for reading kafka messages. >> >> Cheers, >> Jaehyeon >> >> >> On Sun, 17 Mar 2024 at 18:21, Lydian Lee <[email protected]> wrote: >> >>> Hi, >>> >>> I have an issue when setting up a POC of Python SDK with Flink runner >>> to run in docker-compose. The python worker harness was not returning any >>> error but: >>> ``` >>> python-worker-harness-1 | 2024/03/17 07:10:17 Executing: python -m >>> apache_beam.runners.worker.sdk_worker_main >>> python-worker-harness-1 | 2024/03/17 07:10:24 Python exited: <nil> >>> ``` >>> and dead. The error message seems totally unuseful, and I am wondering >>> if there's a way to make the harness script show more debug logging. >>> >>> I started my harness via: >>> ``` >>> /opt/apache/beam/boot --worker_pool >>> ``` >>> and configure my script to use the harness >>> ``` >>> python docker/src/example.py \ >>> --topic test --group test-group --bootstrap-server >>> host.docker.internal:9092 \ >>> --job_endpoint host.docker.internal:8099 \ >>> --artifact_endpoint host.docker.internal:8098 \ >>> --environment_type=EXTERNAL \ >>> --environment_config=host.docker.internal:50000 >>> ``` >>> The full settings is available in: >>> https://github.com/lydian/beam-python-flink-runner-examples >>> Thanks for your help >>> >>>
