Hi, Just FYI, the similar things works on a different image with the one I built using my company’s image as base image. I’ve only replaced the base image with ubuntu. But given that the error log is completely not helpful, it’s really hard for me to continue debugging on the issue though.
The docker is not required on my base image as I’ve already add extra args to ReadFromKafka with default environment to be Process. This is proof to work with my company’s docker image. For the host.internal.docker which is also supported by docker for mac. The only thing i need to do is to configure /etc/hosts so that i can submit the job directly from the laptop and not the flink master. On Sun, Mar 17, 2024 at 2:40 AM Jaehyeon Kim <[email protected]> wrote: > Hello, > > The pipeline runs in host while host.docker.internal would only be > resolved on the containers that run with the host network mode. I guess the > pipeline wouldn't be accessible to host.docker.internal and fails to run. > > If everything before ReadFromKafka works successfully, a docker container > will be launched with the host network mode so that > host.docker.internal:9092 can be resolved inside the container. As far as > I've checked, however, it fails when I start a flink cluster on docker and > I had to rely on a local flink cluster. If you'd like to try to use docker, > you should have docker installed on your custom docker image and > volume-map /var/run/docker.sock to the flink task manager. Otherwise, it > won't be able to launch a Docker container for reading kafka messages. > > Cheers, > Jaehyeon > > > On Sun, 17 Mar 2024 at 18:21, Lydian Lee <[email protected]> wrote: > >> Hi, >> >> I have an issue when setting up a POC of Python SDK with Flink runner to >> run in docker-compose. The python worker harness was not returning any >> error but: >> ``` >> python-worker-harness-1 | 2024/03/17 07:10:17 Executing: python -m >> apache_beam.runners.worker.sdk_worker_main >> python-worker-harness-1 | 2024/03/17 07:10:24 Python exited: <nil> >> ``` >> and dead. The error message seems totally unuseful, and I am wondering >> if there's a way to make the harness script show more debug logging. >> >> I started my harness via: >> ``` >> /opt/apache/beam/boot --worker_pool >> ``` >> and configure my script to use the harness >> ``` >> python docker/src/example.py \ >> --topic test --group test-group --bootstrap-server >> host.docker.internal:9092 \ >> --job_endpoint host.docker.internal:8099 \ >> --artifact_endpoint host.docker.internal:8098 \ >> --environment_type=EXTERNAL \ >> --environment_config=host.docker.internal:50000 >> ``` >> The full settings is available in: >> https://github.com/lydian/beam-python-flink-runner-examples >> Thanks for your help >> >>
