Hi XQ, Sorry to bother you again, but I've tested the same thing again in a linux env, and it is still not working and showing the same error in the python worker harness. (Note that this won't fail immediately, but it is failing after the task is assigned to task manager and the python worker harness is starting to work)
Wondering if you can share what you've changed (maybe a PR) so that I can test again on my linux machine. Thanks so much for your help. There's someone else also pinging me on the same error when testing, and I do want to make this work for everyone. Thanks! On Mon, Mar 18, 2024 at 6:24 PM XQ Hu via user <[email protected]> wrote: > I did not do anything special but ran `docker-compose -f > docker-compose.yaml up` from your repo. > > On Sun, Mar 17, 2024 at 11:38 PM Lydian Lee <[email protected]> > wrote: > >> Hi XQ, >> >> The code is simplified from my previous work and thus it is still using >> the old version. But I've tested with Beam 2.54.0 and the code still works >> (I mean using my company's image.) If this is running well in your linux, >> I guess there could be something related to how I build the docker image. >> Curious if you could share the image you built to docker.io so that I >> can confirm if the problem is related to only the image, thanks. >> >> The goal for this repo is to complete my previous talk: >> https://www.youtube.com/watch?v=XUz90LpGAgc&ab_channel=ApacheBeam >> >> On Sun, Mar 17, 2024 at 8:07 AM XQ Hu via user <[email protected]> >> wrote: >> >>> I cloned your repo on my Linux machine, which is super useful to run. >>> Not sure why you use Beam 2.41 but anyway, I tried this on my Linux machine: >>> >>> python t.py \ >>> --topic test --group test-group --bootstrap-server localhost:9092 \ >>> --job_endpoint localhost:8099 \ >>> --artifact_endpoint localhost:8098 \ >>> --environment_type=EXTERNAL \ >>> --environment_config=localhost:50000 >>> >>> Note I replaced host.docker.internal with localhost and it runs well. >>> >>> I then tried to use host.docker.internal and it also runs well, >>> >>> Maybe this is related to your Mac setting? >>> >>> On Sun, Mar 17, 2024 at 8:34 AM Lydian Lee <[email protected]> >>> wrote: >>> >>>> >>>> Hi, >>>> >>>> Just FYI, the similar things works on a different image with the one I >>>> built using my company’s image as base image. I’ve only replaced the base >>>> image with ubuntu. But given that the error log is completely not helpful, >>>> it’s really hard for me to continue debugging on the issue though. >>>> >>>> The docker is not required on my base image as I’ve already add extra >>>> args to ReadFromKafka with default environment to be Process. This is proof >>>> to work with my company’s docker image. For the host.internal.docker which >>>> is also supported by docker for mac. The only thing i need to do is to >>>> configure /etc/hosts so that i can submit the job directly from the laptop >>>> and not the flink master. >>>> >>>> >>>> >>>> >>>> On Sun, Mar 17, 2024 at 2:40 AM Jaehyeon Kim <[email protected]> wrote: >>>> >>>>> Hello, >>>>> >>>>> The pipeline runs in host while host.docker.internal would only be >>>>> resolved on the containers that run with the host network mode. I guess >>>>> the >>>>> pipeline wouldn't be accessible to host.docker.internal and fails to run. >>>>> >>>>> If everything before ReadFromKafka works successfully, a docker >>>>> container will be launched with the host network mode so that >>>>> host.docker.internal:9092 can be resolved inside the container. As far as >>>>> I've checked, however, it fails when I start a flink cluster on docker and >>>>> I had to rely on a local flink cluster. If you'd like to try to use >>>>> docker, >>>>> you should have docker installed on your custom docker image and >>>>> volume-map /var/run/docker.sock to the flink task manager. Otherwise, it >>>>> won't be able to launch a Docker container for reading kafka messages. >>>>> >>>>> Cheers, >>>>> Jaehyeon >>>>> >>>>> >>>>> On Sun, 17 Mar 2024 at 18:21, Lydian Lee <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I have an issue when setting up a POC of Python SDK with Flink >>>>>> runner to run in docker-compose. The python worker harness was not >>>>>> returning any error but: >>>>>> ``` >>>>>> python-worker-harness-1 | 2024/03/17 07:10:17 Executing: python -m >>>>>> apache_beam.runners.worker.sdk_worker_main >>>>>> python-worker-harness-1 | 2024/03/17 07:10:24 Python exited: <nil> >>>>>> ``` >>>>>> and dead. The error message seems totally unuseful, and I am >>>>>> wondering if there's a way to make the harness script show more debug >>>>>> logging. >>>>>> >>>>>> I started my harness via: >>>>>> ``` >>>>>> /opt/apache/beam/boot --worker_pool >>>>>> ``` >>>>>> and configure my script to use the harness >>>>>> ``` >>>>>> python docker/src/example.py \ >>>>>> --topic test --group test-group --bootstrap-server >>>>>> host.docker.internal:9092 \ >>>>>> --job_endpoint host.docker.internal:8099 \ >>>>>> --artifact_endpoint host.docker.internal:8098 \ >>>>>> --environment_type=EXTERNAL \ >>>>>> --environment_config=host.docker.internal:50000 >>>>>> ``` >>>>>> The full settings is available in: >>>>>> https://github.com/lydian/beam-python-flink-runner-examples >>>>>> Thanks for your help >>>>>> >>>>>>
