Hey, Our team is trying to use beam with connector Kafka and runner flink to gather information and process data. We adopt python sdk and build in java 11 in python 3.7 sdk image as java runtime for kafka expansion service. so : image: beam python 3.7 docker image + build in java 11 connector: kafka runner: flink container: kubernetes
We encounter an docker not found error when running: python3 -m kafka_test --runner=FlinkRunner --flink_master=flink-job-manager:8081 --flink_submit_uber_jar --environment_type=EXTERNAL --environment_config=localhost:50000 We notice that in https://beam.apache.org/roadmap/portability/ it mentioned the prerequisite also includes Docker. We wonder what is the docker usage here? Is there any suggested way to build docker in k8s container? (something maybe like sysbox for docker in docker?) Or maybe we should not use beam sdk+runner in k8s? Thanks, Yilun
