Hi Folks, I am trying to set up the Beam environment to run our Python pipeline which reads data from Kafka. According to some previous thread <https://lists.apache.org/thread/kz47y88t6zr9k4z043mx3wnb9mz5dqpq>, it seems like the Java SDK harness doesn't work with PROCESS environment_type, and thus I can only use either Docker or External. Given that I need to deploy the job to K8s, and there are some security concerns which prevent me from using the DinD approach. It seems like my best solution is to start a sidecar container which starts the Java extension service in Flink Task manager. However, I am not sure what's the command to start the java extension service which supports the approach.
It looks like in the Docker environment, it is trying to call ``` /opt/apache/beam/boot --id=1-2 --provision_endpoint=localhost:33025 ``` But the script requires me to provide `id` and `provision_endpoint`, which I am not sure what I should do if I want to set up an external service. Wondering if someone can help me with this? for context, I am using beam 2.38.0 with flink 1.13, the job is deployed to k8s using lyft/flinkk8soperator <https://github.com/lyft/flinkk8soperator> Thanks!
