Thanks for your suggestions. Please see below. > Option 3) would be to map in the docker binary and socket to allow > the containerized Flink job server to start "sibling" containers on > the host.
Do you mean packaging Docker inside the Job Server container and mounting /var/run/docker.sock from the host inside the container? That looks like a bit of a hack but for testing it could be fine. > notably, if the runner supports auto-scaling or similar non-trivial > configurations, that would be difficult to manage from the SDK side. You're right, it would be unfortunate if the SDK would have to deal with spinning up SDK harness/backend containers. For non-trivial configurations it would probably require an extended protocol. > Option 4) We are also thinking about adding process based SDKHarness. > This will avoid docker in docker scenario. Actually, I had started implementing a process-based SDK harness but figured it might be impractical because it doubles the execution path for UDF code and potentially doesn't work with custom dependencies. > Process based SDKHarness also has other applications and might be > desirable in some of the production use cases. True. Some users might want something more lightweight.
