Hello all, I am interested in reviving SPARK-42411 and wanted to check in to see if maintainers would be open to it before I write the code.
The problem is that, in the sidecar mode for Istio with STRICT mTLS, traffic to a pod IP that is not backed by a Service gets rejected. So although the driver is covered by a headless Service the executors are not and executor-to-executor block and shuffle fetches fail. Istio seems to have also closed its tracking issue ( https://github.com/istio/istio/issues/37431) without a fix and recommends headless Services. I am wondering if we can solve this issue by having an opt-in headless Service per application that is created by the driver and that selects the app's executor pods. I am happy to take any feedback to understand the feasibility of reviving the issue and implementing a fix. Sincerely, Tariq Hasan
