yalindogusahin opened a new pull request, #794: URL: https://github.com/apache/spark-kubernetes-operator/pull/794
### What changes were proposed in this pull request? This PR adds a second ingress rule to the worker `NetworkPolicy` generated for every `SparkCluster`, admitting any source on the worker's `web` port (8081) — the port that serves the worker web UI and, when `PrometheusServlet` is configured via `spark.metrics.conf`, the `/metrics/prometheus` scrape endpoint. The existing rule (cluster label / driver label allow-list, unrestricted ports) is untouched, so RPC, shuffle, and block-manager ports stay locked down exactly as before. ### Why are the changes needed? The worker `NetworkPolicy` generated in `SparkClusterResourceSpec#buildWorkerNetworkPolicy` only admits ingress from pods carrying the cluster label or, since SPARK-58649, the driver role label. Prometheus scraper pods carry neither, so they're indistinguishable from any other unrelated pod and are locked out entirely — including from the metrics endpoint, which is not sensitive and is meant to be broadly scrapable. Verified against a live cluster running an affected build: with only the pre-fix policy in place, `curl http://<worker-ip>:8081/metrics/prometheus` from an unlabeled pod times out. The worker itself serves real metrics fine locally (`PrometheusServlet` works as configured via `sparkConf`); the `NetworkPolicy` is what blocks the scrape. Masters are unaffected since they have no `NetworkPolicy` applied. ### Does this PR introduce _any_ user-facing change? Yes. The generated worker `NetworkPolicy` now also admits ingress on the `web` port from any source. Since NetworkPolicies are additive this can only widen ingress on that one port, so it cannot deny traffic that was previously allowed. Clusters running on a CNI that does not enforce NetworkPolicy are unaffected either way. ### How was this patch tested? Extended `SparkClusterResourceSpecTest#testWorkerNetworkPolicy` to assert the new ingress rule (no `from` restriction, single port named `web`). ``` ./gradlew :spark-submission-worker:test --tests '*SparkClusterResourceSpecTest*' ./gradlew :spark-submission-worker:spotlessCheck ``` Both pass. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Sonnet 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
