alexandrefimov opened a new pull request, #58104: URL: https://github.com/apache/spark/pull/58104
### What changes were proposed in this pull request? Backport of #57892 ([SPARK-58658]) to branch-3.5, requested in https://github.com/apache/spark/pull/57892#issuecomment-5336856708. The cherry-pick does not apply: on 3.5 the handler lives under `connector/connect/server/...` and the file has diverged - `handle` resolves the session inline through `SparkConnectService.getOrCreateIsolatedSession` rather than `SessionHolder.withSession`. The change itself carries over unchanged: - the four read paths - `Get`, `GetWithDefault`, `GetOption`, `GetAll` - withhold entries matching `spark.redaction.regex`, each reporting a withheld entry the way that same operation already reports an unset key; - the pattern is read from the `SparkConf`, not from the session config, which the client can write; - matching covers the key or the value, following `Utils.redact`; - `GetAll` filters before the requested prefix is stripped; - `IsModifiable` is unchanged. One hunk of the original has no counterpart here. `SparkConnectAuthSuite` asserts that `spark.connect.authenticate.token` is not readable, and 3.5 has no Connect authentication at all, so this backport carries the handler and the new `SparkConnectConfigHandlerSuite` only. The suite is adapted to 3.5 APIs: `SessionHolder.forTesting` instead of `SparkConnectTestUtils.createDummySessionHolder`, `SparkConnectService.invalidateAllSessions()`, `getErrorClass` instead of `getCondition`, and `scala.collection.JavaConverters`. ### Why are the changes needed? The Config RPC discloses anything sensitive that reached the `SparkConf`, whatever put it there, because `SQLConf.mergeSparkConf` copies every entry into the session config and no read path applies a denylist. The full rationale, including the Kyuubi deployment where a client reads back a ZooKeeper digest and an internal pre-shared secret it never held, is in #57892. ### Does this PR introduce _any_ user-facing change? Yes, the same change as #57892: a configuration entry whose key or value matches `spark.redaction.regex` is no longer returned by the Config RPC, and each read operation reports it as if it were unset. ### How was this patch tested? ``` build/sbt "connect/testOnly org.apache.spark.sql.connect.service.SparkConnectConfigHandlerSuite" ``` 8 tests, all passing. With the handler reverted to the branch-3.5 version, 7 of the 8 fail; the one that still passes is the `IsModifiable` case, whose behaviour is deliberately unchanged. ### Was this patch authored or co-authored using generative AI tooling? Yes - Claude Code (Opus 5) was used for code reading, drafting and review. Every line was reviewed by the author, who takes responsibility for the patch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
