dongjoon-hyun opened a new pull request, #971: URL: https://github.com/apache/spark-kubernetes-operator/pull/971
### What changes were proposed in this pull request? This PR aims to change the default value of `spark.redaction.regex` of the operator from `(?i)secret|password|token|access[.]?key` to `(?i)secret|password|token|access[.]?key|credential`, which is the default value of Apache Spark 4.3.0. - `SparkOperatorConf.REDACTION_REGEX` has the new default value, and its description now says that it reuses the default pattern of Apache Spark 4.3.0. - `docs/config_properties.md` is regenerated. - `docs/migration_guide.md` has a new item in `Upgrading from 1.0 to 1.1`. ### Why are the changes needed? Apache Spark 4.3.0 expands the default redaction regex with `credential` via the following commit. - https://github.com/apache/spark/commit/663498caf1e ([CORE][MINOR] Expand default redaction regex) The operator reuses the configuration key and the default pattern of Spark's `spark.redaction.regex`. This PR makes the operator redact the configuration properties containing `credential` in its logs too, independently from the Apache Spark 4.3.0 upgrade. ### Does this PR introduce _any_ user-facing change? Yes. Since 1.1.0, the operator also redacts the configuration properties whose keys or values contain `credential` in its logs. To restore the previous behavior, users can set `spark.redaction.regex` to `(?i)secret|password|token|access[.]?key`. This is documented in the migration guide. ### How was this patch tested? Pass the CIs with the updated `UtilsTest`. - `redactionRegexConfigReusesSparkRedactionConfig` compares the default value with the default pattern of Apache Spark 4.3.0, since the current dependency is still Apache Spark 4.2.0. - `redactSensitiveInfoMasksSensitiveKeysWithSparkDefaultPattern` verifies that a key containing `credential` is redacted. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
