szehon-ho opened a new pull request, #59173: URL: https://github.com/apache/spark/pull/59173
### What changes were proposed in this pull request? Backport #57994 (commit 7e8bf895d1cf21676b021a5b225e2e3c1fc3ea70) to `branch-4.2` for SPARK-51830. The commit cherry-picks cleanly, with the same production changes and regression tests. The write command passes `spark.sql.sources.validatePartitionColumns` when constructing partition paths. When validation is disabled, nonnumeric legacy partition values are preserved instead of throwing `NumberFormatException`. ### Why are the changes needed? An existing nonnumeric partition value in a table with a numeric partition schema can prevent valid inserts even when partition validation is explicitly disabled. This restores the configured behavior for both dynamic inserts and static partition metadata refreshes on Spark 4.2. ### Does this PR introduce _any_ user-facing change? Yes. Affected inserts succeed with `spark.sql.sources.validatePartitionColumns=false`. Default strict validation and normalization of valid numeric partition values remain unchanged. ### How was this patch tested? Ran the four SPARK-51830 regression tests locally on the backport: all passed across `PartitioningUtilsSuite`, `ParquetV2PartitionDiscoverySuite`, and `InsertSuite`. ```sh SPARK_LOCAL_IP=127.0.0.1 build/sbt -Dsbt.log.noformat=true 'sql/testOnly org.apache.spark.sql.execution.datasources.PartitioningUtilsSuite org.apache.spark.sql.execution.datasources.parquet.ParquetV2PartitionDiscoverySuite org.apache.spark.sql.sources.InsertSuite -- -z SPARK-51830' ``` The backport's added and removed lines match the original commit in all five files. `git diff --check` and checks of added lines for ASCII and line length pass. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) Codex prepared this backport and PR description. The original patch retains its author attribution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
