srielau opened a new pull request, #59316: URL: https://github.com/apache/spark/pull/59316
### What changes were proposed in this pull request? When `spark.sql.charVarchar.standardSemantics.enabled` is true, `from_xml` and the XML datasource parse CHAR/VARCHAR fields as unbounded STRING and then apply write-side assignment (`CHAR` pad, `EXCEED_LIMIT_LENGTH` on overflow). That matches JSON (`SPARK-59722`) and CSV (`SPARK-60090`). This PR: - Parses XML against a STRING physical schema (`replaceCharVarcharWithStringAlways`) and runs `CharVarcharUtils.assignAfterParse` on the InternalRow. - Treats assignment overflow as a whole-value parse-mode error (PERMISSIVE returns null; FAILFAST raises). - Drops the SPARK-59274 dual-schema walk, `ElementBoundedEventReader`, `assignAttributes`, `convertNestedStruct`, and `DuplicateMapKeyUtils.buildConstrainedMap`. - Leaves exact repeated XML element names last-win, matching ordinary `MAP<STRING, ...>`. Distinct names that pad or trim to the same CHAR/VARCHAR key still honor `spark.sql.mapKeyDedupPolicy`. `DUPLICATED_MAP_KEY` still fails the query under PERMISSIVE. JIRA: https://issues.apache.org/jira/browse/SPARK-60103 ### Why are the changes needed? SPARK-59274 applied CHAR/VARCHAR constraints during the XML walk so a failed map or nested field could leave siblings intact. JSON and CSV now parse as STRING and assign after parse. XML should use the same wrapping: overflow is a parse error for the whole value, and CHAR/VARCHAR map-key collisions are an assignment-time `DUPLICATED_MAP_KEY` instead of a parser-local map builder. ### Does this PR introduce _any_ user-facing change? Yes, for unreleased Spark 4.4 / master when `spark.sql.charVarchar.standardSemantics.enabled` is true: - CHAR/VARCHAR overflow in `from_xml` or the XML datasource is a whole-value error. PERMISSIVE no longer keeps sibling fields such as `tail` next to a failed map or nested value. - Distinct XML element names that become the same key after CHAR padding or VARCHAR trailing-space trimming honor `spark.sql.mapKeyDedupPolicy`. Exact repeated names stay last-win, like `MAP<STRING, ...>`. - `DUPLICATED_MAP_KEY` from that assignment still fails the query under PERMISSIVE. Documented in `docs/sql-migration-guide.md` (Spark SQL 4.3 to 4.4). ### How was this patch tested? Added and updated tests in `BasicCharVarcharTestSuite`: - pad/trim map-key collisions vs exact repeats - overflow as whole-value under PERMISSIVE and FAILFAST - mixed attributes, file reads, and XML `rowTag` Ran: ``` JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 build/sbt -Dsbt.override.build.repos=true \ 'sql/testOnly org.apache.spark.sql.BasicCharVarcharTestSuite -- -z SPARK-60103 -z "SPARK-59274: from_json/csv/xml" -z "SPARK-59274: ordinary STRING" -z "SPARK-59274: collated" -z "SPARK-59274: XML rowTag"' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Grok 4.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
