Serge Rielau created SPARK-60103:
------------------------------------
Summary: Apply XML CHAR/VARCHAR assignment after STRING parse
Key: SPARK-60103
URL: https://issues.apache.org/jira/browse/SPARK-60103
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 5.0.0
Reporter: Serge Rielau
Context
SPARK-59274 already put CHAR/VARCHAR map keys in
StaxXmlParser.convertConstrainedMap, plus attribute and rowTag handling and
sibling recovery on EXCEED_LIMIT_LENGTH. That path is the remaining in-parser
CHAR machinery after SPARK-59722 moved JSON to parse-as-STRING then
assignAfterParse.
XML is last on purpose. Do not combine this with the JSON or CSV PRs.
Proposed change
For from_xml and the XML datasource, use the same wrapping as SPARK-59722 /
SPARK-60090:
* Parse against a physical schema with CHAR/VARCHAR replaced by STRING
(replaceCharVarcharWithStringAlways).
* After parse, apply CharVarcharUtils.assignAfterParse so CHAR pad, overflow,
and CHAR/VARCHAR map keys go through stringLengthCheck / MapFromArrays.
* CHAR/VARCHAR-keyed maps then honor spark.sql.mapKeyDedupPolicy the same way
JSON does after SPARK-59722, including exact repeated names.
* Delete convertConstrainedMap and the MapType(CharType|VarcharType, ...)
branch in convertField.
* Remove applyTextParseSemantics from the XML STRING / empty-element converters.
Attributes, rowTag, valueTag, and attributePrefix stay parser concerns. They
must still be present on the parsed STRING row so assignment can pad or reject
them. The goal is to drop CHAR-specific key draining and sibling recovery, not
to rewrite the StAX walker.
DUPLICATED_MAP_KEY from key assignment must still fail the query under
PERMISSIVE (same as JSON/XML today).
h3. User-facing change
Only when spark.sql.charVarchar.standardSemantics.enabled is true:
* CHAR/VARCHAR fields, attributes, and map keys still pad / length-check.
* Overflow is a parse-mode error for the whole parsed value, not a null sibling
with the rest of the row kept. That drops SPARK-59274 sibling recovery (e.g.
mixed rowTag attributes where one overflows).
* Exact repeated XML element names in a CHAR/VARCHAR-keyed map follow
mapKeyDedupPolicy instead of the convertConstrainedMap last-wins rules.
* Ordinary MAP<STRING, ...> duplicate behavior stays parser-historical.
Document in the 4.3-to-4.4 SQL migration guide and on
spark.sql.mapKeyDedupPolicy (today it already mentions XML CHAR keys).
h3. Tests
Port SPARK-59274 XML CHAR coverage in BasicCharVarcharTestSuite to post-parse
expectations: padding, overflow, attributes, rowTag, nested maps, collated
keys, FAILFAST, DUPLICATED_MAP_KEY vs LAST_WIN, STRING map duplicates
unchanged. Expect whole-value nulls where tests currently keep valid siblings.
h3. Related
* SPARK-59274
* SPARK-59722 (do not land XML on that PR)
* SPARK-60090
* the CSV follow-up (same helpers, no map keys)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
