Serge Rielau created SPARK-60102:
------------------------------------
Summary: Apply CSV CHAR/VARCHAR assignment after STRING parse
Key: SPARK-60102
URL: https://issues.apache.org/jira/browse/SPARK-60102
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 5.0.0
Reporter: Serge Rielau
Context
SPARK-59274 applies CHAR/VARCHAR assignment inside UnivocityParser via
applyTextParseSemantics on each STRING field. SPARK-59722 (JSON) and
SPARK-60090 (Hive TRANSFORM) instead parse as unbounded STRING, then wrap the
result with CharVarcharUtils.assignAfterParse / stringLengthCheck.
CSV is the next format. It has no map keys and little CHAR-specific recovery,
so it is the check that assignAfterParse is not JSON-shaped.
Do not wait on SPARK-60090 if those helpers are already on master via
SPARK-59722 (replaceCharVarcharWithStringAlways,
replaceCharVarcharWithStringInSchemaAlways, assignAfterParse). Do not combine
this with XML.
Proposed change
For from_csv and the CSV datasource:
* Build UnivocityParser converters from a physical schema in which every
CHAR/VARCHAR is replaced by STRING (replaceCharVarcharWithStringAlways),
including when first-class CHAR/VARCHAR types are enabled.
* After a row is parsed, apply CharVarcharUtils.assignAfterParse (write-side
pad / EXCEED_LIMIT_LENGTH).
* Remove applyTextParseSemantics from the CSV STRING converter.
* Skip CSV filter pushdown when the schema contains CHAR/VARCHAR, matching
JacksonParser after SPARK-59722.
Keep the wrapping reusable (not a JSON-only InternalRow shape). CSV has no MAP
keys, so MapFromArrays / mapKeyDedupPolicy is out of scope.
User-facing change
Only when spark.sql.charVarchar.standardSemantics.enabled is true (still false
by default):
* CHAR padding and VARCHAR/CHAR overflow still apply to from_csv and CSV reads.
* Overflow becomes a parse-mode error for the whole parsed row (PERMISSIVE:
null; FAILFAST: EXCEED_LIMIT_LENGTH), matching JSON after SPARK-59722.
SPARK-59274 currently fails the overflowing field only.
Document in the 4.3-to-4.4 SQL migration guide.
Tests
Extend BasicCharVarcharTestSuite for from_csv + CSV datasource: padding,
overflow under PERMISSIVE/FAILFAST, nested STRUCT/ARRAY, no change with the
flag off. Keep SPARK-59274 CSV document-as-CHAR cases (padding vs delimiter).
Related
* SPARK-59274 (in-parser CHAR/VARCHAR for JSON/CSV/XML)
* SPARK-59722 (JSON post-parse assignment)
* SPARK-60090 (Hive TRANSFORM post-parse assignment)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
