srielau opened a new pull request, #59315: URL: https://github.com/apache/spark/pull/59315
### What changes were proposed in this pull request? When `spark.sql.charVarchar.standardSemantics.enabled` is true, parse `from_csv` and the CSV datasource as unbounded STRING, then apply write-side CHAR/VARCHAR assignment (`CharVarcharUtils.assignAfterParse` / `assignParsedValue`). - Build Univocity converters from a physical schema with `replaceCharVarcharWithStringAlways`. - Restore the STRING converter to `UTF8String.fromString` (drop in-parser `applyTextParseSemantics`). - After a row is parsed, pad CHAR and length-check VARCHAR. Overflow is a parse-mode error for the whole row. - Skip CSV filter pushdown when the schema contains CHAR/VARCHAR, matching JacksonParser after SPARK-59722. - Shared helpers are named for JSON, CSV, and Hive TRANSFORM so SPARK-59722 (#59167) and SPARK-60090 (#59302) can rebase without a second copy. CSV has no map keys; `mapKeyDedupPolicy` is unchanged. JIRA: https://issues.apache.org/jira/browse/SPARK-60102 Related: SPARK-59722 (JSON post-parse assignment), SPARK-60090 (TRANSFORM post-parse assignment), SPARK-59274 (in-parser CHAR/VARCHAR). ### Why are the changes needed? SPARK-59274 applied CHAR/VARCHAR assignment inside UnivocityParser per STRING field, so PERMISSIVE overflow nulls only that field. JSON after SPARK-59722 parses as STRING and assigns afterward, so overflow fails the whole parsed value. CSV should use the same model. ### Does this PR introduce _any_ user-facing change? Yes, only when `spark.sql.charVarchar.standardSemantics.enabled` is true (still false by default), relative to current `master` (unreleased SPARK-59274): - CHAR padding and VARCHAR/CHAR overflow still apply to `from_csv` and CSV reads. - Overflow is a parse-mode error for the whole parsed row (PERMISSIVE: null; FAILFAST: `EXCEED_LIMIT_LENGTH`). SPARK-59274 failed only the overflowing field. ```sql SELECT from_csv('abcdef,keep', 'a CHAR(2), b STRING') -- PERMISSIVE: named_struct('a', NULL, 'b', NULL) -- previously: named_struct('a', NULL, 'b', 'keep') ``` Released Spark versions are unchanged while the flag stays off. ### How was this patch tested? - `BasicCharVarcharTestSuite`: SPARK-60102 padding, sibling overflow under PERMISSIVE/FAILFAST, CSV datasource pad + filter, nested `assignParsedValue`; SPARK-59274 `from_json/csv/xml` CHAR document cases. - `UnivocityParserSuite`: CHAR assignment after parse skips filter pushdown; full suite. - `CsvExpressionsSuite`: SPARK-59917 CHAR document padding. ``` build/sbt 'sql/testOnly org.apache.spark.sql.BasicCharVarcharTestSuite -- -z SPARK-60102' build/sbt 'sql/testOnly org.apache.spark.sql.BasicCharVarcharTestSuite -- -z "from_json/csv/xml honor"' build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.csv.UnivocityParserSuite' build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.expressions.CsvExpressionsSuite -- -z SPARK-59917' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Grok 4.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
