[
https://issues.apache.org/jira/browse/SPARK-60090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60090:
-----------------------------------
Labels: pull-request-available (was: )
> TRANSFORM clause and CHAR: V2: Treat like any other STRING source
> -----------------------------------------------------------------
>
> Key: SPARK-60090
> URL: https://issues.apache.org/jira/browse/SPARK-60090
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 4.4.0
> Reporter: Serge Rielau
> Priority: Major
> Labels: pull-request-available
>
> Parent / related: SPARK-59683 / PR
> [#58936|https://github.com/apache/spark/pull/58936]
> h3. What
> Keep Hive {{TRANSFORM}} / {{MAP}} / {{REDUCE}} script I/O as unbounded
> {{STRING}} (and JSON for complex types), the same as sources that have no
> native CHAR/VARCHAR. After the script reader, apply Spark CHAR/VARCHAR
> assignment ({{{}CharVarcharUtils.stringLengthCheck{}}}: pad {{{}CHAR{}}},
> {{EXCEED_LIMIT_LENGTH}} on overflow) in a {{Project}} so the query result
> type is the declared {{{}CHAR{}}}/{{{}VARCHAR{}}} (including nested, e.g.
> {{{}MAP<CHAR(5), CHAR(10)>{}}}).
> Do not teach the no-SerDe writers, Hive unwrappers, or SerDe schema rewrite
> to understand CHAR/VARCHAR.
> h3. Why
> Default Spark (and pre first-class CHAR) already rewrote {{AS (c CHAR(5))}}
> to {{{}STRING{}}}. No pad, no length check, no {{{}Cast{}}}. Hive TRANSFORM
> is STRING-on-the-wire. First-class CHAR made {{CharType}} survive parse, so
> SPARK-59683 built a deep codec inside the script/Hive path.
> That mixed two problems:
> # CHAR/VARCHAR assignment (pad / length) -- same as CSV/JSON/Parquet.
> # TRANSFORM JSON / Hive wire issues that are not CHAR-specific (non-string
> map keys, duplicate JSON keys, SerDe-dependent truncate vs fail). SPARK-59683
> only handled some of those on the CHAR path, which is why {{MAP<STRING,
> STRING>}} vs {{MAP<STRING, CHAR(2)>}} diverged and DATE map keys can
> round-trip wrong.
> TRANSFORM is Hive-compat, uncommon, documented in OSS Spark, not in
> Databricks SQL {{{}SELECT{}}}. The CHAR epic already marked Hive TRANSFORM
> out of scope.
> h3. Proposed plan
> # Script exec output: recursively replace CHAR/VARCHAR with unbounded STRING
> (historical {{replaceCharVarcharWithString}} behavior), including nested
> maps/arrays/structs.
> # Parse delimited / JSON as today for STRING and complex STRING-shaped types.
> # Wrap with {{Project}} using {{stringLengthCheck}} (and CHAR pad) onto the
> declared first-class types when
> {{spark.sql.charVarchar.standardSemantics.enabled}} is true.
> # Flag off: keep today's STRING result ({{{}AS (c CHAR(5)){}}} remains
> {{STRING}} in the schema). No behavior change.
> # Remove or do not land CHAR-specific logic in
> {{BaseScriptTransformationExec}} converters, {{outputTypesForSerDe}}
> LazySimple rewrite, and Hive {{unwrapperFor}} CHAR checks.
> # JSON non-string map keys ({{{}MAP<INT, ...>{}}}, {{{}MAP<DATE, ...>{}}}):
> out of scope. Separate ticket: reject unsafe key types, or a reversible codec
> for all map key types. CHAR map keys are strings on the JSON wire and are
> handled by assignment.
> h3. Tests / acceptance
> * Scalar {{AS (c CHAR(n))}} / {{{}VARCHAR(n){}}}: pad, overflow, null token;
> result type is CHAR/VARCHAR with the flag on; STRING with the flag off.
> * Nested: {{{}ARRAY<CHAR(n)>{}}}, {{{}STRUCT{}}}, {{MAP<CHAR(n),
> VARCHAR(m)>}} via the {{{}Project{}}}, not a custom restorer.
> * Hive SerDe and no-SerDe both go through the same Spark-side assignment
> (script still STRING-shaped).
> * Preserve-only ({{{}preserveCharVarcharTypeInfo{}}} without standard
> semantics): no pad / no {{{}EXCEED_LIMIT_LENGTH{}}}.
> * No CHAR-only change to map-key dedup vs {{{}MAP<STRING, STRING>{}}}.
> h3. Note for implementer
> SPARK-59683 / OSS PR 58936 should be superseded or stripped back to this
> model rather than patched further (bind-at-parse flag, Hive unwrapper gating,
> JSON restorer).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
