Serge Rielau created SPARK-60521:
------------------------------------
Summary: Revert CHAR/VARCHAR collation JSON to __COLLATIONS for
stable schema metadata
Key: SPARK-60521
URL: https://issues.apache.org/jira/browse/SPARK-60521
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 5.0.0
Environment: Summary
SPARK-59276 moved CHAR/VARCHAR collations out of the existing schema JSON key
{{__COLLATIONS}} and onto a new key, {{{}__CHAR_VARCHAR_COLLATIONS{}}}. STRING
collations stayed on {{{}__COLLATIONS{}}}. The split was meant to keep older
readers that reject {{__COLLATIONS}} on non-STRING types from failing: they
would see uncollated {{char(n)}} / {{varchar(n)}} and treat the new key as
opaque metadata.
That makes persisted {{StructType}} JSON release-dependent. The same
CHAR/VARCHAR column is encoded differently depending on which Spark wrote it,
so catalog and view schemas are not stable across the SPARK-59276 boundary.
What goes wrong
Schema JSON is written once and later parsed by {{DataType.fromJson}} /
{{StructType.fromJson}} on another release.
# Pre-SPARK-59276 JSON, post-SPARK-59276 reader. Older writers stored
CHAR/VARCHAR collations as {{"type": "varchar(n)"}} plus {{{}__COLLATIONS{}}}.
The new reader allows {{__COLLATIONS}} only on {{"string"}} and throws
{{{}INVALID_JSON_DATA_TYPE_FOR_COLLATIONS{}}}. The column does not round-trip.
# Post-SPARK-59276 JSON, pre-SPARK-59276 reader. Newer writers store the
collation under {{{}__CHAR_VARCHAR_COLLATIONS{}}}. Older readers do not apply
that key. Parse succeeds, but the CHAR/VARCHAR collation is dropped (default /
uncollated type). SPARK-59276 tests this as the intended “preceding reader”
behavior; it is still a silent metadata incompatibility.
Both are the same bug: the on-disk collation key is not stable, so
mixed-version read/write of the same schema cannot preserve CHAR/VARCHAR
collations.
STRING-only collations are unaffected. Uncollated CHAR/VARCHAR are unaffected.
Same-generation write and read of the new key also round-trips. The failure is
only cross-release CHAR/VARCHAR collation metadata.
Proposal
Stop using a second metadata key.
* Write CHAR/VARCHAR collations back under {{{}__COLLATIONS{}}}, together with
STRING, as before SPARK-59276.
* Read both {{__COLLATIONS}} and {{__CHAR_VARCHAR_COLLATIONS}} for a
transition window so already-written new-key JSON still parses.
* Reject only truly invalid combinations (both keys on one path, collation
metadata on a non-string-family type, inline {{collate}} plus a collation map).
* Do not dual-write both keys as a long-term format; that still changes the
JSON and leaves two encodings.
The persisted schema for collated CHAR/VARCHAR should be one JSON shape across
releases: {{__COLLATIONS}} on {{char(n)}} / {{varchar(n)}} / {{{}string{}}}.
Reporter: Serge Rielau
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
