srielau commented on PR #59302:
URL: https://github.com/apache/spark/pull/59302#issuecomment-6098982881
Follow-up on the two decision challenges:
**Preserve-only (released in 4.0.0).** The assignment Project under
`preserveCharVarcharTypeInfo` is intended: script I/O is STRING-on-the-wire, so
first-class CHAR/VARCHAR still needs Spark assignment. Documented in the
`spark.sql.preserveCharVarcharTypeInfo` config doc, a 4.3-to-4.4
migration-guide bullet, and the PR description (no longer calling preserve-only
unreleased). No legacy switch in this PR.
**Comparison padding vs alias metadata.** Ran `SELECT * FROM (SELECT
TRANSFORM(c) USING 'cat' AS (c CHAR(4)) FROM VALUES ('ab') t(c)) WHERE c =
'ab'` with both first-class flags. The analyzed plan is `cast(c as string
collate UTF8_BINARY) = 'ab'`, so the padded value does not match and the row is
dropped. Keeping `__CHAR_VARCHAR_TYPE_STRING` on the assignment alias (as
`readSidePadding` does) does not restore literal padding:
`ApplyCharTypePaddingHelper` pads from StringType attributes that carry CHAR
metadata, while first-class TRANSFORM output is a CharType column, so `=`
promotes CHAR to STRING. Added a test that pins the empty filter result and the
padded SELECT. The alias still keeps the raw metadata for consistency with
scans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
