srielau opened a new pull request, #58255:
URL: https://github.com/apache/spark/pull/58255
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR:
https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g.,
'[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a
faster review.
7. If you want to add a new configuration, please read the guideline first
for naming configurations in
'common/utils/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the
guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
Follow-up on first-class CHAR/VARCHAR
(`spark.sql.charVarchar.standardSemantics.enabled`) so a few string-family call
sites treat `CharType` / `VarcharType` like `STRING`.
- `V1Writes` applies `Empty2Null` to nullable CHAR/VARCHAR partition columns
(`dataType.isInstanceOf[StringType]`). `CHAR(n>0)` still pads `''` to spaces,
so the empty CHAR case is `CHAR(0)`.
- The V2 text data source accepts CHAR/VARCHAR as a string-family type.
- Hive metastore filter conversion still refuses CHAR/VARCHAR partition keys
(`varcharKeys` in `SupportedAttribute`: Hive's trailing-blank comparison is not
Spark's). Under standard semantics that would scan every partition, so
`prunePartitionsFastFallback` now also runs when a partition schema contains
CHAR/VARCHAR. Pruning is then client-side with Spark's own predicates (CHAR
compared without PAD SPACE; the test literal is `'a '` for `CHAR(5)`).
This is independent of https://github.com/apache/spark/pull/58130.
### Why are the changes needed?
With first-class types, CHAR/VARCHAR stay in the plan instead of being
rewritten to annotated STRING. Equality against `StringType` then skips them:
- empty partition values are not converted to NULL, so they become a
distinct partition directory instead of `__HIVE_DEFAULT_PARTITION__`
- `USING text` rejects a CHAR/VARCHAR schema
- Hive partition filters on CHAR keys fetch every partition
The annotation-skipping idea was tried and dropped: `ApplyCharTypePadding`
uses `__CHAR_VARCHAR_TYPE_STRING` as an idempotence marker, so the annotation
is load-bearing.
### Does this PR introduce _any_ user-facing change?
Yes, only when `spark.sql.charVarchar.standardSemantics.enabled` is true
(still off by default).
- Empty `VARCHAR` / `CHAR(0)` partition values become NULL like STRING.
- `spark.read.schema("value CHAR(n)").text(...)` is accepted.
- Hive CHAR partition filters prune to the matching partitions
(client-side), and `CHAR(5) = 'a'` does not match a stored `'a '` unless the
literal carries the pad or an RTRIM collation is used.
### How was this patch tested?
- `sql/testOnly *CharVarcharTestSuite *V1WriteCommandSuite`: 164 succeeded
- `hive/testOnly *HiveCharVarcharTestSuite`: 58 succeeded
- `hive/testOnly *HivePartitionFilteringSuite*`: 360 succeeded (Hive 2.3-4.1)
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Grok 4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
