LuciferYang opened a new pull request, #13204:
URL: https://github.com/apache/gluten/pull/13204
## What changes are proposed in this pull request?
The native text reader got its field delimiter only from the legacy
`delimiter` option. `sep`, Spark's canonical key, was ignored, so on ClickHouse
`option("sep", "|")` read the file as comma-separated.
The delimiter is now taken the way Spark's `CSVOptions` takes it: `sep` wins
over `delimiter`, the default is a comma, and escapes such as `\t` are decoded
with `CSVExprUtils.toDelimiterStr`. The native reader splits on the first byte
of the delimiter, so a text scan now fails validation and falls back when the
delimiter is not a single ASCII character or cannot be decoded. The `header`
option is also parsed case-insensitively, as Spark does, and a null value no
longer throws.
## How was this patch tested?
`FileSourceScanOptionsSuite` builds the scan transformer and checks the
forwarded options: `sep` maps to the field delimiter and wins over `delimiter`,
an escaped `\t` arrives as a tab, `||`, a non-ASCII and an empty delimiter fail
validation with the delimiter reason, and `header` accepts `TRUE` and null. The
native ClickHouse read is not covered here, since the ClickHouse backend could
not be run locally.
## Was this patch authored or co-authored using generative AI tooling?
Yes. Generated-by: Claude Opus.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
