Rui Mo created SPARK-59182:
------------------------------
Summary: Unify string trimming semantics for CAST from STRING
across target types
Key: SPARK-59182
URL: https://issues.apache.org/jira/browse/SPARK-59182
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Rui Mo
Spark currently uses different trimming semantics when casting STRING values to
different target types.
For casts to BYTE/SHORT/INT/LONG, BOOLEAN, DATE, TIMESTAMP, and TIMESTAMP_NTZ,
Spark uses UTF8String-based paths such as `UTF8String.trimAll()`. These trim
leading/trailing bytes matching Spark's UTF8String whitespace/control
predicate, including `0x00..0x20` and `0x7F`.
For casts to FLOAT/DOUBLE and DECIMAL, Spark uses Java string parsing paths,
such as `String.trim()` or `Decimal.fromString`. These follow Java
`String.trim()` semantics, trimming characters `<= U+0020`, but not `U+007F`.
As a result, the same input string can be accepted or rejected depending only
on the cast target type. For example, a value with leading/trailing `U+007F`
may be accepted when cast to INT/BOOLEAN/DATE/TIMESTAMP, but rejected when cast
to DOUBLE/DECIMAL.
This JIRA proposes clarifying whether Spark SQL should use one common trimming
semantic for CAST from STRING, or document the existing target-type-specific
behavior as compatibility semantics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
