aviyehuda opened a new pull request, #56787:
URL: https://github.com/apache/spark/pull/56787
### What changes were proposed in this pull request?
Add `retryWithTrim` in `UnivocityParser` for integral, boolean, and decimal
value converters. When the first parse attempt fails with
`NumberFormatException` or `IllegalArgumentException`, the parser retries after
trimming leading/trailing whitespace.
This aligns behavior with float/double parsing (which already accepts
surrounding whitespace via Java's parsers) for both `from_csv` and
`spark.read.csv`, since they share `UnivocityParser`.
Reproduction example:
```sql
-- before: b is null
SELECT from_csv('1, 1', 'a INT, b INT');
-- {"a":1,"b":null}
-- after: b is parsed correctly
SELECT from_csv('1, 1', 'a INT, b INT');
-- {"a":1,"b":1}
### Why are the changes needed?
CSV fields often include a space after the delimiter (e.g. "1, 1"). With
default ignoreLeadingWhiteSpace=false, integral columns fail to parse while
float/double columns succeed, producing inconsistent and surprising results:
SELECT from_csv('1, 1', 'a INT, b INT'); -- {"a":1,"b":null}
SELECT from_csv('1, 1', 'a INT, b DOUBLE'); -- {"a":1,"b":1.0}
This is because Scala's toInt/toLong/etc. reject leading/trailing
whitespace, while toDouble/toFloat delegate to Java parsers that accept it.
Related JIRA: https://issues.apache.org/jira/browse/SPARK-50110
### Does this PR introduce any user-facing change?
Yes. Numeric CSV values with leading or trailing whitespace are now parsed
correctly for integral, boolean, and decimal types when reading CSV files and
using from_csv, without requiring ignoreLeadingWhiteSpace=true.
Example:
SELECT from_csv('1, 1', 'a INT, b INT');
Previously returned {"a":1,"b":null}; now returns {"a":1,"b":1}.
### How was this patch tested?
Added unit tests:
CsvFunctionsSuite: SPARK-50110: parse numeric CSV values with surrounding
whitespace (covers from_csv for integral types, decimal, and boolean)
CSVSuite: SPARK-50110: read CSV numeric values with surrounding whitespace
(covers spark.read.csv)
Updated existing corrupt-input tests (SPARK-25387, SPARK-31261, SPARK-32968)
so malformed control-character inputs remain invalid after the trim-retry path.
Locally:
build/sbt 'sql/testOnly *CsvFunctionsSuite -- -z "SPARK-50110"'
build/sbt 'sql/testOnly *CSVv1Suite -- -z "SPARK-50110"'
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor
---
To open the PR from the CLI after pushing to your fork:
```bash
gh pr create --repo apache/spark \
--head aviyehuda:SPARK-50110 \
--base master \
--title "[SPARK-50110][SQL] Fix CSV parsing when numeric values have
surrounding whitespace" \
--body "$(cat <<'EOF'
<paste the body above>
EOF
)"
Adjust the Generated-by line if you prefer to answer No after your own
review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
