kz930 opened a new pull request, #7568:
URL: https://github.com/apache/texera/pull/7568
### What changes were proposed in this PR?
Schema inference and execution disagreed about what a blank cell is, and the
schema side was the one that lost information. Inference set `nullValue("")` on
its parser, so a blank read as an empty string: `tryParseDouble("")` fails,
`tryParseBoolean("")` fails, and `inferField` lands on `tryParseString()`. One
empty cell was enough to type a whole numeric column as STRING. Execution
builds its parser without `nullValue`, so the same blank read as null there,
which is what `AttributeTypeUtils.parseField` is written to pass through.
The effect reaches well past the scan. Every downstream operator that does
arithmetic on such a column then receives strings and fails on rows whose
values are perfectly good numbers, not on the blank one. Hugging Face Iris
Logistic Regression on a three-row file fails at the first row, where numpy is
handed `array([['2.6', '0.75']], dtype='<U32')`.
Dropping the setting leaves both sides reading a blank as null, and
`tryParseDouble(null)` already answers DOUBLE, so the column keeps the type its
values give it.
One corner changes with it: a column that is blank in every sampled row now
infers as INTEGER rather than STRING. Its values are null either way, so this
renames the empty rather than reinterpreting anything.
### Any related issues, documentation, discussions?
Closes #7550
### How was this PR tested?
`CSVScanSourceOpDescSpec` gains a case that writes a two-column file whose
numeric column is blank on one row and asserts the inferred type is DOUBLE. It
fails on the previous behavior, 16 passed / 1 failed before the change and 17 /
0 after. Since this touches inference every scan goes through, the whole module
was run as well: 2187 passed, 0 failed.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
