kz930 opened a new issue, #8674:
URL: https://github.com/apache/texera/issues/8674
### What happened?
The JSONL File Scan reads a JSON `null` as the text `"null"`. One null in a
column is enough to make the whole column STRING, so a column of numbers comes
out as text and its missing value is the word `null` rather than an empty cell.
A sort, a filter or any arithmetic downstream then works on the wrong type.
The cause is `JSONUtils.JSONToMap`: a null is a value node, so it takes the
`child.asText()` branch, and `NullNode.asText()` returns `"null"`. Schema
inference then fails to parse `"null"` as a number and widens the column to
STRING.
Two assertions in `JSONUtilsSpec` expect `"null"` today. They were added in
#4716, whose description says the spec pins the current behavior so that a fix
has to update it on purpose.
### How to reproduce?
1. Upload a JSONL file with these two lines:
```
{"a":1}
{"a":null}
```
2. Add a JSONL File Scan that reads it and run the workflow.
3. Column `a` is inferred as STRING, and the two rows hold the text `"1"`
and the text `"null"`. Expected: INTEGER, with `1` and an empty value.
### Version/Branch
1.4.0-incubating-SNAPSHOT (main)
### Commit Hash (Optional)
53052c482
### What browsers are you seeing the problem on?
_No response_
### Relevant log output
```shell
Schema[Attribute[name=a, type=string]]
row 1: '1' (String)
row 2: 'null' (String)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
