eugenegujing opened a new issue, #6279:
URL: https://github.com/apache/texera/issues/6279
### What happened?
The CSV and JSONL scan source operators infer each column's type from only
the first `INFER_READ_LIMIT` (= 100) rows. When a **later** row holds a value
that does not parse as the inferred type, the operator **silently discards the
entire row** — no warning, no skipped-row count, no console message, no log
entry. The user has no way to learn that data was lost, so every downstream
count, aggregate, join, and model is computed on a silently truncated dataset.
Cells that are simply **empty** are kept as `null` (the row survives), while
cells that **fail to parse** cause the **whole row** to vanish. The operator
treats "missing" and "malformed" differently, and the more destructive path is
the silent one. Rejecting malformed input is defensible but doing it
**silently** would be a bug
**What I expected to happen:** the loss should not be silent. At minimum the
number of skipped rows should be surfaced (execution statistic / console
warning), or the scan should fail loudly and name the offending row and column.
Losing data with zero signal is the defect.
### How to reproduce?
Using the `diabetes` dataset (400 data rows). Type inference only samples
the first 100 rows, so inject a bad value **after** row 100 into a column that
is all-integer in the first 100 rows (e.g. `age`), which makes inference stably
INTEGER:
1. Create a corrupted copy that puts a non-integer into `age` on three rows
past row 100:
```bash
# data rows 150, 250, 350 -> age becomes "55.5" (fails Integer parsing)
awk -F, 'BEGIN{OFS=","} NR==1{print; next}
{n++; if(n==150||n==250||n==350){$8="55.5"} print}' diabetes.csv >
diabetes_b2.csv
```
2. Upload `diabetes_b2.csv` as a dataset and point a **CSV File Scan**
operator at it.
3. Run the workflow and look at the scan operator's output row count.
**Expected:** 403 rows accounted for, or a visible signal that 3 rows were
skipped.
**Actual:** the scan emits **400** rows. The 3 rows containing `55.5` are
gone, with no warning, no skipped-row count, and nothing in the console —
silent data loss. (JSONL, ParallelCSV, and the `csvOld` variant share the same
behavior.)
<img width="1344" height="836" alt="Image"
src="https://github.com/user-attachments/assets/e8fb0c15-130a-4e2f-8069-b4c7fd2d05b0";
/>
### Version/Branch
1.3.0-incubating-SNAPSHOT (main)
### Commit Hash (Optional)
_No response_
### What browsers are you seeing the problem on?
_No response_
### Relevant log output
```shell
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
