jayzhan211 opened a new issue, #26754:
URL: https://github.com/apache/datafusion/issues/26754
### Describe the bug
With `--format csv`, `tsv`, `json` or `nd-json`, `datafusion-cli` can leave
whole record batches out of the output when a query returns more than
`--maxrows` rows (40 by default). It gives no warning, and the footer still
reports the full row count. `--maxrows` is documented as the display limit for
the `table` format, and the table output is correct.
Which batches are lost depends on their sizes and the order they arrive in.
A query over several partitions can therefore print different rows from run to
run.
### To Reproduce
```sh
datafusion-cli --format csv -c "SELECT value FROM generate_series(1, 10000)"
```
### Expected behavior
| Command | DataFusion prints | Expected |
|---|---|---|
| the command above | the values 1 to 8192, then `10000 row(s) fetched.` |
the values 1 to 10000 |
| the same with `--format tsv`, `json` or `nd-json` | the values 1 to 8192 |
the values 1 to 10000 |
`--format table` and `--maxrows inf` both print what is expected.
### Additional context
On a bounded result, `StatementExecutor` in `datafusion-cli/src/exec.rs`
collects the batches before printing them. It keeps a batch only if `row_count
< max_rows.saturating_add(curr_num_rows)`, where `row_count` counts every
earlier batch, including the ones it dropped. So once the total passes
`max_rows` by k rows, every later batch of k rows or fewer is dropped. In the
repro, `generate_series` returns a batch of 8192 rows and then one of 1808, and
the second one fails `8192 < 40 + 1808`. The CSV, TSV and JSON writers then
print every collected batch and ignore `maxrows`, so the missing rows show up
in the output. The table writer prints only the first 40 rows, and those are
always collected.
The condition came in with #14766. #20185 later replaced `+` with
`saturating_add` to fix an overflow.
A fix direction: cap the collected rows only for the `table` format, and use
`row_count < max_rows` there, which matches the comment above the check
("include the last batch that exceeds the limit").
### Willingness to contribute
I'm not planning to fix this myself, so anyone is welcome to pick it up.
Feel free to ping me with questions about the cause or when you'd like a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
