lukekim opened a new issue, #27592:
URL: https://github.com/apache/datafusion/issues/27592
### Describe the bug
A CSV or JSON file compressed with bzip2, xz or zstd that is made of several
concatenated members (streams or frames), such as `cat a.csv.zst b.csv.zst >
ab.csv.zst` produces, returns only the first member's rows when it is read from
any object store other than the local file system (HTTP, S3, GCS, Azure, the
in-memory store). There is no error. The same file read from the local file
system returns every row.
When the first member ends partway through a row, the last row returned is
cut off. A JSON array file fails with `Json error: Truncated record whilst
reading string`.
### To Reproduce
With `datafusion-cli` 55.3.0:
```shell
{ echo a; seq 0 9999; } > a.csv
seq 10000 19999 > b.csv
zstd -q a.csv b.csv
cat a.csv.zst b.csv.zst > ab.csv.zst
zstd -dc ab.csv.zst | wc -l # 20001: the header and 20,000 rows
python3 -m http.server 8000 --bind 127.0.0.1 &
```
```sql
CREATE EXTERNAL TABLE local_file STORED AS CSV LOCATION 'ab.csv.zst'
OPTIONS ('format.compression' 'zstd', 'format.has_header' 'true');
CREATE EXTERNAL TABLE over_http STORED AS CSV LOCATION
'http://127.0.0.1:8000/ab.csv.zst'
OPTIONS ('format.compression' 'zstd', 'format.has_header' 'true');
SELECT count(*), sum(a) FROM local_file;
SELECT count(*), sum(a) FROM over_http;
```
```
+----------+-------------------+
| count(*) | sum(local_file.a) |
+----------+-------------------+
| 20000 | 199990000 |
+----------+-------------------+
+----------+------------------+
| count(*) | sum(over_http.a) |
+----------+------------------+
| 10000 | 49995000 |
+----------+------------------+
```
`bzip2` and `xz` (with `.bz2` and `.xz` files and `'format.compression'` set
to match) return the same two results.
### Expected behavior
Both tables return `20000` and `199990000`: every member of the file, as
`zstd -dc`, `bzip2 -dc` and `xz -dc` decode it, and as DataFusion already does
for a local file.
### Additional context
The file openers decode a `GetResultPayload::File` (local file system) with
`FileCompressionType::convert_read` and a `GetResultPayload::Stream` (every
other object store) with `FileCompressionType::convert_stream`. `convert_read`
decodes every member for all four codecs (#5628). `convert_stream` calls
`multiple_members(true)` only on the gzip decoder (#9175, for block gzip), so
its bzip2, xz and zstd decoders stop after the first member.
I plan to contribute a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]