lukekim opened a new issue, #27592:
URL: https://github.com/apache/datafusion/issues/27592

   ### Describe the bug
   
   A CSV or JSON file compressed with bzip2, xz or zstd that is made of several 
concatenated members (streams or frames), such as `cat a.csv.zst b.csv.zst > 
ab.csv.zst` produces, returns only the first member's rows when it is read from 
any object store other than the local file system (HTTP, S3, GCS, Azure, the 
in-memory store). There is no error. The same file read from the local file 
system returns every row.
   
   When the first member ends partway through a row, the last row returned is 
cut off. A JSON array file fails with `Json error: Truncated record whilst 
reading string`.
   
   ### To Reproduce
   
   With `datafusion-cli` 55.3.0:
   
   ```shell
   { echo a; seq 0 9999; } > a.csv
   seq 10000 19999 > b.csv
   zstd -q a.csv b.csv
   cat a.csv.zst b.csv.zst > ab.csv.zst
   zstd -dc ab.csv.zst | wc -l     # 20001: the header and 20,000 rows
   python3 -m http.server 8000 --bind 127.0.0.1 &
   ```
   
   ```sql
   CREATE EXTERNAL TABLE local_file STORED AS CSV LOCATION 'ab.csv.zst'
   OPTIONS ('format.compression' 'zstd', 'format.has_header' 'true');
   CREATE EXTERNAL TABLE over_http STORED AS CSV LOCATION 
'http://127.0.0.1:8000/ab.csv.zst'
   OPTIONS ('format.compression' 'zstd', 'format.has_header' 'true');
   
   SELECT count(*), sum(a) FROM local_file;
   SELECT count(*), sum(a) FROM over_http;
   ```
   
   ```
   +----------+-------------------+
   | count(*) | sum(local_file.a) |
   +----------+-------------------+
   | 20000    | 199990000         |
   +----------+-------------------+
   
   +----------+------------------+
   | count(*) | sum(over_http.a) |
   +----------+------------------+
   | 10000    | 49995000         |
   +----------+------------------+
   ```
   
   `bzip2` and `xz` (with `.bz2` and `.xz` files and `'format.compression'` set 
to match) return the same two results.
   
   ### Expected behavior
   
   Both tables return `20000` and `199990000`: every member of the file, as 
`zstd -dc`, `bzip2 -dc` and `xz -dc` decode it, and as DataFusion already does 
for a local file.
   
   ### Additional context
   
   The file openers decode a `GetResultPayload::File` (local file system) with 
`FileCompressionType::convert_read` and a `GetResultPayload::Stream` (every 
other object store) with `FileCompressionType::convert_stream`. `convert_read` 
decodes every member for all four codecs (#5628). `convert_stream` calls 
`multiple_members(true)` only on the gzip decoder (#9175, for block gzip), so 
its bzip2, xz and zstd decoders stop after the first member.
   
   I plan to contribute a fix.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to