[
https://issues.apache.org/jira/browse/ARROW-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16887829#comment-16887829
]
Jordan Samuels commented on ARROW-5974:
---------------------------------------
[~pitrou] there are two reasons I'd like it to work. Generally, because it
seems intuitive that reading a gzip'd file with decompression should have the
same end state as gunzip'ing the file on the command line and then reading it -
and this is not the case in my example. Also specifically, I'm working with
some third party data that is evidently of this form - a vendor has
concatenated gzip'd files together as .gz files, and it would be useful to be
able to read them directly with read_csv directly rather than gunzip them as a
separate step.
> read_csv returns truncated read for some valid gzip files
> ---------------------------------------------------------
>
> Key: ARROW-5974
> URL: https://issues.apache.org/jira/browse/ARROW-5974
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.13.0, 0.14.0
> Reporter: Jordan Samuels
> Priority: Minor
>
> If two gzipped files are concatenated together, the result is a valid gzip
> file. However, it appears that pyarrow.csv.read_csv will only read the
> portion related to the first file.
> If the repro script
> [here|https://gist.github.com/jordansamuels/d69f1c22c58418f5dfa0785b9ecd211e]
> is run, the output is:
> {{$ python repro.py}}
> {{pyarrow.csv only reads one row:}}
> {{ x}}
> {{0 1}}
> {{pandas reads two rows:}}
> {{ x}}
> {{0 1}}
> {{1 2}}
> {{pyarrow version: 0.14.0}}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
