sohailshk opened a new issue, #41471:
URL: https://github.com/apache/beam/issues/41471

   ### What happened?
   
   Reading a compressed CSV with `beam.io.ReadFromCsv` (or 
`apache_beam.dataframe.io.read_csv`, and Beam YAML `ReadFromCsv`) returns only 
the first part of the rows and no error. A 200-row file:
   
       import bz2, gzip
       import apache_beam as beam
   
       csv = 'id,name\n' + ''.join(f'{i},row {i}\n' for i in range(200))
       open('in.csv.gz', 'wb').write(gzip.compress(csv.encode()))
       open('in.csv.bz2', 'wb').write(bz2.compress(csv.encode()))
       open('in.csv', 'w').write(csv)
   
       for path in ('in.csv', 'in.csv.gz', 'in.csv.bz2'):
         with beam.Pipeline() as p:
           _ = (p | beam.io.ReadFromCsv(path) | beam.combiners.Count.Globally()
                | beam.Map(print))
   
   Master prints 200, 80 and 53. The cut is at the compressed size of the file: 
the reader's byte range comes from the file size on disk, but the compressed 
file is read through a decompressing handle, whose positions are counted in 
decompressed bytes, so the reader stops once it has read as many decompressed 
bytes as the file has compressed bytes. `splittable=False` has the same result. 
This has been the case since the read became splittable in 2.27.0 (#13443); 
`beam.io.ReadFromCsv` has had it since it was added in 2.47.0.
   
   ### Issue Priority
   
   Priority: 1 (data loss / total loss of function)
   
   ### Issue Components
   
   - [x] Component: Python SDK
   - [x] Component: IO connector
   - [x] Component: Beam YAML
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to