kimotorc opened a new issue, #35894:
URL: https://github.com/apache/arrow/issues/35894

   ### Describe the bug, including details regarding any error messages, 
version, and platform.
   
   It appears that parquet files are opened twice when using 
parquet.read_table.  The first time seems to be to read in the schema, the 
second to do the actual data read.  
   
   This seems to be the case when use_legacy_dataset=False and without 
specifying a schema argument.  It also is the case for datasets and single 
parquet reads.
   
   My use case is that i'd like to use a custom fsspec filesystem that 
interfaces to an s3 like API. It handles the remote download of the parquet 
file and passes to pyarrow a handle of a temporary file downloaded locally.  
Without cached access, this behavior results in files being downloaded twice.  
   
   
[Here](https://github.com/apache/arrow/blob/5a55fb4f66cbd316568085129054b5ff1d2eb089/python/pyarrow/parquet/core.py#L2479)
 is where the file is opened to read in table schema.
   
   Here is code to reproduce:
   ```
   import pandas as pd
   import pyarrow as pa
   import pyarrow.parquet as pq
   from fsspec.implementations.local import LocalFileSystem
   import os
   
   # create a fsspec filesystem to track the number of times files are opened
   class LocalFSWithAccessCounter(LocalFileSystem):
       def __init__(self, *args, **storage_options):
           super().__init__(*args, **storage_options)
           self.access_count = {}
   
       def open(
           self,
           path,
           mode="rb",
           block_size=None,
           cache_options=None,
           compression=None,
           **kwargs,
       ):
           if path in self.access_count:
               self.access_count[path] += 1
           else:
               self.access_count[path] = 1
   
           return super().open(        
               path,
               mode,
               block_size,
               cache_options,
               compression,
               **kwargs,
           )
           
   
   # create a simple test table
   df = pd.DataFrame({'col1':range(2),'col2':range(2)})
   out_table = pa.Table.from_pandas(df)
   
   # write then read as single parquet file.
   # file is opened twice using read_table
   pq.write_table(out_table, 'example.parquet')
   fs = LocalFSWithAccessCounter()
   in_table = pq.read_table('example.parquet', filesystem=fs)
   print('files open count:\n', fs.access_count, '\n\n')
   #files open count:
   # {'example.parquet': 2} 
   
   # write then read as dataset with partitions.
   # one file is opened twice when using read_table
   pq.write_to_dataset(out_table, 'example_ds.parquet', partition_cols=['col1'])
   fs = LocalFSWithAccessCounter()
   in_table = pq.read_table(os.path.join(os.getcwd(), 'example_ds.parquet'), 
filesystem=fs)
   print('dataset file open count:\n',fs.access_count)
   #dataset file open count:
   # {'example.parquet': 2, 
'example_ds.parquet/col1=0/237b18616a4844b3b094c264f6a09c48-0.parquet': 2, 
example_ds.parquet/col1=1/2d70ed145ee742e99f4c5528b053446d-0.parquet': 1, }
   ```
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to