kimotorc opened a new issue, #35894: URL: https://github.com/apache/arrow/issues/35894
### Describe the bug, including details regarding any error messages, version, and platform. It appears that parquet files are opened twice when using parquet.read_table. The first time seems to be to read in the schema, the second to do the actual data read. This seems to be the case when use_legacy_dataset=False and without specifying a schema argument. It also is the case for datasets and single parquet reads. My use case is that i'd like to use a custom fsspec filesystem that interfaces to an s3 like API. It handles the remote download of the parquet file and passes to pyarrow a handle of a temporary file downloaded locally. Without cached access, this behavior results in files being downloaded twice. [Here](https://github.com/apache/arrow/blob/5a55fb4f66cbd316568085129054b5ff1d2eb089/python/pyarrow/parquet/core.py#L2479) is where the file is opened to read in table schema. Here is code to reproduce: ``` import pandas as pd import pyarrow as pa import pyarrow.parquet as pq from fsspec.implementations.local import LocalFileSystem import os # create a fsspec filesystem to track the number of times files are opened class LocalFSWithAccessCounter(LocalFileSystem): def __init__(self, *args, **storage_options): super().__init__(*args, **storage_options) self.access_count = {} def open( self, path, mode="rb", block_size=None, cache_options=None, compression=None, **kwargs, ): if path in self.access_count: self.access_count[path] += 1 else: self.access_count[path] = 1 return super().open( path, mode, block_size, cache_options, compression, **kwargs, ) # create a simple test table df = pd.DataFrame({'col1':range(2),'col2':range(2)}) out_table = pa.Table.from_pandas(df) # write then read as single parquet file. # file is opened twice using read_table pq.write_table(out_table, 'example.parquet') fs = LocalFSWithAccessCounter() in_table = pq.read_table('example.parquet', filesystem=fs) print('files open count:\n', fs.access_count, '\n\n') #files open count: # {'example.parquet': 2} # write then read as dataset with partitions. # one file is opened twice when using read_table pq.write_to_dataset(out_table, 'example_ds.parquet', partition_cols=['col1']) fs = LocalFSWithAccessCounter() in_table = pq.read_table(os.path.join(os.getcwd(), 'example_ds.parquet'), filesystem=fs) print('dataset file open count:\n',fs.access_count) #dataset file open count: # {'example.parquet': 2, 'example_ds.parquet/col1=0/237b18616a4844b3b094c264f6a09c48-0.parquet': 2, example_ds.parquet/col1=1/2d70ed145ee742e99f4c5528b053446d-0.parquet': 1, } ``` ### Component(s) Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
