lucazanna opened a new issue, #34608:
URL: https://github.com/apache/arrow/issues/34608

   ### Describe the bug, including details regarding any error messages, 
version, and platform.
   
   I am accessing a Parquet file on Azure data lake with the following code. 
   (to make the example reproducible, the example has a file publicly 
accessible on Azure)
   
   ```
   import pyarrow.dataset as ds
   from adlfs import AzureBlobFileSystem
   
   abfs_public = AzureBlobFileSystem(
       account_name="azureopendatastorage")
   
   dataset_public = 
ds.dataset('az://nyctlc/yellow/puYear=2010/puMonth=1/part-00000-tid-8898858832658823408-a1de80bd-eed3-4d11-b9d4-fa74bfbd47bc-426339-18.c000.snappy.parquet',
 filesystem=abfs_public)
   ```
   
   The processing time is the same for accessing the full file or only the 
first 5 rows:
   
   ```
   dataset_public.head(5)
   # 5min 11s
   
   dataset_public.to_table()
   # 5min 30s
   
   dataset_public.scanner().head(5)
   # 5min 43s
   ```
   I would expect the time to be less for 5 rows. I am not sure about the 
difference between .scanner().head() and .head().
   
   Regarding reducing the number of columns:
   reducing the number of columns retrieved speeds up the query, but the 
reduction seems small. For example, filtering to only 2 columns out of 21 
reduces the query to 2min 7:
   
   ```
   dataset_public.scanner(columns=['vendorID','passengerCount']).to_table()
   # 2min 7s
   ```
   
   I would have expected that collecting 10% of the columns ( 2 columns instead 
of 21 columns) to reduce the time by more than half. Unless there is an 
overhead to the query for collecting from Azure?
   
   Sources for the code:
   * Apache Arrow website: 
https://arrow.apache.org/docs/python/parquet.html#reading-parquet-and-memory-mapping
   * ADLFS Github page: https://github.com/fsspec/adlfs
   
   Thank you for the outstanding job on the Arrow library
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to