JonnyWaffles commented on issue #34414:
URL: https://github.com/apache/arrow/issues/34414#issuecomment-1452405066
Oh I just wanted to add one more note, when using the dataset API with the
above `fsspec` I noticed it's much slower to use the dataset API versus
`pq.read_table`
Example
```python
def read_table_pq(s3fs):
time_start = time.perf_counter()
num_bytes = pq.read_table(s3_key, filesystem=s3fs).nbytes
print(f"{num_bytes} Took {time.perf_counter() - time_start}")
def read_table_ds(s3fs):
time_start = time.perf_counter()
num_bytes = ds.dataset(s3_key, filesystem=s3fs,
format="parquet").to_table().nbytes
print(f"{num_bytes} Took {time.perf_counter() - time_start}")
```
Result
```
read_table_pq(s3fs1)
301340480 Took 25.171526359998097
read_table_ds(s3fs1)
301340480 Took 46.74551891900046
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
