cjackal opened a new issue, #36272:
URL: https://github.com/apache/arrow/issues/36272
### Describe the bug, including details regarding any error messages,
version, and platform.
xref: pola-rs/polars#9505
We have some huge datasets on S3 (filecount > 100,000), scanning with
`.scanner` works well but not with `polars.scan_pyarrow_dataset`, raising the
following libcurl error.
```bash
---------------------------------------------------------------------------
ComputeError Traceback (most recent call last)
Cell In[2], line 16
14 files = ds.files
15 ds = dataset(files[:300], format="parquet", filesystem=fs)
---> 16 pl.scan_pyarrow_dataset(ds).select(pl.col("area").head()).collect()
File
/mnt/venvs/main/lib/python3.10/site-packages/polars/lazyframe/frame.py:1504, in
LazyFrame.collect(self, type_coercion, predicate_pushdown, projection_pushdown,
simplify_expression, no_optimization, slice_pushdown,
common_subplan_elimination, streaming)
1493 common_subplan_elimination = False
1495 ldf = self._ldf.optimization_toggle(
1496 type_coercion,
1497 predicate_pushdown,
(...)
1502 streaming,
1503 )
-> 1504 return wrap_df(ldf.collect())
ComputeError: OSError: When reading information for key '<redacted - string
of length 190>' in bucket '<redacted - string of length 27>': AWS Error
NETWORK_CONNECTION during HeadObject operation: curlCode 43, A libcurl function
was given a bad argument
```
While it is a `polars` bug, the error is raised from `pyarrow`, so I open an
issue here with polars xref.
This error message above is generated using `.head()` but effectively all
scanner suffer the same issue.
I just observed that the total number of files is a critical factor: when
restricted to <=272 files scanning in `polars` always succeeded, for datasets
with more files `polars` always fails.
Script used:
```python
import polars as pl
from pyarrow.dataset import dataset
from pyarrow.fs import S3FileSystem
fs = S3FileSystem(
access_key=access_key,
secret_key=secret_key,
region=region,
endpoint_override=endpoint_override,
)
ds = dataset("my-bucket/common-prefix", format="parquet",
partitioning="hive", filesystem=fs)
ds.head(10) # succeeded in few secs
pl.scan_pyarrow_dataset(ds).select(pl.col("area").head()).collect() #
failed after >10 mins with libcurl error
files = ds.files # to get the full list of URIs
ds = dataset(files[:300], format="parquet", filesystem=fs) # restrict num
of files
ds.head(10) # still works
pl.scan_pyarrow_dataset(ds).select(pl.col("area").head()).collect() #
failed after >10 mins with libcurl error
ds = dataset(files[:272], format="parquet", filesystem=fs)
pl.scan_pyarrow_dataset(ds).select(pl.col("area").head()).collect() # now
it works
```
Version info:
```bash
--------Version info---------
Polars: 0.18.3
Index type: UInt32
Platform: # it's ubuntu jammy latest, I forgot to copy the output
Python: 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0]
----Optional dependencies----
numpy: 1.24.3
pandas: 1.5.2
pyarrow: 12.0.1
connectorx: <not installed>
deltalake: <not installed>
fsspec: <not installed>
matplotlib: <not installed>
xlsx2csv: <not installed>
xlsxwriter: <not installed>
```
All packages are installed via pip.
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
