shou123 opened a new issue, #37866:
URL: https://github.com/apache/arrow/issues/37866
### Describe the usage question you have. Please include as many useful
details as possible.
I want to use pyarrow.dataset.dataset to open a parquet file and to make it
a dataset. The parquet is 395M, it always pops up an error as "failed to scan
file fragment", but when I cut the parquet to a smaller file. it is working
well. No, error. I assume there is an argument to control the threshold.
My code
```
format = ds.SkyhookFileFormat("parquet", "/etc/ceph/ceph.conf")
data = list()
lineitem = ds.dataset(os.path.join(dataset_path, "lineitem"),
format=format)
print(f"lineitem arrow dataset: {lineitem.to_table().to_pandas()}")
```
Error message:
```
Exception has occurred: ArrowInvalid (note: full exception trace is
shown but execution is paused at: _run_module_as_main)
failed to scan file fragment
File "/usr/local/lib/python3.8/dist-packages/pyarrow/error.pxi", line 99,
in pyarrow.lib.check_status
raise ArrowInvalid(message)
File "/usr/local/lib/python3.8/dist-packages/pyarrow/error.pxi", line 143,
in pyarrow.lib.pyarrow_internal_check_status
return check_status(status)
File "/usr/local/lib/python3.8/dist-packages/pyarrow/_dataset.pyx", line
3070, in pyarrow._dataset.Scanner.to_table
return pyarrow_wrap_table(GetResultValue(result))
File "/usr/local/lib/python3.8/dist-packages/pyarrow/_dataset.pyx", line
466, in pyarrow._dataset.Dataset.to_table
return self.scanner(**kwargs).to_table()
File "/home/yue21/project/NDP_test/bench_single.py", line 29, in <module>
print(f"nation arrow dataset: {lineitem.to_table().to_pandas()}")
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
(Current frame)
```
### Component(s)
Parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
