Antoine Pitrou created ARROW-9814:
-------------------------------------
Summary: [Python] Crash in
test_parquet.py::test_read_partitioned_directory_s3fs
Key: ARROW-9814
URL: https://issues.apache.org/jira/browse/ARROW-9814
Project: Apache Arrow
Issue Type: Bug
Components: C++, Python
Reporter: Antoine Pitrou
This seems to happen with some Minio versions, but is definitely a problem in
Arrow.
The crash message says:
{code}
pyarrow/tests/test_parquet.py::test_read_partitioned_directory_s3fs[False]
../src/arrow/dataset/discovery.cc:188: Check failed: relative.has_value()
GetFileInfo() yielded path outside selector.base_dir
{code}
The underlying problem is that we pass a full URI for the selector base_dir
(such as "s3://bucket/path.") and the S3 filesystem implementation then returns
regular paths (such as "bucket/path/foo/bar").
I think we should do two things:
1) error out rather than crash (and include the path strings in the error
message), which would be more user-friendly
2) fix the issue that full URIs are passed in base_dir
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
