[
https://issues.apache.org/jira/browse/ARROW-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17181304#comment-17181304
]
Antoine Pitrou commented on ARROW-9814:
---------------------------------------
cc [~jorisvandenbossche]
> [Python] Crash in test_parquet.py::test_read_partitioned_directory_s3fs
> -----------------------------------------------------------------------
>
> Key: ARROW-9814
> URL: https://issues.apache.org/jira/browse/ARROW-9814
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, Python
> Reporter: Antoine Pitrou
> Priority: Critical
>
> This seems to happen with some Minio versions, but is definitely a problem in
> Arrow.
> The crash message says:
> {code}
> pyarrow/tests/test_parquet.py::test_read_partitioned_directory_s3fs[False]
> ../src/arrow/dataset/discovery.cc:188: Check failed: relative.has_value()
> GetFileInfo() yielded path outside selector.base_dir
> {code}
> The underlying problem is that we pass a full URI for the selector base_dir
> (such as "s3://bucket/path.") and the S3 filesystem implementation then
> returns regular paths (such as "bucket/path/foo/bar").
> I think we should do two things:
> 1) error out rather than crash (and include the path strings in the error
> message), which would be more user-friendly
> 2) fix the issue that full URIs are passed in base_dir
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
