[
https://issues.apache.org/jira/browse/ARROW-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17394099#comment-17394099
]
Joris Van den Bossche commented on ARROW-5825:
----------------------------------------------
Since this is in the legacy implementation, I would say it's no priority to
fix. But if it's easy to fix, maybe we should just do it. I am not super
familiar with concurrent.futures, but is the first answer in
https://stackoverflow.com/questions/35711160/detect-failed-tasks-in-concurrent-futures
the way to go? (replacing the {{wait}} call with {{for fut in
futures.as_completed(fs): fut.result()}})
> [Python] Exceptions swallowed in ParquetManifest._visit_directories
> -------------------------------------------------------------------
>
> Key: ARROW-5825
> URL: https://issues.apache.org/jira/browse/ARROW-5825
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: George Sakkis
> Priority: Major
> Labels: dataset-parquet-read, parquet
>
> {{ParquetManifest._visit_directories}} uses a {{ThreadPoolExecutor}} to visit
> partitioned parquet datasets concurrently, it waits for them to finish but
> doesn't check if the respective futures have failed or not. This is quite
> tricky to detect and debug as an exception is either raised later as a a
> side-effect or (perhaps worse) it passes silently.
> Observed on 0.12.1 but appears to be on latest master too.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
