[
https://issues.apache.org/jira/browse/ARROW-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17394114#comment-17394114
]
Antoine Pitrou commented on ARROW-5825:
---------------------------------------
It depends what the intent is. If we still want to wait for all futures
(instead of exiting on the first error), you would use:
{code:python}
if futures_list:
futures.wait(futures_list)
for fut in futures:
fut.result()
{code}
> [Python] Exceptions swallowed in ParquetManifest._visit_directories
> -------------------------------------------------------------------
>
> Key: ARROW-5825
> URL: https://issues.apache.org/jira/browse/ARROW-5825
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: George Sakkis
> Priority: Major
> Labels: dataset-parquet-read, parquet
>
> {{ParquetManifest._visit_directories}} uses a {{ThreadPoolExecutor}} to visit
> partitioned parquet datasets concurrently, it waits for them to finish but
> doesn't check if the respective futures have failed or not. This is quite
> tricky to detect and debug as an exception is either raised later as a a
> side-effect or (perhaps worse) it passes silently.
> Observed on 0.12.1 but appears to be on latest master too.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
