[
https://issues.apache.org/jira/browse/ARROW-13088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jayjeet Chakraborty updated ARROW-13088:
----------------------------------------
Description:
Launching nested threads using the C++ ThreadPool causes the Dataset API
(C++/Python using the SyncScanner/AsyncScanner) to stop progressing. The
execution just hangs and keeps waiting. Examples of nested threading:
# Force turn on parallel column reads in ParquetFileFormat by tweaking the
code
[here|https://github.com/apache/arrow/blob/master/cpp/src/arrow/dataset/file_parquet.cc#L327]
to get into nested parallelism (for every scan task launched in parallel to
spawn threads internally).
# Create a simple dataset containing Parquet files of count X (where X >= No.
of logical cores).
# {{Just do a dataset.to_table() call.}}
The execution should stop and go into a kind of a sleep state immediately.
was:
Launching nested threads using the C++ ThreadPool causes the Dataset API
(C++/Python using the SyncScanner/AsyncScanner) to stop progressing. The
execution just hangs and keeps waiting. Examples of nested threading:
# Force turn on parallel column reads in ParquetFileFormat by tweaking the
code
[here|https://github.com/apache/arrow/blob/master/cpp/src/arrow/dataset/file_parquet.cc#L327]
to get into nested parallelism (for every scan task launched in parallel to
spawn threads internally).
# Create a simple dataset containing Parquet files of count X (where X >= No.
of logical cores).
# {{Just to a dataset.to_table() call.}}
The execution should stop and go into a kind of a sleep state immediately.
> Dataset API calls with nested parallelism does not progress
> -----------------------------------------------------------
>
> Key: ARROW-13088
> URL: https://issues.apache.org/jira/browse/ARROW-13088
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Affects Versions: 3.0.0, 4.0.1
> Reporter: Jayjeet Chakraborty
> Priority: Major
>
> Launching nested threads using the C++ ThreadPool causes the Dataset API
> (C++/Python using the SyncScanner/AsyncScanner) to stop progressing. The
> execution just hangs and keeps waiting. Examples of nested threading:
> # Force turn on parallel column reads in ParquetFileFormat by tweaking the
> code
> [here|https://github.com/apache/arrow/blob/master/cpp/src/arrow/dataset/file_parquet.cc#L327]
> to get into nested parallelism (for every scan task launched in parallel to
> spawn threads internally).
> # Create a simple dataset containing Parquet files of count X (where X >=
> No. of logical cores).
> # {{Just do a dataset.to_table() call.}}
> The execution should stop and go into a kind of a sleep state immediately.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
