jonkeane opened a new pull request #11058:
URL: https://github.com/apache/arrow/pull/11058
We've experienced a number of hangs when using our DuckDB-Arrow integration.
This is to test the theory that we're getting these hangs because we're using
parallelism in both DuckDB and Arrow's scanning. This PR disables setting
DuckDB's parallelism.
Using the taxi parquet dataset, the following query will hang some
proportion of the time:
```
library(arrow)
library(dplyr)
ds <- open_dataset("~/repos/ab_store/data/taxi_parquet/", partitioning =
c("year", "month"))
ds %>%
filter(total_amount > 100, year == 2015) %>%
select(tip_amount, total_amount, passenger_count) %>%
mutate(tip_pct = 100 * tip_amount / total_amount) %>%
to_duckdb() %>%
group_by(passenger_count) %>%
summarise(
mean_tip_pct = mean(tip_pct, na.rm = TRUE),
n = n()
) %>%
collect()
```
But with this PR, we do not get that same hanging behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
