Neal Richardson created ARROW-13627:
---------------------------------------
Summary: [C++] ScalarAggregateOptions don't make sense (in hash
aggregation)
Key: ARROW-13627
URL: https://issues.apache.org/jira/browse/ARROW-13627
Project: Apache Arrow
Issue Type: Improvement
Components: C++
Reporter: Neal Richardson
Fix For: 6.0.0
R's aggregation functions have a {{na.rm}} argument that governs how missing
data is handled. Assume {{x <- c(1, 2, NA, 3)}}. {{sum(x, na.rm = TRUE) == 6}}
and {{sum(x, na.rm = FALSE)}} is {{NA}} because there is at least one missing
value.
The ScalarAggregateOptions have two options: skip_nulls and min_count. From
what I can tell reading the source, in the context of sum(), skip_nulls affects
whether each element of the Array is added to "count", and if count <
min_count, you get a null value returned. So to get the expected behavior when
calling "sum" on an Array, when na.rm = TRUE, we pass skip_nulls = false,
min_count = 0. When na.rm = FALSE, we pass skip_nulls = true, min_count =
length(x), the reasoning being that you return a null value unless all values
are non-null (and count == length). See
https://github.com/apache/arrow/blob/master/r/R/compute.R#L125-L130
This doesn't really work in the query engine, though. We don't know how many
rows are in the data to set an appropriate min_count to get the expected
behavior--the dataset being queried may have filtering. And when doing hash
aggregation, each group may have a different number of rows.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
