[
https://issues.apache.org/jira/browse/ARROW-13627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17398871#comment-17398871
]
Keith Kraus commented on ARROW-13627:
-------------------------------------
I would have expected the `na.rm` R argument to map directly to the
`skip_nulls` Arrow C++ argument. Given that Arrow arithmetic functions return
null if any input is null, I would expect `skip_nulls=False` to return `null`
if any value in the input array is `null` which would map to R's expected
behavior without needing to get the size of the array.
> [C++] ScalarAggregateOptions don't make sense (in hash aggregation)
> -------------------------------------------------------------------
>
> Key: ARROW-13627
> URL: https://issues.apache.org/jira/browse/ARROW-13627
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Neal Richardson
> Assignee: David Li
> Priority: Major
> Labels: kernel, query-engine
> Fix For: 6.0.0
>
>
> R's aggregation functions have a {{na.rm}} argument that governs how missing
> data is handled. Assume {{x <- c(1, 2, NA, 3)}}. {{sum(x, na.rm = TRUE) ==
> 6}} and {{sum(x, na.rm = FALSE)}} is {{NA}} because there is at least one
> missing value.
> The ScalarAggregateOptions have two options: skip_nulls and min_count. From
> what I can tell reading the source, in the context of sum(), skip_nulls
> affects whether each element of the Array is added to "count", and if count <
> min_count, you get a null value returned. So to get the expected behavior
> when calling "sum" on an Array, when na.rm = TRUE, we pass skip_nulls =
> false, min_count = 0. When na.rm = FALSE, we pass skip_nulls = true,
> min_count = length(x), the reasoning being that you return a null value
> unless all values are non-null (and count == length). See
> https://github.com/apache/arrow/blob/master/r/R/compute.R#L125-L130
> This doesn't really work in the query engine, though. We don't know how many
> rows are in the data to set an appropriate min_count to get the expected
> behavior--the dataset being queried may have filtering. And when doing hash
> aggregation, each group may have a different number of rows.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
