[
https://issues.apache.org/jira/browse/ARROW-17820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17629225#comment-17629225
]
Weston Pace edited comment on ARROW-17820 at 11/4/22 11:24 PM:
---------------------------------------------------------------
{quote}
Such an approach doesn't really fit our kernels / Acero, I think. One option
could be to have a generic kernel to "map" another kernel on the list values.
where you can pass the function name you want to apply, and a FunctionOptions
object matching the kernel. Would something like this be possible technically?
{quote}
Yes, I think that should be possible for unary kernels. Though I think mapping
a single kernel (as opposed to a single expression) might be a bit limiting,
though maybe it isn't so bad. For example, what if a user wants to do
something like {{map(lambda f: f.upper() * 2, ["a", "b", "c"])}}
Another thing is that it should be valid to use n-ary functions too provided
the other arguments are scalars. This discussion has come up in Substrait with
respect to lambdas (https://github.com/substrait-io/substrait/issues/349).
Perhaps the "map function" for {{List<T>}} could be an expression bound to a
schema of "\{item: T\}" (e.g. so you could do {{field_ref(0)}} or
{{field_ref("item")}}).
Though if the map function is an expression then a kernel would have to execute
an entire expression which may or may not be doable (I've reached the limit of
my imagination for a Friday :)
was (Author: westonpace):
{quote}
Such an approach doesn't really fit our kernels / Acero, I think. One option
could be to have a generic kernel to "map" another kernel on the list values.
where you can pass the function name you want to apply, and a FunctionOptions
object matching the kernel. Would something like this be possible technically?
{quote}
Yes, I think that should be possible for unary kernels. Though I think mapping
a single kernel (as opposed to a single expression) might be a bit limiting,
though maybe it isn't so bad. For example, what if a user wants to do
something like {{map(lambda f: f.upper() * 2, ["a", "b", "c"])}}
Another thing is that it should be valid to use n-ary functions too provided
the other arguments are scalars. This discussion has come up in Substrait with
respect to lambdas (https://github.com/substrait-io/substrait/issues/349).
Perhaps the "map function" for {{List<T>}} could be an expression bound to a
schema of "{item: T}" (e.g. so you could do {{field_ref(0)}} or
{{field_ref("item")}}).
Though if the map function is an expression then a kernel would have to execute
an entire expression which may or may not be doable (I've reached the limit of
my imagination for a Friday :)
> [C++] Implement arithmetic kernels on List(number)
> --------------------------------------------------
>
> Key: ARROW-17820
> URL: https://issues.apache.org/jira/browse/ARROW-17820
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++, Python
> Reporter: Adam Lippai
> Priority: Major
> Labels: kernel, query-engine
>
> eg. rounding in list(float64()), similar to a map or foreach
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
