[
https://issues.apache.org/jira/browse/ARROW-12873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17351945#comment-17351945
]
Ben Kietzman edited comment on ARROW-12873 at 5/26/21, 5:13 PM:
----------------------------------------------------------------
Additionally, we'll have performance regressions in expression evaluation since
we have no way to attach known guarantees to batches. Currently (for example)
when filtering batches from the file {{/a=3/dat.parquet}} with {{a == 3 and b
== 2}} we will only do the work associated with {{b == 2}}. However if a
FilterNode is receiving batches from a ScanNode it has no way to safely skip
the work associated with {{a == 3}} since it can't detect the partition of
origin
was (Author: bkietz):
Additionally, we'll have performance regressions in expression evaluation since
we have no way to attach known guarantees to batches. Currently (for example)
when filtering batches from the file {{/a=3/dat.parquet}} with {{a == 3 and b
== 2}} we will only do the work associated with {{b == 2}}. However if a
FilterNode is receiving batches from a ScanNode it has no way to safely skip
the work associated with {{a == 3}}
> [C++][Compute] Support tagging ExecBatches with arbitrary extra information
> ---------------------------------------------------------------------------
>
> Key: ARROW-12873
> URL: https://issues.apache.org/jira/browse/ARROW-12873
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Ben Kietzman
> Priority: Major
>
> Ideally, ExecBatches could be tagged with arbitrary optional objects for
> tracing purposes and to transmit execution hints from one ExecNode to another.
> These should *not* be explicit members like ExecBatch::selection_vector is,
> since they may not originate from the arrow library. For an example within
> the arrow project: {{libarrow_dataset}} will be used to produce ScanNodes and
> a WriteNodes and it's useful to tag scanned batches with their {{Fragment}}
> of origin. However adding {{ExecBatch::fragment}} would result in a cyclic
> dependency.
> To facilitate this tagging capability, we would need a type erased container
> something like
> {code}
> struct AnySet {
> void* Get(tag_t tag);
> void Set(tag_t tag, void* value, FnOnce<void(void*)> destructor);
> };
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
