[
https://issues.apache.org/jira/browse/ARROW-10183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272696#comment-17272696
]
Weston Pace edited comment on ARROW-10183 at 1/27/21, 9:01 AM:
---------------------------------------------------------------
As maybe more food for though one could consider a classic map-reduce style
summation task. A buffer comes in with 1024 elements, it is split into 8 sum
tasks of 128 and then reduced with 7 reduce tasks. This "fan-out" could be
implemented as a fully reentrant pipeline...
Mapper (Just a flatMap, fits Transformer model, but should be able to be made
reentrant, emits 8 times for each input)
Summer (Takes in a buffer and computes the sum, fits transformer model, but
should be reentrant)
Reducer (Consumes two numbers and outputs the sum, would NOT fit transformer
model (even with skip), curiously though it should also be reentrant)
All of these could be "transformers" but the existing transform model would
make them non-reentrant. However, all of them "should" be able to be
reentrant. I think this argues somewhat to gaps in the transformer model. I
have an improved model in mind, but it is was not compatible with synchronous
iterators so I abandoned it. I may have to revisit.
was (Author: westonpace):
As maybe more food for though one could consider a classic map-reduce style
summation task. A buffer comes in with 1024 elements, it is split into 8 sum
tasks of 128 and then reduced with 7 reduce tasks. This "fan-out" could be
implemented as a fully reentrant pipeline...
Mapper (Just a flatMap, fits Transformer model, but should be able to be made
reentrant, emits 8 times for each input)
Summer (Takes in a buffer and computes the sum, fits transformer model, but
should be reentrant)
Reducer (Consumes two numbers and outputs the sum, could fit transformer model
with skip, curiously though it should be reentrant)
All of these could be "transformers" but the existing transform model would
make them non-reentrant. However, all of them "should" be able to be
reentrant. I think this argues somewhat to gaps in the transformer model. I
have an improved model in mind, but it is was not compatible with synchronous
iterators so I abandoned it. I may have to revisit.
> [C++] Create a ForEach library function that runs on an iterator of futures
> ---------------------------------------------------------------------------
>
> Key: ARROW-10183
> URL: https://issues.apache.org/jira/browse/ARROW-10183
> Project: Apache Arrow
> Issue Type: New Feature
> Components: C++
> Reporter: Weston Pace
> Assignee: Weston Pace
> Priority: Major
> Labels: pull-request-available
> Attachments: arrow-continuation-flow.jpg
>
> Time Spent: 7h 50m
> Remaining Estimate: 0h
>
> This method should take in an iterator of futures and a callback and pull an
> item off the iterator, "await" it, run the callback on it, and then fetch the
> next item from the iterator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
