[
https://issues.apache.org/jira/browse/BEAM-11048?focusedWorklogId=726269&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-726269
]
ASF GitHub Bot logged work on BEAM-11048:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Feb/22 12:45
Start Date: 14/Feb/22 12:45
Worklog Time Spent: 10m
Work Description: github-actions[bot] commented on pull request #13059:
URL: https://github.com/apache/beam/pull/13059#issuecomment-1039044023
This pull request has been closed due to lack of activity. If you think that
is incorrect, or the pull request requires review, you can revive the PR at any
time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 726269)
Time Spent: 50m (was: 40m)
> Add alternate Sorting transform as an implementation of CombineFn
> -----------------------------------------------------------------
>
> Key: BEAM-11048
> URL: https://issues.apache.org/jira/browse/BEAM-11048
> Project: Beam
> Issue Type: Improvement
> Components: extensions-java-sorter
> Reporter: Claire McGinty
> Priority: P3
> Labels: Clarified
> Time Spent: 50m
> Remaining Estimate: 0h
>
> My team has been using the
> [SortValues|https://github.com/apache/beam/blob/master/sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/SortValues.java]
> transform in `extensions-java-sorter` to sort pre-grouped values by a
> secondary sorter key. However, for large key groups, we've run into many OOM
> issues and have to increase disk size quite a bit to accommodate the larger
> key groups spilling to disk, even if there are only a few large key groups
> and most fit in memory.
> I drafted a new iteration of a Sorter that's a distributed merge-sort
> implemented as a `CombineFn`: each Accumulator maintains an always-sorted
> list of elements, and those Accumulators can be merged simply by zipping
> their lists together. This has the extra advantage that `extractOutput` can
> be lazily evaluated as a merging Iterator rather than as a fully materialized
> list. I also observed that this implementation is able to scale more
> effectively than the old SortValues, and for several use cases where
> `SortValues` ran OOM, the CombineFn-based implementation was able to complete
> using only the default Dataflow disk specs.
> Finally, from an API perspective, I think it's a little easier to use,
> because the user doesn't have to extract the sortKey out into the PCollection
> itself, but instead provide a function mapping each element type T to its
> sort key K, which will be evaluated inside the combiner. So I think in that
> sense it's more intuitive and similar to a Comparator-style sort.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
