[
https://issues.apache.org/jira/browse/BEAM-10303?focusedWorklogId=478024&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478024
]
ASF GitHub Bot logged work on BEAM-10303:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 02/Sep/20 18:09
Start Date: 02/Sep/20 18:09
Worklog Time Spent: 10m
Work Description: boyuanzz commented on a change in pull request #12710:
URL: https://github.com/apache/beam/pull/12710#discussion_r482267362
##########
File path:
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java
##########
@@ -1546,50 +1459,32 @@ public Object restriction() {
currentWindowIndex,
stopWindowIndex,
stopWindowIndex,
- splitResult,
+ elementSplitResult,
watermarkAndState);
}
- return KV.of(windowedSplitResult, newWindowStopIndex);
+ return SplitResultsWithStopIndex.of(
+ windowedSplitResult, downstreamSplitResult, newWindowStopIndex);
}
- private HandlesSplits.SplitResult trySplitForElementAndRestriction(
- double fractionOfRemainder, Duration resumeDelay) {
- KV<Instant, WatermarkEstimatorStateT> watermarkAndState;
- WindowedSplitResult windowedSplitResult = null;
- synchronized (splitLock) {
- // There is nothing to split if we are between element and restriction
processing calls.
- if (currentTracker == null) {
- return null;
- }
- // Make sure to get the output watermark before we split to ensure that
the lower bound
- // applies to the residual.
- watermarkAndState = currentWatermarkEstimator.getWatermarkAndState();
- KV<WindowedSplitResult, Integer> splitResult =
- trySplitForProcess(
- currentElement,
- currentRestriction,
- currentWindow,
- currentWindows,
- currentWatermarkEstimatorState,
- fractionOfRemainder,
- currentTracker,
- watermarkAndState,
- windowCurrentIndex,
- windowStopIndex);
- if (splitResult == null) {
- return null;
- }
- windowStopIndex = splitResult.getValue();
- windowedSplitResult =
- calculateRestrictionSize(
- splitResult.getKey(),
-
PTransformTranslation.SPLITTABLE_PROCESS_SIZED_ELEMENTS_AND_RESTRICTIONS_URN
- + "/GetSize");
- }
-
+ @VisibleForTesting
+ static <WatermarkEstimatorStateT> HandlesSplits.SplitResult
constructSplitResult(
+ WindowedSplitResult windowedSplitResult,
+ HandlesSplits.SplitResult downstreamElementSplit,
+ Coder fullInputCoder,
+ Instant initialWatermark,
+ KV<Instant, WatermarkEstimatorStateT> watermarkAndState,
+ String pTransformId,
+ String mainInputId,
+ Collection<String> outputIds,
+ Duration resumeDelay) {
+ // The element split cannot from both windowedSplitResult and
downstreamElementSplit.
+ checkArgument(
Review comment:
Yeah both `windowedSplitResult.getResidualSplitRoot` and
`downstreamElementSplit` can be null. The only illegal state here is both of
them are not null.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 478024)
Time Spent: 9.5h (was: 9h 20m)
> FnApiDoFnRunner window observing optimization
> ---------------------------------------------
>
> Key: BEAM-10303
> URL: https://issues.apache.org/jira/browse/BEAM-10303
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-harness
> Reporter: Luke Cwik
> Assignee: Luke Cwik
> Priority: P2
> Labels: portability
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> Currently the FnApiDoFnRunner processes each element within it's own window.
> There is an easy optimization where we process the element once if and only
> if the function doesn't observe the window (either directly or indirectly via
> side inputs/state/...).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
