[
https://issues.apache.org/jira/browse/BEAM-2939?focusedWorklogId=167821&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-167821
]
ASF GitHub Bot logged work on BEAM-2939:
----------------------------------------
Author: ASF GitHub Bot
Created on: 20/Nov/18 16:43
Start Date: 20/Nov/18 16:43
Worklog Time Spent: 10m
Work Description: lukecwik commented on a change in pull request #6969:
[BEAM-2939] SplittableDoFn Java SDK API Changes
URL: https://github.com/apache/beam/pull/6969#discussion_r234817969
##########
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
##########
@@ -725,14 +731,30 @@ public Duration getAllowedTimestampSkew() {
* href="https://s.apache.org/splittable-do-fn";>splittable</a> {@link DoFn}
into multiple parts to
* be processed in parallel.
*
- * <p>Signature: {@code List<RestrictionT> splitRestriction( InputT element,
RestrictionT
- * restriction);}
+ * <p>The signature of this method must satisfy the following constraints:
+ *
+ * <ul>
+ * <li>One of its parameters must be the {@code InputT} element.
+ * <li>One of its parameters must be the restriction.
+ * <li>One of its input parameters must be of type {@link Backlog}.
Splitting the restriction
+ * should attempt to take the backlog information into account. If the
backlog is known,
+ * each split should return a restriction with an approximate amount
of work bounded by the
+ * backlog. In the case of an unbounded restriction, at most one of
the splits can represent
+ * the unbounded portion of work. If the backlog that is specified is
unknown, it is up to
+ * the SDK to choose a number of splits of approximately equally sized
portions with
+ * potentially one of those splits representing the unbounded portion
of work.
+ * <li>One of its parameters must be the output receiver for restrictions.
+ * </ul>
+ *
+ * <p>Signature: {@code splitRestriction(InputT element, RestrictionT
restriction, Backlog
+ * backlog, OutputReceiver<RestrictionT> receiver);}
*
* <p>Optional: if this method is omitted, the restriction will not be split
(equivalent to
- * defining the method and returning {@code
Collections.singletonList(restriction)}).
+ * defining the method and outputting {@code
Collections.singletonList(restriction)}).
+ *
+ * <p>TODO: Make the InputT parameter optional.
Review comment:
Yes, this refers to the implementation for the invoker part.
I was hoping that I could get the API part done without having the generic
invoker logic done since that allows parallelizing work amongst other
contributors. Will file JIRA about adding support for making these optional and
moving towards a generic invoker / method format once this gets ship it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 167821)
Time Spent: 7h 10m (was: 7h)
> Fn API streaming SDF support
> ----------------------------
>
> Key: BEAM-2939
> URL: https://issues.apache.org/jira/browse/BEAM-2939
> Project: Beam
> Issue Type: Improvement
> Components: beam-model
> Reporter: Henning Rohde
> Assignee: Luke Cwik
> Priority: Major
> Labels: portability
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> The Fn API should support streaming SDF. Detailed design TBD.
> Once design is ready, expand subtasks similarly to BEAM-2822.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
