[
https://issues.apache.org/jira/browse/BEAM-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-3194:
----------------------------------
Comment: was deleted
(was: kennknowles commented on a change in pull request #4135: [BEAM-3194] Add
@RequiresStableInput annotation
URL: https://github.com/apache/beam/pull/4135#discussion_r153614701
##########
File path:
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java
##########
@@ -568,6 +568,29 @@ public Duration getAllowedTimestampSkew() {
public @interface ProcessElement {}
/**
+ * <b><i>Experimental - no backwards compatibility guarantees. The exact
name or usage of this
+ * feature may change.</i></b>
+ *
+ * <p>Annotation that may be added to a {@link ProcessElement} or {@link
OnTimer} method to
+ * indicate that the observable contents of the input {@link PCollection} or
mutable state must be
+ * stable upon retries.
+ *
+ * <p>This is important for exactly-once semantics when writing to a storage
medium outside of
+ * your pipeline. The general pattern is to establish an idempotent protocol
for writing the data,
+ * and then require stable input. Combined, these allow the write to be
freely retried until
+ * success.
+ *
+ * <p>An example of an unstable input would be anything computed using
nondeterministic logic. In
+ * Beam, any user-defined function is permitted to be nondeterministic, and
any {@link
+ * PCollection} is permitted to be recomputed in any manner.
+ */
+ @Documented
+ @Experimental
+ @Retention(RetentionPolicy.RUNTIME)
+ @Target(ElementType.METHOD)
+ public @interface RequiresStableInput {}
Review comment:
I think you could go either way, but I think this one is a bit more flexible
- no need to add the flag to various places you might add it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
)
> Support annotating that a DoFn requires stable / deterministic input for
> replay/retry
> -------------------------------------------------------------------------------------
>
> Key: BEAM-3194
> URL: https://issues.apache.org/jira/browse/BEAM-3194
> Project: Beam
> Issue Type: New Feature
> Components: beam-model
> Reporter: Kenneth Knowles
> Assignee: Eugene Kirpichov
> Priority: Major
>
> See the thread:
> https://lists.apache.org/thread.html/5fd81ce371aeaf642665348f8e6940e308e04275dd7072f380f9f945@%3Cdev.beam.apache.org%3E
> We need this in order to have truly cross-runner end-to-end exactly once via
> replay + idempotence.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
