Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4956#discussion_r26087647
--- Diff: docs/streaming-programming-guide.md ---
@@ -1868,13 +1961,38 @@ Furthermore, there are two kinds of failures that
we should be concerned about:
With this basic knowledge, let us understand the fault-tolerance semantics
of Spark Streaming.
-## Semantics with files as input source
+## Definitions
+{:.no_toc}
+The semantics of streaming systems are often captured in terms of how many
times each record can be processed by the system. There are three types of
guarantees that a system can provide under all possible operating conditions
(despite failures, etc.)
+
+1. *At most once*: Each record will be either processed once or not
processed at all.
+2. *At least once*: Each record will be processed one or more times. This
is stronger than *at-most once* as it ensure that no data will be lost. But
there may be duplicates.
+3. *Exactly once*: Each record will be processed exactly once - no data
will be lost and no data will be processed multiple times. This is obviously
the strongest guarantee of the three.
+
+## Basic Semantics
+{:.no_toc}
+In any stream processing system, broadly speaking, there are three steps
in processing the data.
--- End diff --
There should be an extra blank line here in order for the list to be
formatted properly; currently, it renders like this:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
