Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/15889#discussion_r88597141
--- Diff: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala
---
@@ -1014,7 +1015,7 @@ class PairRDDFunctions[K, V](self: RDD[(K, V)])
* Output the RDD to any Hadoop-supported file system, using a Hadoop
`OutputFormat` class
* supporting the key and value types K and V in this RDD.
*
- * Note that, we should make sure our tasks are idempotent when
speculation is enabled, i.e. do
+ * @note We should make sure our tasks are idempotent when speculation
is enabled, i.e. do
--- End diff --
Ah, I just checked that It seems both
```
We should make sure our tasks are idempotent when speculation is enabled,
i.e. do
not use output committer that writes data directly.
```
and
```
There is an example in https://issues.apache.org/jira/browse/SPARK-10063 to
show the bad
result of using direct output committer with speculation enabled.
```
are relevant (it seems both are related with a direct output committer).
Also, the original documentation also concatenates both sentences.
So. let me leave it as is if it looks okay.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
