[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried
[
https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520752#comment-16520752
]
Apache Spark commented on SPARK-24552:
--
User 'vanzin' has created a pull request for this issue:
https://github.com/apache/spark/pull/21616
> Task attempt numbers are reused when stages are retried
> ---
>
> Key: SPARK-24552
> URL: https://issues.apache.org/jira/browse/SPARK-24552
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.1.1, 2.2.0, 2.2.1, 2.3.0, 2.3.1
>Reporter: Ryan Blue
>Priority: Blocker
>
> When stages are retried due to shuffle failures, task attempt numbers are
> reused. This causes a correctness bug in the v2 data sources write path.
> Data sources (both the original and v2) pass the task attempt to writers so
> that writers can use the attempt number to track and clean up data from
> failed or speculative attempts. In the v2 docs for DataWriterFactory, the
> attempt number's javadoc states that "Implementations can use this attempt
> number to distinguish writers of different task attempts."
> When two attempts of a stage use the same (partition, attempt) pair, two
> tasks can create the same data and attempt to commit. The commit coordinator
> prevents both from committing and will abort the attempt that finishes last.
> When using the (partition, attempt) pair to track data, the aborted task may
> delete data associated with the (partition, attempt) pair. If that happens,
> the data for the task that committed is also deleted as well, which is a
> correctness bug.
> For a concrete example, I have a data source that creates files in place
> named with {{part---.}}. Because these
> files are written in place, both tasks create the same file and the one that
> is aborted deletes the file, leading to data corruption when the file is
> added to the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried
[
https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520742#comment-16520742
]
Apache Spark commented on SPARK-24552:
--
User 'vanzin' has created a pull request for this issue:
https://github.com/apache/spark/pull/21615
> Task attempt numbers are reused when stages are retried
> ---
>
> Key: SPARK-24552
> URL: https://issues.apache.org/jira/browse/SPARK-24552
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.1.1, 2.2.0, 2.2.1, 2.3.0, 2.3.1
>Reporter: Ryan Blue
>Priority: Blocker
>
> When stages are retried due to shuffle failures, task attempt numbers are
> reused. This causes a correctness bug in the v2 data sources write path.
> Data sources (both the original and v2) pass the task attempt to writers so
> that writers can use the attempt number to track and clean up data from
> failed or speculative attempts. In the v2 docs for DataWriterFactory, the
> attempt number's javadoc states that "Implementations can use this attempt
> number to distinguish writers of different task attempts."
> When two attempts of a stage use the same (partition, attempt) pair, two
> tasks can create the same data and attempt to commit. The commit coordinator
> prevents both from committing and will abort the attempt that finishes last.
> When using the (partition, attempt) pair to track data, the aborted task may
> delete data associated with the (partition, attempt) pair. If that happens,
> the data for the task that committed is also deleted as well, which is a
> correctness bug.
> For a concrete example, I have a data source that creates files in place
> named with {{part---.}}. Because these
> files are written in place, both tasks create the same file and the one that
> is aborted deletes the file, leading to data corruption when the file is
> added to the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried
[
https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519723#comment-16519723
]
Apache Spark commented on SPARK-24552:
--
User 'vanzin' has created a pull request for this issue:
https://github.com/apache/spark/pull/21606
> Task attempt numbers are reused when stages are retried
> ---
>
> Key: SPARK-24552
> URL: https://issues.apache.org/jira/browse/SPARK-24552
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.1.1, 2.2.0, 2.2.1, 2.3.0, 2.3.1
>Reporter: Ryan Blue
>Priority: Blocker
>
> When stages are retried due to shuffle failures, task attempt numbers are
> reused. This causes a correctness bug in the v2 data sources write path.
> Data sources (both the original and v2) pass the task attempt to writers so
> that writers can use the attempt number to track and clean up data from
> failed or speculative attempts. In the v2 docs for DataWriterFactory, the
> attempt number's javadoc states that "Implementations can use this attempt
> number to distinguish writers of different task attempts."
> When two attempts of a stage use the same (partition, attempt) pair, two
> tasks can create the same data and attempt to commit. The commit coordinator
> prevents both from committing and will abort the attempt that finishes last.
> When using the (partition, attempt) pair to track data, the aborted task may
> delete data associated with the (partition, attempt) pair. If that happens,
> the data for the task that committed is also deleted as well, which is a
> correctness bug.
> For a concrete example, I have a data source that creates files in place
> named with {{part---.}}. Because these
> files are written in place, both tasks create the same file and the one that
> is aborted deletes the file, leading to data corruption when the file is
> added to the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried
[
https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519445#comment-16519445
]
Thomas Graves commented on SPARK-24552:
---
more details on hadoop committer side:
So I think the commit/delete thing is also an issue for existing v1 and hadoop
committers as well. So this doesn't fully solve the problem. spark uses a file
format like (HadoopMapReduceWriteConfigUtil/HadoopMapRedWriteConfigUtil):
{quote}{{{date}_\{rddid}_\{m/r}_\{partitionid}_\{task attempt number}}}
{quote}
I believe the same fix as the v2 would work using the taskAttemptId instead of
the attemptNumber.
In the case we have the stage failure and a second stage attempt the task
attempt number could be the same and thus both tasks write to the same place.
If one of them fails or is told not to commit it could delete the output which
is being used by both.
Need to think through all the scenarios to make sure its covered.
> Task attempt numbers are reused when stages are retried
> ---
>
> Key: SPARK-24552
> URL: https://issues.apache.org/jira/browse/SPARK-24552
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.1.1, 2.2.0, 2.2.1, 2.3.0, 2.3.1
>Reporter: Ryan Blue
>Priority: Blocker
>
> When stages are retried due to shuffle failures, task attempt numbers are
> reused. This causes a correctness bug in the v2 data sources write path.
> Data sources (both the original and v2) pass the task attempt to writers so
> that writers can use the attempt number to track and clean up data from
> failed or speculative attempts. In the v2 docs for DataWriterFactory, the
> attempt number's javadoc states that "Implementations can use this attempt
> number to distinguish writers of different task attempts."
> When two attempts of a stage use the same (partition, attempt) pair, two
> tasks can create the same data and attempt to commit. The commit coordinator
> prevents both from committing and will abort the attempt that finishes last.
> When using the (partition, attempt) pair to track data, the aborted task may
> delete data associated with the (partition, attempt) pair. If that happens,
> the data for the task that committed is also deleted as well, which is a
> correctness bug.
> For a concrete example, I have a data source that creates files in place
> named with {{part---.}}. Because these
> files are written in place, both tasks create the same file and the one that
> is aborted deletes the file, leading to data corruption when the file is
> added to the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried
[
https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519430#comment-16519430
]
Thomas Graves commented on SPARK-24552:
---
this is actually a problem with hadoop committers, v1 and v2
> Task attempt numbers are reused when stages are retried
> ---
>
> Key: SPARK-24552
> URL: https://issues.apache.org/jira/browse/SPARK-24552
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.1.1, 2.2.0, 2.2.1, 2.3.0, 2.3.1
>Reporter: Ryan Blue
>Priority: Blocker
>
> When stages are retried due to shuffle failures, task attempt numbers are
> reused. This causes a correctness bug in the v2 data sources write path.
> Data sources (both the original and v2) pass the task attempt to writers so
> that writers can use the attempt number to track and clean up data from
> failed or speculative attempts. In the v2 docs for DataWriterFactory, the
> attempt number's javadoc states that "Implementations can use this attempt
> number to distinguish writers of different task attempts."
> When two attempts of a stage use the same (partition, attempt) pair, two
> tasks can create the same data and attempt to commit. The commit coordinator
> prevents both from committing and will abort the attempt that finishes last.
> When using the (partition, attempt) pair to track data, the aborted task may
> delete data associated with the (partition, attempt) pair. If that happens,
> the data for the task that committed is also deleted as well, which is a
> correctness bug.
> For a concrete example, I have a data source that creates files in place
> named with {{part---.}}. Because these
> files are written in place, both tasks create the same file and the one that
> is aborted deletes the file, leading to data corruption when the file is
> added to the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried
[
https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516522#comment-16516522
]
Marcelo Vanzin commented on SPARK-24552:
I forked the output commiter issue into SPARK-24589 so that we have a separate
record of each issue.
> Task attempt numbers are reused when stages are retried
> ---
>
> Key: SPARK-24552
> URL: https://issues.apache.org/jira/browse/SPARK-24552
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.1.1, 2.2.0, 2.2.1, 2.3.0, 2.3.1
>Reporter: Ryan Blue
>Priority: Blocker
>
> When stages are retried due to shuffle failures, task attempt numbers are
> reused. This causes a correctness bug in the v2 data sources write path.
> Data sources (both the original and v2) pass the task attempt to writers so
> that writers can use the attempt number to track and clean up data from
> failed or speculative attempts. In the v2 docs for DataWriterFactory, the
> attempt number's javadoc states that "Implementations can use this attempt
> number to distinguish writers of different task attempts."
> When two attempts of a stage use the same (partition, attempt) pair, two
> tasks can create the same data and attempt to commit. The commit coordinator
> prevents both from committing and will abort the attempt that finishes last.
> When using the (partition, attempt) pair to track data, the aborted task may
> delete data associated with the (partition, attempt) pair. If that happens,
> the data for the task that committed is also deleted as well, which is a
> correctness bug.
> For a concrete example, I have a data source that creates files in place
> named with {{part---.}}. Because these
> files are written in place, both tasks create the same file and the one that
> is aborted deletes the file, leading to data corruption when the file is
> added to the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried
[
https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514526#comment-16514526
]
Apache Spark commented on SPARK-24552:
--
User 'vanzin' has created a pull request for this issue:
https://github.com/apache/spark/pull/21577
> Task attempt numbers are reused when stages are retried
> ---
>
> Key: SPARK-24552
> URL: https://issues.apache.org/jira/browse/SPARK-24552
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.1.1, 2.2.0, 2.2.1, 2.3.0, 2.3.1
>Reporter: Ryan Blue
>Priority: Blocker
>
> When stages are retried due to shuffle failures, task attempt numbers are
> reused. This causes a correctness bug in the v2 data sources write path.
> Data sources (both the original and v2) pass the task attempt to writers so
> that writers can use the attempt number to track and clean up data from
> failed or speculative attempts. In the v2 docs for DataWriterFactory, the
> attempt number's javadoc states that "Implementations can use this attempt
> number to distinguish writers of different task attempts."
> When two attempts of a stage use the same (partition, attempt) pair, two
> tasks can create the same data and attempt to commit. The commit coordinator
> prevents both from committing and will abort the attempt that finishes last.
> When using the (partition, attempt) pair to track data, the aborted task may
> delete data associated with the (partition, attempt) pair. If that happens,
> the data for the task that committed is also deleted as well, which is a
> correctness bug.
> For a concrete example, I have a data source that creates files in place
> named with {{part---.}}. Because these
> files are written in place, both tasks create the same file and the one that
> is aborted deletes the file, leading to data corruption when the file is
> added to the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried
[
https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512519#comment-16512519
]
Thomas Graves commented on SPARK-24552:
---
sorry just realized the v2 api is still marked experiment so downgrading to
critical
> Task attempt numbers are reused when stages are retried
> ---
>
> Key: SPARK-24552
> URL: https://issues.apache.org/jira/browse/SPARK-24552
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.1.1
>Reporter: Ryan Blue
>Priority: Critical
>
> When stages are retried due to shuffle failures, task attempt numbers are
> reused. This causes a correctness bug in the v2 data sources write path.
> Data sources (both the original and v2) pass the task attempt to writers so
> that writers can use the attempt number to track and clean up data from
> failed or speculative attempts. In the v2 docs for DataWriterFactory, the
> attempt number's javadoc states that "Implementations can use this attempt
> number to distinguish writers of different task attempts."
> When two attempts of a stage use the same (partition, attempt) pair, two
> tasks can create the same data and attempt to commit. The commit coordinator
> prevents both from committing and will abort the attempt that finishes last.
> When using the (partition, attempt) pair to track data, the aborted task may
> delete data associated with the (partition, attempt) pair. If that happens,
> the data for the task that committed is also deleted as well, which is a
> correctness bug.
> For a concrete example, I have a data source that creates files in place
> named with {{part---.}}. Because these
> files are written in place, both tasks create the same file and the one that
> is aborted deletes the file, leading to data corruption when the file is
> added to the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried
[
https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512504#comment-16512504
]
Thomas Graves commented on SPARK-24552:
---
Note if this is a correctness bug and can cause data loss/corruption it needs
to be a blocker, changed to blocker, if I am misunderstanding please update.
> Task attempt numbers are reused when stages are retried
> ---
>
> Key: SPARK-24552
> URL: https://issues.apache.org/jira/browse/SPARK-24552
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.1.1
>Reporter: Ryan Blue
>Priority: Blocker
>
> When stages are retried due to shuffle failures, task attempt numbers are
> reused. This causes a correctness bug in the v2 data sources write path.
> Data sources (both the original and v2) pass the task attempt to writers so
> that writers can use the attempt number to track and clean up data from
> failed or speculative attempts. In the v2 docs for DataWriterFactory, the
> attempt number's javadoc states that "Implementations can use this attempt
> number to distinguish writers of different task attempts."
> When two attempts of a stage use the same (partition, attempt) pair, two
> tasks can create the same data and attempt to commit. The commit coordinator
> prevents both from committing and will abort the attempt that finishes last.
> When using the (partition, attempt) pair to track data, the aborted task may
> delete data associated with the (partition, attempt) pair. If that happens,
> the data for the task that committed is also deleted as well, which is a
> correctness bug.
> For a concrete example, I have a data source that creates files in place
> named with {{part---.}}. Because these
> files are written in place, both tasks create the same file and the one that
> is aborted deletes the file, leading to data corruption when the file is
> added to the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried
[
https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512500#comment-16512500
]
Thomas Graves commented on SPARK-24552:
---
I agree, I don't think changing the attempt number at this point does much help
and could cause confusion. I would rather see something like this change if we
do major reworking of the scheduler.
> Task attempt numbers are reused when stages are retried
> ---
>
> Key: SPARK-24552
> URL: https://issues.apache.org/jira/browse/SPARK-24552
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.1.1
>Reporter: Ryan Blue
>Priority: Major
>
> When stages are retried due to shuffle failures, task attempt numbers are
> reused. This causes a correctness bug in the v2 data sources write path.
> Data sources (both the original and v2) pass the task attempt to writers so
> that writers can use the attempt number to track and clean up data from
> failed or speculative attempts. In the v2 docs for DataWriterFactory, the
> attempt number's javadoc states that "Implementations can use this attempt
> number to distinguish writers of different task attempts."
> When two attempts of a stage use the same (partition, attempt) pair, two
> tasks can create the same data and attempt to commit. The commit coordinator
> prevents both from committing and will abort the attempt that finishes last.
> When using the (partition, attempt) pair to track data, the aborted task may
> delete data associated with the (partition, attempt) pair. If that happens,
> the data for the task that committed is also deleted as well, which is a
> correctness bug.
> For a concrete example, I have a data source that creates files in place
> named with {{part---.}}. Because these
> files are written in place, both tasks create the same file and the one that
> is aborted deletes the file, leading to data corruption when the file is
> added to the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried
[
https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511695#comment-16511695
]
Jiang Xingbo commented on SPARK-24552:
--
IIUC stageAttemptId + taskAttemptId shall probably define a unique task
attempt, and it carries enough information to know how many failed attempts you
had previously.
> Task attempt numbers are reused when stages are retried
> ---
>
> Key: SPARK-24552
> URL: https://issues.apache.org/jira/browse/SPARK-24552
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.1.1
>Reporter: Ryan Blue
>Priority: Major
>
> When stages are retried due to shuffle failures, task attempt numbers are
> reused. This causes a correctness bug in the v2 data sources write path.
> Data sources (both the original and v2) pass the task attempt to writers so
> that writers can use the attempt number to track and clean up data from
> failed or speculative attempts. In the v2 docs for DataWriterFactory, the
> attempt number's javadoc states that "Implementations can use this attempt
> number to distinguish writers of different task attempts."
> When two attempts of a stage use the same (partition, attempt) pair, two
> tasks can create the same data and attempt to commit. The commit coordinator
> prevents both from committing and will abort the attempt that finishes last.
> When using the (partition, attempt) pair to track data, the aborted task may
> delete data associated with the (partition, attempt) pair. If that happens,
> the data for the task that committed is also deleted as well, which is a
> correctness bug.
> For a concrete example, I have a data source that creates files in place
> named with {{part---.}}. Because these
> files are written in place, both tasks create the same file and the one that
> is aborted deletes the file, leading to data corruption when the file is
> added to the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried
[
https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511674#comment-16511674
]
Imran Rashid commented on SPARK-24552:
--
I wouldn't call this a bug in the scheduler, though I agree its definitely
confusing and not clearly documented at all. I think this has always been the
meaning of the attempt number. It might be confusing for other cases to change
its meaning now -- eg. its useful to know how many failed attempts you had
within one taskset, and some code out there might be using max(attemptNumber)
now as a proxy for that.
For sure we should improve the documentation.
[~markhamstra] [~tgraves] [~jiangxb1987] for some more opinions on the
scheduler side.
> Task attempt numbers are reused when stages are retried
> ---
>
> Key: SPARK-24552
> URL: https://issues.apache.org/jira/browse/SPARK-24552
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.1.1
>Reporter: Ryan Blue
>Priority: Major
>
> When stages are retried due to shuffle failures, task attempt numbers are
> reused. This causes a correctness bug in the v2 data sources write path.
> Data sources (both the original and v2) pass the task attempt to writers so
> that writers can use the attempt number to track and clean up data from
> failed or speculative attempts. In the v2 docs for DataWriterFactory, the
> attempt number's javadoc states that "Implementations can use this attempt
> number to distinguish writers of different task attempts."
> When two attempts of a stage use the same (partition, attempt) pair, two
> tasks can create the same data and attempt to commit. The commit coordinator
> prevents both from committing and will abort the attempt that finishes last.
> When using the (partition, attempt) pair to track data, the aborted task may
> delete data associated with the (partition, attempt) pair. If that happens,
> the data for the task that committed is also deleted as well, which is a
> correctness bug.
> For a concrete example, I have a data source that creates files in place
> named with {{part---.}}. Because these
> files are written in place, both tasks create the same file and the one that
> is aborted deletes the file, leading to data corruption when the file is
> added to the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried
[
https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511593#comment-16511593
]
Apache Spark commented on SPARK-24552:
--
User 'rdblue' has created a pull request for this issue:
https://github.com/apache/spark/pull/21558
> Task attempt numbers are reused when stages are retried
> ---
>
> Key: SPARK-24552
> URL: https://issues.apache.org/jira/browse/SPARK-24552
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.1.1
>Reporter: Ryan Blue
>Priority: Major
>
> When stages are retried due to shuffle failures, task attempt numbers are
> reused. This causes a correctness bug in the v2 data sources write path.
> Data sources (both the original and v2) pass the task attempt to writers so
> that writers can use the attempt number to track and clean up data from
> failed or speculative attempts. In the v2 docs for DataWriterFactory, the
> attempt number's javadoc states that "Implementations can use this attempt
> number to distinguish writers of different task attempts."
> When two attempts of a stage use the same (partition, attempt) pair, two
> tasks can create the same data and attempt to commit. The commit coordinator
> prevents both from committing and will abort the attempt that finishes last.
> When using the (partition, attempt) pair to track data, the aborted task may
> delete data associated with the (partition, attempt) pair. If that happens,
> the data for the task that committed is also deleted as well, which is a
> correctness bug.
> For a concrete example, I have a data source that creates files in place
> named with {{part---.}}. Because these
> files are written in place, both tasks create the same file and the one that
> is aborted deletes the file, leading to data corruption when the file is
> added to the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried
[
https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16511590#comment-16511590
]
Ryan Blue commented on SPARK-24552:
---
cc [~vanzin], [~henryr], [~cloud_fan]
> Task attempt numbers are reused when stages are retried
> ---
>
> Key: SPARK-24552
> URL: https://issues.apache.org/jira/browse/SPARK-24552
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.1.1
>Reporter: Ryan Blue
>Priority: Major
>
> When stages are retried due to shuffle failures, task attempt numbers are
> reused. This causes a correctness bug in the v2 data sources write path.
> Data sources (both the original and v2) pass the task attempt to writers so
> that writers can use the attempt number to track and clean up data from
> failed or speculative attempts. In the v2 docs for DataWriterFactory, the
> attempt number's javadoc states that "Implementations can use this attempt
> number to distinguish writers of different task attempts."
> When two attempts of a stage use the same (partition, attempt) pair, two
> tasks can create the same data and attempt to commit. The commit coordinator
> prevents both from committing and will abort the attempt that finishes last.
> When using the (partition, attempt) pair to track data, the aborted task may
> delete data associated with the (partition, attempt) pair. If that happens,
> the data for the task that committed is also deleted as well, which is a
> correctness bug.
> For a concrete example, I have a data source that creates files in place
> named with {{part---.}}. Because these
> files are written in place, both tasks create the same file and the one that
> is aborted deletes the file, leading to data corruption when the file is
> added to the table.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
