[
https://issues.apache.org/jira/browse/SPARK-29037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16928674#comment-16928674
]
feiwang edited comment on SPARK-29037 at 9/12/19 3:58 PM:
----------------------------------------------------------
[~advancedxy]
Thanks for your reply.
I just checked the code, as shown below.
https://github.com/apache/spark/blob/c56a012bc839cd2f92c2be41faea91d1acfba4eb/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala#L105-L106
{code:java}
val dynamicPartitionOverwrite = enableDynamicOverwrite && mode ==
SaveMode.Overwrite &&
staticPartitions.size < partitionColumns.length
{code}
When partitionColumns==1, for the operation of inserting overwrite table
partition, dynamicPartitionOverwrite is always false even DynamicOverwrite is
enabled.
I will learn more about dynamic partition. Thanks for your suggestion.
was (Author: hzfeiwang):
[~advancedxy]
Thanks for your reply.
I just checked the code, as shown below.
https://github.com/apache/spark/blob/c56a012bc839cd2f92c2be41faea91d1acfba4eb/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala#L105-L106
{code:java}
val dynamicPartitionOverwrite = enableDynamicOverwrite && mode ==
SaveMode.Overwrite &&
staticPartitions.size < partitionColumns.length
{code}
When partitionColumns==1, for the operation of inserting overwrite table
partition, dynamicPartitionOverwrite is always false even DynamicOverwrite is
enabled.
> [Core] Spark gives duplicate result when an application was killed and rerun
> ----------------------------------------------------------------------------
>
> Key: SPARK-29037
> URL: https://issues.apache.org/jira/browse/SPARK-29037
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.0, 2.3.3
> Reporter: feiwang
> Priority: Major
> Attachments: screenshot-1.png
>
>
> When we insert overwrite a partition of table.
> For a stage, whose tasks commit output, a task saves output to a staging dir
> firstly, when this task complete, it will save output to committedTaskPath,
> when all tasks of this stage success, all task output under committedTaskPath
> will be moved to destination dir.
> However, when we kill an application, which is committing tasks' output,
> parts of tasks' results will be kept in committedTaskPath, which would not be
> cleared gracefully.
> Then we rerun this application and the new application will reuse this
> committedTaskPath dir.
> And when the task commit stage of new application success, all task output
> under this committedTaskPath, which contains parts of old application's task
> output , would be moved to destination dir and the result is duplicated.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
