GitHub user fangshil opened a pull request:
https://github.com/apache/spark/pull/20931
[SPARK-23815][Core]Spark writer dynamic partition overwrite mode may fail
to write output on multi level partition
## What changes were proposed in this pull request?
Spark introduced new writer mode to overwrite only related partitions in
SPARK-20236. While we are using this feature in our production cluster, we
found a bug when writing multi-level partitions on HDFS.
A simple test case to reproduce this issue:
val df = Seq(("1","2","3")).toDF("col1", "col2","col3")
df.write.partitionBy("col1","col2").mode("overwrite").save("/my/hdfs/location")
If HDFS location "/my/hdfs/location" does not exist, there will be no
output.
This seems to be caused by the job commit change in SPARK-20236 in
HadoopMapReduceCommitProtocol.
In the commit job process, the output has been written into staging dir
/my/hdfs/location/.spark-staging.xxx/col1=1/col2=2, and then the code calls
fs.rename to rename /my/hdfs/location/.spark-staging.xxx/col1=1/col2=2 to
/my/hdfs/location/col1=1/col2=2. However, in our case the operation will fail
on HDFS because /my/hdfs/location/col1=1 does not exists. HDFS rename can not
create directory for more than one level.
This does not happen in unit test covered with SPARK-20236 with local file
system.
We are proposing a fix. When cleaning current partition dir
/my/hdfs/location/col1=1/col2=2 before the rename op, if the delete op fails
(because /my/hdfs/location/col1=1/col2=2 may not exist), we call mkdirs op to
create the parent dir /my/hdfs/location/col1=1 (if the parent dir does not
exist) so the following rename op can succeed.
## How was this patch tested?
We have tested this patch on our production cluster and it fixed the problem
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/fangshil/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20931.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20931
----
commit da63c17d7ae7fbf04cc474d946d61a098b3e1ade
Author: Fangshi Li <fli@...>
Date: 2018-03-28T04:25:54Z
Spark writer dynamic partition overwrite mode may fail to write output on
multi level partition
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
