hantiantian created SPARK-25477:
-----------------------------------
Summary: “INSERT OVERWRITE LOCAL DIRECTORY”， the data files
allocated on the non-driver node will not be written to the specified output
directory
Key: SPARK-25477
URL: https://issues.apache.org/jira/browse/SPARK-25477
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.3.1, 2.3.0
Reporter: hantiantian
The "INSERT OVERWRITE LOCAL DIRECTORY" features use the local staging directory
to load data into the specified output directory. If the executor is executed
on a non-driver node, the data files allocated on that node will not be written
to the specified output directory.
The "spark-sql" application has two executors, one on the driver node and one
on the non-driver node.
spark-sql> select * from aa;
1 10000
2 30000
2 10000
3 30000
As follows, view the data file of the aa table on HDFS,
/spark/aa/part-00000-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
/spark/aa/part-00001-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
/spark/aa/part-00002-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
/spark/aa/part-00003-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
spark-sql> insert overwrite local directory "/test" (select * from aa);
As follows, view the contents of the output directory "/test", there are only
two data files,
part-00000-16312194-288a-4085-9cb8-de0a685f7af0-c000
part-00002-16312194-288a-4085-9cb8-de0a685f7af0-c000
_SUCCESS
As follows, view the executor log on non-driver node, the other two data files
is Is assigned to this node. And the execution results of the tasks is saved to
the local staging directory.
INFO org.apache.spark.rdd.HadoopRDD: Input split:
hdfs://nameservice/spark/aa/part-00001-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000:0+8
INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output
of task 'attempt_20180920151705_0002_m_000001_0' to
file:/spark-tmp/stagedir/.hive-staging_hive_2018-09-20_15-17-01_313_4124694630008666159-1/-ext-10000/_temporary/0/task_20180920151705_0002_m_000001
INFO org.apache.spark.rdd.HadoopRDD: Input split:
hdfs://nameservice/spark/aa/part-00003-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000:0+8
INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output
of task 'attempt_20180920151705_0002_m_000003_0' to
file:/spark-tmp/stagedir/.hive-staging_hive_2018-09-20_15-17-01_313_4124694630008666159-1/-ext-10000/_temporary/0/task_20180920151705_0002_m_000003
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
