[
https://issues.apache.org/jira/browse/SPARK-25477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623877#comment-16623877
]
Apache Spark commented on SPARK-25477:
--------------------------------------
User 'httfighter' has created a pull request for this issue:
https://github.com/apache/spark/pull/22487
> “INSERT OVERWRITE LOCAL DIRECTORY”， the data files allocated on the
> non-driver node will not be written to the specified output directory
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-25477
> URL: https://issues.apache.org/jira/browse/SPARK-25477
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.3.0, 2.3.1
> Reporter: hantiantian
> Priority: Major
>
> The "INSERT OVERWRITE LOCAL DIRECTORY" features use the local staging
> directory to load data into the specified output directory. If the executor
> is executed on a non-driver node, the data files allocated on that node will
> not be written to the specified output directory.
> The "spark-sql" application has two executors, one on the driver node and one
> on the non-driver node.
> spark-sql> select * from aa;
> 1 10000
> 2 30000
> 2 10000
> 3 30000
> As follows, view the data file of the aa table on HDFS,
> /spark/aa/part-00000-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
> /spark/aa/part-00001-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
> /spark/aa/part-00002-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
> /spark/aa/part-00003-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000
>
> spark-sql> insert overwrite local directory "/test" (select * from aa);
> As follows, view the contents of the output directory "/test", there are only
> two data files,
> part-00000-16312194-288a-4085-9cb8-de0a685f7af0-c000
> part-00002-16312194-288a-4085-9cb8-de0a685f7af0-c000
> _SUCCESS
> As follows, view the executor log on non-driver node, the other two data
> files is Is assigned to this node. And the execution results of the tasks is
> saved to the local staging directory.
> INFO org.apache.spark.rdd.HadoopRDD: Input split:
> hdfs://nameservice/spark/aa/part-00001-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000:0+8
> INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved
> output of task 'attempt_20180920151705_0002_m_000001_0' to
> file:/spark-tmp/stagedir/.hive-staging_hive_2018-09-20_15-17-01_313_4124694630008666159-1/-ext-10000/_temporary/0/task_20180920151705_0002_m_000001
>
> INFO org.apache.spark.rdd.HadoopRDD: Input split:
> hdfs://nameservice/spark/aa/part-00003-5adc7acc-6e2d-407a-be77-abdfce61e55f-c000:0+8
> INFO org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output
> of task 'attempt_20180920151705_0002_m_000003_0' to
> file:/spark-tmp/stagedir/.hive-staging_hive_2018-09-20_15-17-01_313_4124694630008666159-1/-ext-10000/_temporary/0/task_20180920151705_0002_m_000003
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
