[
https://issues.apache.org/jira/browse/SPARK-54855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18064301#comment-18064301
]
JieFang.He commented on SPARK-54855:
------------------------------------
Based on the information I found, the {{INSERT OVERWRITE PARTITION}} syntax in
Spark is designed to be compatible with Hive. In Hive, the implementation is
partition-level, meaning the {{_temporary}} directory is created under the
target partition. This design seems more reasonable for the partition overwrite
semantics. In Spark, however, {{_temporary}} is placed at the table level. For
this syntax, wouldn’t a partition-level implementation be more appropriate? Why
must {{_temporary}} be table-level instead of partition-level?
> FileNotFoundException on _temporary directory
> ---------------------------------------------
>
> Key: SPARK-54855
> URL: https://issues.apache.org/jira/browse/SPARK-54855
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.4.1
> Reporter: zuotingbing
> Priority: Major
> Attachments: image-2025-12-27-15-23-05-849.png,
> image-2025-12-29-16-28-32-754.png
>
>
> My use case like this:
> spark-sql> CREATE TABLE test_parquet (a INT, b INT, c INT, d INT) USING
> PARQUET PARTITIONED BY (b, c);
> INSERT overwrite TABLE test_parquet PARTITION(b=1, c=1) SELECT 1,2; // app1
> INSERT overwrite TABLE test_parquet PARTITION(b=2, c=2) SELECT 3,4; // app2
> Two apps are writing to the same table but different partitions
> simultaneously. Then, when they commited job/task, they both cleans up
> $dest/_temporary dir by Hadoop FileOutputCommitter and they cause the
> problem. We definitely want to do better for this case.
>
> !image-2025-12-27-15-23-05-849.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
