[
https://issues.apache.org/jira/browse/SPARK-54855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zuotingbing updated SPARK-54855:
--------------------------------
Affects Version/s: 3.4.1
(was: 3.5.6)
> FileNotFoundException on _temporary directory
> ---------------------------------------------
>
> Key: SPARK-54855
> URL: https://issues.apache.org/jira/browse/SPARK-54855
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.4.1
> Reporter: zuotingbing
> Priority: Major
> Attachments: image-2025-12-27-15-23-05-849.png
>
>
> My use case like this:
> spark-sql> CREATE TABLE test_parquet (a INT, b INT, c INT, d INT) USING
> PARQUET PARTITIONED BY (b, c);
> INSERT overwrite TABLE test_parquet PARTITION(b=1, c=1) SELECT 1,2; // app1
> INSERT overwrite TABLE test_parquet PARTITION(b=2, c=2) SELECT 3,4; // app2
> Two apps are writing to the same table but different partitions
> simultaneously. Then, when they commited job/task, they both cleans up
> $dest/_temporary dir by Hadoop FileOutputCommitter and they cause the
> problem. We definitely want to do better for this case.
>
> !image-2025-12-27-15-23-05-849.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
