[
https://issues.apache.org/jira/browse/SPARK-50595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tianqi Wan updated SPARK-50595:
-------------------------------
Attachment: Snipaste_2024-12-17_10-38-42.png
> Temp file storing state doesn't renamed to delta file, but was read by next
> task
> --------------------------------------------------------------------------------
>
> Key: SPARK-50595
> URL: https://issues.apache.org/jira/browse/SPARK-50595
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 3.2.0
> Environment: spark on yarn
> Reporter: Tianqi Wan
> Priority: Major
> Attachments: Snipaste_2024-12-17_10-38-42.png
>
>
> Structured streaming job failed due to delta file does not exist
> !image-2024-12-17-10-35-18-053.png!
>
> we checked the access log for the missing file, and found that the file was
> never created. And we found that the correspond tmp file exists whose file
> name is
> [.506.delta.936315d9-b58b-4f18-b3ed-b413cd92646f.TID3368893.tmp|https://www.cosmos09.osdinfra.net/cosmos/searchDM/raw/AnaheimRawLogs/Anaheim/RealTimeCheckpoints/Execution0/20241215_application_1733980441401_12313/state/0/1721/.506.delta.936315d9-b58b-4f18-b3ed-b413cd92646f.TID3368893.tmp?property=info]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
