[
https://issues.apache.org/jira/browse/HUDI-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17430326#comment-17430326
]
liyuanzhao435 edited comment on HUDI-2576 at 10/19/21, 6:31 AM:
----------------------------------------------------------------
Hoodie table says:
*LOG.info("Removing duplicate data files created due to spark retries before
committing. Paths=" + invalidDataPaths);*
however, the invalid path means have spececial extension: parquet, log, orc,
hfile
but my deleted file is end with the extension parquet, why the jobmanager
delete it? I can't understand it
was (Author: liyuanzhao435):
Hoodie table says:
*LOG.info("Removing duplicate data files created due to spark retries before
committing. Paths=" + invalidDataPaths);*
however, the invalid path means have spececial extension: parquet, log, orc,
hfile
bug my deleted file is end with the extension parquet, why the jobmanager
delete it? I can't understand it
> flink do checkpoint error because parquet file is missing
> ----------------------------------------------------------
>
> Key: HUDI-2576
> URL: https://issues.apache.org/jira/browse/HUDI-2576
> Project: Apache Hudi
> Issue Type: Bug
> Components: Flink Integration
> Affects Versions: 0.10.0
> Reporter: liyuanzhao435
> Priority: Major
> Labels: flink, hudi
> Fix For: 0.10.0
>
> Attachments: error.txt
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> hudi:0.10.0, flink 1.13.1
> some times when flink do checkpoint , error occurs, the error shows a hudi
> parquet file is missing (says file not exists) :
> *2021-10-19 09:20:03,796 INFO
> org.apache.hudi.io.storage.row.HoodieRowDataCreateHandle [] - start close
> hoodie row data*
> *2021-10-19 09:20:03,800 WARN org.apache.hadoop.hdfs.DataStreamer [] -
> DataStreamer Exception*
> *java.io.FileNotFoundException: File does not exist:
> /tmp/test_liyz2/aa/2ff301cc-8db2-478e-b707-e8f2327ba38f-0_0-1-4_20211019091917.parquet
> (inode 32234795) Holder DFSClient_NONMAPREDUCE_633610786_99 does not have
> any open files.*
> *at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2815)*
>
> detail see appendix
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
