HuangZhenQiu commented on issue #19207:
URL: https://github.com/apache/hudi/issues/19207#issuecomment-4949260339
After investigation. the root cause is that the underneath file system is
not able to return the right number of flles. The issue happens when the flink
ingestion job write to 10k base file groups without quick log compaction for
MDT. There are 100k+ log file created after several days. In this situation,
the number of files returned by getAllFilesInPartition is not exactly the
number in remote storage. It could be a bug in file system implementation
rather than the issue in hudi.
tableMetadata.getAllFilesInPartition(partitionPath)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
