deepakpanda93 commented on issue #8810: URL: https://github.com/apache/hudi/issues/8810#issuecomment-4844991653
This is fixed in Hudi 1.2.0. Root cause: Flink can leave intermediate/uncommitted data files prefixed with . (hidden). The file-system-view's partition listing loaded every file in the partition without checking it was a valid Hudi data/log file, so the Spark reader tried to read a stray .-prefixed (incomplete) parquet → is not a Parquet file. expected magic number. Fix: [#18047](https://github.com/apache/hudi/pull/18047) ("Filter stray files when loading partitions in AbstractTableFileSystemView") adds filterValidDataFiles() (using FSUtils.isDataFile()) so partition listing skips hidden/temp/.crc/non-conformant files. As a result, spark.read.format("hudi") no longer attempts to read these stray files. This is in release-1.2.0. On the Flink side, intermediate-file handling/cleanup was also improved (HUDI-1902 [#2949](https://github.com/apache/hudi/pull/2949), HUDI-5223 partial failover). As @danny0405 noted at the time, older releases had no automatic handling (the workaround was deleting the hidden files manually). On 1.2.0+ the reader filters them out automatically. Closing as fixed in 1.2.0. If you still hit is not a Parquet file reading a Hudi table on 1.2.0+, please reopen with the offending file name and how the table was written. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
