[ https://issues.apache.org/jira/browse/HUDI-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Danny Chen resolved HUDI-2923. ------------------------------ > Unable to read from metadata table when a compaction is in progress or has > failed > --------------------------------------------------------------------------------- > > Key: HUDI-2923 > URL: https://issues.apache.org/jira/browse/HUDI-2923 > Project: Apache Hudi > Issue Type: Bug > Reporter: Prashant Wason > Assignee: Prashant Wason > Priority: Blocker > Labels: pull-request-available > Fix For: 0.10.0 > > > When reading from metadata table, the [readers are opened with the latest > file > slices|[https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java#L248].] > > When a compaction is in progress, the latest file slice does not have any > base-file or log file (yet). Hence we are unable to read data from the > metadata table. > > There are two cases here: > # Compaction eventually completes: We will be able to read data from the > metadata table. > # Compaction fails: We will not be able to read data unless the next time > compaction runs. This can be a fatal issue if the next writer tries to > perform an update which requires listing partition from the metadata table. > > Relevant logs from a unit test: > > 13084 [main] INFO > org.apache.hudi.common.table.view.AbstractTableFileSystemView - *Pending > Compaction instant for* (FileSlice > \{fileGroupId=HoodieFileGroupId{partitionPath='files', > fileId='2733a6ef-4bfd-444d-91bd-b42c3b66a84e-0'}, baseCommitTime=002001, > baseFile='null', logFiles='[]'}) is > :Option\{val=(002001,CompactionOperation{baseInstantTime='001', > dataFileCommitTime=Option{val=001}, > deltaFileNames=[.2733a6ef-4bfd-444d-91bd-b42c3b66a84e-0_001.log.1_0-34-36], > dataFileName=Option\{val=2733a6ef-4bfd-444d-91bd-b42c3b66a84e-0_0-16-20_001.hfile}, > id='HoodieFileGroupId\{partitionPath='files', > fileId='2733a6ef-4bfd-444d-91bd-b42c3b66a84e-0'}', metrics={}, > bootstrapFilePath=Optional.empty})} > 13084 [main] INFO > org.apache.hudi.common.table.view.AbstractTableFileSystemView - File Slice > (FileSlice \{fileGroupId=HoodieFileGroupId{partitionPath='files', > fileId='2733a6ef-4bfd-444d-91bd-b42c3b66a84e-0'}, *baseCommitTime=002001, > baseFile='null', logFiles='[]'}) is in pending compaction* > > 13089 [main] INFO > org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - *Number of > log files scanned => 0* > 13089 [main] INFO > org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - > MaxMemoryInBytes allowed for compaction => 0 > 13089 [main] INFO > org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - Number of > entries in MemoryBasedMap in ExternalSpillableMap => 0 > 13089 [main] INFO > org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - Total size > in bytes of MemoryBasedMap in ExternalSpillableMap => 0 > 13089 [main] INFO > org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - Number of > entries in DiskBasedMap in ExternalSpillableMap => 0 > 13089 [main] INFO > org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner - Size of file > spilled to disk => 0 > 13089 [main] INFO org.apache.hudi.metadata.HoodieBackedTableMetadata - > *Opened metadata log files from []* at instant (dataset instant=002, metadata > instant=002) in 2 ms > 13089 [main] INFO org.apache.hudi.metadata.HoodieBackedTableMetadata - > Metadata read for key __all_partitions__ took [baseFileRead, logMerge] [0, 0] > ms > 13090 [main] INFO org.apache.hudi.metadata.BaseTableMetadata - *Listed > partitions from metadata: #partitions=0* -- This message was sent by Atlassian Jira (v8.20.1#820001)