Balaji Varadarajan created HUDI-1309:
----------------------------------------
Summary: Listing Metadata unreadable in S3 as the log block is
deemed corrupted
Key: HUDI-1309
URL: https://issues.apache.org/jira/browse/HUDI-1309
Project: Apache Hudi
Issue Type: Sub-task
Components: Writer Core
Reporter: Balaji Varadarajan
When running metadata list-partitions CLI command, I am seeing the below
messages and the partition list is empty. Was expecting 10K partitions.
36589 [Spring Shell] INFO
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner - Scanning log
file
HoodieLogFile\{pathStr='s3a://robinhood-encrypted-hudi-data-cove/dev_hudi_tables/balaji_varadarajan/benchmark_1M_10K_partitions/.hoodie/metadata/metadata_partition/.f02585bd-bb02-43f6-8bc8-cec71df87d1e-0_00000000000000.log.1_0-23-206045',
fileLen=0}
36590 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFileReader
- Found corrupted block in file
HoodieLogFile\{pathStr='s3a://robinhood-encrypted-hudi-data-cove/dev_hudi_tables/balaji_varadarajan/benchmark_1M_10K_partitions/.hoodie/metadata/metadata_partition/.f02585bd-bb02-43f6-8bc8-cec71df87d1e-0_00000000000000.log.1_0-23-206045',
fileLen=0} with block size(3723305) running past EOF
36684 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFileReader
- Log
HoodieLogFile\{pathStr='s3a://xxxx/dev_hudi_tables/balaji_varadarajan/benchmark_1M_10K_partitions/.hoodie/metadata/metadata_partition/.f02585bd-bb02-43f6-8bc8-cec71df87d1e-0_00000000000000.log.1_0-23-206045',
fileLen=0} has a corrupted block at 14
44515 [Spring Shell] INFO org.apache.hudi.common.table.log.HoodieLogFileReader
- Next available block in
HoodieLogFile\{pathStr='s3a://xxxx/dev_hudi_tables/balaji_varadarajan/benchmark_1M_10K_partitions/.hoodie/metadata/metadata_partition/.f02585bd-bb02-43f6-8bc8-cec71df87d1e-0_00000000000000.log.1_0-23-206045',
fileLen=0} starts at 3723319
44566 [Spring Shell] INFO
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner - Found a
corrupt block in
s3a://xxxx/dev_hudi_tables/balaji_varadarajan/benchmark_1M_10K_partitions/.hoodie/metadata/metadata_partition/.f02585bd-bb02-43f6-8bc8-cec71df87d1e-0_00000000000000.log.1_0-23-206045
44567 [Spring Shell] INFO
org.apache.hudi.common.table.log.AbstractHoodieLogRecordScanner - M
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
