[
https://issues.apache.org/jira/browse/HUDI-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-2952:
--------------------------------------
Description:
Looks like the conversion of empty partition path to "." (dot) is not fixed in
all places in metadata table. like clean metadata and rollback metadtata.
[link|https://github.com/apache/hudi/blob/e8473b9a2b5bf0ad9370377899f6a7ea4d1ceba1/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java#L133].
{code:java}
Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to merge old
record into new file for
key from old file
s3a://testing-427581a9-041b-4bb3-a4dc-28afab79f675/testing_default/ghCfAvroIssueCommen
tEvent/.hoodie/metadata/files/files-0000_0-209-274_20211206213223485001.hfile
to new file s3a://testing-4
27581a9-041b-4bb3-a4dc-28afab79f675/testing_default/ghCfAvroIssueCommentEvent/.hoodie/metadata/files/file
s-0000_0-50-62_20211206213724027001.hfile with writerSchema {
"type" : "record",
"name" : "HoodieMetadataRecord",
"namespace" : "org.apache.hudi.avro.model",
"doc" : "A record saved within the Metadata Table",
"fields" : [ {
"name" : "_hoodie_commit_time",
"type" : [ "null", "string" ],
"doc" : "",
"default" : null
}, {
"name" : "_hoodie_commit_seqno",
"type" : [ "null", "string" ],
"doc" : "",
"default" : null
}, {
"name" : "_hoodie_record_key",
"type" : [ "null", "string" ],
"doc" : "",
"default" : null
}, {
.
.
.
at
org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:356)
at
org.apache.hudi.io.HoodieSortedMergeHandle.write(HoodieSortedMergeHandle.java:105)
at
org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:122)
at
org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:112)
at
org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
at
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.lang.IllegalArgumentException: key length must be > 0
at org.apache.hadoop.util.bloom.HashFunction.hash(HashFunction.java:114)
at org.apache.hadoop.util.bloom.BloomFilter.add(BloomFilter.java:122)
at
org.apache.hudi.common.bloom.InternalDynamicBloomFilter.add(InternalDynamicBloomFilter.java:94)
at
org.apache.hudi.common.bloom.HoodieDynamicBoundedBloomFilter.add(HoodieDynamicBoundedBloomFilter.java:81)
at
org.apache.hudi.io.storage.HoodieHFileWriter.writeAvro(HoodieHFileWriter.java:130)
at
org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:351)
... 9 more
{code}
was:
{code:java}
Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to merge old
record into new file for
key from old file
s3a://testing-427581a9-041b-4bb3-a4dc-28afab79f675/testing_default/ghCfAvroIssueCommen
tEvent/.hoodie/metadata/files/files-0000_0-209-274_20211206213223485001.hfile
to new file s3a://testing-4
27581a9-041b-4bb3-a4dc-28afab79f675/testing_default/ghCfAvroIssueCommentEvent/.hoodie/metadata/files/file
s-0000_0-50-62_20211206213724027001.hfile with writerSchema {
"type" : "record",
"name" : "HoodieMetadataRecord",
"namespace" : "org.apache.hudi.avro.model",
"doc" : "A record saved within the Metadata Table",
"fields" : [ {
"name" : "_hoodie_commit_time",
"type" : [ "null", "string" ],
"doc" : "",
"default" : null
}, {
"name" : "_hoodie_commit_seqno",
"type" : [ "null", "string" ],
"doc" : "",
"default" : null
}, {
"name" : "_hoodie_record_key",
"type" : [ "null", "string" ],
"doc" : "",
"default" : null
}, {
.
.
.
at
org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:356)
at
org.apache.hudi.io.HoodieSortedMergeHandle.write(HoodieSortedMergeHandle.java:105)
at
org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:122)
at
org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:112)
at
org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
at
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.lang.IllegalArgumentException: key length must be > 0
at org.apache.hadoop.util.bloom.HashFunction.hash(HashFunction.java:114)
at org.apache.hadoop.util.bloom.BloomFilter.add(BloomFilter.java:122)
at
org.apache.hudi.common.bloom.InternalDynamicBloomFilter.add(InternalDynamicBloomFilter.java:94)
at
org.apache.hudi.common.bloom.HoodieDynamicBoundedBloomFilter.add(HoodieDynamicBoundedBloomFilter.java:81)
at
org.apache.hudi.io.storage.HoodieHFileWriter.writeAvro(HoodieHFileWriter.java:130)
at
org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:351)
... 9 more
{code}
> Metadata table compaction fails for non partitioned dataset
> -----------------------------------------------------------
>
> Key: HUDI-2952
> URL: https://issues.apache.org/jira/browse/HUDI-2952
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Writer Core
> Affects Versions: 0.10.0
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Major
>
> Looks like the conversion of empty partition path to "." (dot) is not fixed
> in all places in metadata table. like clean metadata and rollback metadtata.
> [link|https://github.com/apache/hudi/blob/e8473b9a2b5bf0ad9370377899f6a7ea4d1ceba1/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java#L133].
>
> {code:java}
> Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to merge
> old record into new file for
> key from old file
> s3a://testing-427581a9-041b-4bb3-a4dc-28afab79f675/testing_default/ghCfAvroIssueCommen
> tEvent/.hoodie/metadata/files/files-0000_0-209-274_20211206213223485001.hfile
> to new file s3a://testing-4
> 27581a9-041b-4bb3-a4dc-28afab79f675/testing_default/ghCfAvroIssueCommentEvent/.hoodie/metadata/files/file
> s-0000_0-50-62_20211206213724027001.hfile with writerSchema {
> "type" : "record",
> "name" : "HoodieMetadataRecord",
> "namespace" : "org.apache.hudi.avro.model",
> "doc" : "A record saved within the Metadata Table",
> "fields" : [ {
> "name" : "_hoodie_commit_time",
> "type" : [ "null", "string" ],
> "doc" : "",
> "default" : null
> }, {
> "name" : "_hoodie_commit_seqno",
> "type" : [ "null", "string" ],
> "doc" : "",
> "default" : null
> }, {
> "name" : "_hoodie_record_key",
> "type" : [ "null", "string" ],
> "doc" : "",
> "default" : null
> }, {
> .
> .
> .
> at
> org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:356)
> at
> org.apache.hudi.io.HoodieSortedMergeHandle.write(HoodieSortedMergeHandle.java:105)
> at
> org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:122)
> at
> org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:112)
> at
> org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
> at
> org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.lang.IllegalArgumentException: key length must be > 0
> at
> org.apache.hadoop.util.bloom.HashFunction.hash(HashFunction.java:114)
> at org.apache.hadoop.util.bloom.BloomFilter.add(BloomFilter.java:122)
> at
> org.apache.hudi.common.bloom.InternalDynamicBloomFilter.add(InternalDynamicBloomFilter.java:94)
> at
> org.apache.hudi.common.bloom.HoodieDynamicBoundedBloomFilter.add(HoodieDynamicBoundedBloomFilter.java:81)
> at
> org.apache.hudi.io.storage.HoodieHFileWriter.writeAvro(HoodieHFileWriter.java:130)
> at
> org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:351)
> ... 9 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
