[
https://issues.apache.org/jira/browse/HUDI-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-2952:
--------------------------------------
Description:
Looks like the conversion of empty partition path to "." (dot) is not fixed in
all places when we convert the records for metadata table. like clean metadata
and rollback metadata.
[link|https://github.com/apache/hudi/blob/e8473b9a2b5bf0ad9370377899f6a7ea4d1ceba1/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java#L133].
{code:java}
Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to merge old
record into new file for
key from old file
s3a://testing-427581a9-041b-4bb3-a4dc-28afab79f675/testing_default/ghCfAvroIssueCommen
tEvent/.hoodie/metadata/files/files-0000_0-209-274_20211206213223485001.hfile
to new file s3a://testing-4
27581a9-041b-4bb3-a4dc-28afab79f675/testing_default/ghCfAvroIssueCommentEvent/.hoodie/metadata/files/file
s-0000_0-50-62_20211206213724027001.hfile with writerSchema {
"type" : "record",
"name" : "HoodieMetadataRecord",
"namespace" : "org.apache.hudi.avro.model",
"doc" : "A record saved within the Metadata Table",
"fields" : [ {
"name" : "_hoodie_commit_time",
"type" : [ "null", "string" ],
"doc" : "",
"default" : null
}, {
"name" : "_hoodie_commit_seqno",
"type" : [ "null", "string" ],
"doc" : "",
"default" : null
}, {
"name" : "_hoodie_record_key",
"type" : [ "null", "string" ],
"doc" : "",
"default" : null
}, {
.
.
.
at
org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:356)
at
org.apache.hudi.io.HoodieSortedMergeHandle.write(HoodieSortedMergeHandle.java:105)
at
org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:122)
at
org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:112)
at
org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
at
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.lang.IllegalArgumentException: key length must be > 0
at org.apache.hadoop.util.bloom.HashFunction.hash(HashFunction.java:114)
at org.apache.hadoop.util.bloom.BloomFilter.add(BloomFilter.java:122)
at
org.apache.hudi.common.bloom.InternalDynamicBloomFilter.add(InternalDynamicBloomFilter.java:94)
at
org.apache.hudi.common.bloom.HoodieDynamicBoundedBloomFilter.add(HoodieDynamicBoundedBloomFilter.java:81)
at
org.apache.hudi.io.storage.HoodieHFileWriter.writeAvro(HoodieHFileWriter.java:130)
at
org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:351)
... 9 more
{code}
was:
Looks like the conversion of empty partition path to "." (dot) is not fixed in
all places in metadata table. like clean metadata and rollback metadtata.
[link|https://github.com/apache/hudi/blob/e8473b9a2b5bf0ad9370377899f6a7ea4d1ceba1/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java#L133].
{code:java}
Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to merge old
record into new file for
key from old file
s3a://testing-427581a9-041b-4bb3-a4dc-28afab79f675/testing_default/ghCfAvroIssueCommen
tEvent/.hoodie/metadata/files/files-0000_0-209-274_20211206213223485001.hfile
to new file s3a://testing-4
27581a9-041b-4bb3-a4dc-28afab79f675/testing_default/ghCfAvroIssueCommentEvent/.hoodie/metadata/files/file
s-0000_0-50-62_20211206213724027001.hfile with writerSchema {
"type" : "record",
"name" : "HoodieMetadataRecord",
"namespace" : "org.apache.hudi.avro.model",
"doc" : "A record saved within the Metadata Table",
"fields" : [ {
"name" : "_hoodie_commit_time",
"type" : [ "null", "string" ],
"doc" : "",
"default" : null
}, {
"name" : "_hoodie_commit_seqno",
"type" : [ "null", "string" ],
"doc" : "",
"default" : null
}, {
"name" : "_hoodie_record_key",
"type" : [ "null", "string" ],
"doc" : "",
"default" : null
}, {
.
.
.
at
org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:356)
at
org.apache.hudi.io.HoodieSortedMergeHandle.write(HoodieSortedMergeHandle.java:105)
at
org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:122)
at
org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:112)
at
org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
at
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.lang.IllegalArgumentException: key length must be > 0
at org.apache.hadoop.util.bloom.HashFunction.hash(HashFunction.java:114)
at org.apache.hadoop.util.bloom.BloomFilter.add(BloomFilter.java:122)
at
org.apache.hudi.common.bloom.InternalDynamicBloomFilter.add(InternalDynamicBloomFilter.java:94)
at
org.apache.hudi.common.bloom.HoodieDynamicBoundedBloomFilter.add(HoodieDynamicBoundedBloomFilter.java:81)
at
org.apache.hudi.io.storage.HoodieHFileWriter.writeAvro(HoodieHFileWriter.java:130)
at
org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:351)
... 9 more
{code}
> Metadata table compaction fails for non partitioned dataset
> -----------------------------------------------------------
>
> Key: HUDI-2952
> URL: https://issues.apache.org/jira/browse/HUDI-2952
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Writer Core
> Affects Versions: 0.10.0
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Major
>
> Looks like the conversion of empty partition path to "." (dot) is not fixed
> in all places when we convert the records for metadata table. like clean
> metadata and rollback metadata.
> [link|https://github.com/apache/hudi/blob/e8473b9a2b5bf0ad9370377899f6a7ea4d1ceba1/hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java#L133].
>
> {code:java}
> Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to merge
> old record into new file for
> key from old file
> s3a://testing-427581a9-041b-4bb3-a4dc-28afab79f675/testing_default/ghCfAvroIssueCommen
> tEvent/.hoodie/metadata/files/files-0000_0-209-274_20211206213223485001.hfile
> to new file s3a://testing-4
> 27581a9-041b-4bb3-a4dc-28afab79f675/testing_default/ghCfAvroIssueCommentEvent/.hoodie/metadata/files/file
> s-0000_0-50-62_20211206213724027001.hfile with writerSchema {
> "type" : "record",
> "name" : "HoodieMetadataRecord",
> "namespace" : "org.apache.hudi.avro.model",
> "doc" : "A record saved within the Metadata Table",
> "fields" : [ {
> "name" : "_hoodie_commit_time",
> "type" : [ "null", "string" ],
> "doc" : "",
> "default" : null
> }, {
> "name" : "_hoodie_commit_seqno",
> "type" : [ "null", "string" ],
> "doc" : "",
> "default" : null
> }, {
> "name" : "_hoodie_record_key",
> "type" : [ "null", "string" ],
> "doc" : "",
> "default" : null
> }, {
> .
> .
> .
> at
> org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:356)
> at
> org.apache.hudi.io.HoodieSortedMergeHandle.write(HoodieSortedMergeHandle.java:105)
> at
> org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:122)
> at
> org.apache.hudi.table.action.commit.AbstractMergeHelper$UpdateHandler.consumeOneRecord(AbstractMergeHelper.java:112)
> at
> org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37)
> at
> org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.lang.IllegalArgumentException: key length must be > 0
> at
> org.apache.hadoop.util.bloom.HashFunction.hash(HashFunction.java:114)
> at org.apache.hadoop.util.bloom.BloomFilter.add(BloomFilter.java:122)
> at
> org.apache.hudi.common.bloom.InternalDynamicBloomFilter.add(InternalDynamicBloomFilter.java:94)
> at
> org.apache.hudi.common.bloom.HoodieDynamicBoundedBloomFilter.add(HoodieDynamicBoundedBloomFilter.java:81)
> at
> org.apache.hudi.io.storage.HoodieHFileWriter.writeAvro(HoodieHFileWriter.java:130)
> at
> org.apache.hudi.io.HoodieMergeHandle.write(HoodieMergeHandle.java:351)
> ... 9 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
