[
https://issues.apache.org/jira/browse/IMPALA-10886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452187#comment-17452187
]
Yu-Wen Lai edited comment on IMPALA-10886 at 12/2/21, 5:42 AM:
---------------------------------------------------------------
Good find. Now that it is an issue related to event processor, I suppose the
catalogd log from a failed job might be helpful.
{code:java}
I1130 21:42:29.674419 116548 HdfsTable.java:1197] Reloading metadata for 1
partition(s) of test_reuse_partition_meta_8304193a.alltypes (INSERT)
I1130 21:42:29.674573 116548 HdfsTable.java:2904] Load Valid Write Id List
Done. Time taken: 7.260us
I1130 21:42:29.689234 116548 HdfsTable.java:783] Loaded file and block metadata
for test_reuse_partition_meta_8304193a.alltypes partitions: year=2009/month=1.
Time taken: 1.753ms
I1130 21:42:29.689532 116548 HdfsTable.java:783] Loaded file and block metadata
for test_reuse_partition_meta_8304193a.alltypes partitions: . Time taken:
69.852us
I1130 21:42:29.689572 116548 HdfsTable.java:1234] Incrementally loaded table
metadata for: test_reuse_partition_meta_8304193a.alltypes
I1130 21:42:29.698971 115115 MetastoreEvents.java:534] EventId: 41653
EventType: ADD_PARTITION Incremented skipped metric to 3
I1130 21:42:29.700379 115115 MetastoreEvents.java:523] EventId: 41654
EventType: ALTER_TABLE Incremented events skipped counter to 4
I1130 21:42:29.700445 115115 MetastoreEvents.java:523] EventId: 41654
EventType: ALTER_TABLE Not processing the event as it is a self-event
I1130 21:42:29.700765 115115 CatalogOpExecutor.java:4083] EventId: 41655
Skipping removal of 0/1 partitions since they don't exist orwere created later
in table test_reuse_partition_meta_8304193a.alltypes.
I1130 21:42:29.700970 115115 MetastoreEvents.java:523] EventId: 41655
EventType: DROP_PARTITION 1 partitions dropped from table
test_reuse_partition_meta_8304193a.alltypes {code}
It looks like the drop event was received after it handled insert and then 1
partition was dropped.
was (Author: hsnusonic):
Good find. Now that it is an issue related to event processor, I suppose the
catalogd log from a failed job might be helpful.
{code:java}
I1130 21:42:29.674419 116548 HdfsTable.java:1197] Reloading metadata for 1
partition(s) of test_reuse_partition_meta_8304193a.alltypes (INSERT)
I1130 21:42:29.674573 116548 HdfsTable.java:2904] Load Valid Write Id List
Done. Time taken: 7.260us
I1130 21:42:29.689234 116548 HdfsTable.java:783] Loaded file and block metadata
for test_reuse_partition_meta_8304193a.alltypes partitions: year=2009/month=1.
Time taken: 1.753ms
I1130 21:42:29.689532 116548 HdfsTable.java:783] Loaded file and block metadata
for test_reuse_partition_meta_8304193a.alltypes partitions: . Time taken:
69.852us
I1130 21:42:29.689572 116548 HdfsTable.java:1234] Incrementally loaded table
metadata for: test_reuse_partition_meta_8304193a.alltypes
I1130 21:42:29.698971 115115 MetastoreEvents.java:534] EventId: 41653
EventType: ADD_PARTITION Incremented skipped metric to 3
I1130 21:42:29.700379 115115 MetastoreEvents.java:523] EventId: 41654
EventType: ALTER_TABLE Incremented events skipped counter to 4
I1130 21:42:29.700445 115115 MetastoreEvents.java:523] EventId: 41654
EventType: ALTER_TABLE Not processing the event as it is a self-event
I1130 21:42:29.700765 115115 CatalogOpExecutor.java:4083] EventId: 41655
Skipping removal of 0/1 partitions since they don't exist orwere created later
in table test_reuse_partition_meta_8304193a.alltypes.
I1130 21:42:29.700970 115115 MetastoreEvents.java:523] EventId: 41655
EventType: DROP_PARTITION 1 partitions dropped from table
test_reuse_partition_meta_8304193a.alltypes {code}
It looks like the drop event was received after it handled insert and then 1
partition is dropped.
> TestReusePartitionMetadata.test_reuse_partition_meta fails
> ----------------------------------------------------------
>
> Key: IMPALA-10886
> URL: https://issues.apache.org/jira/browse/IMPALA-10886
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
> Labels: broken-build
> Attachments: test_local_catalog.patch
>
>
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/14670/testReport/junit/custom_cluster.test_local_catalog/TestReusePartitionMetadata/test_reuse_partition_meta/
> {code}
> custom_cluster/test_local_catalog.py:586: in test_reuse_partition_meta
> self.check_missing_partitions(unique_database, 1)
> custom_cluster/test_local_catalog.py:595: in check_missing_partitions
> assert match.group(1) == str(partition_misses)
> E assert '0' == '1'
> E - 0
> E + 1
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
