[
https://issues.apache.org/jira/browse/IMPALA-10886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17452389#comment-17452389
]
Quanlong Huang commented on IMPALA-10886:
-----------------------------------------
Yeah, this issue happens when the INSERT finishes before the DROP_PARTITION
event is processed.
I think there are two approaches to fix this. One is detecting the
DROP_PARTITION event as a self-event and skip it. The other way is make sure
the new partition has a larger createEventId so the DROP_PARTITION event won't
be processed:
{code:java}
private boolean canDropPartitionFromEvent(long eventId, HdfsTable hdfsTable,
List<String> values) throws CatalogException {
...
// if the partition has been created since the event was generated, skip
// dropping the event.
if (hdfsPartition.getCreateEventId() > eventId) { // The CreateEventId
is -1 in this case.
LOG.info("Not dropping partition {} of table {} since it's create event
id {} is "
+ "higher than eventid {}", hdfsPartition.getPartitionName(),
hdfsTable.getFullName(), hdfsPartition.getCreateEventId(), eventId);
return false;
}
return true;
}
{code}
In this case, the partition is reloaded in executing updateCatalog() request
for the INSERT. Its CreateEventId is -1 so the check fail.
[~vihangk1], what do you think?
> TestReusePartitionMetadata.test_reuse_partition_meta fails
> ----------------------------------------------------------
>
> Key: IMPALA-10886
> URL: https://issues.apache.org/jira/browse/IMPALA-10886
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Critical
> Labels: broken-build
> Attachments: test_local_catalog.patch
>
>
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/14670/testReport/junit/custom_cluster.test_local_catalog/TestReusePartitionMetadata/test_reuse_partition_meta/
> {code}
> custom_cluster/test_local_catalog.py:586: in test_reuse_partition_meta
> self.check_missing_partitions(unique_database, 1)
> custom_cluster/test_local_catalog.py:595: in check_missing_partitions
> assert match.group(1) == str(partition_misses)
> E assert '0' == '1'
> E - 0
> E + 1
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
