[
https://issues.apache.org/jira/browse/IMPALA-10502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17453317#comment-17453317
]
ASF subversion and git services commented on IMPALA-10502:
----------------------------------------------------------
Commit 6e2134ebdf0771fff74e89bfd223ae24937cfadf in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=6e2134e ]
IMPALA-10886: Fix loss of createEventId for INSERT created partitions
Coordinator calls updateCatalog RPC to catalogd to finalize the INSERT
statement. Catalogd will create any new partitions and reload metadata
of the updated partitions (including new partitions).
After IMPALA-10502, each new partition has a createEventId which is used
to detect how "fresh" it is. If the event processor receives a
DROP_PARTITION event, it will compare its event id with the
createEventId of the partition. Only DROP_PARTITION events happen after
the createEventId will be evaluated.
There is a bug in CatalogOpExecutor#updateCatalog() that we loss the
createEventId of the new partitions. It should be used in the final call
of loadTableMetadata(). This bug causes intermittent failures in
TestReusePartitionMetadata.test_reuse_partition_meta. The last two DMLs
of the test is dropping a partition and then creating it back by an
INSERT. If the DROP_PARTITION event is processed after the INSERT
finishes, the partition will be dropped incorrectly.
Tests
- Ran the test 100 times locally. Without the fix, it fails in 10 runs.
Change-Id: I2622c28a5ce6084fc77f6ea475d2633445c7f8dd
Reviewed-on: http://gerrit.cloudera.org:8080/18066
Reviewed-by: Vihang Karajgaonkar <[email protected]>
Tested-by: Vihang Karajgaonkar <[email protected]>
> delayed 'Invalidated objects in cache' cause 'Table already exists'
> -------------------------------------------------------------------
>
> Key: IMPALA-10502
> URL: https://issues.apache.org/jira/browse/IMPALA-10502
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog, Clients, Frontend
> Affects Versions: Impala 3.4.0
> Reporter: Adriano
> Assignee: Vihang Karajgaonkar
> Priority: Critical
> Fix For: Impala 4.1.0
>
>
> In fast paced environment where the interval between the step 1 and 2 is # <
> 100ms (a simplified pipeline looks like):
> 0- catalog 'on demand' in use and disableHmsSync (enabled or disabled: no
> difference)
> 1- open session to coord A -> DROP TABLE X -> close session
> 2- open session to coord A -> CREATE TABLE X-> close session
> Results: the step -2- can fail with table already exist.
> During the internal investigation was discovered that IMPALA-9913 will
> regress the issue in almost all scenarios.
> However considering that the investigation are internally ongoing it is nice
> to have the event tracked also here.
> Once we are sure that IMPALA-9913 fix these events we can close this as
> duplicate, in alternative carry on the investigation.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
