Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/17380 )
Change subject: IMPALA-10692: Fix acid insert when event polling is enabled ...................................................................... Patch Set 3: (3 comments) Thanks for the review! I am unsure about how exactly event sending/processing should behave, this clearly needs more thought and work. It is possible that Hive used to send different events when event processing was implemented in Impala and may also change in the future, making my comments stale. Carrying +2 from Zoltan. http://gerrit.cloudera.org:8080/#/c/17380/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17380/2//COMMIT_MSG@24 PS2, Line 24: ACID inserts from Hive work differently, they always cause an : ALTER_TABLE or ALTER_PARTITION > Shouldn't Impala behave the same? My goal was to make replication work and it replication manager listens to COMMIT event AFAIK. I don't know whether we should send these ALTER events and whether we actually need to reload the table when we see them, e.g. a simply stat change shouldn't lead to table reload. http://gerrit.cloudera.org:8080/#/c/17380/2//COMMIT_MSG@29 PS2, Line 29: (before the commit is finished). > So if the following happens: Yes, I think that we should listen to the commit event. I didn't reproduce the bug above and I don't know how exactly Hive sends its events, so it is possible that this is not an issue, e.g. if Hive sends all these events during the commit and by the time we can get a new validWriteIdList the whole commit will be finished. Btw our code does seem to expect INSERT events for transactional tables: https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L793 http://gerrit.cloudera.org:8080/#/c/17380/2/tests/custom_cluster/test_event_processing.py File tests/custom_cluster/test_event_processing.py: http://gerrit.cloudera.org:8080/#/c/17380/2/tests/custom_cluster/test_event_processing.py@534 PS2, Line 534: writes to > "writes to"? Done -- To view, visit http://gerrit.cloudera.org:8080/17380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c2d0702232538a746410539ad55f87b7fde57e7 Gerrit-Change-Number: 17380 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Vihang Karajgaonkar <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Mon, 03 May 2021 17:23:58 +0000 Gerrit-HasComments: Yes
