Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21663 )
Change subject: IMPALA-13126: Obtain table read lock in EP to process partitioned event ...................................................................... Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/21663/3/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java File fe/src/main/java/org/apache/impala/catalog/FeFsTable.java: http://gerrit.cloudera.org:8080/#/c/21663/3/fe/src/main/java/org/apache/impala/catalog/FeFsTable.java@628 PS3, Line 628: if (isPartitionReload) { It might not be worth adding a parameter just for tests. I think we can try using a table with many partitions to make this for-loop slow so concurrent INSERT statements can modify the partition map. I tried a custom-cluster test like this and it works in my env (failed before the fix, succeeded after the fix): @CustomClusterTestSuite.with_args( catalogd_args="--enable_reload_events=true " "--invalidate_metadata_on_event_processing_failure=false") def test_reload_events_modified_concurrently(self): """IMPALA-13126: This test verifies that the event processor successfully consumes or ignores the RELOAD event triggered by refresh operation on a partitioned table if the partitions are modified concurrently on the table.""" tbl = "scale_db.num_partitions_1234_blocks_per_partition_1" refresh_stmt = "refresh {} partition(j=0)".format(tbl) for i in range(32): self.client.execute_async(refresh_stmt) for i in range(100): self.client.execute( "alter table {} add if not exists partition(j=-1)".format(tbl, i, i)) self.client.execute( "alter table {} drop partition(j=-1)".format(tbl, i, i)) try: EventProcessorUtils.wait_for_event_processing(self, 1000) # bigger timeout required assert EventProcessorUtils.get_event_processor_status() == "ACTIVE" finally: # Make sure the table doesn't change after this test self.execute_query("alter table {} drop if exists partition(j=-1)".format(tbl)) -- To view, visit http://gerrit.cloudera.org:8080/21663 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26933f98556736f66df986f9440ebb64be395bc1 Gerrit-Change-Number: 21663 Gerrit-PatchSet: 3 Gerrit-Owner: Sai Hemanth Gantasala <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]> Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]> Gerrit-Comment-Date: Mon, 16 Dec 2024 04:07:02 +0000 Gerrit-HasComments: Yes
