[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Jason Fehr has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. IMPALA-14949: Fix Catalogd Startup Deadlock The getOrLoadTable functionality now waits for the initial database metadata to be loaded for the given database before it takes the versionLock_ read lock. This order of operations is reversed from its previous ordering. There was a deadlock condition during a slow Catalogd startup because the thread that performs the initial global reset periodically releases the versionLock_ write lock to allow for metadata operations on already loaded databases to proceed. If the database was not yet loaded, the read thread waited until it was loaded holding a read lock on versionLock_. However, the initial global reset thread could not proceed with loading any databases because it needed a write lock on versionLock_. Testing accomplished with a new custom cluster test. This new test was run on a local Impala dev machine without the fix in place to demonstrate it consistently failed because of the deadlock. It was also successfully run with the fix in place both on a local Impala dev machine and in an automated build. Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Reviewed-on: http://gerrit.cloudera.org:8080/24275 Reviewed-by: Yida Wu Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M tests/custom_cluster/test_catalog_wait.py 2 files changed, 137 insertions(+), 11 deletions(-) Approvals: Yida Wu: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 10 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 9 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 13 May 2026 04:14:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 9: > Patch Set 9: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13458/ Build failed with known error downloading artifacts. -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 9 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 12 May 2026 23:57:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13464/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 9 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 12 May 2026 23:57:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13458/ -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 9 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 12 May 2026 23:15:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 9 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 12 May 2026 22:39:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/22394/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 9 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 12 May 2026 18:53:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13458/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 9 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 12 May 2026 18:32:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Jason Fehr has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. IMPALA-14949: Fix Catalogd Startup Deadlock The getOrLoadTable functionality now waits for the initial database metadata to be loaded for the given database before it takes the versionLock_ read lock. This order of operations is reversed from its previous ordering. There was a deadlock condition during a slow Catalogd startup because the thread that performs the initial global reset periodically releases the versionLock_ write lock to allow for metadata operations on already loaded databases to proceed. If the database was not yet loaded, the read thread waited until it was loaded holding a read lock on versionLock_. However, the initial global reset thread could not proceed with loading any databases because it needed a write lock on versionLock_. Testing accomplished with a new custom cluster test. This new test was run on a local Impala dev machine without the fix in place to demonstrate it consistently failed because of the deadlock. It was also successfully run with the fix in place both on a local Impala dev machine and in an automated build. Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M tests/custom_cluster/test_catalog_wait.py 2 files changed, 137 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24275/9 -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 9 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 8: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py File tests/custom_cluster/test_catalog_wait.py: http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py@170 PS8, Line 170: repetedly nit. repeatedly http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py@193 PS8, Line 193: CREEATED nit. CREATED -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 8 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 12 May 2026 17:46:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13457/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 8 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 12 May 2026 17:01:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py File tests/custom_cluster/test_catalog_wait.py: http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py@186 PS8, Line 186: func=count_completed_queries, max_attempts=60, sleep_time_s=2, backoff=1), \ > I don't think ASAN jobs will run exhaustive tests. I specifically marked t Ack -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 8 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 12 May 2026 15:56:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py File tests/custom_cluster/test_catalog_wait.py: http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py@186 PS8, Line 186: func=count_completed_queries, max_attempts=60, sleep_time_s=2, backoff=1), \ > Should we check these in an asan run too if they need lots of time? I don't think ASAN jobs will run exhaustive tests. I specifically marked this test as exhaustive only since it takes so long to run (40+ seconds locally), I was concerned about it failing in ASAN runs, and it is checking for a deadlock in Java thus ASAN/TSAN won't apply. -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 8 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 12 May 2026 14:41:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 8: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py File tests/custom_cluster/test_catalog_wait.py: http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py@186 PS8, Line 186: func=count_completed_queries, max_attempts=60, sleep_time_s=2, backoff=1), \ Should we check these in an asan run too if they need lots of time? -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 8 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 11 May 2026 21:10:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/22381/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 8 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 11 May 2026 21:06:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 8: > Patch Set 8: > > Build Failed > > https://jenkins.impala.io/job/gerrit-code-review-checks/22380/ : Initial code > review checks failed. See linked job for details on the failure. Build failed because it could not download a jar from the native-toolchain S3 bucket. -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 8 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 11 May 2026 20:43:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 8: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/22380/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 8 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Mon, 11 May 2026 20:11:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Jason Fehr has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. IMPALA-14949: Fix Catalogd Startup Deadlock The getOrLoadTable functionality now waits for the initial database metadata to be loaded for the given database before it takes the versionLock_ read lock. This order of operations is reversed from its previous ordering. There was a deadlock condition during a slow Catalogd startup because the thread that performs the initial global reset periodically releases the versionLock_ write lock to allow for metadata operations on already loaded databases to proceed. If the database was not yet loaded, the read thread waited until it was loaded holding a read lock on versionLock_. However, the initial global reset thread could not proceed with loading any databases because it needed a write lock on versionLock_. Testing accomplished with a new custom cluster test. This new test was run on a local Impala dev machine without the fix in place to demonstrate it consistently failed because of the deadlock. It was also successfully run with the fix in place both on a local Impala dev machine and in an automated build. Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M tests/custom_cluster/test_catalog_wait.py 2 files changed, 137 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24275/8 -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 8 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Jason Fehr has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. IMPALA-14949: Fix Catalogd Startup Deadlock The getOrLoadTable functionality now waits for the initial database metadata to be loaded for the given database before it takes the versionLock_ read lock. This order of operations is reversed from its previous ordering. There was a deadlock condition during a slow Catalogd startup because the thread that performs the initial global reset periodically releases the versionLock_ write lock to allow for metadata operations on already loaded databases to proceed. If the database was not yet loaded, the read thread waited until it was loaded holding a read lock on versionLock_. However, the initial global reset thread could not proceed with loading any databases because it needed a write lock on versionLock_. Testing accomplished with a new custom cluster test. This new test was run on a local Impala dev machine without the fix in place to demonstrate it consistently failed because of the deadlock. It was also successfully run with the fix in place both on a local Impala dev machine and in an automated build. Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M tests/custom_cluster/test_catalog_wait.py 2 files changed, 137 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24275/5 -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 5 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 4: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py File tests/custom_cluster/test_catalog_wait.py: http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py@179 PS3, Line 179: def count_completed_queries(): > Replaced with waiting for 100 queries to complete. Done http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py@185 PS3, Line 185: func=count_completed_queries, max_attempts=10, sleep_time_s=2, backoff=1), \ > There needs to be time for queries that need Catalogd interaction to queue Done -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 4 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 08 May 2026 18:52:53 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/22364/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 4 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 08 May 2026 18:07:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Jason Fehr has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. IMPALA-14949: Fix Catalogd Startup Deadlock The getOrLoadTable functionality now waits for the initial database metadata to be loaded for the given database before it takes the versionLock_ read lock. This order of operations is reversed from its previous ordering. There was a deadlock condition during a slow Catalogd startup because the thread that performs the initial global reset periodically releases the versionLock_ write lock to allow for metadata operations on already loaded databases to proceed. If the database was not yet loaded, the read thread waited until it was loaded holding a read lock on versionLock_. However, the initial global reset thread could not proceed with loading any databases because it needed a write lock on versionLock_. Testing accomplished with a new custom cluster test. This new test was run on a local Impala dev machine without the fix in place to demonstrate it consistently failed because of the deadlock. It was also successfully run with the fix in place both on a local Impala dev machine and in an automated build. Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M tests/custom_cluster/test_catalog_wait.py 2 files changed, 137 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24275/4 -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 4 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 3: (4 comments) http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py File tests/custom_cluster/test_catalog_wait.py: http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py@142 PS3, Line 142: lock = threading.Lock() > The lock isn't needed; Python GIL ensures basic assignment is atomic. But i Yeah, I think we don't need the lock. We have other examples like https://github.com/apache/impala/blob/f7b4601482ab698ff524a68e89c9485299879062/tests/custom_cluster/test_local_catalog.py#L441-L462 https://github.com/apache/impala/blob/f7b4601482ab698ff524a68e89c9485299879062/tests/custom_cluster/test_concurrent_ddls.py#L181-L198 http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py@179 PS3, Line 179: sleep(5) > Could this be a check for inflight queries from a metric or the /queries pa I think we can let the thread add a marker and wait for the marker to appear. http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py@218 PS3, Line 218: sleep(1) By not killing the impalads, I found the queries all finish at the end. The longest duration is 34s. If I increase the sleep interval to 5s, the test passes without the fix. http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py@226 PS3, Line 226: impalad.kill() Why should we explicitly kill the impalads? -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 3 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Fri, 08 May 2026 13:18:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py File tests/custom_cluster/test_catalog_wait.py: http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py@141 PS3, Line 141: test_catalog_startup_deadlock How about adding a comment like this? Regression test for IMPALA-14949: Verifies no deadlock between the initial global reset and concurrent metadata reload/query workload. -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 3 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 07 May 2026 23:09:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/22357/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 3 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 07 May 2026 22:01:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Jason Fehr has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. IMPALA-14949: Fix Catalogd Startup Deadlock The getOrLoadTable functionality now waits for the initial database metadata to be loaded for the given database before it takes the versionLock_ read lock. This order of operations is reversed from its previous ordering. There was a deadlock condition during a slow Catalogd startup because the thread that performs the initial global reset periodically releases the versionLock_ write lock to allow for metadata operations on already loaded databases to proceed. If the database was not yet loaded, the read thread waited until it was loaded holding a read lock on versionLock_. However, the initial global reset thread could not proceed with loading any databases because it needed a write lock on versionLock_. Manual testing replicated this issue and demonstrated the fix effectiveness. The replication required adding an arbitrary sleep in the CatalogServiceCatalog.rebuildDbCache function's while loop. Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M tests/custom_cluster/test_catalog_wait.py 2 files changed, 111 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24275/3 -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 3 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 2 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 07 May 2026 14:49:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 2: (1 comment) The fix looks good to me. Do you plan to add the test? http://gerrit.cloudera.org:8080/#/c/24275/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/24275/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2609 PS2, Line 2609: or if the initial reset is in progress. nit: this is stale now -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 2 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Thu, 07 May 2026 14:49:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 2: Have you tried to inject a new DebugAction to force the sleep for the custom test? There are some existing ones like DebugUtils.RESET_METADATA_LOOP_LOCKED in CatalogServiceCatalog.java used in custom_cluster/test_catalogd_ha.py, maybe a new DebugAction would help? -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 2 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 06 May 2026 23:44:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/22341/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 2 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 06 May 2026 23:33:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 2 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 06 May 2026 23:14:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Jason Fehr has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. IMPALA-14949: Fix Catalogd Startup Deadlock The getOrLoadTable functionality now waits for the initial database metadata to be loaded for the given database before it takes the versionLock_ read lock. This order of operations is reversed from its previous ordering. There was a deadlock condition during a slow Catalogd startup because the thread that performs the initial global reset periodically releases the versionLock_ write lock to allow for metadata operations on already loaded databases to proceed. If the database was not yet loaded, the read thread waited until it was loaded holding a read lock on versionLock_. However, the initial global reset thread could not proceed with loading any databases because it needed a write lock on versionLock_. Manual testing replicated this issue and demonstrated the fix effectiveness. The replication required adding an arbitrary sleep in the CatalogServiceCatalog.rebuildDbCache function's while loop. Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java 1 file changed, 8 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24275/2 -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 2 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
..
Patch Set 1:
(2 comments)
Thanks for reproducing the issue and the quick fix! The patch is safe and
correct but I hope we can keep the optimization we did for IMPALA-13850 (part
4).
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2611
PS1, Line 2611: triggeredInitialReset_
> same here.
This reverts the optimization we did in IMPALA-13850 (part 4) so catalogd
requests still need to wait for the initial reset to finish, no matter whether
their dbs have been loaded.
I think we can fix the issue like IMPALA-14400 by using ReadLockAndLookupDb.
Left a comment in getOrLoadTable().
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2808
PS1, Line 2808: acquireVersionReadLock(catalogTimeline);
The problem is we acquire the lock first before the waiting. I think we should
actually wait before the lock by using ReadLockAndLookupDb:
-acquireVersionReadLock(catalogTimeline);
-try {
- tbl = getTable(dbName, tblName);
+try (ReadLockAndLookupDb result = new ReadLockAndLookupDb(dbName)) {
+ catalogTimeline.markEvent(GOT_CATALOG_VERSION_READ_LOCK);
+ Db db = result.getDb();
+ if (db == null) {
+throw new DatabaseNotFoundException("Database '" + dbName + "' not
found");
+ }
+ tbl = db.getTable(tblName);
So once the initial reset finishes loading the db of this request, this thread
can move forward when the initial reset temporarily releases the versionLock_
write lock.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 1
Gerrit-Owner: Jason Fehr
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jason Fehr
Gerrit-Reviewer: Michael Smith
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sai Hemanth Gantasala
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Wed, 06 May 2026 03:28:46 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2611 PS1, Line 2611: triggeredInitialReset_ > That makes sense. I'm ok with this change, but I'd like Quanlong or Sai to same here. -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 1 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 06 May 2026 00:10:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 1: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2611 PS1, Line 2611: triggeredInitialReset_ > In a failover situation today, the deadlock won't happen even if CatalogSer That makes sense. I'm ok with this change, but I'd like Quanlong or Sai to take a look since they helped with the IMPALA-13850. -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 1 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 05 May 2026 23:52:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2611 PS1, Line 2611: triggeredInitialReset_ > The behavior matches syncDdl, which semantically makes sense for startup. H In a failover situation today, the deadlock won't happen even if CatalogServiceCatalog.rebuildDbCache releases the versionLock_ write lock because triggeredInitialReset_ will be `true` and thus other threads will not sleep in CatalogServiceCatalog.waitInitialResetCompletion. I also considered making the reset that happens after a Catalogd failover not yield versionLock_ write lock, but that is a behavior change from what happens today. Maybe it is a change we want to make though? -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 1 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 05 May 2026 23:45:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2611 PS1, Line 2611: triggeredInitialReset_ > Yeah, it will be blocked, but the initial refresh should be light since the The behavior matches syncDdl, which semantically makes sense for startup. However I wonder how it interacts with HA failover. I think __set_sync_ddl(true) at https://github.com/apache/impala/blob/master/be/src/catalog/catalog-server.cc#L1050 would be semantically equivalent, and I think it would also be used for catalog reset after failover. -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 1 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Michael Smith Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 05 May 2026 22:33:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 1: I think we probably need a regression test case for this -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 1 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 05 May 2026 20:02:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2611 PS1, Line 2611: triggeredInitialReset_ > Thanks for the quick fix. One concern is that, it seems getCatalogDelta() w Yeah, it will be blocked, but the initial refresh should be light since there won't be any data loaded. Also, a refresh caused by a failover won't block on this operation since both the active and passive catalogd run the initial refresh all. -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 1 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 05 May 2026 19:49:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2611 PS1, Line 2611: triggeredInitialReset_ Thanks for the quick fix. One concern is that, it seems getCatalogDelta() will be completely blocked for the entire duration of the initial global reset. Do you see a potential problem there? Or is there any lighter way for the fix? -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 1 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 05 May 2026 17:36:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 ) Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/22331/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 1 Gerrit-Owner: Jason Fehr Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jason Fehr Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Tue, 05 May 2026 17:34:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Jason Fehr has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24275 Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock .. IMPALA-14949: Fix Catalogd Startup Deadlock Reset metadata handling code no longer yields up the write lock on versionLock_ in CatalogServiceCatalog during the Catalogd startup initial reset metadata. When Catalogd runs a global reset, the rebuildDbCache() function will pause and unlock versionLock_ at intervals defined by the --reset_metadata_lock_duration_ms startup flag. The purpose of this unlocking is to ensure a long running invalidate metadata does not block other metadata operations. During startup, Catalogd runs a global metadata reset. Other threads must wait until this initial reset finishes before they can process operations such as getPartialCatalogObject. If the initial reset takes longer than the duration in the --reset_metadata_lock_duration_ms startup flag, it will release its ownership of versionLock_ followed by immediately re-obtaining the write lock. In between, other threads take a read lock on versionLock_ but sleep until the initial reset completes. At that point, a deadlock situation occurs where the initial reset thread is waiting on a versionLock_ write lock while other threads that hold the versionLock_ read lock are waiting on the initial reset thread to complete. Manual testing replicated this issue and demonstrated the fix effectiveness. The replication required adding an arbitrary sleep in the CatalogServiceCatalog.rebuildDbCache function's while loop. Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24275/1 -- To view, visit http://gerrit.cloudera.org:8080/24275 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404 Gerrit-Change-Number: 24275 Gerrit-PatchSet: 1 Gerrit-Owner: Jason Fehr
