[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Kurt Deschler has posted comments on this change. (
http://gerrit.cloudera.org:8080/21071 )
Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to
aborted write id is cleaned up
..
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21071/5/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/21071/5/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@842
PS5, Line 842: LOG.info("EventId: {} EventType: COMMIT_TXN transaction
id: {}", getEventId(),
> These are very helpful in connecting the COMMIT/ABORT_TXN events with the t
Thanks for the clarification.
http://gerrit.cloudera.org:8080/#/c/21071/5/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@842
PS5, Line 842: LOG.info("EventId: {} EventType: COMMIT_TXN transaction
id: {}", getEventId(),
> Why are these (and other) messages INFO level now? This may create a lot of
Done
--
To view, visit http://gerrit.cloudera.org:8080/21071
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681
Gerrit-Change-Number: 21071
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sai Hemanth Gantasala
Gerrit-Comment-Date: Thu, 29 Feb 2024 00:12:26 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/21071 )
Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to
aborted write id is cleaned up
..
Patch Set 6:
(1 comment)
> Patch Set 5:
>
> (1 comment)
http://gerrit.cloudera.org:8080/#/c/21071/5/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/21071/5/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@842
PS5, Line 842: LOG.info("EventId: {} EventType: COMMIT_TXN transaction
id: {}", getEventId(),
> Why are these (and other) messages INFO level now? This may create a lot of
These are very helpful in connecting the COMMIT/ABORT_TXN events with the
transaction ids. Before this, the logs have no info about that and it's hard to
debug when issue happens.
Note that we already have logs for each event. It won't introduce too many logs.
--
To view, visit http://gerrit.cloudera.org:8080/21071
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681
Gerrit-Change-Number: 21071
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sai Hemanth Gantasala
Gerrit-Comment-Date: Wed, 28 Feb 2024 22:14:55 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up HdfsTable tracks the ValidWriteIdList from HMS. When the table is reloaded, the ValidWriteIdList is updated to the latest state. An ABORT_TXN event that is lagging behind could match to aborted write ids that have already been cleaned up by the HMS housekeeping thread. Such write ids can't be found in the cached ValidWriteIdList as opened or aborted write ids. This hits a Precondition check and fails the event processing. This patch fixes the check to allow this case. Also adds more logs for dealing with write ids. Tests - Add custom-cluster test to start Hive with the housekeeping thread turned on and verified that such ABORT_TXN event is processed correctly. Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Reviewed-on: http://gerrit.cloudera.org:8080/21071 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/create-test-configuration.sh M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java M fe/src/test/resources/hive-site.xml.py M tests/common/impala_test_suite.py M tests/custom_cluster/test_events_custom_configs.py 8 files changed, 84 insertions(+), 11 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 6 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 17:54:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Kurt Deschler has posted comments on this change. (
http://gerrit.cloudera.org:8080/21071 )
Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to
aborted write id is cleaned up
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21071/5/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/21071/5/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@842
PS5, Line 842: LOG.info("EventId: {} EventType: COMMIT_TXN transaction
id: {}", getEventId(),
Why are these (and other) messages INFO level now? This may create a lot of
chatter in the logs.
--
To view, visit http://gerrit.cloudera.org:8080/21071
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681
Gerrit-Change-Number: 21071
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Anonymous Coward
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kurt Deschler
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sai Hemanth Gantasala
Gerrit-Comment-Date: Tue, 27 Feb 2024 15:35:07 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 5: > Patch Set 5: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10312/ This seems to be IMPALA-12416. -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 13:18:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10313/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 13:19:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10312/ -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 12:07:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 4: > Patch Set 4: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10310/ The failure is due to IMPALA-12621 which is unrelated to this patch. Rerun the test. -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 07:24:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10312/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 07:24:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 07:24:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10310/ -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 05:59:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 4: Code-Review+2 Carry +2 from Csaba -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 01:23:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10310/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Tue, 27 Feb 2024 01:24:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Sai Hemanth Gantasala has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 4: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 26 Feb 2024 22:33:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15326/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 26 Feb 2024 22:24:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 4: (3 comments) http://gerrit.cloudera.org:8080/#/c/21071/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21071/3//COMMIT_MSG@14 PS3, Line 14: aborted > typo Done http://gerrit.cloudera.org:8080/#/c/21071/3//COMMIT_MSG@22 PS3, Line 22: verifi > nit: verified that Done http://gerrit.cloudera.org:8080/#/c/21071/3/tests/custom_cluster/test_events_custom_configs.py File tests/custom_cluster/test_events_custom_configs.py: http://gerrit.cloudera.org:8080/#/c/21071/3/tests/custom_cluster/test_events_custom_configs.py@1191 PS3, Line 1191: # > flake8: E265 block comment should start with '# ' Done -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 26 Feb 2024 21:58:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Hello k.venureddy2...@gmail.com, Sai Hemanth Gantasala, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21071 to look at the new patch set (#4). Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up HdfsTable tracks the ValidWriteIdList from HMS. When the table is reloaded, the ValidWriteIdList is updated to the latest state. An ABORT_TXN event that is lagging behind could match to aborted write ids that have already been cleaned up by the HMS housekeeping thread. Such write ids can't be found in the cached ValidWriteIdList as opened or aborted write ids. This hits a Precondition check and fails the event processing. This patch fixes the check to allow this case. Also adds more logs for dealing with write ids. Tests - Add custom-cluster test to start Hive with the housekeeping thread turned on and verified that such ABORT_TXN event is processed correctly. Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 --- M bin/create-test-configuration.sh M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java M fe/src/test/resources/hive-site.xml.py M tests/common/impala_test_suite.py M tests/custom_cluster/test_events_custom_configs.py 8 files changed, 84 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/21071/4 -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 3: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/21071/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/21071/3//COMMIT_MSG@14 PS3, Line 14: aboorted typo http://gerrit.cloudera.org:8080/#/c/21071/3//COMMIT_MSG@22 PS3, Line 22: verify nit: verified that -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 26 Feb 2024 18:07:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15314/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 26 Feb 2024 09:23:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/21071/3/tests/custom_cluster/test_events_custom_configs.py File tests/custom_cluster/test_events_custom_configs.py: http://gerrit.cloudera.org:8080/#/c/21071/3/tests/custom_cluster/test_events_custom_configs.py@1191 PS3, Line 1191: # flake8: E265 block comment should start with '# ' -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 26 Feb 2024 08:57:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Hello k.venureddy2...@gmail.com, Sai Hemanth Gantasala, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21071 to look at the new patch set (#3). Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up HdfsTable tracks the ValidWriteIdList from HMS. When the table is reloaded, the ValidWriteIdList is updated to the latest state. An ABORT_TXN event that is lagging behind could match to aborted write ids that have already been cleaned up by the HMS housekeeping thread. Such write ids can't be found in the cached ValidWriteIdList as opened or aboorted write ids. This hits a Precondition check and fails the event processing. This patch fixes the check to allow this case. Also adds more logs for dealing with write ids. Tests - Add custom-cluster test to start Hive with the housekeeping thread turned on and verify such ABORT_TXN event is processed correctly. Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 --- M bin/create-test-configuration.sh M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java M fe/src/test/resources/hive-site.xml.py M tests/common/impala_test_suite.py M tests/custom_cluster/test_events_custom_configs.py 8 files changed, 84 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/21071/3 -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 26 Feb 2024 08:51:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10304/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sai Hemanth Gantasala Gerrit-Comment-Date: Mon, 26 Feb 2024 04:16:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15312/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 26 Feb 2024 03:19:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21071 ) Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/15311/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 26 Feb 2024 03:04:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/21071 to look at the new patch set (#2). Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up HdfsTable tracks the ValidWriteIdList from HMS. When the table is reloaded, the ValidWriteIdList is updated to the latest state. An ABORT_TXN event that is lagging behind could match to aborted write ids that have already been cleaned up by the HMS housekeeping thread. Such write ids can't be found in the cached ValidWriteIdList as opened or aboorted write ids. This hits a Precondition check and fails the event processing. This patch fixes the check to allow this case. Also adds more logs for dealing with write ids. Tests - Add custom-cluster test to start Hive with the housekeeping thread turned on and verify such ABORT_TXN event is processed correctly. Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 --- M bin/create-test-configuration.sh M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java M fe/src/test/resources/hive-site.xml.py M tests/common/impala_test_suite.py M tests/custom_cluster/test_events_custom_configs.py 8 files changed, 80 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/21071/2 -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21071 Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up .. IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up HdfsTable tracks the ValidWriteIdList from HMS. When the table is reloaded, the ValidWriteIdList is updated to the latest state. An ABORT_TXN event that is lagging behind could match to aborted write ids that have already been cleaned up by the HMS housekeeping thread. Such write ids can't be found in the cached ValidWriteIdList as opened or aboorted write ids. This hits a Precondition check and fails the event processing. This patch fixes the check to allow this case. Also adds more logs for dealing with write ids. Tests - Add custom-cluster test to start Hive with the housekeeping thread turned on and verify such ABORT_TXN event is processed correctly. Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 --- M bin/create-test-configuration.sh M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java M fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java M fe/src/test/resources/hive-site.xml.py M tests/common/impala_test_suite.py M tests/custom_cluster/test_events_custom_configs.py 8 files changed, 78 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/21071/1 -- To view, visit http://gerrit.cloudera.org:8080/21071 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681 Gerrit-Change-Number: 21071 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/21071 )
Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to
aborted write id is cleaned up
..
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21071/1/bin/create-test-configuration.sh
File bin/create-test-configuration.sh:
http://gerrit.cloudera.org:8080/#/c/21071/1/bin/create-test-configuration.sh@158
PS1, Line 158: ln -s "${CONFIG_DIR}/hive-site_housekeeping_on.xml"
hive-site-housekeeping-on/hive-site.xml
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/21071/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/21071/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@842
PS1, Line 842: LOG.info("EventId: {} EventType: COMMIT_TXN transaction
id: {}", getEventId(), txnId_);
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/21071/1/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:
http://gerrit.cloudera.org:8080/#/c/21071/1/tests/custom_cluster/test_events_custom_configs.py@1188
PS1, Line 1188: \
flake8: E502 the backslash is redundant between brackets
--
To view, visit http://gerrit.cloudera.org:8080/21071
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681
Gerrit-Change-Number: 21071
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Mon, 26 Feb 2024 02:39:41 +
Gerrit-HasComments: Yes
