[Impala-ASF-CR] IMPALA-7714: try to avoid be test crash in statestore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16341 ) Change subject: IMPALA-7714: try to avoid be test crash in statestore .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6297/ -- To view, visit http://gerrit.cloudera.org:8080/16341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4363a93addb8a808d292906cac44ebd25c16889 Gerrit-Change-Number: 16341 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 15 Aug 2020 04:49:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated HMS partitions
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16308 ) Change subject: IMPALA-4364: Query option to refresh updated HMS partitions .. IMPALA-4364: Query option to refresh updated HMS partitions This patch introduces a new boolean query option REFRESH_UPDATED_HMS_PARTITIONS. When this query option is set the refresh table command reloads the partitions which have been modified in HMS in addition to adding [removing] the new [removed] partitions. In order to do this the refresh table command needs to fetch all the partitions instead of the just the partition names which can cause the performance of refresh table to degrade when the query option is set. However for certain use-cases currently there is no way to detect changed partitions using refresh table command. For instance, if certain partition locations have been changed, a refresh table will not update those partitions. Testing: 1. Added a new test which sets the query option and makes sure that the updated partitions from hive are reloaded after refresh table command. 2. Ran exhaustive tests with the patch. Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Reviewed-on: http://gerrit.cloudera.org:8080/16308 Reviewed-by: Quanlong Huang Tested-by: Impala Public Jenkins --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/CatalogService.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/metadata/test_reset_metadata.py 11 files changed, 464 insertions(+), 106 deletions(-) Approvals: Quanlong Huang: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/16308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Gerrit-Change-Number: 16308 Gerrit-PatchSet: 9 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated HMS partitions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16308 ) Change subject: IMPALA-4364: Query option to refresh updated HMS partitions .. Patch Set 8: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Gerrit-Change-Number: 16308 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 15 Aug 2020 01:57:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9382: part 1: transposed profile prototype
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15798 ) Change subject: IMPALA-9382: part 1: transposed profile prototype .. Patch Set 16: (3 comments) I made it through most of the runtime-profile.h. The transformation of the averaged profile to the aggregated profile and resulting class hierarchy seems reasonable to me. http://gerrit.cloudera.org:8080/#/c/15798/16//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15798/16//COMMIT_MSG@30 PS16, Line 30: with RuntimeProfile and AggregatedRuntimeProfile : base classes "base classes" -> "subclasses"? http://gerrit.cloudera.org:8080/#/c/15798/16//COMMIT_MSG@44 PS16, Line 44: These stats can now be : automatically as aggregations of counters. Some words seem to be missing. http://gerrit.cloudera.org:8080/#/c/15798/16/common/thrift/RuntimeProfile.thrift File common/thrift/RuntimeProfile.thrift: http://gerrit.cloudera.org:8080/#/c/15798/16/common/thrift/RuntimeProfile.thrift@249 PS16, Line 249: an averaged profile : // for the fragment is also included with averaged counter values. Does the thrift for V1's average profile now also contain the TAggregatedRuntimeProfileNode struct? If so, are there any size concerns? -- To view, visit http://gerrit.cloudera.org:8080/15798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0838c6a0872f57c696267ff4e92d29c08748eb7a Gerrit-Change-Number: 15798 Gerrit-PatchSet: 16 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 15 Aug 2020 00:44:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. Patch Set 45: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6295/ -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 45 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 15 Aug 2020 00:44:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated HMS partitions
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16308 ) Change subject: IMPALA-4364: Query option to refresh updated HMS partitions .. Patch Set 8: Code-Review+2 Thanks for working on this! -- To view, visit http://gerrit.cloudera.org:8080/16308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Gerrit-Change-Number: 16308 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 15 Aug 2020 00:40:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7714: try to avoid be test crash in statestore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16341 ) Change subject: IMPALA-7714: try to avoid be test crash in statestore .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6942/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4363a93addb8a808d292906cac44ebd25c16889 Gerrit-Change-Number: 16341 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 23:41:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9989 Improve admission control pool stats logging
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/16220 ) Change subject: IMPALA-9989 Improve admission control pool stats logging .. Patch Set 38: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/16220 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 Gerrit-Change-Number: 16220 Gerrit-PatchSet: 38 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 23:39:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7714: try to avoid be test crash in statestore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16341 ) Change subject: IMPALA-7714: try to avoid be test crash in statestore .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6297/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4363a93addb8a808d292906cac44ebd25c16889 Gerrit-Change-Number: 16341 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 23:35:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7714: try to avoid be test crash in statestore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16341 ) Change subject: IMPALA-7714: try to avoid be test crash in statestore .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4363a93addb8a808d292906cac44ebd25c16889 Gerrit-Change-Number: 16341 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 23:35:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7714: try to avoid be test crash in statestore
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/16341 ) Change subject: IMPALA-7714: try to avoid be test crash in statestore .. Patch Set 2: Code-Review+2 LGTM -- To view, visit http://gerrit.cloudera.org:8080/16341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4363a93addb8a808d292906cac44ebd25c16889 Gerrit-Change-Number: 16341 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 23:31:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7714: try to avoid be test crash in statestore
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16341 ) Change subject: IMPALA-7714: try to avoid be test crash in statestore .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/16341/1/be/src/statestore/statestore.cc File be/src/statestore/statestore.cc: http://gerrit.cloudera.org:8080/#/c/16341/1/be/src/statestore/statestore.cc@396 PS1, Line 396: // IMPALA-7714: log warning to aid debugging. > Should the logging go before the DCHECK? This was mainly targeted at release builds, but yeah, makes sense. I also added the topic id to the DCHECK cause that'd be useful. -- To view, visit http://gerrit.cloudera.org:8080/16341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4363a93addb8a808d292906cac44ebd25c16889 Gerrit-Change-Number: 16341 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 23:20:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7714: try to avoid be test crash in statestore
Hello Andrew Sherman, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16341 to look at the new patch set (#2). Change subject: IMPALA-7714: try to avoid be test crash in statestore .. IMPALA-7714: try to avoid be test crash in statestore We didn't get to a clear root cause for this, so I'm going to try two things. First, under the theory that the problem is somehow the destruction of the strings, convert them to char char* which does not require destruction on process teardown. Second, add some logging if the map lookup fails so we can better understand what may have happened. Change-Id: Id4363a93addb8a808d292906cac44ebd25c16889 --- M be/src/statestore/statestore.cc M be/src/statestore/statestore.h 2 files changed, 9 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/16341/2 -- To view, visit http://gerrit.cloudera.org:8080/16341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id4363a93addb8a808d292906cac44ebd25c16889 Gerrit-Change-Number: 16341 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9989 Improve admission control pool stats logging
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16220 ) Change subject: IMPALA-9989 Improve admission control pool stats logging .. Patch Set 38: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6941/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16220 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 Gerrit-Change-Number: 16220 Gerrit-PatchSet: 38 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 22:59:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10039 (part 2): Fixed Expr-test crash due to race condition
Thomas Tauber-Marshall has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16313 ) Change subject: IMPALA-10039 (part 2): Fixed Expr-test crash due to race condition .. IMPALA-10039 (part 2): Fixed Expr-test crash due to race condition The root cause for crash is that QueryState::Cancel() was called before thread unsafe function QueryState::Init() was completed. This patch fixs the race condition between QueryState::Cancel() and QueryState::Init(). QueryState::Init() is safe to be called at any time. Testing: - The issue could be reproduced by running expr-test for 10-20 iterations. Verified the fixing by running expr-test over 1000 iterations without crash. - Passed TestProcessFailures::test_kill_coordinator. - Passed core tests. Change-Id: Ib0d3b9c59924a25b70fa20afeb6e8ca93016eca9 Reviewed-on: http://gerrit.cloudera.org:8080/16313 Tested-by: Impala Public Jenkins Reviewed-by: Thomas Tauber-Marshall --- M be/src/runtime/query-state.cc M be/src/runtime/query-state.h 2 files changed, 32 insertions(+), 7 deletions(-) Approvals: Impala Public Jenkins: Verified Thomas Tauber-Marshall: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/16313 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib0d3b9c59924a25b70fa20afeb6e8ca93016eca9 Gerrit-Change-Number: 16313 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10039 (part 2): Fixed Expr-test crash due to race condition
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16313 ) Change subject: IMPALA-10039 (part 2): Fixed Expr-test crash due to race condition .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16313 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib0d3b9c59924a25b70fa20afeb6e8ca93016eca9 Gerrit-Change-Number: 16313 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 14 Aug 2020 22:32:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10039 (part 2): Fixed Expr-test crash due to race condition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16313 ) Change subject: IMPALA-10039 (part 2): Fixed Expr-test crash due to race condition .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16313 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib0d3b9c59924a25b70fa20afeb6e8ca93016eca9 Gerrit-Change-Number: 16313 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 14 Aug 2020 22:29:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7779 Parquet Scanner can write binary data into profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16331 ) Change subject: IMPALA-7779 Parquet Scanner can write binary data into profile .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6293/ -- To view, visit http://gerrit.cloudera.org:8080/16331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I281d6fa7cb2f88f04588110943e3e768678b9cf1 Gerrit-Change-Number: 16331 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 14 Aug 2020 22:27:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7714: try to avoid be test crash in statestore
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/16341 ) Change subject: IMPALA-7714: try to avoid be test crash in statestore .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/16341/1/be/src/statestore/statestore.cc File be/src/statestore/statestore.cc: http://gerrit.cloudera.org:8080/#/c/16341/1/be/src/statestore/statestore.cc@396 PS1, Line 396: // IMPALA-7714: log warning to aid debugging. Should the logging go before the DCHECK? -- To view, visit http://gerrit.cloudera.org:8080/16341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4363a93addb8a808d292906cac44ebd25c16889 Gerrit-Change-Number: 16341 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 14 Aug 2020 22:10:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9989 Improve admission control pool stats logging
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/16220 ) Change subject: IMPALA-9989 Improve admission control pool stats logging .. Patch Set 37: (2 comments) http://gerrit.cloudera.org:8080/#/c/16220/37/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/16220/37/be/src/scheduling/admission-controller.cc@911 PS37, Line 911: if ( not_admitted_details ) : *not_admitted_details = ""; nit: dont think this is necessary since a string object has a default initialization of an empty string. But in any case if you want to explicitly init it, it would be better to do this during QueueNode's initializer list http://gerrit.cloudera.org:8080/#/c/16220/36/common/thrift/generate_error_codes.py File common/thrift/generate_error_codes.py: http://gerrit.cloudera.org:8080/#/c/16220/36/common/thrift/generate_error_codes.py@337 PS36, Line 337: Details: > Addressed in AdmissionController::CanAdmitRequest() by init the details str What I meant was that if I see an error message like: "Admission for query exceeded timeout 6ms in pool root.poolA. Queued reason: Queue non-empty Details: " Here the details is empty, so i was proposing to get rid of "Details" if it does not exist and only print the error msg like: "Admission for query exceeded timeout 6ms in pool root.poolA. Queued reason: Queue non-empty" -- To view, visit http://gerrit.cloudera.org:8080/16220 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 Gerrit-Change-Number: 16220 Gerrit-PatchSet: 37 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 21:45:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9989 Improve admission control pool stats logging
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16220 ) Change subject: IMPALA-9989 Improve admission control pool stats logging .. Patch Set 37: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6940/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16220 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 Gerrit-Change-Number: 16220 Gerrit-PatchSet: 37 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 21:30:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated HMS partitions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16308 ) Change subject: IMPALA-4364: Query option to refresh updated HMS partitions .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6939/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Gerrit-Change-Number: 16308 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 14 Aug 2020 21:16:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9989 Improve admission control pool stats logging
Qifan Chen has uploaded a new patch set (#37). ( http://gerrit.cloudera.org:8080/16220 ) Change subject: IMPALA-9989 Improve admission control pool stats logging .. IMPALA-9989 Improve admission control pool stats logging This work addresses the current limitation in admission controller by appending the last known memory consumption statistics about a pool to the existing memory exhaustion message. The statistics is logged in impalad.INFO when a query is queued or timed out due to memory pressure in the pool or on the host. The statistics can also be part of the query profile. The BNF of the new memory consumption statistics is as follows. topN_query_stats ::= queries: a list of query Ids and memory consumed for up to 5 queries with top memory consumptions total_consumed: total memory consumed by these topN queries fraction_of_pool_total_mem: total memory consumed divided by pool memory usage (if feasible to report) all_query_stats ::= num_running: the total number of queries running min: the minimal memory consumption of all running queries max: the maximal memory consumption of all running queries pool_total_mem: the total memory consumption of all running queries average: the average memory consumption of all running queries (if feasible to report) pool_stats ::= ":" stats_on_host ::= "Stats for host " List of aggregated_pool_stats ::= "Aggregated stats for pool " memory_consumption_statistics ::= | The stats_on_host describes memory consumption for every pool on a host and is useful in analyzing memory exhaustion on that host. The aggregated_pool_stats describes the aggregated memory consumption on all hosts for a pool for a set of queries and is useful in analyzing memory exhaustion in that pool. Example of stats_on_host for pool root.queueB and root.queueC on host host1:25000. Stats for host host1:25000 pool_name=root.queueB: topN_query_stats: queries=[ id=0001:0004, consumed=20.00 MB, id=0001:0003, consumed=19.00 MB, id=0001:0002, consumed=8.00 MB ], total_consumed=47.00 MB fraction_of_pool_total_mem=0.47 all_query_stats: num_running=4, min=5.00 MB, max=20.00 MB, pool_total_mem=100.00 MB, average=25.00 MB pool_name=root.queueC: topN_query_stats: queries=[ id=0002:, consumed=18.00 MB, id=0002:0001, consumed=12.00 MB ], total_consumed=30.00 MB fraction_of_pool_total_mem=0.06 all_query_stats: num_running=40, min=10.00 MB, max=200.00 MB, pool_total_mem=500.00 MB, average=12.50 MB Example of aggregated_pool_stats over all hosts for pool root.queueC: Aggregated stats for pool root.queueC: topN_query_stats: queries=[ id=0002:0001, consumed=32.00 MB, id=0002:0004, consumed=26.00 MB, id=0002:, consumed=21.00 MB, id=0002:0002, consumed=17.00 MB, id=0002:000e, consumed=9.00 MB ], total_consumed=105.00 MB fraction_of_pool_total_mem=0.82 When a query request is queued due to memory exhaustion, the above memory_consumption_statistics is logged when the logging is set at level 2 or higher. When a query request is timed out due to memory exhaustion, the above memory_consumption_statistics is reported when the logging is set at level 1 or higher. Testing: 1. Added a new test TopNQueryCheck in admission-controller-test.cc to verify that the topN query memory consumption details are reported correctly. 2. Add two new tests in test_admission_controller.py to simulate queries being queued and then timed out due to pool or host memory pressure. 3. Added a new test TopN in mem-tracker-test.cc to verify that the topN query memory consumption details are computed correctly from a mem tracker hierarchy. 4. Ran Core tests successfully. Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 --- M be/src/runtime/mem-tracker-test.cc M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/util/container-util.h M common/thrift/StatestoreService.thrift M common/thrift/generate_error_codes.py M tests/custom_cluster/test_admission_controller.py 10 files changed, 916 insertions(+), 47
[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated HMS partitions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16308 ) Change subject: IMPALA-4364: Query option to refresh updated HMS partitions .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6938/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Gerrit-Change-Number: 16308 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 14 Aug 2020 21:09:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated HMS partitions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16308 ) Change subject: IMPALA-4364: Query option to refresh updated HMS partitions .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6937/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Gerrit-Change-Number: 16308 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 14 Aug 2020 21:06:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated HMS partitions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16308 ) Change subject: IMPALA-4364: Query option to refresh updated HMS partitions .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6936/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Gerrit-Change-Number: 16308 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 14 Aug 2020 21:01:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated HMS partitions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16308 ) Change subject: IMPALA-4364: Query option to refresh updated HMS partitions .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6935/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Gerrit-Change-Number: 16308 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 14 Aug 2020 20:59:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated HMS partitions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16308 ) Change subject: IMPALA-4364: Query option to refresh updated HMS partitions .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6296/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Gerrit-Change-Number: 16308 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 14 Aug 2020 20:50:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated HMS partitions
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16308 ) Change subject: IMPALA-4364: Query option to refresh updated HMS partitions .. Patch Set 7: rebased to latest master. -- To view, visit http://gerrit.cloudera.org:8080/16308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Gerrit-Change-Number: 16308 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 14 Aug 2020 20:50:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated HMS partitions
Vihang Karajgaonkar has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/16308 ) Change subject: IMPALA-4364: Query option to refresh updated HMS partitions .. IMPALA-4364: Query option to refresh updated HMS partitions This patch introduces a new boolean query option REFRESH_UPDATED_HMS_PARTITIONS. When this query option is set the refresh table command reloads the partitions which have been modified in HMS in addition to adding [removing] the new [removed] partitions. In order to do this the refresh table command needs to fetch all the partitions instead of the just the partition names which can cause the performance of refresh table to degrade when the query option is set. However for certain use-cases currently there is no way to detect changed partitions using refresh table command. For instance, if certain partition locations have been changed, a refresh table will not update those partitions. Testing: 1. Added a new test which sets the query option and makes sure that the updated partitions from hive are reloaded after refresh table command. 2. Ran exhaustive tests with the patch. Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/CatalogService.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/metadata/test_reset_metadata.py 11 files changed, 464 insertions(+), 106 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/16308/8 -- To view, visit http://gerrit.cloudera.org:8080/16308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Gerrit-Change-Number: 16308 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated HMS partitions
Vihang Karajgaonkar has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/16308 ) Change subject: IMPALA-4364: Query option to refresh updated HMS partitions .. IMPALA-4364: Query option to refresh updated HMS partitions This patch introduces a new boolean query option REFRESH_UPDATED_HMS_PARTITIONS. When this query option is set the refresh table command reloads the partitions which have been modified in HMS in addition to adding [removing] the new [removed] partitions. In order to do this the refresh table command needs to fetch all the partitions instead of the just the partition names which can cause the performance of refresh table to degrade when the query option is set. However for certain use-cases currently there is no way to detect changed partitions using refresh table command. For instance, if certain partition locations have been changed, a refresh table will not update those partitions. Testing: 1. Added a new test which sets the query option and makes sure that the updated partitions from hive are reloaded after refresh table command. 2. Ran exhaustive tests with the patch. Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/CatalogService.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/metadata/test_reset_metadata.py 11 files changed, 464 insertions(+), 106 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/16308/7 -- To view, visit http://gerrit.cloudera.org:8080/16308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Gerrit-Change-Number: 16308 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated HMS partitions
Vihang Karajgaonkar has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/16308 ) Change subject: IMPALA-4364: Query option to refresh updated HMS partitions .. IMPALA-4364: Query option to refresh updated HMS partitions This patch introduces a new boolean query option REFRESH_UPDATED_HMS_PARTITIONS. When this query option is set the refresh table command reloads the partitions which have been modified in HMS in addition to adding [removing] the new [removed] partitions. In order to do this the refresh table command needs to fetch all the partitions instead of the just the partition names which can cause the performance of refresh table to degrade when the query option is set. However for certain use-cases currently there is no way to detect changed partitions using refresh table command. For instance, if certain partition locations have been changed, a refresh table will not update those partitions. Testing: 1. Added a new test which sets the query option and makes sure that the updated partitions from hive are reloaded after refresh table command. 2. Ran exhaustive tests with the patch. Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af --- M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/CatalogService.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/metadata/test_reset_metadata.py 11 files changed, 460 insertions(+), 106 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/16308/6 -- To view, visit http://gerrit.cloudera.org:8080/16308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Gerrit-Change-Number: 16308 Gerrit-PatchSet: 6 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated HMS partitions
Vihang Karajgaonkar has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/16308 ) Change subject: IMPALA-4364: Query option to refresh updated HMS partitions .. IMPALA-4364: Query option to refresh updated HMS partitions This patch introduces a new boolean query option REFRESH_UPDATED_HMS_PARTITIONS. When this query option is set the refresh table command reloads the partitions which have been modified in HMS in addition to adding [removing] the new [removed] partitions. In order to do this the refresh table command needs to fetch all the partitions instead of the just the partition names which can cause the performance of refresh table to degrade when the query option is set. However for certain use-cases currently there is no way to detect changed partitions using refresh table command. For instance, if certain partition locations have been changed, a refresh table will not update those partitions. Testing: 1. Added a new test which sets the query option and makes sure that the updated partitions from hive are reloaded after refresh table command. 2. Ran exhaustive tests with the patch. Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af --- M be/src/service/client-request-state.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/CatalogService.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/metadata/test_reset_metadata.py 12 files changed, 461 insertions(+), 107 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/16308/5 -- To view, visit http://gerrit.cloudera.org:8080/16308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Gerrit-Change-Number: 16308 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-4364: Query option to refresh updated partitions
Vihang Karajgaonkar has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/16308 ) Change subject: IMPALA-4364: Query option to refresh updated partitions .. IMPALA-4364: Query option to refresh updated partitions This patch introduces a new boolean query option REFRESH_UPDATED_HMS_PARTITIONS. When this query option is set the refresh table command reloads the partitions which have been modified in HMS in addition to adding [removing] the new [removed] partitions. In order to do this the refresh table command needs to fetch all the partitions instead of the just the partition names which can cause the performance of refresh table to degrade when the query option is set. However for certain use-cases currently there is no way to detect changed partitions using refresh table command. For instance, if certain partition locations have been changed, a refresh table will not update those partitions. Testing: 1. Added a new test which sets the query option and makes sure that the updated partitions from hive are reloaded after refresh table command. 2. Ran exhaustive tests with the patch. Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af --- M be/src/service/client-request-state.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/CatalogService.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M tests/metadata/test_reset_metadata.py 12 files changed, 461 insertions(+), 107 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/16308/4 -- To view, visit http://gerrit.cloudera.org:8080/16308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I50e8680509f4eb0712e7bb3de44df5f2952179af Gerrit-Change-Number: 16308 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9955,IMPALA-9957: Fix not enough reservation for large pages in GroupingAggregator
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/16240 )
Change subject: IMPALA-9955,IMPALA-9957: Fix not enough reservation for large
pages in GroupingAggregator
..
Patch Set 9:
(3 comments)
Ok, I think I'm nearly happy with this.
I thought a bit more about the other operators and I don't think there are
similar bugs. AnalyticEvalNode and the plan root sink only have single streams,
so there's no complicated logic moving reservations between streams.
PartitionedHashJoinNode doesn't have the same problem here - it can always
immediately unpin a stream to spill each partition. I.e. it doesn't have the
problem where it needs to allocate additional memory to spill a partition.
http://gerrit.cloudera.org:8080/#/c/16240/9/be/src/exec/grouping-aggregator-partition.cc
File be/src/exec/grouping-aggregator-partition.cc:
http://gerrit.cloudera.org:8080/#/c/16240/9/be/src/exec/grouping-aggregator-partition.cc@234
PS9, Line 234: --parent->num_pinned_hash_partitions_;
If we were going to keep maintaining this, I think we'd want a wrapper that
decremented this along with setting hash_tbl to null, to maintain the invariant
that it's equal to the number of partitions where is_spilled() is true.
http://gerrit.cloudera.org:8080/#/c/16240/9/be/src/exec/grouping-aggregator-partition.cc@243
PS9, Line 243: void GroupingAggregator::Partition::Close(bool finalize_rows) {
Do we need to decrement num_pinned_hash_partitions here too, if it was counted
as pinned before?
I looked and I guess by the time we're closing partitions we won't spill any
more, so it might not matter as far as the code working, but it's confusing if
the count can get out of sync.
http://gerrit.cloudera.org:8080/#/c/16240/9/be/src/exec/grouping-aggregator.cc
File be/src/exec/grouping-aggregator.cc:
http://gerrit.cloudera.org:8080/#/c/16240/9/be/src/exec/grouping-aggregator.cc@651
PS9, Line 651: while (num_pinned_hash_partitions_ > 0) {
I think I actually preferred computing num_pinned_hash_partitions in this
function because there was less state to reason about - I don't have to check
that the counter is updated whenever a partition is spilled.. I think
num_pinned_hash_partitions_ gets inconsistent when you close the hash
partitions so it's a little confusing.
--
To view, visit http://gerrit.cloudera.org:8080/16240
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I3d9c3a2e7f0da60071b920dec979729e86459775
Gerrit-Change-Number: 16240
Gerrit-PatchSet: 9
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 14 Aug 2020 20:33:52 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10073: Create shaded dependency for S3A and aws-java-sdk-bundle
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16342 ) Change subject: IMPALA-10073: Create shaded dependency for S3A and aws-java-sdk-bundle .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6934/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16342 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0939f73be986f83cc1fd07921563b4d9201780f2 Gerrit-Change-Number: 16342 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 14 Aug 2020 20:01:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. Patch Set 45: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6933/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 45 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 19:58:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7714: try to avoid be test crash in statestore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16341 ) Change subject: IMPALA-7714: try to avoid be test crash in statestore .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6932/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id4363a93addb8a808d292906cac44ebd25c16889 Gerrit-Change-Number: 16341 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 14 Aug 2020 19:48:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10073: Create shaded dependency for S3A and aws-java-sdk-bundle
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16342 Change subject: IMPALA-10073: Create shaded dependency for S3A and aws-java-sdk-bundle .. IMPALA-10073: Create shaded dependency for S3A and aws-java-sdk-bundle The aws-java-sdk-bundle is one of the largest dependencies in the Impala Docker images and continues to grow. The jar includes SDKs for every single AWS service. This patch removes most of the unnecessary SDKs from the aws-java-sdk-bundle, thus drastically decreasing the size of the dependency. The Maven shade plugin is used to do this, and the implementation is similar to what is currently done for the hive-exec jar. This patch takes a conservative approach to removing packages from the aws-java-sdk-bundle jar, and I ensured no direct dependencies of the S3 SDK were removed. The idea is to only remove dependencies that S3A would never conceivably need. Given the huge number of AWS services, I only focused on removing the largest SDKs (the size of each SDK is estimated by the number of classes in the SDK). This decreases the size of the Docker images by about 100 MB. Testing: * Ran core tests against S3 Change-Id: I0939f73be986f83cc1fd07921563b4d9201780f2 --- M CMakeLists.txt M ext-data-source/CMakeLists.txt M fe/CMakeLists.txt M fe/pom.xml D shaded-deps/.gitignore C shaded-deps/hive-exec/CMakeLists.txt R shaded-deps/hive-exec/pom.xml R shaded-deps/s3a-aws-sdk/CMakeLists.txt A shaded-deps/s3a-aws-sdk/pom.xml 9 files changed, 212 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/16342/1 -- To view, visit http://gerrit.cloudera.org:8080/16342 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0939f73be986f83cc1fd07921563b4d9201780f2 Gerrit-Change-Number: 16342 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. Patch Set 45: (8 comments) http://gerrit.cloudera.org:8080/#/c/15531/45/be/src/util/sse2neon.h File be/src/util/sse2neon.h: http://gerrit.cloudera.org:8080/#/c/15531/45/be/src/util/sse2neon.h@213 PS45, Line 213: // https://msdn.microsoft.com/en-us/library/bb514059%28v=vs.120%29.aspx?f=255=-2147217396 line too long (99 > 90) http://gerrit.cloudera.org:8080/#/c/15531/45/be/src/util/sse2neon.h@406 PS45, Line 406: // https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2010/whtfzhzk(v=vs.100) line too long (104 > 90) http://gerrit.cloudera.org:8080/#/c/15531/45/be/src/util/sse2neon.h@413 PS45, Line 413: // https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_set1_epi64x=4961 line too long (97 > 90) http://gerrit.cloudera.org:8080/#/c/15531/45/be/src/util/sse2neon.h@1054 PS45, Line 1054: // https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_shuffle_epi8=5146 line too long (98 > 90) http://gerrit.cloudera.org:8080/#/c/15531/45/be/src/util/sse2neon.h@1199 PS45, Line 1199: // https://docs.microsoft.com/en-us/previous-versions/visualstudio/visual-studio-2010/y41dkk37(v=vs.100) line too long (104 > 90) http://gerrit.cloudera.org:8080/#/c/15531/45/be/src/util/sse2neon.h@1645 PS45, Line 1645: // https://software.intel.com/sites/landingpage/IntrinsicsGuide/#text=_mm_test_all_zeros=5871 line too long (100 > 90) http://gerrit.cloudera.org:8080/#/c/15531/45/be/src/util/sse2neon.h@3581 PS45, Line 3581: // https://github.com/ColinIanKing/linux-next-mirror/blob/b5f466091e130caaf0735976648f72bd5e09aa84/crypto/aegis128-neon-inner.c#L52 line too long (131 > 90) http://gerrit.cloudera.org:8080/#/c/15531/45/be/src/util/sse2neon.h@3681 PS45, Line 3681: // cpp-compiler-developer-guide-and-reference-allocating-and-freeing-aligned-memory-blocks line too long (98 > 90) -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 45 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 19:37:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. Patch Set 45: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6295/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 45 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 19:37:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. Patch Set 45: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 45 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 19:37:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. Patch Set 44: (1 comment) I looked at the startup failure and the problem was pretty clear, so I will rerun with that part of the change removed. http://gerrit.cloudera.org:8080/#/c/15531/44/be/src/codegen/gen_ir_descriptions.py File be/src/codegen/gen_ir_descriptions.py: http://gerrit.cloudera.org:8080/#/c/15531/44/be/src/codegen/gen_ir_descriptions.py@245 PS44, Line 245: if platform.processor() != "aarch64": I missed that this function was removed in http://gerrit.cloudera.org:8080/15683. So I don't think we need to modify this file. That's causing the startup failure cause the file no longer exists -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 44 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 19:37:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
Tim Armstrong has uploaded a new patch set (#45) to the change originally created by zhaoren...@hotmail.com. ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions Replace Intel's SSE instructions with ARM's NEON instructions Replace Intel's crc32 instructions with ARM's instructions Replace Intel's popcntq instruction with ARM's mechanism Replace Intel's pcmpestri and pcmpestrm instructions with ARM mechanism Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 --- M CMakeLists.txt M be/CMakeLists.txt M be/src/benchmarks/bswap-benchmark.cc M be/src/benchmarks/int-hash-benchmark.cc M be/src/codegen/CMakeLists.txt M be/src/codegen/llvm-codegen-test.cc M be/src/exec/delimited-text-parser.inline.h M be/src/kudu/util/block_bloom_filter.cc M be/src/kudu/util/group_varint-inl.h M be/src/kudu/util/group_varint-test.cc A be/src/kudu/util/sse2neon.h M be/src/util/bit-util-test.cc M be/src/util/bit-util.cc M be/src/util/bit-util.h M be/src/util/bloom-filter.cc M be/src/util/bloom-filter.h M be/src/util/hash-util-ir.cc M be/src/util/hash-util.h M be/src/util/sse-util.h A be/src/util/sse2neon.h M bin/rat_exclude_files.txt 21 files changed, 3,966 insertions(+), 35 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/15531/45 -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 45 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7714: try to avoid be test crash in statestore
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16341 Change subject: IMPALA-7714: try to avoid be test crash in statestore .. IMPALA-7714: try to avoid be test crash in statestore We didn't get to a clear root cause for this, so I'm going to try two things. First, under the theory that the problem is somehow the destruction of the strings, convert them to char char* which does not require destruction on process teardown. Second, add some logging if the map lookup fails so we can better understand what may have happened. Change-Id: Id4363a93addb8a808d292906cac44ebd25c16889 --- M be/src/statestore/statestore.cc M be/src/statestore/statestore.h 2 files changed, 8 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/16341/1 -- To view, visit http://gerrit.cloudera.org:8080/16341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id4363a93addb8a808d292906cac44ebd25c16889 Gerrit-Change-Number: 16341 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-9989 Improve admission control pool stats logging
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/16220 ) Change subject: IMPALA-9989 Improve admission control pool stats logging .. Patch Set 36: (4 comments) http://gerrit.cloudera.org:8080/#/c/16220/34/be/src/runtime/mem-tracker.cc File be/src/runtime/mem-tracker.cc: http://gerrit.cloudera.org:8080/#/c/16220/34/be/src/runtime/mem-tracker.cc@461 PS34, Line 461: heavMemoryQuery.__set_queryId(tracker->query_id_); > nit: can you print the mem-tracker->label here so thats its easy to debug i can you address this too http://gerrit.cloudera.org:8080/#/c/16220/36/be/src/scheduling/admission-controller.h File be/src/scheduling/admission-controller.h: http://gerrit.cloudera.org:8080/#/c/16220/36/be/src/scheduling/admission-controller.h@640 PS36, Line 640: friend class MemTracker; why do we need to add this as a friend class? http://gerrit.cloudera.org:8080/#/c/16220/36/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/16220/36/be/src/scheduling/admission-controller.cc@1626 PS36, Line 1626: << " Details:" << queue_node->not_admitted_details; see comment in generate_error_codes.py http://gerrit.cloudera.org:8080/#/c/16220/36/common/thrift/generate_error_codes.py File common/thrift/generate_error_codes.py: http://gerrit.cloudera.org:8080/#/c/16220/36/common/thrift/generate_error_codes.py@337 PS36, Line 337: Details: nit: details might not exist. So can get confusing if its left empty. You can probably just add $3 and append an empty string if it doesnt exist. -- To view, visit http://gerrit.cloudera.org:8080/16220 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 Gerrit-Change-Number: 16220 Gerrit-PatchSet: 36 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 18:48:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9225: Query option for retryable queries to spool all results before returning any to the client
Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/16323 )
Change subject: IMPALA-9225: Query option for retryable queries to spool all
results before returning any to the client
..
Patch Set 6:
(10 comments)
starting to look good
http://gerrit.cloudera.org:8080/#/c/16323/6//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16323/6//COMMIT_MSG@20
PS6, Line 20: spool_all_results_for_retries
might be worth mentioning that if retry_failed_queries = false or if
spool_query_results = false, this option won't take any effect either.
http://gerrit.cloudera.org:8080/#/c/16323/6//COMMIT_MSG@23
PS6, Line 23: To implement this, this patch defers the time when a retryable
query
: runs into the FINISHED stat
I think this merits a bit more clarification. in the normal case, a query is in
the FINISHED state as soon as rows are ready to be fetched. conceptually, we
aren't really changing anything about this state, even when this feature is
triggered , a transition to the FINISHED state indicates that rows are ready to
be fetched.
the only difference is what it means for "rows are ready to be fetched". when
this feature is *not* triggered, rows are ready to be fetched as soon as *any*
rows are available. when this feature is triggered, rows are only ready to be
fetched when *all* rows are available.
http://gerrit.cloudera.org:8080/#/c/16323/6//COMMIT_MSG@25
PS6, Line 25: results spooling will be
: implicitly enabled
I think this feature should also be dependent on result spooling being enabled
by the user (e.g. spool_query_results = true). otherwise users might find it a
bit confusing that result spooling is used even when spool_query_results is
false.
I think users might want to set spool_all_results_for_retries to false if they
find that spooling all results introduces too much overhead.
I think it would be rare that they set spool_query_results = false, the only
reason I can see that happening is if there is a bug in the implementation.
http://gerrit.cloudera.org:8080/#/c/16323/6/be/src/exec/buffered-plan-root-sink.cc
File be/src/exec/buffered-plan-root-sink.cc:
http://gerrit.cloudera.org:8080/#/c/16323/6/be/src/exec/buffered-plan-root-sink.cc@82
PS6, Line 82: if (state->query_options().retry_failed_queries
: &&
state->query_options().spool_all_results_for_retries) {
: state->LogError(ErrorMsg(TErrorCode::GENERAL, "Cannot
spool all results. "
: "Query retry will be skipped if any results have
been returned."));
it would be nice if this log statement was moved elsewhere (perhaps the
coordinator?) to avoid leaking the query retry feature into the result spooling
implementation. right now the two are separate, and it would be nice to keep it
that way.
http://gerrit.cloudera.org:8080/#/c/16323/6/be/src/exec/buffered-plan-root-sink.cc@87
PS6, Line 87: discard_result(spooled_all_results_.Set(Status::OK()));
so I think what is happening here is that if the queue is full, and no more
results can be spooled. we set spooled_all_results_ in order to unblock the
coordinator, and as long as no results have been fetched, the query can still
be retried, is that correct?
that is definitely a nice property to have here. the one nit is that perhaps we
should rename spooled_all_results_ to something else then? technically this
line is setting spooled_all_results_ to true, even though all results have not
been spooled.
http://gerrit.cloudera.org:8080/#/c/16323/6/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/16323/6/be/src/runtime/coordinator.cc@803
PS6, Line 803: Status status =
UpdateExecState(coord_instance_->WaitForOpen(),
nit: the typical pattern is to wrap the call to `UpdateExecState` in
`RETURN_IF_ERROR`, and then return Status::OK() on line 814
http://gerrit.cloudera.org:8080/#/c/16323/6/be/src/runtime/fragment-instance-state.cc
File be/src/runtime/fragment-instance-state.cc:
http://gerrit.cloudera.org:8080/#/c/16323/6/be/src/runtime/fragment-instance-state.cc@91
PS6, Line 91: VLOG_QUERY << "Set opened_promise_ after Prepare()";
nit: delete?
http://gerrit.cloudera.org:8080/#/c/16323/6/be/src/runtime/fragment-instance-state.cc@100
PS6, Line 100: VLOG_QUERY << "Set opended_promise_ after Open";
nit: delete?
http://gerrit.cloudera.org:8080/#/c/16323/6/be/src/runtime/fragment-instance-state.cc@112
PS6, Line 112:
reinterpret_cast(sink_)->MarkSpooledAllResults(status);
can you just do this in the FlushFinal method of BufferedPlanRootSink? I don't
think setting the spooled_all_results_ flag has to be done specifically when
query retries are enabled either
http://gerrit.cloudera.org:8080/#/c/16323/6/be/src/runtime/fragment-instance-state.cc@438
PS6, Line 438:
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. Patch Set 44: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6292/ -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 44 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 17:25:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10039 (part 2): Fixed Expr-test crash due to race condition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16313 ) Change subject: IMPALA-10039 (part 2): Fixed Expr-test crash due to race condition .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6294/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16313 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib0d3b9c59924a25b70fa20afeb6e8ca93016eca9 Gerrit-Change-Number: 16313 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 14 Aug 2020 17:19:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7779 Parquet Scanner can write binary data into profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16331 ) Change subject: IMPALA-7779 Parquet Scanner can write binary data into profile .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I281d6fa7cb2f88f04588110943e3e768678b9cf1 Gerrit-Change-Number: 16331 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 14 Aug 2020 17:18:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7779 Parquet Scanner can write binary data into profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16331 ) Change subject: IMPALA-7779 Parquet Scanner can write binary data into profile .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6293/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I281d6fa7cb2f88f04588110943e3e768678b9cf1 Gerrit-Change-Number: 16331 Gerrit-PatchSet: 4 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 14 Aug 2020 17:18:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7779 Parquet Scanner can write binary data into profile
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16331 ) Change subject: IMPALA-7779 Parquet Scanner can write binary data into profile .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I281d6fa7cb2f88f04588110943e3e768678b9cf1 Gerrit-Change-Number: 16331 Gerrit-PatchSet: 3 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 14 Aug 2020 17:18:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10039 (part 2): Fixed Expr-test crash due to race condition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16313 ) Change subject: IMPALA-10039 (part 2): Fixed Expr-test crash due to race condition .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6931/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16313 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib0d3b9c59924a25b70fa20afeb6e8ca93016eca9 Gerrit-Change-Number: 16313 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 14 Aug 2020 17:17:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10039 (part 2): Fixed Expr-test crash due to race condition
Wenzhe Zhou has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/16313 ) Change subject: IMPALA-10039 (part 2): Fixed Expr-test crash due to race condition .. IMPALA-10039 (part 2): Fixed Expr-test crash due to race condition The root cause for crash is that QueryState::Cancel() was called before thread unsafe function QueryState::Init() was completed. This patch fixs the race condition between QueryState::Cancel() and QueryState::Init(). QueryState::Init() is safe to be called at any time. Testing: - The issue could be reproduced by running expr-test for 10-20 iterations. Verified the fixing by running expr-test over 1000 iterations without crash. - Passed TestProcessFailures::test_kill_coordinator. - Passed core tests. Change-Id: Ib0d3b9c59924a25b70fa20afeb6e8ca93016eca9 --- M be/src/runtime/query-state.cc M be/src/runtime/query-state.h 2 files changed, 32 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/16313/3 -- To view, visit http://gerrit.cloudera.org:8080/16313 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib0d3b9c59924a25b70fa20afeb6e8ca93016eca9 Gerrit-Change-Number: 16313 Gerrit-PatchSet: 3 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-7779 Parquet Scanner can write binary data into profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16331 ) Change subject: IMPALA-7779 Parquet Scanner can write binary data into profile .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6930/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I281d6fa7cb2f88f04588110943e3e768678b9cf1 Gerrit-Change-Number: 16331 Gerrit-PatchSet: 3 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 14 Aug 2020 16:45:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7782: fix constant NOT IN subqueries that can return 0 rows
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16338 ) Change subject: IMPALA-7782: fix constant NOT IN subqueries that can return 0 rows .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6929/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I66c726f0f66ce2f609e6ba44057191f5929a67fc Gerrit-Change-Number: 16338 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 14 Aug 2020 16:31:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7779 Parquet Scanner can write binary data into profile
Qifan Chen has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/16331 ) Change subject: IMPALA-7779 Parquet Scanner can write binary data into profile .. IMPALA-7779 Parquet Scanner can write binary data into profile This fix addresses the current limitation in that an ill-formatted Parquet version string is not properly formatted before appearing in an error message or impalad.INFO. With the fix, any such string is converted to a hex string first. The hex string is a sequence of four hex digit groups separated by spaces and each group is one or two hex digits, such as "6c 65 2e a". Testing: Ran "core" tests successfully. Change-Id: I281d6fa7cb2f88f04588110943e3e768678b9cf1 --- M be/src/exec/parquet/hdfs-parquet-scanner.cc M common/thrift/generate_error_codes.py M testdata/workloads/functional-query/queries/QueryTest/parquet.test 3 files changed, 4 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/16331/3 -- To view, visit http://gerrit.cloudera.org:8080/16331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I281d6fa7cb2f88f04588110943e3e768678b9cf1 Gerrit-Change-Number: 16331 Gerrit-PatchSet: 3 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. Patch Set 44: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6292/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 44 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 16:16:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. Patch Set 44: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 44 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 16:16:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15531 ) Change subject: IMPALA-9544 Replace Intel's SSE instructions with ARM's NEON instructions .. Patch Set 43: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15531 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7dfe17125b2910ece54e7dd18b4e4b25d7de8b9 Gerrit-Change-Number: 15531 Gerrit-PatchSet: 43 Gerrit-Owner: Anonymous Coward Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 16:16:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7782: fix constant NOT IN subqueries that can return 0 rows
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16338 Change subject: IMPALA-7782: fix constant NOT IN subqueries that can return 0 rows .. IMPALA-7782: fix constant NOT IN subqueries that can return 0 rows The bug was the the statement rewriter converted NOT IN predicates to != predicates when the subquery could be an empty set. This was invalid, because NOT IN () is true, but != () is false. Testing: Added targeted planner and end-to-end tests. Ran exhaustive tests. Change-Id: I66c726f0f66ce2f609e6ba44057191f5929a67fc --- M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test M testdata/workloads/functional-query/queries/QueryTest/subquery-in-constant-lhs.test 3 files changed, 115 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/16338/1 -- To view, visit http://gerrit.cloudera.org:8080/16338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I66c726f0f66ce2f609e6ba44057191f5929a67fc Gerrit-Change-Number: 16338 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-9989 Improve admission control pool stats logging
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16220 ) Change subject: IMPALA-9989 Improve admission control pool stats logging .. Patch Set 36: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6928/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16220 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 Gerrit-Change-Number: 16220 Gerrit-PatchSet: 36 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 14:26:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9989 Improve admission control pool stats logging
Qifan Chen has uploaded a new patch set (#36). ( http://gerrit.cloudera.org:8080/16220 ) Change subject: IMPALA-9989 Improve admission control pool stats logging .. IMPALA-9989 Improve admission control pool stats logging This work addresses the current limitation in admission controller by appending the last known memory consumption statistics about a pool to the existing memory exhaustion message. The statistics is logged in impalad.INFO when a query is queued or timed out due to memory pressure in the pool or on the host. The statistics can also be part of the query profile. The BNF of the new memory consumption statistics is as follows. topN_query_stats ::= queries: a list of query Ids and memory consumed for up to 5 queries with top memory consumptions total_consumed: total memory consumed by these topN queries fraction_of_pool_total_mem: total memory consumed divided by pool memory usage (if feasible to report) all_query_stats ::= num_running: the total number of queries running min: the minimal memory consumption of all running queries max: the maximal memory consumption of all running queries pool_total_mem: the total memory consumption of all running queries average: the average memory consumption of all running queries (if feasible to report) pool_stats ::= ":" stats_on_host ::= "Stats for host " List of aggregated_pool_stats ::= "Aggregated stats for pool " memory_consumption_statistics ::= | The stats_on_host describes memory consumption for every pool on a host and is useful in analyzing memory exhaustion on that host. The aggregated_pool_stats describes the aggregated memory consumption on all hosts for a pool for a set of queries and is useful in analyzing memory exhaustion in that pool. Example of stats_on_host for pool root.queueB and root.queueC on host host1:25000. Stats for host host1:25000 pool_name=root.queueB: topN_query_stats: queries=[ id=0001:0004, consumed=20.00 MB, id=0001:0003, consumed=19.00 MB, id=0001:0002, consumed=8.00 MB ], total_consumed=47.00 MB fraction_of_pool_total_mem=0.47 all_query_stats: num_running=4, min=5.00 MB, max=20.00 MB, pool_total_mem=100.00 MB, average=25.00 MB pool_name=root.queueC: topN_query_stats: queries=[ id=0002:, consumed=18.00 MB, id=0002:0001, consumed=12.00 MB ], total_consumed=30.00 MB fraction_of_pool_total_mem=0.06 all_query_stats: num_running=40, min=10.00 MB, max=200.00 MB, pool_total_mem=500.00 MB, average=12.50 MB Example of aggregated_pool_stats over all hosts for pool root.queueC: Aggregated stats for pool root.queueC: topN_query_stats: queries=[ id=0002:0001, consumed=32.00 MB, id=0002:0004, consumed=26.00 MB, id=0002:, consumed=21.00 MB, id=0002:0002, consumed=17.00 MB, id=0002:000e, consumed=9.00 MB ], total_consumed=105.00 MB fraction_of_pool_total_mem=0.82 When a query request is queued due to memory exhaustion, the above memory_consumption_statistics is logged when the logging is set at level 2 or higher. When a query request is timed out due to memory exhaustion, the above memory_consumption_statistics is reported when the logging is set at level 1 or higher. Testing: 1. Added a new test TopNQueryCheck in admission-controller-test.cc to verify that the topN query memory consumption details are reported correctly. 2. Add two new tests in test_admission_controller.py to simulate queries being queued and then timed out due to pool or host memory pressure. 3. Added a new test TopN in mem-tracker-test.cc to verify that the topN query memory consumption details are computed correctly from a mem tracker hierarchy. 4. Ran Core tests successfully. Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 --- M be/src/runtime/mem-tracker-test.cc M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/util/container-util.h M common/thrift/StatestoreService.thrift M common/thrift/generate_error_codes.py M tests/custom_cluster/test_admission_controller.py 10 files changed, 914 insertions(+), 47
[Impala-ASF-CR] IMPALA-7658: Proper codegen for HiveUdfCall
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16314 ) Change subject: IMPALA-7658: Proper codegen for HiveUdfCall .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/6927/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16314 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2f994dac550f297ed3c88491816403f237d4d747 Gerrit-Change-Number: 16314 Gerrit-PatchSet: 7 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 09:45:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7658: Proper codegen for HiveUdfCall
Daniel Becker has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/16314 ) Change subject: IMPALA-7658: Proper codegen for HiveUdfCall .. IMPALA-7658: Proper codegen for HiveUdfCall Implementing codegen for HiveUdfCall. TODO: Testing TODO: Benchmarks Change-Id: I2f994dac550f297ed3c88491816403f237d4d747 --- M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/impala-ir.cc M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/exprs/CMakeLists.txt A be/src/exprs/hive-udf-call-ir.cc M be/src/exprs/hive-udf-call.cc M be/src/exprs/hive-udf-call.h 8 files changed, 471 insertions(+), 39 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/16314/7 -- To view, visit http://gerrit.cloudera.org:8080/16314 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2f994dac550f297ed3c88491816403f237d4d747 Gerrit-Change-Number: 16314 Gerrit-PatchSet: 7 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7658: Proper codegen for HiveUdfCall
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16314 ) Change subject: IMPALA-7658: Proper codegen for HiveUdfCall .. Patch Set 6: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/6926/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/16314 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2f994dac550f297ed3c88491816403f237d4d747 Gerrit-Change-Number: 16314 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 14 Aug 2020 08:25:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7658: Proper codegen for HiveUdfCall
Daniel Becker has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/16314 ) Change subject: IMPALA-7658: Proper codegen for HiveUdfCall .. IMPALA-7658: Proper codegen for HiveUdfCall Implementing codegen for HiveUdfCall. TODO: Testing TODO: Benchmarks Change-Id: I2f994dac550f297ed3c88491816403f237d4d747 --- M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/impala-ir.cc M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/exprs/CMakeLists.txt A be/src/exprs/hive-udf-call-ir.cc M be/src/exprs/hive-udf-call.cc M be/src/exprs/hive-udf-call.h 8 files changed, 470 insertions(+), 39 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/16314/6 -- To view, visit http://gerrit.cloudera.org:8080/16314 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2f994dac550f297ed3c88491816403f237d4d747 Gerrit-Change-Number: 16314 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-5022: Outer join simplification
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16266 ) Change subject: IMPALA-5022: Outer join simplification .. Patch Set 8: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6291/ -- To view, visit http://gerrit.cloudera.org:8080/16266 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa7804033fac68e93f33c387dc68ef67f803e93e Gerrit-Change-Number: 16266 Gerrit-PatchSet: 8 Gerrit-Owner: Xianqing He Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Xianqing He Gerrit-Comment-Date: Fri, 14 Aug 2020 06:41:40 + Gerrit-HasComments: No
