[Impala-ASF-CR] IMPALA-7667: sentry.db.explicit.grants.permitted does not accept empty value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11592 ) Change subject: IMPALA-7667: sentry.db.explicit.grants.permitted does not accept empty value .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/957/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcb80da6ba37405ce59e2d5cd79916f8eb004ec Gerrit-Change-Number: 11592 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 05 Oct 2018 04:42:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query With this patch the per host memory limit of a query is automatically set using the mem_limit set in the query options and the mem_estimate calculated by the planner based on the following pseudo code: if mem_limit is set in query options: use that and if 'clamp-mem-limit-query-option' is true: enforce the min/max query mem limits defined in the pool config. else: mem_limit = max(mem_estiamte, min_mem_limit_required_to_accomodate_largest_initial_reservation) finally, enforce min/max query mem limits defined in the pool config on this value. This calculated mem limit will also be used for admission accounting and consequently for admission control. Moreover, three new pool configuration options have been added to enable this behaviour: "min-query-mem-limit" & "max-query-mem-limit" => help clamp the per host memory limit for a query. If both these limits are not configured, then the estimates from planning are not used as a memory limit and only used for making admission decisions. Moreover the estimates will no longer have a lower bound based on the largest initial reservation. "clamp-mem-limit-query-option" => if false, the mem_limit defined in the query options is used directly and the max/min query mem limits are not enforced on it. Testing: Added e2e test cases. Added frontend tests for changes to RequestPoolService. Successfully passed exhaustive tests. Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Reviewed-on: http://gerrit.cloudera.org:8080/11157 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/benchmarks/process-wide-locks-benchmark.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator.cc M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h M be/src/runtime/query-exec-mgr.cc M be/src/runtime/query-exec-mgr.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-state.cc M be/src/runtime/test-env.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/request-pool-service.cc M be/src/scheduling/scheduler.cc M be/src/service/client-request-state.cc M common/thrift/ImpalaInternalService.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/util/RequestPoolService.java M fe/src/test/java/org/apache/impala/util/TestRequestPoolService.java M fe/src/test/resources/fair-scheduler-test.xml M fe/src/test/resources/llama-site-test.xml A fe/src/test/resources/mem-limit-test-fair-scheduler.xml A fe/src/test/resources/mem-limit-test-llama-site.xml A testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test M testdata/workloads/functional-query/queries/QueryTest/admission-reject-min-reservation.test A tests/common/resource_pool_config.py M tests/custom_cluster/test_admission_controller.py 30 files changed, 1,075 insertions(+), 266 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 20 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. Patch Set 19: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 19 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 05 Oct 2018 04:38:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7667: sentry.db.explicit.grants.permitted does not accept empty value
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11592 Change subject: IMPALA-7667: sentry.db.explicit.grants.permitted does not accept empty value .. IMPALA-7667: sentry.db.explicit.grants.permitted does not accept empty value Empty value in sentry.db.explicit.grants.permitted means allow all privileges to be explicitly granted. However a bug in SENTRY-2424 causes empty value to be ignored. The patch uses a workaround to add a space to mean the same thing as an empty value. The patch also upgrades CDH_BUILD_NUMBER to 618371 to enable sentry.db.explicit.grants.permitted configuration. Testing: - Ran all core tests Change-Id: Iddcb80da6ba37405ce59e2d5cd79916f8eb004ec --- M bin/impala-config.sh M fe/src/test/resources/sentry-site.xml.template M fe/src/test/resources/sentry-site_no_oo.xml.template M fe/src/test/resources/sentry-site_oo.xml.template M fe/src/test/resources/sentry-site_oo_nogrant.xml.template 5 files changed, 10 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/11592/1 -- To view, visit http://gerrit.cloudera.org:8080/11592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iddcb80da6ba37405ce59e2d5cd79916f8eb004ec Gerrit-Change-Number: 11592 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-7667: sentry.db.explicit.grants.permitted does not accept empty value
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11592 ) Change subject: IMPALA-7667: sentry.db.explicit.grants.permitted does not accept empty value .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3277/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/11592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddcb80da6ba37405ce59e2d5cd79916f8eb004ec Gerrit-Change-Number: 11592 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 05 Oct 2018 04:12:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7648 part 1: add expected out-of-memory tests
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11564 ) Change subject: IMPALA-7648 part 1: add expected out-of-memory tests .. IMPALA-7648 part 1: add expected out-of-memory tests This adds test coverage for some cases where queries are currently expected to fail with out-of-memory: * memory limit exceeded in exchange node * aggregation with large var-len intermediate values * top N with large limit * hash join with many duplicates on right side * analytic with a large window that needs to be buffered in-memory Note that it's not always totally deterministic where the query hits 'memory limit exceeded' so we don't include the node ID or name in the expected error message. Testing: * ran exhaustive tests * looped modified tests locally overnight Change-Id: Icd1a7eb97837b742a967c260cafb5a7f4f45412e Reviewed-on: http://gerrit.cloudera.org:8080/11564 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- A testdata/workloads/functional-query/queries/QueryTest/exchange-mem-scaling.test M testdata/workloads/functional-query/queries/QueryTest/spilling-no-debug-action.test M tests/query_test/test_mem_usage_scaling.py 3 files changed, 104 insertions(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11564 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Icd1a7eb97837b742a967c260cafb5a7f4f45412e Gerrit-Change-Number: 11564 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7626: Throttle catalog partial RPC requests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11561 ) Change subject: IMPALA-7626: Throttle catalog partial RPC requests .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/956/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11561 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11f77a16cfa38ada42d8b7c859850198ea7dd142 Gerrit-Change-Number: 11561 Gerrit-PatchSet: 8 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 05 Oct 2018 02:34:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7659: Simplify expression to collect NULLs count
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11565 ) Change subject: IMPALA-7659: Simplify expression to collect NULLs count .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3271/ -- To view, visit http://gerrit.cloudera.org:8080/11565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a Gerrit-Change-Number: 11565 Gerrit-PatchSet: 5 Gerrit-Owner: piotr.findei...@gmail.com Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: piotr.findei...@gmail.com Gerrit-Comment-Date: Fri, 05 Oct 2018 02:29:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7484: Do not interpret unrecognized hints as straight join hints.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3276/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 10 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 05 Oct 2018 02:14:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7626: Throttle catalog partial RPC requests
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11561 ) Change subject: IMPALA-7626: Throttle catalog partial RPC requests .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11561 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11f77a16cfa38ada42d8b7c859850198ea7dd142 Gerrit-Change-Number: 11561 Gerrit-PatchSet: 8 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 05 Oct 2018 02:12:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7484: Do not interpret unrecognized hints as straight join hints.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 10 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 05 Oct 2018 02:14:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7626: Throttle catalog partial RPC requests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11561 ) Change subject: IMPALA-7626: Throttle catalog partial RPC requests .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3275/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/11561 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11f77a16cfa38ada42d8b7c859850198ea7dd142 Gerrit-Change-Number: 11561 Gerrit-PatchSet: 8 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 05 Oct 2018 02:13:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7484: Do not interpret unrecognized hints as straight join hints.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 9 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 05 Oct 2018 02:12:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7626: Throttle catalog partial RPC requests
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/11561 )
Change subject: IMPALA-7626: Throttle catalog partial RPC requests
..
Patch Set 7:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/11561/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/11561/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2119
PS7, Line 2119: (
> nit: weird formatting
Oops IDE issues.
http://gerrit.cloudera.org:8080/#/c/11561/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2131
PS7, Line 2131: // this method.
> is there a chance that we get interrupted right on this line and then miss
Like we discussed offline, there is a very very minor chance of the thread
getting interrupted at this point and if that happens, the permit gets blocked
forever. Good part though is that the subsequent requests should throw the
time-out exception and is easy to diagnose and if someone looks at the queue
length after seeing those exceptions.
There is another variant of this method which uses an Atomic variable to see if
it has been set. But that is equally suscpetible to this problem if the thread
gets interrupted right after tryAcquire() is successful but getAndSet is not
run.
I'll add a comment here about it.
public TGetPartialCatalogObjectResponse getPartialCatalogObject
(TGetPartialCatalogObjectRequest req) throws CatalogException {
AtomicBoolean isPermitAcquired = new AtomicBoolean(false);
try {
isPermitAcquired.getAndSet(partialObjectFetchAccess_.tryAcquire(1,
PARTIAL_FETCH_RPC_QUEUE_TIMEOUT_S, TimeUnit.SECONDS));
if (!isPermitAcquired.get()) {
// Timed out trying to acquire the semaphore permit.
throw new CatalogException("Timed out while fetching partial object
metadata. " +
"Please check the metric 'catalog.partial-fetch-rpc.queue-len' for
the " +
"current queue length and consider increasing " +
"'catalog_partial_fetch_rpc_queue_timeout_s' and/or " +
"'catalog_max_parallel_partial_fetch_rpc'");
}
return doGetPartialCatalogObject(req);
} catch (InterruptedException e) {
throw new CatalogException("Error running getPartialCatalogObject(): ",
e);
} finally {
if (isPermitAcquired.get()) partialObjectFetchAccess_.release();
}
}
--
To view, visit http://gerrit.cloudera.org:8080/11561
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11f77a16cfa38ada42d8b7c859850198ea7dd142
Gerrit-Change-Number: 11561
Gerrit-PatchSet: 7
Gerrit-Owner: Bharath Vissapragada
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Fri, 05 Oct 2018 01:52:42 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7626: Throttle catalog partial RPC requests
Hello Tianyi Wang, Philip Zeyliger, Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11561 to look at the new patch set (#8). Change subject: IMPALA-7626: Throttle catalog partial RPC requests .. IMPALA-7626: Throttle catalog partial RPC requests With more coordinators running in local catalog mode, the expected RPC traffic on the Catalog server is higher compared to the non-local-catalog mode. Each such RPC is handled in its own thread and consumes some non-trivial CPU for serializing and deserializing the metadata. With this change, the maximum number of threads performing the actual work are capped to a certain limit at any point and the remaining requests (if any) are blocked until the current requests are serviced or they they exceed the configured timeout and abort. Adds the following parameters for controlling this behavior. --catalog_max_parallel_partial_fetch_rpc --catalog_partial_fetch_rpc_queue_timeout_s --catalog_partial_fetch_rpc_queue_timeout_s controls the timeout for queued requests. Added some basic supportability to examine the queue length via metrics. Added a unit test to make sure the concurrent requests for this RPC method does not exceed the configured value. Change-Id: I11f77a16cfa38ada42d8b7c859850198ea7dd142 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/JniCatalog.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoTest.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java 13 files changed, 275 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/11561/8 -- To view, visit http://gerrit.cloudera.org:8080/11561 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I11f77a16cfa38ada42d8b7c859850198ea7dd142 Gerrit-Change-Number: 11561 Gerrit-PatchSet: 8 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7626: Throttle catalog partial RPC requests
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11561 ) Change subject: IMPALA-7626: Throttle catalog partial RPC requests .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/11561/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/11561/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2131 PS7, Line 2131: // this method. > Like we discussed offline, there is a very very minor chance of the thread I google this but don't see any other interesting approaches (may be my search skills are bad too). Phil, what do you think? -- To view, visit http://gerrit.cloudera.org:8080/11561 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11f77a16cfa38ada42d8b7c859850198ea7dd142 Gerrit-Change-Number: 11561 Gerrit-PatchSet: 7 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 05 Oct 2018 01:56:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7661: Increase the sleep time in test reconnect
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11589 ) Change subject: IMPALA-7661: Increase the sleep time in test_reconnect .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/955/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11589 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia009808adac0da1cfa00b9e9dd41cc276d49c6eb Gerrit-Change-Number: 11589 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 05 Oct 2018 01:46:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7626: Throttle catalog partial RPC requests
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11561 ) Change subject: IMPALA-7626: Throttle catalog partial RPC requests .. Patch Set 7: (3 comments) lgtm. question about a possible race. perhaps its not an issue. http://gerrit.cloudera.org:8080/#/c/11561/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/11561/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2115 PS7, Line 2115:* A wrapper around doGetPartialCatalogObject() that controls the number concurrent nit: of http://gerrit.cloudera.org:8080/#/c/11561/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2119 PS7, Line 2119: ( nit: weird formatting http://gerrit.cloudera.org:8080/#/c/11561/7/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2131 PS7, Line 2131: // this method. is there a chance that we get interrupted right on this line and then miss the release in the finally clause? -- To view, visit http://gerrit.cloudera.org:8080/11561 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11f77a16cfa38ada42d8b7c859850198ea7dd142 Gerrit-Change-Number: 11561 Gerrit-PatchSet: 7 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 05 Oct 2018 01:29:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7648 part 1: add expected out-of-memory tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11564 ) Change subject: IMPALA-7648 part 1: add expected out-of-memory tests .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11564 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icd1a7eb97837b742a967c260cafb5a7f4f45412e Gerrit-Change-Number: 11564 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 05 Oct 2018 01:36:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7660: Support ECDH ciphers for debug webserver
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11585 ) Change subject: IMPALA-7660: Support ECDH ciphers for debug webserver .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3273/ -- To view, visit http://gerrit.cloudera.org:8080/11585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I80a6b370d5860812cde13229b5bcb2977814c73c Gerrit-Change-Number: 11585 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 05 Oct 2018 01:22:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7661: Increase the sleep time in test reconnect
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11589 Change subject: IMPALA-7661: Increase the sleep time in test_reconnect .. IMPALA-7661: Increase the sleep time in test_reconnect test_reconnect is flaky in ASAN because the time waited for impala shell to connect to impalad is not enough. This patch increases the sleep time from 2 secs to 5 secs. Change-Id: Ia009808adac0da1cfa00b9e9dd41cc276d49c6eb --- M tests/shell/test_shell_interactive.py 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/11589/1 -- To view, visit http://gerrit.cloudera.org:8080/11589 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia009808adac0da1cfa00b9e9dd41cc276d49c6eb Gerrit-Change-Number: 11589 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. Patch Set 18: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/954/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 18 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 05 Oct 2018 00:41:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7543: Enhance scan ranges to support sub-ranges
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11520 ) Change subject: IMPALA-7543: Enhance scan ranges to support sub-ranges .. Patch Set 3: (4 comments) I'm making my way through this. First round of comments. At a high level, I think another dimension of the design is how it interacts with future efforts to manage IO patterns (e.g. IMPALA-6673). I don't think that should hold up this review, and I don't see a problem with the SubRange abstraction. http://gerrit.cloudera.org:8080/#/c/11520/3/be/src/runtime/io/file-reader.h File be/src/runtime/io/file-reader.h: http://gerrit.cloudera.org:8080/#/c/11520/3/be/src/runtime/io/file-reader.h@49 PS3, Line 49: 'eosr' is set to true when end of file has reached, : /// or the file reader has read all the bytes needed by 'scan_range_'. Depending on what we do with eosr, we may need to update this comment. http://gerrit.cloudera.org:8080/#/c/11520/3/be/src/runtime/io/hdfs-file-reader.cc File be/src/runtime/io/hdfs-file-reader.cc: http://gerrit.cloudera.org:8080/#/c/11520/3/be/src/runtime/io/hdfs-file-reader.cc@155 PS3, Line 155: *eosr = true; eosr is a bit weird now. It can detect end of file, but not end of scan range. I'll be taking another look at eosr and how it gets set, but we might end up renaming this variable to be eof. I might be missing something, but the eosr logic seems like it could be implemented here. We have access to the scan_range_ and can increment bytes_read_. The ScanRange has bytes_to_read_ which is similar to len() today. http://gerrit.cloudera.org:8080/#/c/11520/3/be/src/runtime/io/request-ranges.h File be/src/runtime/io/request-ranges.h: http://gerrit.cloudera.org:8080/#/c/11520/3/be/src/runtime/io/request-ranges.h@416 PS3, Line 416: /// They need to be ordered by offset and cannot overlap. I think it would be useful to combine contiguous SubRanges into a single SubRange. (It doesn't necessarily need to happen in this code change.) Should we require that the SubRanges are non-contiguous (forcing the caller to merge them) or should we merge them ourselves? (Is there a time when we wouldn't want to merge contiguous SubRanges?) http://gerrit.cloudera.org:8080/#/c/11520/3/be/src/runtime/io/scan-range.cc File be/src/runtime/io/scan-range.cc: http://gerrit.cloudera.org:8080/#/c/11520/3/be/src/runtime/io/scan-range.cc@268 PS3, Line 268: "There should be no partial reads for sub-ranges." > Yeah, I think it's not necessary to test eosr in the loop condition, but ot About this DCHECK. We have a condition in ReadFromPos() that raises some questions for me: https://github.com/apache/impala/blob/master/be/src/runtime/io/hdfs-file-reader.cc#L152 I'm reading into what would trigger that condition. My current thought: If we issue an hdfsRead() call where the offset is near the end of the file and the length goes beyond the end of the file, does hdfsRead() produce an error or does it do a partial read and return less than the requested amount of data? -- To view, visit http://gerrit.cloudera.org:8080/11520 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea26ba386713990f7671aab5a372cf449b8d51e4 Gerrit-Change-Number: 11520 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 05 Oct 2018 00:28:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7660: Support ECDH ciphers for debug webserver
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11585 ) Change subject: IMPALA-7660: Support ECDH ciphers for debug webserver .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/953/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I80a6b370d5860812cde13229b5bcb2977814c73c Gerrit-Change-Number: 11585 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 05 Oct 2018 00:19:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. Patch Set 18: Code-Review+2 Carrying over +2 -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 18 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 05 Oct 2018 00:10:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7166: ExecSummary should be a first class object.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11555 ) Change subject: IMPALA-7166: ExecSummary should be a first class object. .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/951/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/11555 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4791237a5579f16c9efda8e57876d48980739e13 Gerrit-Change-Number: 11555 Gerrit-PatchSet: 3 Gerrit-Owner: Yongjun Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Yongjun Zhang Gerrit-Comment-Date: Thu, 04 Oct 2018 23:56:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Hello Philip Zeyliger, Tim Armstrong, Impala Public Jenkins, Dan Hecht, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11157 to look at the new patch set (#18). Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query With this patch the per host memory limit of a query is automatically set using the mem_limit set in the query options and the mem_estimate calculated by the planner based on the following pseudo code: if mem_limit is set in query options: use that and if 'clamp-mem-limit-query-option' is true: enforce the min/max query mem limits defined in the pool config. else: mem_limit = max(mem_estiamte, min_mem_limit_required_to_accomodate_largest_initial_reservation) finally, enforce min/max query mem limits defined in the pool config on this value. This calculated mem limit will also be used for admission accounting and consequently for admission control. Moreover, three new pool configuration options have been added to enable this behaviour: "min-query-mem-limit" & "max-query-mem-limit" => help clamp the per host memory limit for a query. If both these limits are not configured, then the estimates from planning are not used as a memory limit and only used for making admission decisions. Moreover the estimates will no longer have a lower bound based on the largest initial reservation. "clamp-mem-limit-query-option" => if false, the mem_limit defined in the query options is used directly and the max/min query mem limits are not enforced on it. Testing: Added e2e test cases. Added frontend tests for changes to RequestPoolService. Successfully passed exhaustive tests. Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 --- M be/src/benchmarks/process-wide-locks-benchmark.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator.cc M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h M be/src/runtime/query-exec-mgr.cc M be/src/runtime/query-exec-mgr.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-state.cc M be/src/runtime/test-env.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/request-pool-service.cc M be/src/scheduling/scheduler.cc M be/src/service/client-request-state.cc M common/thrift/ImpalaInternalService.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/util/RequestPoolService.java M fe/src/test/java/org/apache/impala/util/TestRequestPoolService.java M fe/src/test/resources/fair-scheduler-test.xml M fe/src/test/resources/llama-site-test.xml A fe/src/test/resources/mem-limit-test-fair-scheduler.xml A fe/src/test/resources/mem-limit-test-llama-site.xml A testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test M testdata/workloads/functional-query/queries/QueryTest/admission-reject-min-reservation.test A tests/common/resource_pool_config.py M tests/custom_cluster/test_admission_controller.py 30 files changed, 1,075 insertions(+), 266 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/11157/18 -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 18 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. Patch Set 16: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/952/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 16 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 05 Oct 2018 00:02:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11587 ) Change subject: IMPALA-7643: report # queries actually executing in stress test .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/950/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11587 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335 Gerrit-Change-Number: 11587 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 23:45:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6661 Make NaN values equal for grouping purposes.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11535 ) Change subject: IMPALA-6661 Make NaN values equal for grouping purposes. .. Patch Set 6: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/11535 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I996c4a2e1934fd887046ed0c55457b7285375086 Gerrit-Change-Number: 11535 Gerrit-PatchSet: 6 Gerrit-Owner: Michal Ostrowski Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Michal Ostrowski Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 23:18:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Hello Philip Zeyliger, Tim Armstrong, Impala Public Jenkins, Dan Hecht, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11157 to look at the new patch set (#16). Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query With this patch the per host memory limit of a query is automatically set using the mem_limit set in the query options and the mem_estimate calculated by the planner based on the following pseudo code: if mem_limit is set in query options: use that and if 'clamp-mem-limit-query-option' is true: enforce the min/max query mem limits defined in the pool config. else: mem_limit = max(mem_estiamte, min_mem_limit_required_to_accomodate_largest_initial_reservation) finally, enforce min/max query mem limits defined in the pool config on this value. This calculated mem limit will also be used for admission accounting and consequently for admission control. Moreover, three new pool configuration options have been added to enable this behaviour: "min-query-mem-limit" & "max-query-mem-limit" => help clamp the per host memory limit for a query. If both these limits are not configured, then the estimates from planning are not used as a memory limit and only used for making admission decisions. Moreover the estimates will no longer have a lower bound based on the largest initial reservation. "clamp-mem-limit-query-option" => if false, the mem_limit defined in the query options is used directly and the max/min query mem limits are not enforced on it. Testing: Added e2e test cases. Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 --- M be/src/benchmarks/process-wide-locks-benchmark.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator.cc M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h M be/src/runtime/query-exec-mgr.cc M be/src/runtime/query-exec-mgr.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-state.cc M be/src/runtime/test-env.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/request-pool-service.cc M be/src/scheduling/scheduler.cc M be/src/service/client-request-state.cc M common/thrift/ImpalaInternalService.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/util/RequestPoolService.java M fe/src/test/java/org/apache/impala/util/TestRequestPoolService.java M fe/src/test/resources/fair-scheduler-test.xml M fe/src/test/resources/llama-site-test.xml A fe/src/test/resources/mem-limit-test-fair-scheduler.xml A fe/src/test/resources/mem-limit-test-llama-site.xml A testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test M testdata/workloads/functional-query/queries/QueryTest/admission-reject-min-reservation.test A tests/common/resource_pool_config.py M tests/custom_cluster/test_admission_controller.py 30 files changed, 1,072 insertions(+), 265 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/11157/16 -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 16 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7660: Support ECDH ciphers for debug webserver
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11585 ) Change subject: IMPALA-7660: Support ECDH ciphers for debug webserver .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3273/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I80a6b370d5860812cde13229b5bcb2977814c73c Gerrit-Change-Number: 11585 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 23:44:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7660: Support ECDH ciphers for debug webserver
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/11585 ) Change subject: IMPALA-7660: Support ECDH ciphers for debug webserver .. Patch Set 2: Code-Review+2 (2 comments) carrying forward http://gerrit.cloudera.org:8080/#/c/11585/1/tests/custom_cluster/test_client_ssl.py File tests/custom_cluster/test_client_ssl.py: http://gerrit.cloudera.org:8080/#/c/11585/1/tests/custom_cluster/test_client_ssl.py@116 PS1, Line 116: : > flake8: E203 whitespace before ':' Done http://gerrit.cloudera.org:8080/#/c/11585/1/tests/custom_cluster/test_client_ssl.py@134 PS1, Line 134: : > flake8: E203 whitespace before ':' Done -- To view, visit http://gerrit.cloudera.org:8080/11585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I80a6b370d5860812cde13229b5bcb2977814c73c Gerrit-Change-Number: 11585 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 23:43:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7660: Support ECDH ciphers for debug webserver
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11585 to look at the new patch set (#2). Change subject: IMPALA-7660: Support ECDH ciphers for debug webserver .. IMPALA-7660: Support ECDH ciphers for debug webserver A recent change (IMPALA-7519) added support for ecdh ciphers for the beeswax/hs2 server. This patch pulls in a recent change on squeasel to extend that support to the debug webserver. It also fixes a bug that prevented start-impala-cluster.py from completing successfully when the webserver is launched with ssl, due to it trying to verify the availablitiy of the webserver over http. Testing: - Added a custom cluster test that verifies start-impala-cluster.py runs successfully with webserver ssl enabled. - Adds the webserver to an existing test for ecdh ciphers. Change-Id: I80a6b370d5860812cde13229b5bcb2977814c73c --- M be/src/thirdparty/squeasel/squeasel.c M tests/common/impala_cluster.py M tests/common/impala_service.py M tests/custom_cluster/test_client_ssl.py 4 files changed, 82 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/11585/2 -- To view, visit http://gerrit.cloudera.org:8080/11585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I80a6b370d5860812cde13229b5bcb2977814c73c Gerrit-Change-Number: 11585 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. Patch Set 17: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 17 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 23:41:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. Patch Set 17: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3272/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 17 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 23:41:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11587 ) Change subject: IMPALA-7643: report # queries actually executing in stress test .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/949/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11587 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335 Gerrit-Change-Number: 11587 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 23:40:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. Patch Set 16: Code-Review+2 (1 comment) Carrying over +2 http://gerrit.cloudera.org:8080/#/c/11157/15/testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test File testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test: http://gerrit.cloudera.org:8080/#/c/11157/15/testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test@9 PS15, Line 9: set num_scanner_threads=2; > I think we should set this to 2 or 3, so that it works on smaller machines Done -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 16 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 23:40:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11587 Change subject: IMPALA-7643: report # queries actually executing in stress test .. IMPALA-7643: report # queries actually executing in stress test With admission control it's interesting to separate out two categories of queries: 1. Queries that have started up and are executing 2. Queries that have not made it that far yet, e.g. are waiting to establish a client connection (hitting --fe_service_threads limit), are in the planner, are queued in admission control or are starting up. We now call 1+2 "Active" and 2 "Executing". Example output: Done | Active | Executing | Mem Lmt Ex | AC Reject | AC Timeout | Time Out | Cancel | Err | Incorrect | Next Qry Mem Lmt | Tot Qry Mem Lmt | Tracked Mem | RSS Mem 0 | 0 | 0 | 0 | 0 | 0 |0 | 0 | 0 | 0 |0 | 0 | | 0 | 10 | 3 | 0 | 0 | 0 |0 | 0 | 0 | 0 | 510 |3922 | 158 | 4541 0 | 20 |10 | 0 | 0 | 0 |0 | 0 | 0 | 0 | 390 |8534 | 570 | 4517 Refactored QueryRunner.run_query() to reduce nesting and make it more readable. Testing: Ran local stress tests with and without --test_admission_control set. Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335 --- M tests/stress/concurrent_select.py 1 file changed, 105 insertions(+), 53 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/11587/1 -- To view, visit http://gerrit.cloudera.org:8080/11587 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335 Gerrit-Change-Number: 11587 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7166: ExecSummary should be a first class object.
Yongjun Zhang has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/11555 ) Change subject: IMPALA-7166: ExecSummary should be a first class object. .. IMPALA-7166: ExecSummary should be a first class object. Impala RuntimeProfile currently contains "ExecSummary" as a string. We should make it a first class thrift object, so that tools can extract these fields (Est rows etc..). Testing: Modified unit test. Change-Id: I4791237a5579f16c9efda8e57876d48980739e13 --- M be/src/service/impala-server.cc M be/src/util/runtime-profile-test.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/thrift/RuntimeProfile.thrift 5 files changed, 48 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/11555/3 -- To view, visit http://gerrit.cloudera.org:8080/11555 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4791237a5579f16c9efda8e57876d48980739e13 Gerrit-Change-Number: 11555 Gerrit-PatchSet: 3 Gerrit-Owner: Yongjun Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Yongjun Zhang
[Impala-ASF-CR] Report the top 5 slowest test steps when running on S3
Philip Zeyliger has posted comments on this change. (
http://gerrit.cloudera.org:8080/11574 )
Change subject: Report the top 5 slowest test steps when running on S3
..
Patch Set 1:
Seems reasonable. I've got no objection here as long as it works :)
If you're interested, I have the following questionable code hanging out.
Combined with a little bit of sorting, you might be able to get a sense of what
tests are slow. There's been some question about whether the "time" attribute
is accurate (and some JIRAs about it that I've not dug up). Jenkins definitely
shows wonky-looking times for some of our tests, but I've not investigated.
If you think this is a useful direction, feel free to take this over and we can
stick it in lib/py.
$python ~/src/impala/parsexunit.py $(find . -name '*.xml') | sort -n -k 1 -r |
head
623.787399769 results TEST-impala-parallel.xml
query_test.test_decimal_fuzz.TestDecimalFuzz
query_test.test_decimal_fuzz.TestDecimalFuzz.test_decimal_ops[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0}] success
476.49873209results TEST-impala-custom-cluster.xml
custom_cluster.test_permanent_udfs.TestUdfPersistence
custom_cluster.test_permanent_udfs.TestUdfPersistence.test_java_udfs_hive_integration
success
286.006864071 results TEST-impala-parallel.xml
query_test.test_udfs.TestUdfExecution
query_test.test_udfs.TestUdfExecution.test_native_functions[exec_option:
{'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'exec_single_node_rows_threshold': 100, 'enable_expr_rewrites': True} |
table_format: text/none] success
276.820947886 results TEST-impala-parallel.xml
query_test.test_udfs.TestUdfExecution
query_test.test_udfs.TestUdfExecution.test_native_functions[exec_option:
{'disable_codegen_rows_threshold': 0, 'disable_codegen': False,
'exec_single_node_rows_threshold': 0, 'enable_expr_rewrites': True} |
table_format: text/none] success
256.676662683 results TEST-impala-parallel.xml
metadata.test_ddl.TestDdlStatements
metadata.test_ddl.TestDdlStatements.test_alter_table[exec_option: {'sync_ddl':
0, 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format:
text/none-unique_database0]success
247.263021231 results TEST-impala-parallel.xml
query_test.test_nested_types.TestNestedTypes
query_test.test_nested_types.TestNestedTypes.test_subplan[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
success
227.564936876 results TEST-impala-parallel.xml
query_test.test_spilling.TestSpillingDebugActionDimensions
query_test.test_spilling.TestSpillingDebugActionDimensions.test_spilling[exec_option:
{'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0',
'default_spillable_buffer_size': '256k'} | table_format: parquet/none] success
225.499986172 results TEST-impala-parallel.xml
metadata.test_show_create_table.TestShowCreateTable
metadata.test_show_create_table.TestShowCreateTable.test_show_create_table[table_format:
text/none] success
222.650696039 results TEST-impala-parallel.xml
query_test.test_spilling.TestSpillingDebugActionDimensions
query_test.test_spilling.TestSpillingDebugActionDimensions.test_spilling[exec_option:
{'debug_action': None, 'default_spillable_buffer_size': '256k'} |
table_format: parquet/none] success
211.495414019 results TEST-impala-parallel.xml
query_test.test_udfs.TestUdfTargeted
query_test.test_udfs.TestUdfTargeted.test_concurrent_jar_drop_use[exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': False, 'abort_on_error': 1, 'debug_action': None,
'exec_single_node_rows_threshold': 0} | table_format: text/none] success
$cat parsexunit.py
import sys
import xml.etree.ElementTree as ET
import os
MAX_MESSAGE_LENGTH = 70
def work():
for f in sys.argv[1:]:
tree = ET.parse(file(f))
# TODO: how do we find xfail?
# expected test failure
for testcase in tree.findall(".//testcase"):
state_seen = False
# Default state is SUCCESS
testcase_state = "success"
message = ""
STATES = ("error", "failure", "skipped")
OTHER_CHILDREN = ("system-out", "system-err")
for child in testcase.getchildren():
assert child.tag in STATES or child.tag in OTHER_CHILDREN, child.tag
for state in STATES:
child = testcase.findall(state)
if child:
[Impala-ASF-CR] IMPALA-7166: ExecSummary should be a first class object.
Yongjun Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11555 ) Change subject: IMPALA-7166: ExecSummary should be a first class object. .. Patch Set 3: Thanks for the review Tim. I uploaded new rev to address other comments for completeness here. But I think let me fix IMPALA-6742 first. -- To view, visit http://gerrit.cloudera.org:8080/11555 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4791237a5579f16c9efda8e57876d48980739e13 Gerrit-Change-Number: 11555 Gerrit-PatchSet: 3 Gerrit-Owner: Yongjun Zhang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Yongjun Zhang Gerrit-Comment-Date: Thu, 04 Oct 2018 23:19:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7659: Simplify expression to collect NULLs count
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11565 ) Change subject: IMPALA-7659: Simplify expression to collect NULLs count .. Patch Set 5: I'll start a dry run just to see what tests fail. -- To view, visit http://gerrit.cloudera.org:8080/11565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a Gerrit-Change-Number: 11565 Gerrit-PatchSet: 5 Gerrit-Owner: piotr.findei...@gmail.com Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: piotr.findei...@gmail.com Gerrit-Comment-Date: Thu, 04 Oct 2018 22:46:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11587 ) Change subject: IMPALA-7643: report # queries actually executing in stress test .. Patch Set 1: Fixed bot comments -- To view, visit http://gerrit.cloudera.org:8080/11587 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335 Gerrit-Change-Number: 11587 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 23:13:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Hello Michael Brown, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11587 to look at the new patch set (#2). Change subject: IMPALA-7643: report # queries actually executing in stress test .. IMPALA-7643: report # queries actually executing in stress test With admission control it's interesting to separate out two categories of queries: 1. Queries that have started up and are executing 2. Queries that have not made it that far yet, e.g. are waiting to establish a client connection (hitting --fe_service_threads limit), are in the planner, are queued in admission control or are starting up. We now call 1+2 "Active" and 2 "Executing". Example output: Done | Active | Executing | Mem Lmt Ex | AC Reject | AC Timeout | Time Out | Cancel | Err | Incorrect | Next Qry Mem Lmt | Tot Qry Mem Lmt | Tracked Mem | RSS Mem 0 | 0 | 0 | 0 | 0 | 0 |0 | 0 | 0 | 0 |0 | 0 | | 0 | 10 | 3 | 0 | 0 | 0 |0 | 0 | 0 | 0 | 510 |3922 | 158 | 4541 0 | 20 |10 | 0 | 0 | 0 |0 | 0 | 0 | 0 | 390 |8534 | 570 | 4517 Refactored QueryRunner.run_query() to reduce nesting and make it more readable. Testing: Ran local stress tests with and without --test_admission_control set. Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335 --- M tests/stress/concurrent_select.py 1 file changed, 109 insertions(+), 56 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/11587/2 -- To view, visit http://gerrit.cloudera.org:8080/11587 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335 Gerrit-Change-Number: 11587 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown
[Impala-ASF-CR] IMPALA-7660: Support ECDH ciphers for debug webserver
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11585 ) Change subject: IMPALA-7660: Support ECDH ciphers for debug webserver .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/948/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I80a6b370d5860812cde13229b5bcb2977814c73c Gerrit-Change-Number: 11585 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 23:12:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7575: [DOCS] FMOD() is not the same as the % operator
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11586 ) Change subject: IMPALA-7575: [DOCS] FMOD() is not the same as the % operator .. IMPALA-7575: [DOCS] FMOD() is not the same as the % operator - Removed the text about FMOD being equivalent to %. - Added a note that MOD will show as % in the query plan. Change-Id: I3b02d3e3f556d93e1d651eaee12217d6b0e3f9e0 Reviewed-on: http://gerrit.cloudera.org:8080/11586 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M docs/topics/impala_math_functions.xml 1 file changed, 45 insertions(+), 50 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/11586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3b02d3e3f556d93e1d651eaee12217d6b0e3f9e0 Gerrit-Change-Number: 11586 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Yongjun Zhang
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11587 ) Change subject: IMPALA-7643: report # queries actually executing in stress test .. Patch Set 1: (7 comments) http://gerrit.cloudera.org:8080/#/c/11587/1/tests/stress/concurrent_select.py File tests/stress/concurrent_select.py: http://gerrit.cloudera.org:8080/#/c/11587/1/tests/stress/concurrent_select.py@431 PS1, Line 431: d flake8: E501 line too long (91 > 90 characters) http://gerrit.cloudera.org:8080/#/c/11587/1/tests/stress/concurrent_select.py@465 PS1, Line 465: , flake8: E501 line too long (101 > 90 characters) http://gerrit.cloudera.org:8080/#/c/11587/1/tests/stress/concurrent_select.py@688 PS1, Line 688: a flake8: E501 line too long (94 > 90 characters) http://gerrit.cloudera.org:8080/#/c/11587/1/tests/stress/concurrent_select.py@1078 PS1, Line 1078: e flake8: E501 line too long (94 > 90 characters) http://gerrit.cloudera.org:8080/#/c/11587/1/tests/stress/concurrent_select.py@1113 PS1, Line 1113: d flake8: E303 too many blank lines (2) http://gerrit.cloudera.org:8080/#/c/11587/1/tests/stress/concurrent_select.py@1141 PS1, Line 1141: d flake8: E303 too many blank lines (2) http://gerrit.cloudera.org:8080/#/c/11587/1/tests/stress/concurrent_select.py@1153 PS1, Line 1153: : flake8: E501 line too long (91 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/11587 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335 Gerrit-Change-Number: 11587 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Comment-Date: Thu, 04 Oct 2018 23:08:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7660: Support ECDH ciphers for debug webserver
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11585 ) Change subject: IMPALA-7660: Support ECDH ciphers for debug webserver .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I80a6b370d5860812cde13229b5bcb2977814c73c Gerrit-Change-Number: 11585 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 22:49:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7575: [DOCS] FMOD() is not the same as the % operator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11586 ) Change subject: IMPALA-7575: [DOCS] FMOD() is not the same as the % operator .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/90/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/11586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b02d3e3f556d93e1d651eaee12217d6b0e3f9e0 Gerrit-Change-Number: 11586 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Yongjun Zhang Gerrit-Comment-Date: Thu, 04 Oct 2018 23:02:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7575: [DOCS] FMOD() is not the same as the % operator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11586 ) Change subject: IMPALA-7575: [DOCS] FMOD() is not the same as the % operator .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/90/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b02d3e3f556d93e1d651eaee12217d6b0e3f9e0 Gerrit-Change-Number: 11586 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 04 Oct 2018 22:48:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7575: [DOCS] FMOD() is not the same as the % operator
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11586 ) Change subject: IMPALA-7575: [DOCS] FMOD() is not the same as the % operator .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b02d3e3f556d93e1d651eaee12217d6b0e3f9e0 Gerrit-Change-Number: 11586 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Yongjun Zhang Gerrit-Comment-Date: Thu, 04 Oct 2018 22:52:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7575: [DOCS] FMOD() is not the same as the % operator
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11586 Change subject: IMPALA-7575: [DOCS] FMOD() is not the same as the % operator .. IMPALA-7575: [DOCS] FMOD() is not the same as the % operator - Removed the text about FMOD being equivalent to %. - Added a note that MOD will show as % in the query plan. Change-Id: I3b02d3e3f556d93e1d651eaee12217d6b0e3f9e0 --- M docs/topics/impala_math_functions.xml 1 file changed, 45 insertions(+), 50 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/11586/1 -- To view, visit http://gerrit.cloudera.org:8080/11586 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3b02d3e3f556d93e1d651eaee12217d6b0e3f9e0 Gerrit-Change-Number: 11586 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-7659: Simplify expression to collect NULLs count
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11565 ) Change subject: IMPALA-7659: Simplify expression to collect NULLs count .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3271/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/11565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a Gerrit-Change-Number: 11565 Gerrit-PatchSet: 5 Gerrit-Owner: piotr.findei...@gmail.com Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: piotr.findei...@gmail.com Gerrit-Comment-Date: Thu, 04 Oct 2018 22:46:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7659: Simplify expression to collect NULLs count
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11565 ) Change subject: IMPALA-7659: Simplify expression to collect NULLs count .. Patch Set 5: The change in expression makes sense to me. I think we need someone who knows the planner better to evaluate whether we need more testing here. I suspect that tests will need modification to reflect the behaviour change (otherwise we have a testing gap). -- To view, visit http://gerrit.cloudera.org:8080/11565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a Gerrit-Change-Number: 11565 Gerrit-PatchSet: 5 Gerrit-Owner: piotr.findei...@gmail.com Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: piotr.findei...@gmail.com Gerrit-Comment-Date: Thu, 04 Oct 2018 22:46:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7660: Support ECDH ciphers for debug webserver
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11585 ) Change subject: IMPALA-7660: Support ECDH ciphers for debug webserver .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/11585/1/tests/custom_cluster/test_client_ssl.py File tests/custom_cluster/test_client_ssl.py: http://gerrit.cloudera.org:8080/#/c/11585/1/tests/custom_cluster/test_client_ssl.py@116 PS1, Line 116: flake8: E203 whitespace before ':' http://gerrit.cloudera.org:8080/#/c/11585/1/tests/custom_cluster/test_client_ssl.py@134 PS1, Line 134: flake8: E203 whitespace before ':' -- To view, visit http://gerrit.cloudera.org:8080/11585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I80a6b370d5860812cde13229b5bcb2977814c73c Gerrit-Change-Number: 11585 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 22:41:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7660: Support ECDH ciphers for debug webserver
Thomas Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11585 Change subject: IMPALA-7660: Support ECDH ciphers for debug webserver .. IMPALA-7660: Support ECDH ciphers for debug webserver A recent change (IMPALA-7519) added support for ecdh ciphers for the beeswax/hs2 server. This patch pulls in a recent change on squeasel to extend that support to the debug webserver. It also fixes a bug that prevented start-impala-cluster.py from completing successfully when the webserver is launched with ssl, due to it trying to verify the availablitiy of the webserver over http. Testing: - Added a custom cluster test that verifies start-impala-cluster.py runs successfully with webserver ssl enabled. - Adds the webserver to an existing test for ecdh ciphers. Change-Id: I80a6b370d5860812cde13229b5bcb2977814c73c --- M be/src/thirdparty/squeasel/squeasel.c M tests/common/impala_cluster.py M tests/common/impala_service.py M tests/custom_cluster/test_client_ssl.py 4 files changed, 82 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/11585/1 -- To view, visit http://gerrit.cloudera.org:8080/11585 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I80a6b370d5860812cde13229b5bcb2977814c73c Gerrit-Change-Number: 11585 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall
[Impala-ASF-CR] IMPALA-7651: [DOCS] Kudu support to scheduler-related query hints and options
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11584 ) Change subject: IMPALA-7651: [DOCS] Kudu support to scheduler-related query hints and options .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/89/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/11584 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I481d2a002edc1a18491bf9fc249e868005b42fa5 Gerrit-Change-Number: 11584 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 04 Oct 2018 22:28:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7651: [DOCS] Kudu support to scheduler-related query hints and options
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11584 ) Change subject: IMPALA-7651: [DOCS] Kudu support to scheduler-related query hints and options .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/89/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11584 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I481d2a002edc1a18491bf9fc249e868005b42fa5 Gerrit-Change-Number: 11584 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 04 Oct 2018 22:21:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7651: [DOCS] Kudu support to scheduler-related query hints and options
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11584 Change subject: IMPALA-7651: [DOCS] Kudu support to scheduler-related query hints and options .. IMPALA-7651: [DOCS] Kudu support to scheduler-related query hints and options The SCHEDULE_RANDOM_REPLICA query option and the RANDOM_REPLICA hint support Kudu as well as HDFS. Change-Id: I481d2a002edc1a18491bf9fc249e868005b42fa5 --- M docs/topics/impala_hints.xml M docs/topics/impala_schedule_random_replica.xml 2 files changed, 50 insertions(+), 67 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/11584/1 -- To view, visit http://gerrit.cloudera.org:8080/11584 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I481d2a002edc1a18491bf9fc249e868005b42fa5 Gerrit-Change-Number: 11584 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-7351: Improve memory estimates for Kudu Scan Nodes
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11440 ) Change subject: IMPALA-7351: Improve memory estimates for Kudu Scan Nodes .. IMPALA-7351: Improve memory estimates for Kudu Scan Nodes This patch adds memory estimates for kudu scan nodes based on empirically derived estimates for the scan's memory consumption that were added in IMPALA-7096. Testing: Modified resource requirements planner test. Change-Id: If9bb52530271b0bff91311a67d222a2e9fac1229 Reviewed-on: http://gerrit.cloudera.org:8080/11440 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/kudu-selectivity.test M testdata/workloads/functional-planner/queries/PlannerTest/min-max-runtime-filters.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test 9 files changed, 205 insertions(+), 79 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11440 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If9bb52530271b0bff91311a67d222a2e9fac1229 Gerrit-Change-Number: 11440 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7351: Improve memory estimates for Kudu Scan Nodes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11440 ) Change subject: IMPALA-7351: Improve memory estimates for Kudu Scan Nodes .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11440 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9bb52530271b0bff91311a67d222a2e9fac1229 Gerrit-Change-Number: 11440 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 22:04:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7581: timeout backend tests after 2 hours
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11533 ) Change subject: IMPALA-7581: timeout backend tests after 2 hours .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e4ef9cb0549ead0bae57b11489f6a4d9b44ef95 Gerrit-Change-Number: 11533 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 21:51:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7648 part 1: add expected out-of-memory tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11564 ) Change subject: IMPALA-7648 part 1: add expected out-of-memory tests .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11564 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icd1a7eb97837b742a967c260cafb5a7f4f45412e Gerrit-Change-Number: 11564 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 21:53:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7648 part 1: add expected out-of-memory tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11564 ) Change subject: IMPALA-7648 part 1: add expected out-of-memory tests .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3270/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11564 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icd1a7eb97837b742a967c260cafb5a7f4f45412e Gerrit-Change-Number: 11564 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 21:54:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7581: timeout backend tests after 2 hours
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11533 ) Change subject: IMPALA-7581: timeout backend tests after 2 hours .. IMPALA-7581: timeout backend tests after 2 hours This will abort the process if a backend test is taking too long, which we assume is because of a hang. This makes job failures easier to triage and may make it easier to debug failures if we collect coredumps or minidumps. Also disable the death tests for ASAN under the theory that the probability of the hang is higher than a regular DEBUG build. Testing: Reduced the timeout to 5s and confirmed that it worked. Change-Id: I2e4ef9cb0549ead0bae57b11489f6a4d9b44ef95 Reviewed-on: http://gerrit.cloudera.org:8080/11533 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong --- M be/src/common/init.cc M be/src/testutil/death-test-util.h M be/src/util/test-info.h 3 files changed, 25 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/11533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2e4ef9cb0549ead0bae57b11489f6a4d9b44ef95 Gerrit-Change-Number: 11533 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7648 part 1: add expected out-of-memory tests
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/11564 ) Change subject: IMPALA-7648 part 1: add expected out-of-memory tests .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11564 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icd1a7eb97837b742a967c260cafb5a7f4f45412e Gerrit-Change-Number: 11564 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 21:34:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7581: timeout backend tests after 2 hours
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11533 ) Change subject: IMPALA-7581: timeout backend tests after 2 hours .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11533 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e4ef9cb0549ead0bae57b11489f6a4d9b44ef95 Gerrit-Change-Number: 11533 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 21:08:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7654: [DOCS] TRUNCATE does not put HDFS data files to the trash
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11583 ) Change subject: IMPALA-7654: [DOCS] TRUNCATE does not put HDFS data files to the trash .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/88/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/11583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4930b2345515abf9b086575a476d20c87117e5c4 Gerrit-Change-Number: 11583 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 20:56:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7484: Do not interpret unrecognized hints as straight join hints.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/947/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 9 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 04 Oct 2018 21:02:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7484: Do not interpret unrecognized hints as straight join hints.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/946/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 04 Oct 2018 20:54:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6658: improve Parquet RLE for low bit widths
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11582 ) Change subject: IMPALA-6658: improve Parquet RLE for low bit widths .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/944/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I191a581d3f699b6669e48ac9dc39c76ed77c4a76 Gerrit-Change-Number: 11582 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 04 Oct 2018 20:41:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7654: [DOCS] TRUNCATE does not put HDFS data files to the trash
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11583 Change subject: IMPALA-7654: [DOCS] TRUNCATE does not put HDFS data files to the trash .. IMPALA-7654: [DOCS] TRUNCATE does not put HDFS data files to the trash Change-Id: I4930b2345515abf9b086575a476d20c87117e5c4 --- M docs/topics/impala_truncate_table.xml 1 file changed, 0 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/11583/1 -- To view, visit http://gerrit.cloudera.org:8080/11583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I4930b2345515abf9b086575a476d20c87117e5c4 Gerrit-Change-Number: 11583 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-7484: Do not interpret unrecognized hints as straight join hints.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/945/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 7 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 04 Oct 2018 20:45:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7484: Do not interpret unrecognized hints as straight join hints.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. .. Patch Set 9: Fixed formatting. -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 9 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 04 Oct 2018 20:29:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7484: Do not interpret unrecognized hints as straight join hints.
Anurag Mantripragada has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. .. IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. Wrapped setIsStraightJoin() in else clause. Testing: Added a wrapper for testing the state of straight_join hint. Modified existing tests to use this wrapper for +ve and -ve test cases. Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 --- M fe/src/main/java/org/apache/impala/analysis/SelectList.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 2 files changed, 26 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/11568/9 -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 9 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7659: Simplify expression to collect NULLs count
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11565 ) Change subject: IMPALA-7659: Simplify expression to collect NULLs count .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/943/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a Gerrit-Change-Number: 11565 Gerrit-PatchSet: 5 Gerrit-Owner: piotr.findei...@gmail.com Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: piotr.findei...@gmail.com Gerrit-Comment-Date: Thu, 04 Oct 2018 20:26:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7484: Do not interpret unrecognized hints as straight join hints.
Anurag Mantripragada has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/11568 ) Change subject: IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. .. IMPALA-7484: Do not interpret unrecognized hints as straight_join hints. Wrapped setIsStraightJoin() in else clause. Testing: Added a wrapper for testing the state of straight_join hint. Modified existing tests to use this wrapper for +ve and -ve test cases. Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 --- M fe/src/main/java/org/apache/impala/analysis/SelectList.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java 2 files changed, 28 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/11568/8 -- To view, visit http://gerrit.cloudera.org:8080/11568 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf600ebbfefc7398e0896df143a0ab91545cae04 Gerrit-Change-Number: 11568 Gerrit-PatchSet: 8 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-6658: improve Parquet RLE for low bit widths
Andrew Sherman has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11582 Change subject: IMPALA-6658: improve Parquet RLE for low bit widths .. IMPALA-6658: improve Parquet RLE for low bit widths RleEncoder buffers values in its own cache to detect run lengths that can be efficiently encoded. When a run is detected it is written with an indicator byte which encodes the length of the run. So a run is encoded as 2 bytes, but the actual cost of a run of small bit width values in the middle of a literal is really 3 bytes, as a new indicator is needed after the run. Change RleEncoder to have the ability to use run lengths other than 8. A new parameter to the constructor (min_run_length) allows test callers to set the minimum run length. By default RleEncoder will now use run length encoding for runs of length 24 for single bit values, and of length 16 for 2 bit wide values. All other bit widths will use the existing length 8 runs. Internally RleEncoder must buffer more values so that the longer runs can be detected. The internal buffer “buffered_values_” is larger and is now a circular buffer so that the first 8 bytes of the buffer can be separately flushed to BitWriter. Testing: All unit tests pass The unit test rle-test is enhanced to run all tests against RleEncoders using all possible values of min_run_length. In Addition, rle-test is refactored so that the rle tests are in a class that inherits from ::testing::Test so that a SetUp() method can be used. Change-Id: I191a581d3f699b6669e48ac9dc39c76ed77c4a76 --- M be/src/util/rle-encoding.h M be/src/util/rle-test.cc 2 files changed, 497 insertions(+), 254 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/11582/1 -- To view, visit http://gerrit.cloudera.org:8080/11582 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I191a581d3f699b6669e48ac9dc39c76ed77c4a76 Gerrit-Change-Number: 11582 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-7659: Simplify expression to collect NULLs count
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11565 ) Change subject: IMPALA-7659: Simplify expression to collect NULLs count .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/942/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a Gerrit-Change-Number: 11565 Gerrit-PatchSet: 4 Gerrit-Owner: piotr.findei...@gmail.com Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: piotr.findei...@gmail.com Gerrit-Comment-Date: Thu, 04 Oct 2018 20:18:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. Patch Set 15: (1 comment) http://gerrit.cloudera.org:8080/#/c/11157/15/testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test File testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test: http://gerrit.cloudera.org:8080/#/c/11157/15/testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test@9 PS15, Line 9: set num_scanner_threads=8; > I think we should set this to 2 or 3, so that it works on smaller machines Yea I though that too, but the problem with 2-3 is that then the estimates is less than 32mb and that triggers the minReservation lower bound. I tried a couple of tables to get something more that that for a lower core count, but couldnt find anything. I wanted to keep the query simple enough so that the estimates dont change much with new changes to the planner. Lemme try to cook up a different query. -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 15 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 19:34:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7659: Simplify expression to collect NULLs count
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11565 ) Change subject: IMPALA-7659: Simplify expression to collect NULLs count .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/941/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a Gerrit-Change-Number: 11565 Gerrit-PatchSet: 3 Gerrit-Owner: piotr.findei...@gmail.com Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: piotr.findei...@gmail.com Gerrit-Comment-Date: Thu, 04 Oct 2018 20:03:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7659: Simplify expression to collect NULLs count
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/11565 )
Change subject: IMPALA-7659: Simplify expression to collect NULLs count
..
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/11565/4/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java
File fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java:
http://gerrit.cloudera.org:8080/#/c/11565/4/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java@250
PS4, Line 250: columnStatsSelectList.add("COUNT(CASE WHEN " + colRefSql +
" IS NULL THEN 1 ELSE NULL END)");
line too long (99 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/11565
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a
Gerrit-Change-Number: 11565
Gerrit-PatchSet: 4
Gerrit-Owner: piotr.findei...@gmail.com
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: piotr.findei...@gmail.com
Gerrit-Comment-Date: Thu, 04 Oct 2018 19:47:51 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7659: Simplify expression to collect NULLs count
piotr.findei...@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/11565 ) Change subject: IMPALA-7659: Simplify expression to collect NULLs count .. Patch Set 5: Thanks Tim for your help -- To view, visit http://gerrit.cloudera.org:8080/11565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a Gerrit-Change-Number: 11565 Gerrit-PatchSet: 5 Gerrit-Owner: piotr.findei...@gmail.com Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: piotr.findei...@gmail.com Gerrit-Comment-Date: Thu, 04 Oct 2018 19:51:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7659: Simplify expression to collect NULLs count
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11565 to look at the new patch set (#5). Change subject: IMPALA-7659: Simplify expression to collect NULLs count .. IMPALA-7659: Simplify expression to collect NULLs count For reasoning behind replacing COUNT(IF(...)) with COUNT(CASE...) see IMPALA-7655. Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a --- M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java 1 file changed, 3 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/11565/5 -- To view, visit http://gerrit.cloudera.org:8080/11565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a Gerrit-Change-Number: 11565 Gerrit-PatchSet: 5 Gerrit-Owner: piotr.findei...@gmail.com Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: piotr.findei...@gmail.com
[Impala-ASF-CR] IMPALA-7659: Simplify expression to collect NULLs count
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11565 to look at the new patch set (#4). Change subject: IMPALA-7659: Simplify expression to collect NULLs count .. IMPALA-7659: Simplify expression to collect NULLs count For reasoning behind replacing COUNT(IF(...)) with COUNT(CASE...) see IMPALA-7655. Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a --- M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java 1 file changed, 2 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/11565/4 -- To view, visit http://gerrit.cloudera.org:8080/11565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a Gerrit-Change-Number: 11565 Gerrit-PatchSet: 4 Gerrit-Owner: piotr.findei...@gmail.com Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: piotr.findei...@gmail.com
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. Patch Set 15: (1 comment) One minor request http://gerrit.cloudera.org:8080/#/c/11157/15/testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test File testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test: http://gerrit.cloudera.org:8080/#/c/11157/15/testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test@9 PS15, Line 9: set num_scanner_threads=8; I think we should set this to 2 or 3, so that it works on smaller machines (e.g. this might fail on a dual core laptop). -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 15 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 19:27:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7659: Simplify expression to collect NULLs count
piotr.findei...@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/11565 ) Change subject: IMPALA-7659: Simplify expression to collect NULLs count .. Patch Set 3: > If you rewrite with a case expression instead of if you get > something thats much faster again. You something like: COUNT(CASE WHEN symbol IS NULL THEN NULL ELSE 1 END) ? -- To view, visit http://gerrit.cloudera.org:8080/11565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a Gerrit-Change-Number: 11565 Gerrit-PatchSet: 3 Gerrit-Owner: piotr.findei...@gmail.com Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: piotr.findei...@gmail.com Gerrit-Comment-Date: Thu, 04 Oct 2018 19:30:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7626: Throttle catalog partial RPC requests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11561 ) Change subject: IMPALA-7626: Throttle catalog partial RPC requests .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/940/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11561 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11f77a16cfa38ada42d8b7c859850198ea7dd142 Gerrit-Change-Number: 11561 Gerrit-PatchSet: 7 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 04 Oct 2018 19:24:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7659: Simplify expression to collect NULLs count
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11565 to look at the new patch set (#3). Change subject: IMPALA-7659: Simplify expression to collect NULLs count .. IMPALA-7659: Simplify expression to collect NULLs count Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a --- M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java 1 file changed, 2 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/11565/3 -- To view, visit http://gerrit.cloudera.org:8080/11565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a Gerrit-Change-Number: 11565 Gerrit-PatchSet: 3 Gerrit-Owner: piotr.findei...@gmail.com Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: piotr.findei...@gmail.com
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. Patch Set 15: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 15 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 19:27:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. Patch Set 15: > Hi Bikram! > > Quick question on this stuff: is it obvious from the profile what > memory limit was in use and how it got there? > > Thanks! Yes in the query profile I have added a new info string named "Cluster Memory Admitted" which tells us what value was used to admit the query. If either of the min/max query mem limit attributes are set then that value also corresponds to the mem_limit set. As to how it got there, it is calculated based on the pseduo code I mentioned in the commit message, so it might not be obvious unless the person looking at it knows that part. Open to suggestions on improving either -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 15 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 19:03:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. Patch Set 15: > "Cluster Memory Admitted" which tells us what value was used to admit the quer Ok, thanks. I think this is a good recent example for our continuing conversation about how to do "typing" on the profile strings. I think I wish that we had a Thrift structure for the query profile where we added this sort of thing rather than having a bunch of key/value pairs. It's got nothing to do with this change, though. Anyway, thanks for the info! (You're all clear as far as I'm concerned.) -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 15 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 19:21:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Hello Tim Armstrong, Impala Public Jenkins, Dan Hecht, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11157 to look at the new patch set (#15). Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query With this patch the per host memory limit of a query is automatically set using the mem_limit set in the query options and the mem_estimate calculated by the planner based on the following pseudo code: if mem_limit is set in query options: use that and if 'clamp-mem-limit-query-option' is true: enforce the min/max query mem limits defined in the pool config. else: mem_limit = max(mem_estiamte, min_mem_limit_required_to_accomodate_largest_initial_reservation) finally, enforce min/max query mem limits defined in the pool config on this value. This calculated mem limit will also be used for admission accounting and consequently for admission control. Moreover, three new pool configuration options have been added to enable this behaviour: "min-query-mem-limit" & "max-query-mem-limit" => help clamp the per host memory limit for a query. If both these limits are not configured, then the estimates from planning are not used as a memory limit and only used for making admission decisions. Moreover the estimates will no longer have a lower bound based on the largest initial reservation. "clamp-mem-limit-query-option" => if false, the mem_limit defined in the query options is used directly and the max/min query mem limits are not enforced on it. Testing: Added e2e test cases. Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 --- M be/src/benchmarks/process-wide-locks-benchmark.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator.cc M be/src/runtime/mem-tracker.cc M be/src/runtime/mem-tracker.h M be/src/runtime/query-exec-mgr.cc M be/src/runtime/query-exec-mgr.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-state.cc M be/src/runtime/test-env.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/request-pool-service.cc M be/src/scheduling/scheduler.cc M be/src/service/client-request-state.cc M common/thrift/ImpalaInternalService.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/util/RequestPoolService.java M fe/src/test/java/org/apache/impala/util/TestRequestPoolService.java M fe/src/test/resources/fair-scheduler-test.xml M fe/src/test/resources/llama-site-test.xml A fe/src/test/resources/mem-limit-test-fair-scheduler.xml A fe/src/test/resources/mem-limit-test-llama-site.xml A testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test M testdata/workloads/functional-query/queries/QueryTest/admission-reject-min-reservation.test A tests/common/resource_pool_config.py M tests/custom_cluster/test_admission_controller.py 30 files changed, 1,072 insertions(+), 266 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/11157/15 -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 15 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Prettify the timeline produced by test-with-docker.py
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11578 ) Change subject: Prettify the timeline produced by test-with-docker.py .. Patch Set 2: Code-Review+2 I think you could move removing a common prefix from the names into Python by adding "common_prefix=..." to Timeline.__init__() or even stripping it when we do the monitoring. That said, it's not a big deal, and the layers are pretty tightly coupled, so I'm fine with the way you did it too. Thanks for the improvement! -- To view, visit http://gerrit.cloudera.org:8080/11578 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7a41bea762b0e33f3d71b0be57eedbacb19c680c Gerrit-Change-Number: 11578 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Thu, 04 Oct 2018 19:03:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7626: Throttle catalog partial RPC requests
Hello Tianyi Wang, Philip Zeyliger, Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11561 to look at the new patch set (#7). Change subject: IMPALA-7626: Throttle catalog partial RPC requests .. IMPALA-7626: Throttle catalog partial RPC requests With more coordinators running in local catalog mode, the expected RPC traffic on the Catalog server is higher compared to the non-local-catalog mode. Each such RPC is handled in its own thread and consumes some non-trivial CPU for serializing and deserializing the metadata. With this change, the maximum number of threads performing the actual work are capped to a certain limit at any point and the remaining requests (if any) are blocked until the current requests are serviced or they they exceed the configured timeout and abort. Adds the following parameters for controlling this behavior. --catalog_max_parallel_partial_fetch_rpc --catalog_partial_fetch_rpc_queue_timeout_s --catalog_partial_fetch_rpc_queue_timeout_s controls the timeout for queued requests. Added some basic supportability to examine the queue length via metrics. Added a unit test to make sure the concurrent requests for this RPC method does not exceed the configured value. Change-Id: I11f77a16cfa38ada42d8b7c859850198ea7dd142 --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/JniCatalog.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/test/java/org/apache/impala/catalog/PartialCatalogInfoTest.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java 13 files changed, 268 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/11561/7 -- To view, visit http://gerrit.cloudera.org:8080/11561 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I11f77a16cfa38ada42d8b7c859850198ea7dd142 Gerrit-Change-Number: 11561 Gerrit-PatchSet: 7 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] Simplify expression to collect NULLs count
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11565 ) Change subject: Simplify expression to collect NULLs count .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/11565/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11565/2//COMMIT_MSG@7 PS2, Line 7: Simplify expression to collect NULLs count Any chance you could file a JIRA to track the work and mention it in the commit message? https://issues.apache.org/jira/secure/Dashboard.jspa It would be interesting to understand how you stumbled on this since it seems like your motivation is to emit the null count as part of stats. -- To view, visit http://gerrit.cloudera.org:8080/11565 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic68f8b4c3756eb1980ce299a602a7d56db1e507a Gerrit-Change-Number: 11565 Gerrit-PatchSet: 2 Gerrit-Owner: piotr.findei...@gmail.com Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: piotr.findei...@gmail.com Gerrit-Comment-Date: Thu, 04 Oct 2018 18:45:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. Patch Set 15: Hi Bikram! Quick question on this stuff: is it obvious from the profile what memory limit was in use and how it got there? Thanks! -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 15 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 18:48:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11157 ) Change subject: IMPALA-7349: Add Admission control support for automatically setting per host memory limit for a query .. Patch Set 15: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/939/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11157 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6 Gerrit-Change-Number: 11157 Gerrit-PatchSet: 15 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 18:46:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7351: Improve memory estimates for Kudu Scan Nodes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11440 ) Change subject: IMPALA-7351: Improve memory estimates for Kudu Scan Nodes .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3269/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11440 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If9bb52530271b0bff91311a67d222a2e9fac1229 Gerrit-Change-Number: 11440 Gerrit-PatchSet: 7 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 04 Oct 2018 18:17:30 + Gerrit-HasComments: No
