[Impala-ASF-CR] IMPALA-9456: allow disabling kerberos selectively
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15351 ) Change subject: IMPALA-9456: allow disabling kerberos selectively .. Patch Set 2: (5 comments) http://gerrit.cloudera.org:8080/#/c/15351/2/be/src/common/global-flags.cc File be/src/common/global-flags.cc: http://gerrit.cloudera.org:8080/#/c/15351/2/be/src/common/global-flags.cc@55 PS2, Line 55: DEFINE_bool(skip_internal_kerberos_auth, false, : "(Advanced) skip kerberos authentication for incoming internal connections from " : "other daemons within the Impala cluster (i.e. impalads, statestored, catalogd). " : "Must be set to the same value across all daemons. Only has an effect if --principal " : "is set, i.e. Kerberos is enabled."); : DEFINE_bool(skip_external_kerberos_auth, false, : "(Advanced) skip kerberos authentication for incoming external connections to " : "this daemon, e.g. clients connecting to the HS2 interface. Only has an effect " : "if --principal is set, i.e. Kerberos is enabled."); what category does impalad-HMS connections fall into? http://gerrit.cloudera.org:8080/#/c/15351/2/be/src/rpc/authentication.cc File be/src/rpc/authentication.cc: http://gerrit.cloudera.org:8080/#/c/15351/2/be/src/rpc/authentication.cc@a958 PS2, Line 958: i think the docs for this method in authentication.h need to be updated to reflect this change http://gerrit.cloudera.org:8080/#/c/15351/2/be/src/rpc/authentication.cc@1231 PS2, Line 1231: // Flags | Internal | External : // - | | : // None | NoAuth | NoAuth : // LDAP only | NoAuth | Sasl(ldap) : // Kerb only | Sasl(be) | Sasl(fe) : // Both | Sasl(be) | Sasl(fe+ldap) does this table need to be updated? it seems there are a few more options now since --skip_internal_kerberos_auth and --skip_external_kerberos_auth can change if Internal or External auth are enabled http://gerrit.cloudera.org:8080/#/c/15351/2/be/src/rpc/authentication.cc@1293 PS2, Line 1293: "and" http://gerrit.cloudera.org:8080/#/c/15351/2/be/src/rpc/authentication.cc@1295 PS2, Line 1295: kerberos_internal_principal what about the renew thread for the external principal? -- To view, visit http://gerrit.cloudera.org:8080/15351 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3b1c641e05e588287e4d9d9cd8389d96fc71cf74 Gerrit-Change-Number: 15351 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 06 Mar 2020 23:48:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9156: share broadcast join builds
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15096 ) Change subject: IMPALA-9156: share broadcast join builds .. Patch Set 11: PS9->PS12 adds some tests and fixes some accounting bugs that were discovered with the tests, specifically * THe spilled partitions got out of sync between the join nodes because of the empty probe side optimisation. I just disabled that optimisation because i don't think it's worth the complexity. * I didn't multiply the probe reservation by the number of threads in one place * num_spilled_probe_rows wasn't added together correctly across threads * NodeDebugString had an invalid dcheck that got hit when I set -vmodule=partitioned-hash-join-node=3 -- To view, visit http://gerrit.cloudera.org:8080/15096 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c67e4b2c87ed0fba648f1e1710addb885d66dc7 Gerrit-Change-Number: 15096 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 07 Mar 2020 00:57:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9156: share broadcast join builds
Hello Csaba Ringhofer, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15096 to look at the new patch set (#12). Change subject: IMPALA-9156: share broadcast join builds .. IMPALA-9156: share broadcast join builds The scheduler will only create one join build finstance per backend in cases where this is supported. The builder is aware of the number of finstances executing the probe and hands off the build data structures to the builders. Nested loop join requires minimal modifications because the build data structures are read-only after initial construction. The only significant change is that memory can't be transferred to the multiple consumers, so MarkNeedsDeepCopy() needs to be used instead. Hash join requires additional synchronisation because the spilling algorithm mutates build-side data structures. This patch adds synchronisation so that rebuilding spilled partitions is done in a thread-safe manner, using a single thread. This uses the CyclicBarrier added in an earlier patch. Threads blocked on CyclicBarrier need to be cancellable, which is handled by cancelling the barrier when closing the join builder. Update planner to cost broadcast join and estimate memory consumption based on a single instance per node. Planner estimates of number of instances are improved. Instead of assuming mt_dop instances per node, use the total number of input splits (also called scan ranges in places) as an upper bound on the number of instances generated by scans. These instance estimates from the scan nodes are then propagated up the plan tree in the same was as the numNodes estimates. The instance estimate for the join build fragment is fixed to be based on the destination fragment. The profile now correctly accounts for time waiting for the builder, counting it in inactive time and showing it in the node timeline. Additional improvements/cleanup to the time accounting are deferring until IMPALA-9422. Testing: * Updated planner tests * Ran a single node stress test with TPC-H and TPC-DS * Add a targeted test for spilling broadcast joins, both repartitioning and not repartitioning. * Add a targeted test for a spilling broadcast join with empty probe * Add a targeted test for spilling broadcast join with empty build partitions. * Add a broadcast join to test_cancellation and test_failpoints. Perf: I did a single node run on my desktop: +--+---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +--+---+-++++ | TPCH(30) | parquet / none / none | 6.26| -15.70%| 4.63 | -16.16%| +--+---+-++++ +--+--+---++-++---++---++-+-+ | Workload | Query| File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval| +--+--+---++-++---++---++-+-+ | TPCH(30) | TPCH-Q21 | parquet / none / none | 24.97 | 23.25 | R +7.38% | 0.51% | 0.22%| 5 | R +6.95% | 2.31| 27.93 | | TPCH(30) | TPCH-Q4 | parquet / none / none | 2.83 | 2.79| +1.31% | 1.86% | 0.36%| 5 | +1.88% | 1.15| 1.53| | TPCH(30) | TPCH-Q6 | parquet / none / none | 1.28 | 1.28| -0.01% | 1.64% | 1.63%| 5 | -0.11% | -0.58 | -0.01 | | TPCH(30) | TPCH-Q22 | parquet / none / none | 2.65 | 2.68| -0.94% | 0.84% | 1.46%| 5 | -0.21% | -0.87 | -1.25 | | TPCH(30) | TPCH-Q1 | parquet / none / none | 4.69 | 4.72| -0.56% | 1.29% | 0.52%| 5 | -1.04% | -1.15 | -0.89 | | TPCH(30) | TPCH-Q13 | parquet / none / none | 10.64 | 10.80 | -1.48% | 0.61% | 0.60%| 5 | -1.39% | -1.73 | -3.91 | | TPCH(30) | TPCH-Q15 | parquet / none / none | 4.11 | 4.32| -4.92% | 0.05% | 0.40%| 5 | -4.93% | -2.31 | -27.46 | | TPCH(30) | TPCH-Q20 | parquet / none / none | 3.47 | 3.67| I -5.41% | 0.81% | 0.03%| 5 | I -5.70% | -2.31 | -15.75 | | TPCH(30) | TPCH-Q17 | parquet / none / none | 7.58 | 8.14| I -6.93% | 3.13% | 2.62%| 5 | I -9.31% | -2.02 | -3.96 | | TPCH(30) | TPCH-Q9 | parquet / none / none | 15.59 | 17.02 | I -8.38% | 0.95% | 0.43%
[Impala-ASF-CR] IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15284 ) Change subject: IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5446/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4153968551acd58b25c7923c2ebf75ee29a7e76b Gerrit-Change-Number: 15284 Gerrit-PatchSet: 8 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Mar 2020 20:15:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9156: share broadcast join builds
Hello Csaba Ringhofer, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15096 to look at the new patch set (#11). Change subject: IMPALA-9156: share broadcast join builds .. IMPALA-9156: share broadcast join builds The scheduler will only create one join build finstance per backend in cases where this is supported. The builder is aware of the number of finstances executing the probe and hands off the build data structures to the builders. Nested loop join requires minimal modifications because the build data structures are read-only after initial construction. The only significant change is that memory can't be transferred to the multiple consumers, so MarkNeedsDeepCopy() needs to be used instead. Hash join requires additional synchronisation because the spilling algorithm mutates build-side data structures. This patch adds synchronisation so that rebuilding spilled partitions is done in a thread-safe manner, using a single thread. This uses the CyclicBarrier added in an earlier patch. Threads blocked on CyclicBarrier need to be cancellable, which is handled by cancelling the barrier when closing the join builder. Update planner to cost broadcast join and estimate memory consumption based on a single instance per node. Planner estimates of number of instances are improved. Instead of assuming mt_dop instances per node, use the total number of input splits (also called scan ranges in places) as an upper bound on the number of instances generated by scans. These instance estimates from the scan nodes are then propagated up the plan tree in the same was as the numNodes estimates. The instance estimate for the join build fragment is fixed to be based on the destination fragment. The profile now correctly accounts for time waiting for the builder, counting it in inactive time and showing it in the node timeline. Additional improvements/cleanup to the time accounting are deferring until IMPALA-9422. Testing: * Updated planner tests * Ran a single node stress test with TPC-H and TPC-DS * Add a targeted test for spilling broadcast joins, both repartitioning and not repartitioning. * Add a targeted test for a spilling broadcast join with empty probe * Add a targeted test for spilling broadcast join with empty build partitions. * Add a broadcast join to test_cancellation and test_failpoints. Perf: I did a single node run on my desktop: +--+---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +--+---+-++++ | TPCH(30) | parquet / none / none | 6.26| -15.70%| 4.63 | -16.16%| +--+---+-++++ +--+--+---++-++---++---++-+-+ | Workload | Query| File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval| +--+--+---++-++---++---++-+-+ | TPCH(30) | TPCH-Q21 | parquet / none / none | 24.97 | 23.25 | R +7.38% | 0.51% | 0.22%| 5 | R +6.95% | 2.31| 27.93 | | TPCH(30) | TPCH-Q4 | parquet / none / none | 2.83 | 2.79| +1.31% | 1.86% | 0.36%| 5 | +1.88% | 1.15| 1.53| | TPCH(30) | TPCH-Q6 | parquet / none / none | 1.28 | 1.28| -0.01% | 1.64% | 1.63%| 5 | -0.11% | -0.58 | -0.01 | | TPCH(30) | TPCH-Q22 | parquet / none / none | 2.65 | 2.68| -0.94% | 0.84% | 1.46%| 5 | -0.21% | -0.87 | -1.25 | | TPCH(30) | TPCH-Q1 | parquet / none / none | 4.69 | 4.72| -0.56% | 1.29% | 0.52%| 5 | -1.04% | -1.15 | -0.89 | | TPCH(30) | TPCH-Q13 | parquet / none / none | 10.64 | 10.80 | -1.48% | 0.61% | 0.60%| 5 | -1.39% | -1.73 | -3.91 | | TPCH(30) | TPCH-Q15 | parquet / none / none | 4.11 | 4.32| -4.92% | 0.05% | 0.40%| 5 | -4.93% | -2.31 | -27.46 | | TPCH(30) | TPCH-Q20 | parquet / none / none | 3.47 | 3.67| I -5.41% | 0.81% | 0.03%| 5 | I -5.70% | -2.31 | -15.75 | | TPCH(30) | TPCH-Q17 | parquet / none / none | 7.58 | 8.14| I -6.93% | 3.13% | 2.62%| 5 | I -9.31% | -2.02 | -3.96 | | TPCH(30) | TPCH-Q9 | parquet / none / none | 15.59 | 17.02 | I -8.38% | 0.95% | 0.43%
[Impala-ASF-CR] IMPALA-9156: share broadcast join builds
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/15096 )
Change subject: IMPALA-9156: share broadcast join builds
..
Patch Set 10:
(17 comments)
http://gerrit.cloudera.org:8080/#/c/15096/10/be/src/exec/partitioned-hash-join-builder.h
File be/src/exec/partitioned-hash-join-builder.h:
http://gerrit.cloudera.org:8080/#/c/15096/10/be/src/exec/partitioned-hash-join-builder.h@444
PS10, Line 444: void IncrementNumSpilledProbeRows(int64_t count) {
num_spilled_probe_rows_.Add(count); }
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/15096/10/be/src/exec/partitioned-hash-join-builder.cc
File be/src/exec/partitioned-hash-join-builder.cc:
http://gerrit.cloudera.org:8080/#/c/15096/10/be/src/exec/partitioned-hash-join-builder.cc@601
PS10, Line 601: VLOG(3) << "PHJ(node_id=" << join_node_id_ << ")
probe_stream_reservation_=" << probe_stream_reservation_.GetReservation();
line too long (125 > 90)
http://gerrit.cloudera.org:8080/#/c/15096/10/be/src/exec/partitioned-hash-join-builder.cc@603
PS10, Line 603: VLOG(3) << "PHJ(node_id=" << join_node_id_ << ")
probe_stream_reservation_=" << probe_stream_reservation_.GetReservation();
line too long (125 > 90)
http://gerrit.cloudera.org:8080/#/c/15096/10/be/src/exec/partitioned-hash-join-builder.cc@627
PS10, Line 627: VLOG(3) << "PHJ(node_id=" << join_node_id_ << ")
probe_stream_reservation_=" << probe_stream_reservation_.GetReservation();
line too long (125 > 90)
http://gerrit.cloudera.org:8080/#/c/15096/10/be/src/exec/partitioned-hash-join-builder.cc@629
PS10, Line 629: VLOG(3) << "PHJ(node_id=" << join_node_id_ << ")
probe_stream_reservation_=" << probe_stream_reservation_.GetReservation();
line too long (125 > 90)
http://gerrit.cloudera.org:8080/#/c/15096/10/be/src/exec/partitioned-hash-join-builder.cc@632
PS10, Line 632: VLOG(3) << "PHJ(node_id=" << join_node_id_ << ")
transferring " << probe_reservation << " back to builder.";
line too long (112 > 90)
http://gerrit.cloudera.org:8080/#/c/15096/10/be/src/exec/partitioned-hash-join-builder.cc@712
PS10, Line 712: VLOG(3) << "PHJ(node_id=" << join_node_id_ << ")
transferring " << probe_reservation << " back to builder.";
line too long (112 > 90)
http://gerrit.cloudera.org:8080/#/c/15096/10/be/src/exec/partitioned-hash-join-builder.cc@776
PS10, Line 776: VLOG(3) << "PHJ(node_id=" << join_node_id_ << ")
transferring " << probe_reservation << " back to builder.";
line too long (112 > 90)
http://gerrit.cloudera.org:8080/#/c/15096/10/be/src/exec/partitioned-hash-join-builder.cc@940
PS10, Line 940: int64_t saved_probe_reservation = need_probe_buffer ?
max_row_buffer_size_ * num_probe_threads_ : 0;
line too long (102 > 90)
http://gerrit.cloudera.org:8080/#/c/15096/10/be/src/exec/partitioned-hash-join-builder.cc@941
PS10, Line 941: VLOG(3) << "PHJ(node_id=" << join_node_id_ << ")
probe_stream_reservation_=" << probe_stream_reservation_.GetReservation();
line too long (125 > 90)
http://gerrit.cloudera.org:8080/#/c/15096/10/be/src/exec/partitioned-hash-join-builder.cc@944
PS10, Line 944: VLOG(3) << "PHJ(node_id=" << join_node_id_ << ")
probe_stream_reservation_=" << probe_stream_reservation_.GetReservation();
line too long (125 > 90)
http://gerrit.cloudera.org:8080/#/c/15096/10/be/src/exec/partitioned-hash-join-builder.cc@966
PS10, Line 966: VLOG(3) << "PHJ(node_id=" << join_node_id_ << ")
probe_stream_reservation_=" << probe_stream_reservation_.GetReservation();
line too long (125 > 90)
http://gerrit.cloudera.org:8080/#/c/15096/10/be/src/exec/partitioned-hash-join-builder.cc@969
PS10, Line 969: VLOG(3) << "PHJ(node_id=" << join_node_id_ << ")
probe_stream_reservation_=" << probe_stream_reservation_.GetReservation();
line too long (125 > 90)
http://gerrit.cloudera.org:8080/#/c/15096/10/be/src/exec/partitioned-hash-join-builder.cc@1051
PS10, Line 1051: VLOG(3) << "PHJ(node_id=" << join_node_id_ << ")
transferring " << bytes << " back to builder.";
line too long (98 > 90)
http://gerrit.cloudera.org:8080/#/c/15096/10/fe/src/main/java/org/apache/impala/planner/PlanFragment.java
File fe/src/main/java/org/apache/impala/planner/PlanFragment.java:
http://gerrit.cloudera.org:8080/#/c/15096/10/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@201
PS10, Line 201: DataStreamSink streamSink = new
DataStreamSink((ExchangeNode)destNode_, outputPartition_);
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/15096/10/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@371
PS10, Line 371: // join. ParallelPlanner sets the destination fragment
when adding the JoinBuildSink.
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/15096/10/fe/src/main/java/org/apache/impala/planner/PlanFragment.java@400
PS10, Line 400: // join. ParallelPlanner sets the destination fragment
when adding the
[Impala-ASF-CR] IMPALA-9156: share broadcast join builds
Hello Csaba Ringhofer, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15096 to look at the new patch set (#10). Change subject: IMPALA-9156: share broadcast join builds .. IMPALA-9156: share broadcast join builds The scheduler will only create one join build finstance per backend in cases where this is supported. The builder is aware of the number of finstances executing the probe and hands off the build data structures to the builders. Nested loop join requires minimal modifications because the build data structures are read-only after initial construction. The only significant change is that memory can't be transferred to the multiple consumers, so MarkNeedsDeepCopy() needs to be used instead. Hash join requires additional synchronisation because the spilling algorithm mutates build-side data structures. This patch adds synchronisation so that rebuilding spilled partitions is done in a thread-safe manner, using a single thread. This uses the CyclicBarrier added in an earlier patch. Threads blocked on CyclicBarrier need to be cancellable, which is handled by cancelling the barrier when closing the join builder. Update planner to cost broadcast join and estimate memory consumption based on a single instance per node. Planner estimates of number of instances are improved. Instead of assuming mt_dop instances per node, use the total number of input splits (also called scan ranges in places) as an upper bound on the number of instances generated by scans. These instance estimates from the scan nodes are then propagated up the plan tree in the same was as the numNodes estimates. The instance estimate for the join build fragment is fixed to be based on the destination fragment. The profile now correctly accounts for time waiting for the builder, counting it in inactive time and showing it in the node timeline. Additional improvements/cleanup to the time accounting are deferring until IMPALA-9422. Testing: * Updated planner tests * Ran a single node stress test with TPC-H and TPC-DS * Add a targeted test for spilling broadcast joins, both repartitioning and not repartitioning. * Add a targeted test for a spilling broadcast join with empty probe * Add a targeted test for spilling broadcast join with empty build partitions. * Add a broadcast join to test_cancellation and test_failpoints. Perf: I did a single node run on my desktop: +--+---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +--+---+-++++ | TPCH(30) | parquet / none / none | 6.26| -15.70%| 4.63 | -16.16%| +--+---+-++++ +--+--+---++-++---++---++-+-+ | Workload | Query| File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval| +--+--+---++-++---++---++-+-+ | TPCH(30) | TPCH-Q21 | parquet / none / none | 24.97 | 23.25 | R +7.38% | 0.51% | 0.22%| 5 | R +6.95% | 2.31| 27.93 | | TPCH(30) | TPCH-Q4 | parquet / none / none | 2.83 | 2.79| +1.31% | 1.86% | 0.36%| 5 | +1.88% | 1.15| 1.53| | TPCH(30) | TPCH-Q6 | parquet / none / none | 1.28 | 1.28| -0.01% | 1.64% | 1.63%| 5 | -0.11% | -0.58 | -0.01 | | TPCH(30) | TPCH-Q22 | parquet / none / none | 2.65 | 2.68| -0.94% | 0.84% | 1.46%| 5 | -0.21% | -0.87 | -1.25 | | TPCH(30) | TPCH-Q1 | parquet / none / none | 4.69 | 4.72| -0.56% | 1.29% | 0.52%| 5 | -1.04% | -1.15 | -0.89 | | TPCH(30) | TPCH-Q13 | parquet / none / none | 10.64 | 10.80 | -1.48% | 0.61% | 0.60%| 5 | -1.39% | -1.73 | -3.91 | | TPCH(30) | TPCH-Q15 | parquet / none / none | 4.11 | 4.32| -4.92% | 0.05% | 0.40%| 5 | -4.93% | -2.31 | -27.46 | | TPCH(30) | TPCH-Q20 | parquet / none / none | 3.47 | 3.67| I -5.41% | 0.81% | 0.03%| 5 | I -5.70% | -2.31 | -15.75 | | TPCH(30) | TPCH-Q17 | parquet / none / none | 7.58 | 8.14| I -6.93% | 3.13% | 2.62%| 5 | I -9.31% | -2.02 | -3.96 | | TPCH(30) | TPCH-Q9 | parquet / none / none | 15.59 | 17.02 | I -8.38% | 0.95% | 0.43%
[Impala-ASF-CR] IMPALA-9156: share broadcast join builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15096 ) Change subject: IMPALA-9156: share broadcast join builds .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5449/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15096 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c67e4b2c87ed0fba648f1e1710addb885d66dc7 Gerrit-Change-Number: 15096 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 07 Mar 2020 01:28:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9466: impala-shell client retry for hs2-http protocol
Abhishek Rawat has posted comments on this change. (
http://gerrit.cloudera.org:8080/15378 )
Change subject: IMPALA-9466: impala-shell client retry for hs2-http protocol
..
Patch Set 3:
(11 comments)
Address flake comments.
http://gerrit.cloudera.org:8080/#/c/15378/3/shell/impala_client.py
File shell/impala_client.py:
http://gerrit.cloudera.org:8080/#/c/15378/3/shell/impala_client.py@657
PS3, Line 657: r
> flake8: E731 do not assign a lambda expression, use a def
I think in this case we need the lazy evaluation which lambda expressions
provide and so cannot use def.
http://gerrit.cloudera.org:8080/#/c/15378/3/shell/impala_client.py@685
PS3, Line 685: ;
> flake8: E703 statement ends with a semicolon
Done.
http://gerrit.cloudera.org:8080/#/c/15378/3/shell/impala_client.py@715
PS3, Line 715: r
> flake8: E731 do not assign a lambda expression, use a def
Need lazy evaluation and so cannot use def.
http://gerrit.cloudera.org:8080/#/c/15378/3/shell/impala_client.py@733
PS3, Line 733: r
> flake8: E731 do not assign a lambda expression, use a def
Need lazy evaluation and so cannot use def.
http://gerrit.cloudera.org:8080/#/c/15378/3/shell/impala_client.py@744
PS3, Line 744: r
> flake8: E731 do not assign a lambda expression, use a def
Need lazy evaluation and so cannot use def.
http://gerrit.cloudera.org:8080/#/c/15378/3/shell/impala_client.py@812
PS3, Line 812: r
> flake8: E731 do not assign a lambda expression, use a def
Need lazy evaluation and so cannot use def.
http://gerrit.cloudera.org:8080/#/c/15378/3/shell/impala_client.py@829
PS3, Line 829: r
> flake8: E731 do not assign a lambda expression, use a def
Need lazy evaluation and so cannot use def.
http://gerrit.cloudera.org:8080/#/c/15378/3/shell/impala_client.py@849
PS3, Line 849: r
> flake8: E731 do not assign a lambda expression, use a def
Need lazy evaluation and so cannot use def.
http://gerrit.cloudera.org:8080/#/c/15378/3/shell/impala_client.py@858
PS3, Line 858: r
> flake8: E731 do not assign a lambda expression, use a def
Need lazy evaluation and so cannot use def.
http://gerrit.cloudera.org:8080/#/c/15378/3/tests/custom_cluster/test_hs2_fault_injection.py
File tests/custom_cluster/test_hs2_fault_injection.py:
http://gerrit.cloudera.org:8080/#/c/15378/3/tests/custom_cluster/test_hs2_fault_injection.py@65
PS3, Line 65: /
> flake8: E226 missing whitespace around arithmetic operator
Done.
http://gerrit.cloudera.org:8080/#/c/15378/3/tests/custom_cluster/test_hs2_fault_injection.py@223
PS3, Line 223:
> flake8: E211 whitespace before '('
Done.
--
To view, visit http://gerrit.cloudera.org:8080/15378
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0da9e9e8d34a340eaf763397cc095ff6260d65d5
Gerrit-Change-Number: 15378
Gerrit-PatchSet: 3
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Fri, 06 Mar 2020 23:31:33 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header
Hello David Knupp, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15284 to look at the new patch set (#8). Change subject: IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header .. IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header The 'Expect: 100-continue' http header allows http clients to send only the headers for their request, get a confirmation back from the server that the headers are valid, and only then send the body of the request, avoiding the overhead of sending large requests that will ultimately fail. This patch adds support for this in the HS2 HTTP server by having THttpServer look for the header, and if it's present and the request is validated returning a '100 Continue' response before reading the body of the request. It also adds supports for using this header on large requests sent by impala-shell. Testing: - This case is covered by the existing test_large_sql, however that test was previously broken and passing spuriously. This patch fixes the test. - Passed all other shell tests. Change-Id: I4153968551acd58b25c7923c2ebf75ee29a7e76b --- M be/src/transport/THttpServer.cpp M be/src/transport/THttpTransport.cpp M be/src/transport/THttpTransport.h R shell/ImpalaHttpClient.py M shell/impala_client.py M shell/impala_shell.py M shell/make_shell_tarball.sh M shell/packaging/make_python_package.sh A shell/shell_exceptions.py M tests/shell/test_shell_commandline.py 10 files changed, 133 insertions(+), 70 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/15284/8 -- To view, visit http://gerrit.cloudera.org:8080/15284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4153968551acd58b25c7923c2ebf75ee29a7e76b Gerrit-Change-Number: 15284 Gerrit-PatchSet: 8 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9156: share broadcast join builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15096 ) Change subject: IMPALA-9156: share broadcast join builds .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5451/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15096 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c67e4b2c87ed0fba648f1e1710addb885d66dc7 Gerrit-Change-Number: 15096 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 07 Mar 2020 01:38:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9472,IMPALA-9473: Add per-partition metrics for data cache
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15382 ) Change subject: IMPALA-9472,IMPALA-9473: Add per-partition metrics for data cache .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15382/1/be/src/runtime/io/data-cache.cc File be/src/runtime/io/data-cache.cc: http://gerrit.cloudera.org:8080/#/c/15382/1/be/src/runtime/io/data-cache.cc@567 PS1, Line 567: eviction_latency_ = ImpaladMetrics::IO_MGR_METRICS->RegisterMetric(new HistogramMetric( line too long (91 > 90) -- To view, visit http://gerrit.cloudera.org:8080/15382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a57d75ff11f00ebc85b85bcaf104fb8108c478 Gerrit-Change-Number: 15382 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 06 Mar 2020 22:06:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9472,IMPALA-9473: Add per-partition metrics for data cache
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15382 to look at the new patch set (#2). Change subject: IMPALA-9472,IMPALA-9473: Add per-partition metrics for data cache .. IMPALA-9472,IMPALA-9473: Add per-partition metrics for data cache This adds two sets of metrics. The first is per-partition metrics to track the performance of the underlying filesystem for the data cache. It keeps histograms of read, write, and eviction latency for each data cache partition along with another metric recording the path for the partition. These are exposed as the following metrics: impala-server.io-mgr.remote-data-cache-partition-$0.path impala-server.io-mgr.remote-data-cache-partition-$0.read-latency impala-server.io-mgr.remote-data-cache-partition-$0.write-latency impala-server.io-mgr.remote-data-cache-partition-$0.eviction-latency This also adds metrics to keep counts of hits, misses, and entries in the data cache. Since reducing the latency of IO is an important feature of the data cache, the absolute count of hits and misses is as important as the hit bytes and miss bytes. This adds the following metrics: impala-server.io-mgr.remote-data-cache-hit-count impala-server.io-mgr.remote-data-cache-miss-count impala-server.io-mgr.remote-data-cache-num-entries Testing: - Hand testing to verify the per-partition latency histograms - Modified custom_cluster/test_data_cache.py to also test the counts. Change-Id: I56a57d75ff11f00ebc85b85bcaf104fb8108c478 --- M be/src/runtime/io/data-cache.cc M be/src/runtime/io/data-cache.h M be/src/runtime/io/hdfs-file-reader.cc M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h M common/thrift/metrics.json M tests/custom_cluster/test_data_cache.py 7 files changed, 205 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/15382/2 -- To view, visit http://gerrit.cloudera.org:8080/15382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I56a57d75ff11f00ebc85b85bcaf104fb8108c478 Gerrit-Change-Number: 15382 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-3343: Make impala-shell compatible with python 3.
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/15132 ) Change subject: IMPALA-3343: Make impala-shell compatible with python 3. .. Patch Set 17: (1 comment) http://gerrit.cloudera.org:8080/#/c/15132/15/tests/shell/test_shell_commandline.py File tests/shell/test_shell_commandline.py: http://gerrit.cloudera.org:8080/#/c/15132/15/tests/shell/test_shell_commandline.py@485 PS15, Line 485: if SHELL_IS_PYTHON_2: > Just a note to let you know this hasn't been languishing. I'm stuck in a we Ah, and it's failing in the shell code, not actually in the test code. -- To view, visit http://gerrit.cloudera.org:8080/15132 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb75e162bac0faeae3e12106c15da39cbfb8b462 Gerrit-Change-Number: 15132 Gerrit-PatchSet: 17 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Mar 2020 19:45:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9156: share broadcast join builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15096 ) Change subject: IMPALA-9156: share broadcast join builds .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5450/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15096 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c67e4b2c87ed0fba648f1e1710addb885d66dc7 Gerrit-Change-Number: 15096 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 07 Mar 2020 01:37:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9384: Improve Impala shell usability by enabling live progress in interactive mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15219 ) Change subject: IMPALA-9384: Improve Impala shell usability by enabling live_progress in interactive mode .. Patch Set 10: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15219 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3765b775f663fa227e59728acffe4d5ea9a5e2d3 Gerrit-Change-Number: 15219 Gerrit-PatchSet: 10 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 07 Mar 2020 02:52:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9472,IMPALA-9473: Add per-partition metrics for data cache
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15382 Change subject: IMPALA-9472,IMPALA-9473: Add per-partition metrics for data cache .. IMPALA-9472,IMPALA-9473: Add per-partition metrics for data cache This adds two sets of metrics. The first is per-partition metrics to track the performance of the underlying filesystem for the data cache. It keeps histograms of read, write, and eviction latency for each data cache partition along with another metric recording the path for the partition. These are exposed as the following metrics: impala-server.io-mgr.remote-data-cache-partition-$0.path impala-server.io-mgr.remote-data-cache-partition-$0.read-latency impala-server.io-mgr.remote-data-cache-partition-$0.write-latency impala-server.io-mgr.remote-data-cache-partition-$0.eviction-latency This also adds metrics to keep counts of hits, misses, and entries in the data cache. Since reducing the latency of IO is an important feature of the data cache, the absolute count of hits and misses is as important as the hit bytes and miss bytes. This adds the following metrics: impala-server.io-mgr.remote-data-cache-hit-count impala-server.io-mgr.remote-data-cache-miss-count impala-server.io-mgr.remote-data-cache-num-entries Testing: - Hand testing to verify the per-partition latency histograms - Modified custom_cluster/test_data_cache.py to also test the counts. Change-Id: I56a57d75ff11f00ebc85b85bcaf104fb8108c478 --- M be/src/runtime/io/data-cache.cc M be/src/runtime/io/data-cache.h M be/src/runtime/io/hdfs-file-reader.cc M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h M common/thrift/metrics.json M tests/custom_cluster/test_data_cache.py 7 files changed, 204 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/15382/1 -- To view, visit http://gerrit.cloudera.org:8080/15382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I56a57d75ff11f00ebc85b85bcaf104fb8108c478 Gerrit-Change-Number: 15382 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-9384: Improve Impala shell usability by enabling live progress in interactive mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15219 ) Change subject: IMPALA-9384: Improve Impala shell usability by enabling live_progress in interactive mode .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5459/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15219 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3765b775f663fa227e59728acffe4d5ea9a5e2d3 Gerrit-Change-Number: 15219 Gerrit-PatchSet: 10 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Mar 2020 21:57:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9430: always pass through kerberos configs
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/15340 )
Change subject: IMPALA-9430: always pass through kerberos configs
..
Patch Set 5:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/15340/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15340/5//COMMIT_MSG@15
PS5, Line 15: Having them pick up different
: kerberos settings if internal communication is disabled is weird
It'll fall back to whatever the default behaviour of libkrb5 and the Java
kerberos implementation is - which first look at environment vars or JVM args,
then fall back to whatever their defaults are.
> does that mean this patch is basically an improvement to how Kerberos get's
> configured? e.g. before this patch, you couldn't use Impala flags to
> configure external Kerberos authentication (unless --principal is set), but
> after this patch you can?
Yeah exactly. It means you can configure the kerberos libraries *without*
setting --principal. Which is a good thing because those kerberos libraries
*can* be used for outgoing connections even if --principal is not set.
http://gerrit.cloudera.org:8080/#/c/15340/5//COMMIT_MSG@19
PS5, Line 19: It matches the documentation of the flags
I updated --principal to explain better what it does.
The key thing is that it's implicitly referring to incoming connections. E.g.
incoming external connections means client connections to HS2 or beeswax,
whereas incoming internal connections means the various internal interfaces -
KRPC, the backend thrift stuff, statestore, catalog.
Outgoing connections are kinda glossed over in those comments, and those are
handled internally in different libraries. The kinit behaviour *does* affect
those - it's the difference between acquiring a new TGT as --principal and
picking up whatever principal was in the credential cache from a kinit.
> impala-HMS connections can be kerberized without --principal being set
This was the case before and after this change. But before this change you'd
have to configure it indirectly by setting JAVA_TOOL_OPTIONS, and the various
KRB5 env vars.
http://gerrit.cloudera.org:8080/#/c/15340/5/be/src/rpc/rpc-mgr-kerberized-test.cc
File be/src/rpc/rpc-mgr-kerberized-test.cc:
http://gerrit.cloudera.org:8080/#/c/15340/5/be/src/rpc/rpc-mgr-kerberized-test.cc@206
PS5, Line 206: // Check that the above changes went into the appropriate env
variables.
: EXPECT_EQ("/tmp/DisabledKerberosConfigsKeytab",
string(getenv("KRB5_KTNAME")));
: EXPECT_EQ("/tmp/DisabledKerberosConfigsCC",
string(getenv("KRB5CCNAME")));
: EXPECT_EQ("/tmp/DisabledKerberosConfigsConf",
string(getenv("KRB5_CONFIG")));
: EXPECT_EQ("/tmp/DisabledKerberosConfigsDebug",
string(getenv("KRB5_TRACE")));
> just to clarify, the kerberos config flags get passed to Hadoop via these e
yup. enhanced the comment.
--
To view, visit http://gerrit.cloudera.org:8080/15340
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If4bb311c7ab7173232aab36c5ed801f93f38f5b9
Gerrit-Change-Number: 15340
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Sat, 07 Mar 2020 01:26:02 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9430: always pass through kerberos configs
Hello Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15340 to look at the new patch set (#7). Change subject: IMPALA-9430: always pass through kerberos configs .. IMPALA-9430: always pass through kerberos configs The behaviour of kerberos-related command line flags is changed so that their values are always passed through to underlying libraries, even if Kerberos isn't enabled for internal communication in Impala. This is good because: * Various libraries that communicate with external systems may use kerberos for outgoing connections, if *incoming* connections are not authenticated. e.g. it might just be enabled for HMS. Having them pick up different kerberos settings for outgoing connections if kerberos is disabled for incoming connections is a little weird. This is a safer default that reduces chances of inadvertant misconfigurations. * It matches the documentation of the flags. Some validations are still disabled when --principal is not set, e.g. we don't check the replay cache directory. This is to avoid any potential regressions or startup failures on non-kerberised clusters. Testing: Added unit tests for flag validation and env var setting on the code paths that I touched. Change-Id: If4bb311c7ab7173232aab36c5ed801f93f38f5b9 --- M be/src/common/global-flags.cc M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/rpc/rpc-mgr-kerberized-test.cc 4 files changed, 124 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/15340/7 -- To view, visit http://gerrit.cloudera.org:8080/15340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If4bb311c7ab7173232aab36c5ed801f93f38f5b9 Gerrit-Change-Number: 15340 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9430: always pass through kerberos configs
Hello Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15340 to look at the new patch set (#6). Change subject: IMPALA-9430: always pass through kerberos configs .. IMPALA-9430: always pass through kerberos configs The behaviour of kerberos-related command line flags is changed so that their values are always passed through to underlying libraries, even if Kerberos isn't enabled for internal communication in Impala. This is good because: * Various libraries that communicate with external systems may use kerberos for outgoing connections, if *incoming* connections are not authenticated. e.g. it might just be enabled for HMS. Having them pick up different kerberos settings for outgoing connections if kerberos is disabled for incoming connections is a little weird. This is a safer default that reduces chances of inadvertant misconfigurations. * It matches the documentation of the flags. Some validations are still disabled when --principal is not set, e.g. we don't check the replay cache directory. This is to avoid any potential regressions or startup failures on non-kerberised clusters. Testing: Added unit tests for flag validation and env var setting on the code paths that I touched. Change-Id: If4bb311c7ab7173232aab36c5ed801f93f38f5b9 --- M be/src/common/global-flags.cc M be/src/rpc/authentication.cc M be/src/rpc/authentication.h M be/src/rpc/rpc-mgr-kerberized-test.cc 4 files changed, 124 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/15340/6 -- To view, visit http://gerrit.cloudera.org:8080/15340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If4bb311c7ab7173232aab36c5ed801f93f38f5b9 Gerrit-Change-Number: 15340 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-9156: share broadcast join builds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15096 ) Change subject: IMPALA-9156: share broadcast join builds .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5454/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15096 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c67e4b2c87ed0fba648f1e1710addb885d66dc7 Gerrit-Change-Number: 15096 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 07 Mar 2020 02:43:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9430: always pass through kerberos configs
Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/15340 )
Change subject: IMPALA-9430: always pass through kerberos configs
..
Patch Set 5:
(3 comments)
generally LGTM, mostly questions about what the desired behavior is suppose to
be.
http://gerrit.cloudera.org:8080/#/c/15340/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/15340/5//COMMIT_MSG@15
PS5, Line 15: Having them pick up different
: kerberos settings if internal communication is disabled is weird
what does this mean exactly? if Impala doesn't set the necessary env variables
, then will the Hadoop components use some fallback from core-site.xml?
does that mean this patch is basically an improvement to how Kerberos get's
configured? e.g. before this patch, you couldn't use Impala flags to configure
external Kerberos authentication (unless --principal is set), but after this
patch you can?
http://gerrit.cloudera.org:8080/#/c/15340/5//COMMIT_MSG@19
PS5, Line 19: It matches the documentation of the flags
"Kerberos will not be used for internal or external connections if this is not
set." (from the docs for --principal) not sure what external communication is
suppose to be in this case, but should the docs for --principal be updated
here? the docs of --principal make it sounds like Kerberos isn't use for any
connections unless --principal is set, but things changes things right?
impala-HMS connections can be kerberized without --principal being set
http://gerrit.cloudera.org:8080/#/c/15340/5/be/src/rpc/rpc-mgr-kerberized-test.cc
File be/src/rpc/rpc-mgr-kerberized-test.cc:
http://gerrit.cloudera.org:8080/#/c/15340/5/be/src/rpc/rpc-mgr-kerberized-test.cc@206
PS5, Line 206: // Check that the above changes went into the appropriate env
variables.
: EXPECT_EQ("/tmp/DisabledKerberosConfigsKeytab",
string(getenv("KRB5_KTNAME")));
: EXPECT_EQ("/tmp/DisabledKerberosConfigsCC",
string(getenv("KRB5CCNAME")));
: EXPECT_EQ("/tmp/DisabledKerberosConfigsConf",
string(getenv("KRB5_CONFIG")));
: EXPECT_EQ("/tmp/DisabledKerberosConfigsDebug",
string(getenv("KRB5_TRACE")));
just to clarify, the kerberos config flags get passed to Hadoop via these env
variables?
--
To view, visit http://gerrit.cloudera.org:8080/15340
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If4bb311c7ab7173232aab36c5ed801f93f38f5b9
Gerrit-Change-Number: 15340
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Comment-Date: Fri, 06 Mar 2020 23:02:49 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3343: Make impala-shell compatible with python 3.
David Knupp has posted comments on this change. (
http://gerrit.cloudera.org:8080/15132 )
Change subject: IMPALA-3343: Make impala-shell compatible with python 3.
..
Patch Set 17:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15132/15/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:
http://gerrit.cloudera.org:8080/#/c/15132/15/tests/shell/test_shell_commandline.py@485
PS15, Line 485: if SHELL_IS_PYTHON_2:
> Still digging into this. The proposed change of cname.decode('utf8') fails
Just a note to let you know this hasn't been languishing. I'm stuck in a weird
loop that I'm still trying to figure out.
It turns out that your suggestion was definitely on the right path after all.
What was misleading We seem to need to call decode('utf-8') in python 2, and
forego in python 3. I also needed to revert a change I had made to
https://github.com/apache/impala/blob/master/shell/impala_shell.py#L1221 --
essentially, we need to do the same thing. I've confirmed that with those
changes, on both py2 and py3, the intended behavior works. Here's an example
with a smaller set of chars.
$ impala-shell.sh -q "select '?, ?, ?, ?, ?, ?, ?'"
Starting Impala with no authentication using Python 2.7.12
Opened TCP connection to localhost:21000
Connected to localhost:21000
Server version: impalad version 3.4.0-SNAPSHOT DEBUG (build
b0634b1341f6189b8c7e0c1ba4cce810e947c52a)
Query: select '?, ?, ?, ?, ?, ?, ?'
Query submitted at: 2020-03-06 11:26:31 (Coordinator:
http://dknupp-desktop:25000)
Query progress can be monitored at: http://dknupp-desktop:25000/query_plan?
query_id=b54072afd6be4181:9ea5acc2
+---+
| '?, ?, ?, ?, ?, ?, ?' |
+---+
| ?, ?, ?, ?, ?, ?, ? |
+---+
Fetched 1 row(s) in 0.12s
And it works interactively.
$ impala-shell.sh
Starting Impala with no authentication using Python 2.7.12
Opened TCP connection to localhost:21000
Connected to localhost:21000
Server version: impalad version 3.4.0-SNAPSHOT DEBUG (build
b0634b1341f6189b8c7e0c1ba4cce810e947c52a)
***
Welcome to the Impala shell.
(Impala Shell v3.4.0-SNAPSHOT (b0634b1) built on Fri Feb 21 13:49:48 PST 2020)
Press TAB twice to see a list of available commands.
***
[localhost:21000] default> select '?, ?, ?, ?, ?, ?, ?';
Query: select '?, ?, ?, ?, ?, ?, ?'
Query submitted at: 2020-03-06 11:31:23 (Coordinator:
http://dknupp-desktop:25000)
Query progress can be monitored at: http://dknupp-desktop:25000/query_plan?
query_id=8b4b4528051650f5:95206507
+---+
| '?, ?, ?, ?, ?, ?, ?' |
+---+
| ?, ?, ?, ?, ?, ?, ? |
+---+
Fetched 1 row(s) in 0.12s
[localhost:21000] default>
However (and ignoring the added complication of tabs being present),
test_international_characters_prettyprint is now failing with a
UnicodeDecodeError. On the other hand, that test is passing on my original
version -- but then actual usage fails. It's weird.
--
To view, visit http://gerrit.cloudera.org:8080/15132
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb75e162bac0faeae3e12106c15da39cbfb8b462
Gerrit-Change-Number: 15132
Gerrit-PatchSet: 17
Gerrit-Owner: David Knupp
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 06 Mar 2020 19:33:24 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/15284 ) Change subject: IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header .. Patch Set 8: (3 comments) http://gerrit.cloudera.org:8080/#/c/15284/7/shell/ImpalaHttpClient.py File shell/ImpalaHttpClient.py: http://gerrit.cloudera.org:8080/#/c/15284/7/shell/ImpalaHttpClient.py@47 PS7, Line 47: > flake8: E501 line too long (118 > 90 characters) Done http://gerrit.cloudera.org:8080/#/c/15284/7/shell/ImpalaHttpClient.py@51 PS7, Line 51: > flake8: E501 line too long (132 > 90 characters) Done http://gerrit.cloudera.org:8080/#/c/15284/7/shell/ImpalaHttpClient.py@78 PS7, Line 78: > flake8: E501 line too long (113 > 90 characters) Done -- To view, visit http://gerrit.cloudera.org:8080/15284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4153968551acd58b25c7923c2ebf75ee29a7e76b Gerrit-Change-Number: 15284 Gerrit-PatchSet: 8 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Mar 2020 19:31:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15284 ) Change subject: IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5445/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4153968551acd58b25c7923c2ebf75ee29a7e76b Gerrit-Change-Number: 15284 Gerrit-PatchSet: 7 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Mar 2020 19:29:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9156: share broadcast join builds
Hello Csaba Ringhofer, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15096 to look at the new patch set (#13). Change subject: IMPALA-9156: share broadcast join builds .. IMPALA-9156: share broadcast join builds The scheduler will only create one join build finstance per backend in cases where this is supported. The builder is aware of the number of finstances executing the probe and hands off the build data structures to the builders. Nested loop join requires minimal modifications because the build data structures are read-only after initial construction. The only significant change is that memory can't be transferred to the multiple consumers, so MarkNeedsDeepCopy() needs to be used instead. Hash join requires additional synchronisation because the spilling algorithm mutates build-side data structures. This patch adds synchronisation so that rebuilding spilled partitions is done in a thread-safe manner, using a single thread. This uses the CyclicBarrier added in an earlier patch. Threads blocked on CyclicBarrier need to be cancellable, which is handled by cancelling the barrier when closing the join builder. Update planner to cost broadcast join and estimate memory consumption based on a single instance per node. Planner estimates of number of instances are improved. Instead of assuming mt_dop instances per node, use the total number of input splits (also called scan ranges in places) as an upper bound on the number of instances generated by scans. These instance estimates from the scan nodes are then propagated up the plan tree in the same was as the numNodes estimates. The instance estimate for the join build fragment is fixed to be based on the destination fragment. The profile now correctly accounts for time waiting for the builder, counting it in inactive time and showing it in the node timeline. Additional improvements/cleanup to the time accounting are deferring until IMPALA-9422. Testing: * Updated planner tests * Ran a single node stress test with TPC-H and TPC-DS * Add a targeted test for spilling broadcast joins, both repartitioning and not repartitioning. * Add a targeted test for a spilling broadcast join with empty probe * Add a targeted test for spilling broadcast join with empty build partitions. * Add a broadcast join to test_cancellation and test_failpoints. Perf: I did a single node run on my desktop: +--+---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +--+---+-++++ | TPCH(30) | parquet / none / none | 6.26| -15.70%| 4.63 | -16.16%| +--+---+-++++ +--+--+---++-++---++---++-+-+ | Workload | Query| File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Iters | Median Diff(%) | MW Zval | Tval| +--+--+---++-++---++---++-+-+ | TPCH(30) | TPCH-Q21 | parquet / none / none | 24.97 | 23.25 | R +7.38% | 0.51% | 0.22%| 5 | R +6.95% | 2.31| 27.93 | | TPCH(30) | TPCH-Q4 | parquet / none / none | 2.83 | 2.79| +1.31% | 1.86% | 0.36%| 5 | +1.88% | 1.15| 1.53| | TPCH(30) | TPCH-Q6 | parquet / none / none | 1.28 | 1.28| -0.01% | 1.64% | 1.63%| 5 | -0.11% | -0.58 | -0.01 | | TPCH(30) | TPCH-Q22 | parquet / none / none | 2.65 | 2.68| -0.94% | 0.84% | 1.46%| 5 | -0.21% | -0.87 | -1.25 | | TPCH(30) | TPCH-Q1 | parquet / none / none | 4.69 | 4.72| -0.56% | 1.29% | 0.52%| 5 | -1.04% | -1.15 | -0.89 | | TPCH(30) | TPCH-Q13 | parquet / none / none | 10.64 | 10.80 | -1.48% | 0.61% | 0.60%| 5 | -1.39% | -1.73 | -3.91 | | TPCH(30) | TPCH-Q15 | parquet / none / none | 4.11 | 4.32| -4.92% | 0.05% | 0.40%| 5 | -4.93% | -2.31 | -27.46 | | TPCH(30) | TPCH-Q20 | parquet / none / none | 3.47 | 3.67| I -5.41% | 0.81% | 0.03%| 5 | I -5.70% | -2.31 | -15.75 | | TPCH(30) | TPCH-Q17 | parquet / none / none | 7.58 | 8.14| I -6.93% | 3.13% | 2.62%| 5 | I -9.31% | -2.02 | -3.96 | | TPCH(30) | TPCH-Q9 | parquet / none / none | 15.59 | 17.02 | I -8.38% | 0.95% | 0.43%
[Impala-ASF-CR] IMPALA-9156: share broadcast join builds
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15096 ) Change subject: IMPALA-9156: share broadcast join builds .. Patch Set 13: Rebased and fix an overly strict dcheck that was triggered by TestSpillingNoDebugActionDimensions::test_spilling_naaj_no_deny_reservation -- To view, visit http://gerrit.cloudera.org:8080/15096 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c67e4b2c87ed0fba648f1e1710addb885d66dc7 Gerrit-Change-Number: 15096 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 07 Mar 2020 01:59:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9472,IMPALA-9473: Add per-partition metrics for data cache
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15382 ) Change subject: IMPALA-9472,IMPALA-9473: Add per-partition metrics for data cache .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5448/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a57d75ff11f00ebc85b85bcaf104fb8108c478 Gerrit-Change-Number: 15382 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 06 Mar 2020 22:54:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15308 ) Change subject: IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82a5a776103abd0a6d73336bebc65e22b4e13fef Gerrit-Change-Number: 15308 Gerrit-PatchSet: 7 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 07 Mar 2020 00:00:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9430: always pass through kerberos configs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15340 ) Change subject: IMPALA-9430: always pass through kerberos configs .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5452/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4bb311c7ab7173232aab36c5ed801f93f38f5b9 Gerrit-Change-Number: 15340 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 07 Mar 2020 02:07:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9472,IMPALA-9473: Add per-partition metrics for data cache
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15382 ) Change subject: IMPALA-9472,IMPALA-9473: Add per-partition metrics for data cache .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5447/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56a57d75ff11f00ebc85b85bcaf104fb8108c478 Gerrit-Change-Number: 15382 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 06 Mar 2020 22:49:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15308 ) Change subject: IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled .. IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled When a runtime filter has remote target, coordinator will Disable the FilterState upon arrival of the last filter update to prevent another update towards that filter. As consequence, such runtime filter will always be displayed as disabled in runtime profile (Enabled column is equal to false in Final filter table), when in reality the runtime filter has heard back from all pending backends and complete. The Enabled column should correctly distinguish between failed runtime filter vs complete runtime filter. To do so, we add all_updates_received_ flag in FilterState class and set it to true after filter received enough filter update from pending backends to proceed. If all_updates_received_ is true, then that runtime filter is considered as enabled. Testing: - Add row regex in runtime_filters.test, query 6, to verify REMOTE runtime filter is marked as enabled in final filter table - Run and pass test_runtime_filters.py - Run and pass core tests Change-Id: I82a5a776103abd0a6d73336bebc65e22b4e13fef Reviewed-on: http://gerrit.cloudera.org:8080/15308 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/runtime/coordinator-filter-state.h M be/src/runtime/coordinator.cc M testdata/workloads/functional-query/queries/QueryTest/runtime_filters.test 3 files changed, 32 insertions(+), 15 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I82a5a776103abd0a6d73336bebc65e22b4e13fef Gerrit-Change-Number: 15308 Gerrit-PatchSet: 8 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9430: always pass through kerberos configs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15340 ) Change subject: IMPALA-9430: always pass through kerberos configs .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5453/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15340 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4bb311c7ab7173232aab36c5ed801f93f38f5b9 Gerrit-Change-Number: 15340 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 07 Mar 2020 02:14:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9414 (part 1): Copy THttpClient from Thrift into Impala
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/15283 ) Change subject: IMPALA-9414 (part 1): Copy THttpClient from Thrift into Impala .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15283 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2662f1d4d455120442ef7c0c198685c07207aeed Gerrit-Change-Number: 15283 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Mar 2020 19:18:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15308 ) Change subject: IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5458/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82a5a776103abd0a6d73336bebc65e22b4e13fef Gerrit-Change-Number: 15308 Gerrit-PatchSet: 7 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Mar 2020 19:08:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15308 ) Change subject: IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82a5a776103abd0a6d73336bebc65e22b4e13fef Gerrit-Change-Number: 15308 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Mar 2020 19:08:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15308 ) Change subject: IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82a5a776103abd0a6d73336bebc65e22b4e13fef Gerrit-Change-Number: 15308 Gerrit-PatchSet: 7 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Mar 2020 19:08:36 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-8800: part 2: Bump Kudu version to 4ed0dbbd1
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/15328 ) Change subject: IMPALA-8800: part 2: Bump Kudu version to 4ed0dbbd1 .. Patch Set 3: Verified+1 Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15328 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf6245e53816c9477b5863a0fb2ada917ad103a7 Gerrit-Change-Number: 15328 Gerrit-PatchSet: 3 Gerrit-Owner: Volodymyr Verovkin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Volodymyr Verovkin Gerrit-Comment-Date: Fri, 06 Mar 2020 18:56:24 + Gerrit-HasComments: No
[native-toolchain-CR] IMPALA-8800: part 2: Bump Kudu version to 4ed0dbbd1
Volodymyr Verovkin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15328 ) Change subject: IMPALA-8800: part 2: Bump Kudu version to 4ed0dbbd1 .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/15328/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15328/2//COMMIT_MSG@7 PS2, Line 7: 4ed0dbbd1 > This is outdated now Done -- To view, visit http://gerrit.cloudera.org:8080/15328 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf6245e53816c9477b5863a0fb2ada917ad103a7 Gerrit-Change-Number: 15328 Gerrit-PatchSet: 3 Gerrit-Owner: Volodymyr Verovkin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Volodymyr Verovkin Gerrit-Comment-Date: Fri, 06 Mar 2020 18:51:31 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-8800: part 2: Bump Kudu version to 4ed0dbbd1
Hello Thomas Tauber-Marshall, Grant Henke, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15328 to look at the new patch set (#3). Change subject: IMPALA-8800: part 2: Bump Kudu version to 4ed0dbbd1 .. IMPALA-8800: part 2: Bump Kudu version to 4ed0dbbd1 This pulls in Kudu DATE support which is needed for the Impala side of the Kudu/Impala DATE integration. Testing: - Built Impala locally with kudu-01b667407 and ran test_kudu.py tests. Change-Id: Icf6245e53816c9477b5863a0fb2ada917ad103a7 --- M buildall.sh M source/kudu/build.sh 2 files changed, 3 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/28/15328/3 -- To view, visit http://gerrit.cloudera.org:8080/15328 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf6245e53816c9477b5863a0fb2ada917ad103a7 Gerrit-Change-Number: 15328 Gerrit-PatchSet: 3 Gerrit-Owner: Volodymyr Verovkin Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15284 ) Change subject: IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header .. Patch Set 7: (3 comments) http://gerrit.cloudera.org:8080/#/c/15284/7/shell/ImpalaHttpClient.py File shell/ImpalaHttpClient.py: http://gerrit.cloudera.org:8080/#/c/15284/7/shell/ImpalaHttpClient.py@47 PS7, Line 47: i flake8: E501 line too long (118 > 90 characters) http://gerrit.cloudera.org:8080/#/c/15284/7/shell/ImpalaHttpClient.py@51 PS7, Line 51: y flake8: E501 line too long (132 > 90 characters) http://gerrit.cloudera.org:8080/#/c/15284/7/shell/ImpalaHttpClient.py@78 PS7, Line 78: n flake8: E501 line too long (113 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/15284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4153968551acd58b25c7923c2ebf75ee29a7e76b Gerrit-Change-Number: 15284 Gerrit-PatchSet: 7 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Mar 2020 18:46:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header
Hello David Knupp, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15284 to look at the new patch set (#7). Change subject: IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header .. IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header The 'Expect: 100-continue' http header allows http clients to send only the headers for their request, get a confirmation back from the server that the headers are valid, and only then send the body of the request, avoiding the overhead of sending large requests that will ultimately fail. This patch adds support for this in the HS2 HTTP server by having THttpServer look for the header, and if it's present and the request is validated returning a '100 Continue' response before reading the body of the request. It also adds supports for using this header on large requests sent by impala-shell. Testing: - This case is covered by the existing test_large_sql, however that test was previously broken and passing spuriously. This patch fixes the test. - Passed all other shell tests. Change-Id: I4153968551acd58b25c7923c2ebf75ee29a7e76b --- M be/src/transport/THttpServer.cpp M be/src/transport/THttpTransport.cpp M be/src/transport/THttpTransport.h R shell/ImpalaHttpClient.py M shell/impala_client.py M shell/impala_shell.py M shell/make_shell_tarball.sh M shell/packaging/make_python_package.sh A shell/shell_exceptions.py M tests/shell/test_shell_commandline.py 10 files changed, 129 insertions(+), 69 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/15284/7 -- To view, visit http://gerrit.cloudera.org:8080/15284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4153968551acd58b25c7923c2ebf75ee29a7e76b Gerrit-Change-Number: 15284 Gerrit-PatchSet: 7 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/15284 ) Change subject: IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/15284/6/shell/THttpClient.py File shell/THttpClient.py: http://gerrit.cloudera.org:8080/#/c/15284/6/shell/THttpClient.py@35 PS6, Line 35: > Actually, I'm wrong about prettytable. I was thinking of sqlparse. Of course, this kind of stuff is why I separated it into two patches - a straight copy and then whatever changes we want to make, so that its clear in the git history, but always good to leave more info directing people to what's going on. I went ahead and did the rename + comment. -- To view, visit http://gerrit.cloudera.org:8080/15284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4153968551acd58b25c7923c2ebf75ee29a7e76b Gerrit-Change-Number: 15284 Gerrit-PatchSet: 7 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Mar 2020 18:45:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/15308 ) Change subject: IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled .. Patch Set 6: Code-Review+1 Thanks Riza for the patch! The patch looks good to me after you added an additional input argument when calling Disable(). -- To view, visit http://gerrit.cloudera.org:8080/15308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82a5a776103abd0a6d73336bebc65e22b4e13fef Gerrit-Change-Number: 15308 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Mar 2020 18:39:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4080 [part 3]: Moved codegen code for TupleRowComparator into a Config object
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/15281 ) Change subject: IMPALA-4080 [part 3]: Moved codegen code for TupleRowComparator into a Config object .. Patch Set 8: Just noticed that I forgot to update the commit message regarding the TODO. So just to keep track, I had ran the exhaustive tests and they had completed successfully. -- To view, visit http://gerrit.cloudera.org:8080/15281 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I265604ecbae2c80ff59e697fddfeea4976b25cd9 Gerrit-Change-Number: 15281 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 06 Mar 2020 18:06:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/15284 ) Change subject: IMPALA-9414 (part 2): Support the 'Expect: 100-continue' http header .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/15284/6/shell/THttpClient.py File shell/THttpClient.py: http://gerrit.cloudera.org:8080/#/c/15284/6/shell/THttpClient.py@35 PS6, Line 35: class THttpClient(TTransportBase): > I guess I'm not convinced that subclassing is better than straight copying, Actually, I'm wrong about prettytable. I was thinking of sqlparse. Anyway, not a requirement. -- To view, visit http://gerrit.cloudera.org:8080/15284 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4153968551acd58b25c7923c2ebf75ee29a7e76b Gerrit-Change-Number: 15284 Gerrit-PatchSet: 6 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Mar 2020 18:05:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled
Riza Suminto has posted comments on this change. (
http://gerrit.cloudera.org:8080/15308 )
Change subject: IMPALA-8674: fix bug where REMOTE runtime filter always marked
disabled
..
Patch Set 6:
(3 comments)
Hi Tim,
Thanks for your feedback. Patch set 5..6 should address your comment.
Let me know if there is something else I missed or can be improved.
http://gerrit.cloudera.org:8080/#/c/15308/4/be/src/runtime/coordinator-filter-state.h
File be/src/runtime/coordinator-filter-state.h:
http://gerrit.cloudera.org:8080/#/c/15308/4/be/src/runtime/coordinator-filter-state.h@100
PS4, Line 100: bool received_all_updates() const { return
all_updates_received_; }
> I'd consider making this an argument to Disable() and DisableAndRelease(),
Done
http://gerrit.cloudera.org:8080/#/c/15308/4/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/15308/4/be/src/runtime/coordinator.cc@605
PS4, Line 605: // In case of remote filter, we might intentionally disable
the filter upon
> Maybe simplify to:
Done
http://gerrit.cloudera.org:8080/#/c/15308/4/be/src/runtime/coordinator.cc@1366
PS4, Line 1366: void Coordinator::FilterState::DisableAndRelease(
> nit: the coding style is usually to put conditions on a single line if they
Done. This is removed now since the flag setting is done through function
Disable().
--
To view, visit http://gerrit.cloudera.org:8080/15308
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I82a5a776103abd0a6d73336bebc65e22b4e13fef
Gerrit-Change-Number: 15308
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto
Gerrit-Reviewer: Fang-Yu Rao
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Riza Suminto
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 06 Mar 2020 17:42:30 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15310 ) Change subject: IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files .. IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files Mentioned deflate support on the following lines of impala_txtfile.xml: - modified text to include deflate info - removed redundant paragraph Mentioned deflate support in impala_file_formats.xml. Change-Id: I9e1205e4e408f2c20fd8642cccd6c74e7ba9eb40 Reviewed-on: http://gerrit.cloudera.org:8080/15310 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M docs/topics/impala_file_formats.xml M docs/topics/impala_txtfile.xml 2 files changed, 23 insertions(+), 27 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9e1205e4e408f2c20fd8642cccd6c74e7ba9eb40 Gerrit-Change-Number: 15310 Gerrit-PatchSet: 7 Gerrit-Owner: Kristine Hahn Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiaomeng Zhang
[Impala-ASF-CR] IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15310 ) Change subject: IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e1205e4e408f2c20fd8642cccd6c74e7ba9eb40 Gerrit-Change-Number: 15310 Gerrit-PatchSet: 6 Gerrit-Owner: Kristine Hahn Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Fri, 06 Mar 2020 17:37:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15310 ) Change subject: IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e1205e4e408f2c20fd8642cccd6c74e7ba9eb40 Gerrit-Change-Number: 15310 Gerrit-PatchSet: 6 Gerrit-Owner: Kristine Hahn Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Fri, 06 Mar 2020 17:30:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15310 ) Change subject: IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/364/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e1205e4e408f2c20fd8642cccd6c74e7ba9eb40 Gerrit-Change-Number: 15310 Gerrit-PatchSet: 6 Gerrit-Owner: Kristine Hahn Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Fri, 06 Mar 2020 17:30:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/15310 ) Change subject: IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e1205e4e408f2c20fd8642cccd6c74e7ba9eb40 Gerrit-Change-Number: 15310 Gerrit-PatchSet: 5 Gerrit-Owner: Kristine Hahn Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Fri, 06 Mar 2020 17:28:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15310 ) Change subject: IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files .. Patch Set 5: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/551/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/15310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e1205e4e408f2c20fd8642cccd6c74e7ba9eb40 Gerrit-Change-Number: 15310 Gerrit-PatchSet: 5 Gerrit-Owner: Kristine Hahn Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Fri, 06 Mar 2020 17:25:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files
Hello Andrew Sherman, Abhishek Rawat, Xiaomeng Zhang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15310 to look at the new patch set (#5). Change subject: IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files .. IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files Mentioned deflate support on the following lines of impala_txtfile.xml: - modified text to include deflate info - removed redundant paragraph Mentioned deflate support in impala_file_formats.xml. Change-Id: I9e1205e4e408f2c20fd8642cccd6c74e7ba9eb40 --- M docs/topics/impala_file_formats.xml M docs/topics/impala_txtfile.xml 2 files changed, 23 insertions(+), 27 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/15310/5 -- To view, visit http://gerrit.cloudera.org:8080/15310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9e1205e4e408f2c20fd8642cccd6c74e7ba9eb40 Gerrit-Change-Number: 15310 Gerrit-PatchSet: 5 Gerrit-Owner: Kristine Hahn Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiaomeng Zhang
[Impala-ASF-CR] IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15310 ) Change subject: IMPALA-9431 [DOCS] Update docs to reflect Deflate is supported for text files .. Patch Set 5: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/551/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/15310 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e1205e4e408f2c20fd8642cccd6c74e7ba9eb40 Gerrit-Change-Number: 15310 Gerrit-PatchSet: 5 Gerrit-Owner: Kristine Hahn Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Comment-Date: Fri, 06 Mar 2020 17:17:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15308 ) Change subject: IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5444/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82a5a776103abd0a6d73336bebc65e22b4e13fef Gerrit-Change-Number: 15308 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Mar 2020 16:43:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled
Hello Fang-Yu Rao, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15308 to look at the new patch set (#6). Change subject: IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled .. IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled When a runtime filter has remote target, coordinator will Disable the FilterState upon arrival of the last filter update to prevent another update towards that filter. As consequence, such runtime filter will always be displayed as disabled in runtime profile (Enabled column is equal to false in Final filter table), when in reality the runtime filter has heard back from all pending backends and complete. The Enabled column should correctly distinguish between failed runtime filter vs complete runtime filter. To do so, we add all_updates_received_ flag in FilterState class and set it to true after filter received enough filter update from pending backends to proceed. If all_updates_received_ is true, then that runtime filter is considered as enabled. Testing: - Add row regex in runtime_filters.test, query 6, to verify REMOTE runtime filter is marked as enabled in final filter table - Run and pass test_runtime_filters.py - Run and pass core tests Change-Id: I82a5a776103abd0a6d73336bebc65e22b4e13fef --- M be/src/runtime/coordinator-filter-state.h M be/src/runtime/coordinator.cc M testdata/workloads/functional-query/queries/QueryTest/runtime_filters.test 3 files changed, 32 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/15308/6 -- To view, visit http://gerrit.cloudera.org:8080/15308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I82a5a776103abd0a6d73336bebc65e22b4e13fef Gerrit-Change-Number: 15308 Gerrit-PatchSet: 6 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15308 ) Change subject: IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5443/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82a5a776103abd0a6d73336bebc65e22b4e13fef Gerrit-Change-Number: 15308 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Mar 2020 15:59:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled
Hello Fang-Yu Rao, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15308 to look at the new patch set (#5). Change subject: IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled .. IMPALA-8674: fix bug where REMOTE runtime filter always marked disabled When a runtime filter has remote target, coordinator will Disable the FilterState upon arrival of the last filter update to prevent another update towards that filter. As consequence, such runtime filter will always be displayed as disabled in runtime profile (Enabled column is equal to false in Final filter table), when in reality the runtime filter has heard back from all pending backends and complete. The Enabled column should correctly distinguish between failed runtime filter vs complete runtime filter. To do so, we add all_updates_received_ flag in FilterState class and set it to true after filter received enough filter update from pending backends to proceed. If all_updates_received_ is true, then that runtime filter is considered as enabled. Testing: - Add row regex in runtime_filters.test, query 6, to verify REMOTE runtime filter is marked as enabled in final filter table - Run and pass test_runtime_filters.py - Run and pass core tests Change-Id: I82a5a776103abd0a6d73336bebc65e22b4e13fef --- M be/src/runtime/coordinator-filter-state.h M be/src/runtime/coordinator.cc M testdata/workloads/functional-query/queries/QueryTest/runtime_filters.test 3 files changed, 29 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/15308/5 -- To view, visit http://gerrit.cloudera.org:8080/15308 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I82a5a776103abd0a6d73336bebc65e22b4e13fef Gerrit-Change-Number: 15308 Gerrit-PatchSet: 5 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] WIP: Asynchronous code generation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. Patch Set 21: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5442/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 21 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 06 Mar 2020 14:31:53 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Asynchronous code generation
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. Patch Set 21: When unnesting PhjBuilder::Partition and thereby also PhjBuilder::Partition::InsertBatch (the function, not the typedef) I should have also changed be/src/codegen/gen_ir_descriptions.py. I'm doing that now together with rebasing and conflict resolution. -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 21 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 06 Mar 2020 13:48:21 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Asynchronous code generation
Daniel Becker has uploaded a new patch set (#21). ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. WIP: Asynchronous code generation This commit introduces optional asynchronous code generation. Asynchronous code generation means that instead of waiting for codegen to finish, the query starts in interpreted mode while codegen is done on another thread. All the function pointers that point to codegen'd functions are changed to be atomic, wrapped in a CodegenFnPtr. These are initialised to nullptr and as long as they are nullptr, the corresponding interpreted functions are used (as before). When code generation is ready, the funtion pointers are set by the codegen thread. No synchronisation is needed as the function pointers are atomic and it is not a problem if, at a given moment, only a subset of the codegen'd function pointers are set and the rest are interpreted. Asynchronous code generation can be turned on using the ASYNC_CODEGEN boolean query option. TODO: The default should be synchronous codegen for now. TODO: Testing. TODO: Benchmarks. Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b --- M be/src/benchmarks/hash-benchmark.cc A be/src/codegen/codegen-fn-ptr.h M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/llvm-codegen-test.cc M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hdfs-avro-scanner.cc M be/src/exec/hdfs-avro-scanner.h M be/src/exec/hdfs-columnar-scanner.cc M be/src/exec/hdfs-columnar-scanner.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-scanner.h M be/src/exec/hdfs-sequence-scanner.cc M be/src/exec/hdfs-text-scanner.cc M be/src/exec/non-grouping-aggregator.cc M be/src/exec/non-grouping-aggregator.h M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/partitioned-hash-join-builder-ir.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node-ir.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/select-node.cc M be/src/exec/select-node.h M be/src/exec/topn-node.cc M be/src/exec/topn-node.h M be/src/exec/union-node.cc M be/src/exec/union-node.h M be/src/exprs/expr-codegen-test.cc M be/src/exprs/scalar-expr.cc M be/src/exprs/scalar-expr.h M be/src/exprs/scalar-expr.inline.h M be/src/exprs/scalar-fn-call.cc M be/src/exprs/scalar-fn-call.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/runtime/runtime-state.h M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/tuple-row-compare.cc M be/src/util/tuple-row-compare.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/query_test/test_queries.py M tests/query_test/test_query_mem_limit.py 52 files changed, 725 insertions(+), 414 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/15105/21 -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 21 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9384: Improve Impala shell usability by enabling live progress in interactive mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15219 ) Change subject: IMPALA-9384: Improve Impala shell usability by enabling live_progress in interactive mode .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5457/ -- To view, visit http://gerrit.cloudera.org:8080/15219 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3765b775f663fa227e59728acffe4d5ea9a5e2d3 Gerrit-Change-Number: 15219 Gerrit-PatchSet: 9 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Mar 2020 13:24:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala WebUI by default
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala WebUI by default .. Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/15288/12/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/15288/12/tests/webserver/test_web_pages.py@419 PS12, Line 419: I miss a test to check that the length of the query statement is configurable. Could you look around in the tests and check if there is a way to test these kind of configs as well? -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 12 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Fri, 06 Mar 2020 13:15:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: Asynchronous code generation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. Patch Set 20: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5441/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 20 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 06 Mar 2020 11:34:12 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Asynchronous code generation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. Patch Set 19: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5440/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 19 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 06 Mar 2020 11:14:54 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: Asynchronous code generation
Daniel Becker has uploaded a new patch set (#20). ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. WIP: Asynchronous code generation This commit introduces optional asynchronous code generation. Asynchronous code generation means that instead of waiting for codegen to finish, the query starts in interpreted mode while codegen is done on another thread. All the function pointers that point to codegen'd functions are changed to be atomic, wrapped in a CodegenFnPtr. These are initialised to nullptr and as long as they are nullptr, the corresponding interpreted functions are used (as before). When code generation is ready, the funtion pointers are set by the codegen thread. No synchronisation is needed as the function pointers are atomic and it is not a problem if, at a given moment, only a subset of the codegen'd function pointers are set and the rest are interpreted. Asynchronous code generation can be turned on using the ASYNC_CODEGEN boolean query option. TODO: The default should be synchronous codegen for now. TODO: Testing. TODO: Benchmarks. Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b --- M be/src/benchmarks/hash-benchmark.cc A be/src/codegen/codegen-fn-ptr.h M be/src/codegen/llvm-codegen-test.cc M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hdfs-avro-scanner.cc M be/src/exec/hdfs-avro-scanner.h M be/src/exec/hdfs-columnar-scanner.cc M be/src/exec/hdfs-columnar-scanner.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-scanner.h M be/src/exec/hdfs-sequence-scanner.cc M be/src/exec/hdfs-text-scanner.cc M be/src/exec/non-grouping-aggregator.cc M be/src/exec/non-grouping-aggregator.h M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/partitioned-hash-join-builder-ir.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node-ir.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/select-node.cc M be/src/exec/select-node.h M be/src/exec/topn-node.cc M be/src/exec/topn-node.h M be/src/exec/union-node.cc M be/src/exec/union-node.h M be/src/exprs/expr-codegen-test.cc M be/src/exprs/scalar-expr.cc M be/src/exprs/scalar-expr.h M be/src/exprs/scalar-expr.inline.h M be/src/exprs/scalar-fn-call.cc M be/src/exprs/scalar-fn-call.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/runtime/runtime-state.h M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/tuple-row-compare.cc M be/src/util/tuple-row-compare.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/query_test/test_queries.py M tests/query_test/test_query_mem_limit.py 51 files changed, 723 insertions(+), 410 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/15105/20 -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 20 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] WIP: Asynchronous code generation
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/15105 )
Change subject: WIP: Asynchronous code generation
..
Patch Set 19:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15105/19/be/src/exec/partitioned-hash-join-builder.cc
File be/src/exec/partitioned-hash-join-builder.cc:
http://gerrit.cloudera.org:8080/#/c/15105/19/be/src/exec/partitioned-hash-join-builder.cc@659
PS19, Line 659: int PhjBuilder::GetNumSpilledPartitions(const
vector>& partitions) {
line too long (100 > 90)
http://gerrit.cloudera.org:8080/#/c/15105/19/be/src/exec/partitioned-hash-join-builder.cc@979
PS19, Line 979: PhjBuilderPartition::PhjBuilderPartition(RuntimeState* state,
PhjBuilder* parent, int level)
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/15105
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b
Gerrit-Change-Number: 15105
Gerrit-PatchSet: 19
Gerrit-Owner: Daniel Becker
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Daniel Becker
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Fri, 06 Mar 2020 10:31:45 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: Asynchronous code generation
Daniel Becker has uploaded a new patch set (#19). ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. WIP: Asynchronous code generation This commit introduces optional asynchronous code generation. Asynchronous code generation means that instead of waiting for codegen to finish, the query starts in interpreted mode while codegen is done on another thread. All the function pointers that point to codegen'd functions are changed to be atomic, wrapped in a CodegenFnPtr. These are initialised to nullptr and as long as they are nullptr, the corresponding interpreted functions are used (as before). When code generation is ready, the funtion pointers are set by the codegen thread. No synchronisation is needed as the function pointers are atomic and it is not a problem if, at a given moment, only a subset of the codegen'd function pointers are set and the rest are interpreted. Asynchronous code generation can be turned on using the ASYNC_CODEGEN boolean query option. TODO: The default should be synchronous codegen for now. TODO: Testing. TODO: Benchmarks. Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b --- M be/src/benchmarks/hash-benchmark.cc A be/src/codegen/codegen-fn-ptr.h M be/src/codegen/llvm-codegen-test.cc M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hdfs-avro-scanner.cc M be/src/exec/hdfs-avro-scanner.h M be/src/exec/hdfs-columnar-scanner.cc M be/src/exec/hdfs-columnar-scanner.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scanner.cc M be/src/exec/hdfs-scanner.h M be/src/exec/hdfs-sequence-scanner.cc M be/src/exec/hdfs-text-scanner.cc M be/src/exec/non-grouping-aggregator.cc M be/src/exec/non-grouping-aggregator.h M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/partitioned-hash-join-builder-ir.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node-ir.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/select-node.cc M be/src/exec/select-node.h M be/src/exec/topn-node.cc M be/src/exec/topn-node.h M be/src/exec/union-node.cc M be/src/exec/union-node.h M be/src/exprs/expr-codegen-test.cc M be/src/exprs/scalar-expr.cc M be/src/exprs/scalar-expr.h M be/src/exprs/scalar-expr.inline.h M be/src/exprs/scalar-fn-call.cc M be/src/exprs/scalar-fn-call.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/runtime/runtime-state.h M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/tuple-row-compare.cc M be/src/util/tuple-row-compare.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M tests/query_test/test_queries.py M tests/query_test/test_query_mem_limit.py 50 files changed, 720 insertions(+), 408 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/15105/19 -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 19 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] WIP: Asynchronous code generation
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/15105 ) Change subject: WIP: Asynchronous code generation .. Patch Set 18: Thanks Tim. I still can't unnest the typedef of InsertBatchFn before the definition of PhjBuilder::Partition because it takes it as a parameter. I think I'll unnest PhjBuilder::Partition to a top level PhjBuilderPartition class. -- To view, visit http://gerrit.cloudera.org:8080/15105 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia7cbfa7c6734dcf03641629429057d6a4194aa6b Gerrit-Change-Number: 15105 Gerrit-PatchSet: 18 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 06 Mar 2020 09:32:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9466: impala-shell client retry for hs2-http protocol
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15378 ) Change subject: IMPALA-9466: impala-shell client retry for hs2-http protocol .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5439/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15378 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0da9e9e8d34a340eaf763397cc095ff6260d65d5 Gerrit-Change-Number: 15378 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 06 Mar 2020 08:27:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9384: Improve Impala shell usability by enabling live progress in interactive mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15219 ) Change subject: IMPALA-9384: Improve Impala shell usability by enabling live_progress in interactive mode .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5457/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15219 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3765b775f663fa227e59728acffe4d5ea9a5e2d3 Gerrit-Change-Number: 15219 Gerrit-PatchSet: 9 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Mar 2020 08:08:25 + Gerrit-HasComments: No
