[Impala-ASF-CR] IMPALA-4784: Remove InProcessStatestore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10843 ) Change subject: IMPALA-4784: Remove InProcessStatestore .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2769/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2621873e593b36c9612a6402ac6c5d8e3b49cde9 Gerrit-Change-Number: 10843 Gerrit-PatchSet: 6 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Jul 2018 05:00:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4784: Remove InProcessStatestore
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10843 to look at the new patch set (#6). Change subject: IMPALA-4784: Remove InProcessStatestore .. IMPALA-4784: Remove InProcessStatestore InProcessStatestore was only used by statestore-test, expr-test and session-expiry-test. With a slight refactor of the Statestore class, InProcessStatestore becomes obsolete. This patch moves the ownership of the ThriftServer into the Statestore class and Statestore::Init() now takes a 'port' parameter instead of using the FLAGS_state_store_port directly. We also remove the InProcessStatestore completely. A follow on patch will get rid of the InProcessImpalaServer too (IMPALA-6013) Testing: Ran 'core' tests. Change-Id: I2621873e593b36c9612a6402ac6c5d8e3b49cde9 --- M be/src/exprs/expr-test.cc M be/src/service/session-expiry-test.cc M be/src/statestore/statestore-test.cc M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M be/src/statestore/statestored-main.cc M be/src/testutil/in-process-servers.cc M be/src/testutil/in-process-servers.h 8 files changed, 95 insertions(+), 125 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/10843/6 -- To view, visit http://gerrit.cloudera.org:8080/10843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2621873e593b36c9612a6402ac6c5d8e3b49cde9 Gerrit-Change-Number: 10843 Gerrit-PatchSet: 6 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-4784: Remove InProcessStatestore
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/10843 ) Change subject: IMPALA-4784: Remove InProcessStatestore .. Patch Set 6: Code-Review+2 Hit a clang-tidy issue. Rebase, carry +2. -- To view, visit http://gerrit.cloudera.org:8080/10843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2621873e593b36c9612a6402ac6c5d8e3b49cde9 Gerrit-Change-Number: 10843 Gerrit-PatchSet: 6 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Jul 2018 04:59:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10704 ) Change subject: IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans .. Patch Set 13: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2768/ -- To view, visit http://gerrit.cloudera.org:8080/10704 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d6c26d42db090c8a15c602f6419ad6399c329e7 Gerrit-Change-Number: 10704 Gerrit-PatchSet: 13 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Jul 2018 04:22:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1760: Implement shutdown command
Lars Volker has posted comments on this change. (
http://gerrit.cloudera.org:8080/10744 )
Change subject: IMPALA-1760: Implement shutdown command
..
Patch Set 7:
(19 comments)
http://gerrit.cloudera.org:8080/#/c/10744/7//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10744/7//COMMIT_MSG@46
PS7, Line 46: does
extra word?
http://gerrit.cloudera.org:8080/#/c/10744/7//COMMIT_MSG@56
PS7, Line 56: Limitations:
I think it would be good to mention that the shutdown cannot be aborted. Even
better would be to make it abortable, e.g. add another cancel_shutdown()
command. That would also help validate the design of the admin fn utility code.
Another minor limitation seems to be that we cannot quiesce a daemon without
shutting it down. Should we expose the two-stage nature of the process through
explicit commands? That could help with debugging, i.e. we could isolate a
misbehaving machine without killing it. This could be a follow up change, too.
http://gerrit.cloudera.org:8080/#/c/10744/7/be/src/service/client-request-state.h
File be/src/service/client-request-state.h:
http://gerrit.cloudera.org:8080/#/c/10744/7/be/src/service/client-request-state.h@a451
PS7, Line 451:
What happened to LOAD DATA?
http://gerrit.cloudera.org:8080/#/c/10744/7/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/10744/7/be/src/service/client-request-state.cc@612
PS7, Line 612: avoid
nit: avoids
http://gerrit.cloudera.org:8080/#/c/10744/7/be/src/service/client-request-state.cc@626
PS7, Line 626: RETURN_IF_ERROR(client_status);
Should we retry if we get a stale client (similar to here:
https://github.com/apache/impala/blob/4ce2af9ff2340bba1bfb40a1ed3b5b39d6011f87/be/src/runtime/coordinator-backend-state.cc#L363)
?
http://gerrit.cloudera.org:8080/#/c/10744/7/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/10744/7/be/src/service/impala-server.cc@216
PS7, Line 216: drain
I'd rephrase it as "wait for queries" since they might not even have arrived
yet. "drain" reads as if they're already running.
Also, mention the unit of time in the help string, too?
http://gerrit.cloudera.org:8080/#/c/10744/7/be/src/service/impala-server.cc@2328
PS7, Line 2328: while (deadline_val == 0 || deadline_val >
absolute_deadline_ms) {
maybe name these "curr_deadline_val" and "new_deadline_val"? or something
similar?
http://gerrit.cloudera.org:8080/#/c/10744/7/common/thrift/ImpalaInternalService.thrift
File common/thrift/ImpalaInternalService.thrift:
http://gerrit.cloudera.org:8080/#/c/10744/7/common/thrift/ImpalaInternalService.thrift@881
PS7, Line 881: struct TShutdownStatus {
Why not fold this into the Result RPC below?
http://gerrit.cloudera.org:8080/#/c/10744/7/common/thrift/Types.thrift
File common/thrift/Types.thrift:
http://gerrit.cloudera.org:8080/#/c/10744/7/common/thrift/Types.thrift@104
PS7, Line 104: comment
comment? command? maybe rename to ADMIN_CMD then?
http://gerrit.cloudera.org:8080/#/c/10744/7/fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java
File fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java:
http://gerrit.cloudera.org:8080/#/c/10744/7/fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java@95
PS7, Line 95:* Supports optionally specify the backend and the deadline,
either shutdown(),
nit: grammar
http://gerrit.cloudera.org:8080/#/c/10744/7/fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java@128
PS7, Line 128: private Pair getShutdownArgs() throws
AnalysisException {
I think this might be more readable with a loop, parsing each expr and setting
the class members as a side effect.
http://gerrit.cloudera.org:8080/#/c/10744/7/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:
http://gerrit.cloudera.org:8080/#/c/10744/7/fe/src/main/java/org/apache/impala/analysis/Expr.java@1546
PS7, Line 1546:* Analyzes and evaluates expression to a non-negative
integral value, returned as a long.
nit: long line
http://gerrit.cloudera.org:8080/#/c/10744/7/fe/src/test/java/org/apache/impala/analysis/ParserTest.java
File fe/src/test/java/org/apache/impala/analysis/ParserTest.java:
http://gerrit.cloudera.org:8080/#/c/10744/7/fe/src/test/java/org/apache/impala/analysis/ParserTest.java@3807
PS7, Line 3807: ParsesOk(":foobar()");
Can you add some with arguments?
http://gerrit.cloudera.org:8080/#/c/10744/7/fe/src/test/java/org/apache/impala/analysis/ParserTest.java@3831
PS7, Line 3831: ParserError(": shutdown() :other()");
How about ": shutdown('hostA'); : shutdown('hostB'); ?
http://gerrit.cloudera.org:8080/#/c/10744/7/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:
[Impala-ASF-CR] IMPALA-7190: Remove unsupported format writer support
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10823 ) Change subject: IMPALA-7190: Remove unsupported format writer support .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I821dc7495a901f1658daa500daf3791b386c7185 Gerrit-Change-Number: 10823 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Jul 2018 02:57:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6352: Dump backtrace on failure of TestTableSample
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10851 ) Change subject: IMPALA-6352: Dump backtrace on failure of TestTableSample .. IMPALA-6352: Dump backtrace on failure of TestTableSample TestTableSample is a flaky test which has been failing very rarely due to a possible hung thread. Therefore this patch adds a timeout to the test and logs the backtrace of all impalads if timeout occurs, so we can get more information on the state of those threads. Change-Id: I73fcdd30863cee105584c947bb0c48cf872809c1 Reviewed-on: http://gerrit.cloudera.org:8080/10851 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/beeswax/impala_beeswax.py M tests/common/impala_connection.py M tests/query_test/test_tablesample.py 3 files changed, 49 insertions(+), 5 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I73fcdd30863cee105584c947bb0c48cf872809c1 Gerrit-Change-Number: 10851 Gerrit-PatchSet: 4 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6352: Dump backtrace on failure of TestTableSample
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10851 ) Change subject: IMPALA-6352: Dump backtrace on failure of TestTableSample .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73fcdd30863cee105584c947bb0c48cf872809c1 Gerrit-Change-Number: 10851 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Jul 2018 02:48:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7190: Remove unsupported format writer support
Hello Lars Volker, Tim Armstrong, Csaba Ringhofer, Dan Hecht, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10823 to look at the new patch set (#3). Change subject: IMPALA-7190: Remove unsupported format writer support .. IMPALA-7190: Remove unsupported format writer support This patch removes write support for unsupported formats like Sequence, Avro and compressed text. Also, the related query options ALLOW_UNSUPPORTED_FORMATS and SEQ_COMPRESSION_MODE have been migrated to the REMOVED query options type. Testing: Ran exhaustive build. Change-Id: I821dc7495a901f1658daa500daf3791b386c7185 --- M be/src/exec/CMakeLists.txt D be/src/exec/hdfs-avro-table-writer.cc D be/src/exec/hdfs-avro-table-writer.h D be/src/exec/hdfs-sequence-table-writer.cc D be/src/exec/hdfs-sequence-table-writer.h M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-text-table-writer.cc M be/src/exec/hdfs-text-table-writer.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java M testdata/bad_avro_snap/README D testdata/workloads/functional-query/queries/QueryTest/avro-writer.test D testdata/workloads/functional-query/queries/QueryTest/seq-writer.test M testdata/workloads/functional-query/queries/QueryTest/set.test D testdata/workloads/functional-query/queries/QueryTest/text-writer.test A testdata/workloads/functional-query/queries/QueryTest/unsupported-writers.test M tests/common/test_dimensions.py M tests/hs2/test_hs2.py M tests/metadata/test_partition_metadata.py M tests/query_test/test_compressed_formats.py M tests/shell/test_shell_interactive.py 25 files changed, 121 insertions(+), 1,607 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/10823/3 -- To view, visit http://gerrit.cloudera.org:8080/10823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I821dc7495a901f1658daa500daf3791b386c7185 Gerrit-Change-Number: 10823 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7190: Remove unsupported format writer support
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/10823 ) Change subject: IMPALA-7190: Remove unsupported format writer support .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/10823/2/testdata/workloads/functional-query/queries/QueryTest/unsupported-writers.test File testdata/workloads/functional-query/queries/QueryTest/unsupported-writers.test: http://gerrit.cloudera.org:8080/#/c/10823/2/testdata/workloads/functional-query/queries/QueryTest/unsupported-writers.test@3 PS2, Line 3: drop table if exists __text_write; > These drop tables shouldn't be needed for the unique database, should they? oops, missed this. Done. -- To view, visit http://gerrit.cloudera.org:8080/10823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I821dc7495a901f1658daa500daf3791b386c7185 Gerrit-Change-Number: 10823 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Jul 2018 01:49:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1760: Implement shutdown command
Bikramjeet Vig has posted comments on this change. (
http://gerrit.cloudera.org:8080/10744 )
Change subject: IMPALA-1760: Implement shutdown command
..
Patch Set 8:
(13 comments)
still need to review tests
http://gerrit.cloudera.org:8080/#/c/10744/8//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10744/8//COMMIT_MSG@39
PS8, Line 39: which is
how about: which ideally is set to the AC
http://gerrit.cloudera.org:8080/#/c/10744/6/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/10744/6/be/src/service/client-request-state.cc@616
PS6, Line 616:
nit:avoids
http://gerrit.cloudera.org:8080/#/c/10744/6/be/src/service/impala-hs2-server.cc
File be/src/service/impala-hs2-server.cc:
http://gerrit.cloudera.org:8080/#/c/10744/6/be/src/service/impala-hs2-server.cc@381
PS6, Line 381: HS2_RETURN_IF_ERROR(return_val, CheckNotShuttingDown(),
SQLSTATE_GENERAL_ERROR);
we can probably remove the check here since its not starting any new operation.
http://gerrit.cloudera.org:8080/#/c/10744/8/be/src/service/impala-server.h
File be/src/service/impala-server.h:
http://gerrit.cloudera.org:8080/#/c/10744/8/be/src/service/impala-server.h@385
PS8, Line 385: an
nit: a
http://gerrit.cloudera.org:8080/#/c/10744/6/be/src/service/impala-server.h
File be/src/service/impala-server.h:
http://gerrit.cloudera.org:8080/#/c/10744/6/be/src/service/impala-server.h@354
PS6, Line 354: otherwi
nit: shutdown_status
http://gerrit.cloudera.org:8080/#/c/10744/6/be/src/service/impala-server.h@358
PS6, Line 358:
dont mention private vars in public method comments. also at L367
http://gerrit.cloudera.org:8080/#/c/10744/6/be/src/service/impala-server.h@361
PS6, Line 361: ft
nit: a
http://gerrit.cloudera.org:8080/#/c/10744/6/be/src/service/impala-server.h@362
PS6, Line 362: ly.
nit:before
http://gerrit.cloudera.org:8080/#/c/10744/6/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/10744/6/be/src/service/impala-server.cc@212
PS6, Line 212:
nit: long line
http://gerrit.cloudera.org:8080/#/c/10744/8/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/10744/8/be/src/service/impala-server.cc@1772
PS8, Line 1772: && is_shutting_down == it->second.is_shutting_down) {
maybe mention briefly that is_shutting_down is the only part that can change
during the lifecycle of the impalad
http://gerrit.cloudera.org:8080/#/c/10744/6/common/thrift/ImpalaInternalService.thrift
File common/thrift/ImpalaInternalService.thrift:
http://gerrit.cloudera.org:8080/#/c/10744/6/common/thrift/ImpalaInternalService.thrift@865
PS6, Line 865: 5: optional TBloomFilter bloom_filter
not sure if mentioned elsewhere in the code, but maybe we can explain the
difference in "quiesce" and "deadline" here
http://gerrit.cloudera.org:8080/#/c/10744/6/common/thrift/ImpalaInternalService.thrift@867
PS6, Line 867: er
nit: deadline
http://gerrit.cloudera.org:8080/#/c/10744/8/fe/src/main/java/org/apache/impala/analysis/Expr.java
File fe/src/main/java/org/apache/impala/analysis/Expr.java:
http://gerrit.cloudera.org:8080/#/c/10744/8/fe/src/main/java/org/apache/impala/analysis/Expr.java@1546
PS8, Line 1546: ong
nit:long line
--
To view, visit http://gerrit.cloudera.org:8080/10744
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d5606ccfec84db4482c1e7f0f198103aad141a0
Gerrit-Change-Number: 10744
Gerrit-PatchSet: 8
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Dan Hecht
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 03 Jul 2018 01:36:55 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6034: Add Cpu and scanned bytes limits per query
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10415 ) Change subject: IMPALA-6034: Add Cpu and scanned bytes limits per query .. Patch Set 6: I'm going to take over this patch from Mostafa. Need to do some cleanup but I'll post a new patchset soon. -- To view, visit http://gerrit.cloudera.org:8080/10415 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c6015e21da684bb9f33e236d71309dd4c178a20 Gerrit-Change-Number: 10415 Gerrit-PatchSet: 6 Gerrit-Owner: Mostafa Mokhtar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Jul 2018 01:14:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4784: Remove InProcessStatestore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10843 ) Change subject: IMPALA-4784: Remove InProcessStatestore .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2766/ -- To view, visit http://gerrit.cloudera.org:8080/10843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2621873e593b36c9612a6402ac6c5d8e3b49cde9 Gerrit-Change-Number: 10843 Gerrit-PatchSet: 5 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Jul 2018 01:14:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10704 ) Change subject: IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans .. Patch Set 13: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10704 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d6c26d42db090c8a15c602f6419ad6399c329e7 Gerrit-Change-Number: 10704 Gerrit-PatchSet: 13 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Jul 2018 01:00:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10704 ) Change subject: IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2768/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10704 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d6c26d42db090c8a15c602f6419ad6399c329e7 Gerrit-Change-Number: 10704 Gerrit-PatchSet: 13 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Jul 2018 01:00:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5486: Port ReportExecStatus() RPC to use KRPC
Michael Ho has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10855 Change subject: IMPALA-5486: Port ReportExecStatus() RPC to use KRPC .. IMPALA-5486: Port ReportExecStatus() RPC to use KRPC This change converts ReportExecStatus() RPC from thrift based RPC to KRPC. This is done in part of the preparation for fixing IMPALA-2990 as we can take advantage of TCP connection multiplexing in KRPC to avoid overwhelming the coordinator with too many connections by reducing the number of TCP connection to one for each executor. This patch also introduces a new service pool for all query execution control related RPCs in the future so they control commands from coordinators aren't blocked by long-running DataStream services' RPCs. The majority of this patch is mechanical convertion of some Thrift structures to Protobuf. Note that the runtime profile is still retained as Thrift structure as Impala clients will still fetch query profile in Thrift. This avoids duplicated serialization implementation in both Thrift and Protobuf for the runtime profile. The Thrift runtime profiles are serialized and sent as a sidecar in ReportExecStatus() RPC. Change-Id: I7638583b433dcac066b87198e448743d90415ebe --- M be/src/benchmarks/expr-benchmark.cc M be/src/exec/data-sink.cc M be/src/exec/data-sink.h M be/src/exec/hbase-table-sink.cc M be/src/exec/hdfs-parquet-table-writer.cc M be/src/exec/hdfs-parquet-table-writer.h M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-writer.cc M be/src/exec/hdfs-table-writer.h M be/src/rpc/thrift-util.h M be/src/runtime/backend-client.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/data-stream-test.cc M be/src/runtime/dml-exec-state.cc M be/src/runtime/dml-exec-state.h M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/query-state.cc M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/service/CMakeLists.txt M be/src/service/client-request-state.cc M be/src/service/client-request-state.h A be/src/service/control-service.cc A be/src/service/control-service.h M be/src/service/impala-internal-service.cc M be/src/service/impala-internal-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/testutil/in-process-servers.cc M be/src/util/container-util.h A be/src/util/error-util-internal.h M be/src/util/error-util-test.cc M be/src/util/error-util.cc M be/src/util/error-util.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M be/src/util/uid-util.h M bin/bootstrap_toolchain.py M bin/impala-config.sh M common/protobuf/CMakeLists.txt M common/protobuf/common.proto A common/protobuf/control_service.proto M common/protobuf/data_stream_service.proto M common/protobuf/row_batch.proto M common/protobuf/rpc_test.proto M common/thrift/ImpalaInternalService.thrift 52 files changed, 942 insertions(+), 555 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/10855/1 -- To view, visit http://gerrit.cloudera.org:8080/10855 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7638583b433dcac066b87198e448743d90415ebe Gerrit-Change-Number: 10855 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho
[Impala-ASF-CR] IMPALA-7237: handle hex digits in ParseSmaps()
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10853 Change subject: IMPALA-7237: handle hex digits in ParseSmaps() .. IMPALA-7237: handle hex digits in ParseSmaps() Testing: Manual. Added some temporary logging to print out which branch it took with each line and confirmed it took the right branch for a line starting with 'f'. Change-Id: I3dad846dafb25b414bee1858eb63f3eda31d59ac --- M be/src/util/mem-info.cc 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/10853/1 -- To view, visit http://gerrit.cloudera.org:8080/10853 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3dad846dafb25b414bee1858eb63f3eda31d59ac Gerrit-Change-Number: 10853 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] [DOCS] Clarification on admission control and DDL statements
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10829 ) Change subject: [DOCS] Clarification on admission control and DDL statements .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10829 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715 Gerrit-Change-Number: 10829 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 23:51:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5981: [DOCS] Documented SET=""
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10816 ) Change subject: IMPALA-5981: [DOCS] Documented SET="" .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10816 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7211405d5cc0a548c05ea5218798591873c14417 Gerrit-Change-Number: 10816 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 23:51:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6352: Dump backtrace on failure of TestTableSample
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10851 ) Change subject: IMPALA-6352: Dump backtrace on failure of TestTableSample .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73fcdd30863cee105584c947bb0c48cf872809c1 Gerrit-Change-Number: 10851 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 23:26:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6352: Dump backtrace on failure of TestTableSample
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10851 ) Change subject: IMPALA-6352: Dump backtrace on failure of TestTableSample .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2767/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73fcdd30863cee105584c947bb0c48cf872809c1 Gerrit-Change-Number: 10851 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 23:26:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/10704 ) Change subject: IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans .. Patch Set 12: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10704 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d6c26d42db090c8a15c602f6419ad6399c329e7 Gerrit-Change-Number: 10704 Gerrit-PatchSet: 12 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 23:25:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7190: Remove unsupported format writer support
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10823 ) Change subject: IMPALA-7190: Remove unsupported format writer support .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/10823/2/testdata/workloads/functional-query/queries/QueryTest/unsupported-writers.test File testdata/workloads/functional-query/queries/QueryTest/unsupported-writers.test: http://gerrit.cloudera.org:8080/#/c/10823/2/testdata/workloads/functional-query/queries/QueryTest/unsupported-writers.test@3 PS2, Line 3: drop table if exists __text_write; These drop tables shouldn't be needed for the unique database, should they? -- To view, visit http://gerrit.cloudera.org:8080/10823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I821dc7495a901f1658daa500daf3791b386c7185 Gerrit-Change-Number: 10823 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 23:21:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10704 ) Change subject: IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans .. Patch Set 12: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/10704 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d6c26d42db090c8a15c602f6419ad6399c329e7 Gerrit-Change-Number: 10704 Gerrit-PatchSet: 12 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 23:19:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6352: Dump backtrace on failure of TestTableSample
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10851 ) Change subject: IMPALA-6352: Dump backtrace on failure of TestTableSample .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73fcdd30863cee105584c947bb0c48cf872809c1 Gerrit-Change-Number: 10851 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 23:19:13 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Corrected the supported values for parquet array resolution
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10840 ) Change subject: [DOCS] Corrected the supported values for parquet_array_resolution .. [DOCS] Corrected the supported values for parquet_array_resolution Change-Id: Icfc1b7209d0f6b28c814be4149b0bacfebad2356 Reviewed-on: http://gerrit.cloudera.org:8080/10840 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins --- M docs/topics/impala_parquet_array_resolution.xml 1 file changed, 3 insertions(+), 2 deletions(-) Approvals: Alex Rodoni: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10840 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Icfc1b7209d0f6b28c814be4149b0bacfebad2356 Gerrit-Change-Number: 10840 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] [DOCS] Corrected the supported values for parquet array resolution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10840 ) Change subject: [DOCS] Corrected the supported values for parquet_array_resolution .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10840 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfc1b7209d0f6b28c814be4149b0bacfebad2356 Gerrit-Change-Number: 10840 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 23:06:51 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Corrected the supported values for parquet array resolution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10840 ) Change subject: [DOCS] Corrected the supported values for parquet_array_resolution .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/333/ -- To view, visit http://gerrit.cloudera.org:8080/10840 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfc1b7209d0f6b28c814be4149b0bacfebad2356 Gerrit-Change-Number: 10840 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 22:56:54 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Corrected the supported values for parquet array resolution
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10840 ) Change subject: [DOCS] Corrected the supported values for parquet_array_resolution .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10840 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfc1b7209d0f6b28c814be4149b0bacfebad2356 Gerrit-Change-Number: 10840 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 22:56:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans
Pooja Nilangekar has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/10704 ) Change subject: IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans .. IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans This change ensures that the planner computes parquet conjuncts only when for scans containing parquet files. Additionally, it also handles PARQUET_DICTIONARY_FILTERING and PARQUET_READ_STATISTICS query options in the planner. Testing was carried out independently on parquet and non-parquet scans: 1. Parquet scans were tested via the existing parquet-filtering planner test. Additionally, a new test [parquet-filtering-disabled] was added to ensure that the explain plan generated skips parquet predicates based on the query options. 2. Non-parquet scans were tested manually to ensure that the functions to compute parquet conjucts were not invoked. Additional test cases were added to the parquet-filtering planner test to scan non parquet tables and ensure that the plans do not contain conjuncts based on parquet statistics. 3. A parquet partition was added to the alltypesmixedformat table in the functional database. Planner tests were added to ensure that Parquet conjuncts are constructed only when the Parquet partition is included in the query. Change-Id: I9d6c26d42db090c8a15c602f6419ad6399c329e7 --- M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M testdata/bin/create-load-data.sh M testdata/bin/load-dependent-tables.sql M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test A testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering-disabled.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tablesample.test M testdata/workloads/functional-query/queries/QueryTest/mixed-format.test M testdata/workloads/functional-query/queries/QueryTest/show-stats.test M tests/query_test/test_rows_availability.py 14 files changed, 484 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/10704/12 -- To view, visit http://gerrit.cloudera.org:8080/10704 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9d6c26d42db090c8a15c602f6419ad6399c329e7 Gerrit-Change-Number: 10704 Gerrit-PatchSet: 12 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans
Pooja Nilangekar has posted comments on this change. ( http://gerrit.cloudera.org:8080/10704 ) Change subject: IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans .. Patch Set 12: (2 comments) http://gerrit.cloudera.org:8080/#/c/10704/11/tests/query_test/test_rows_availability.py File tests/query_test/test_rows_availability.py: http://gerrit.cloudera.org:8080/#/c/10704/11/tests/query_test/test_rows_availability.py@109 PS11, Line 109: matches = re.findall(r'([0-9]+h)?([0-9]+m)?([0-9]+s)?'\ > nit: long line Done http://gerrit.cloudera.org:8080/#/c/10704/11/tests/query_test/test_rows_availability.py@127 PS11, Line 127: milliseconds = int(float(matches[0][3][:-2])) : return hours * 60 * 60 * 1000 + minutes * 60 * 1000 + seconds * 1000 + milliseconds : : : > i think we can do away with not parsing this here since the method only ret Done -- To view, visit http://gerrit.cloudera.org:8080/10704 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d6c26d42db090c8a15c602f6419ad6399c329e7 Gerrit-Change-Number: 10704 Gerrit-PatchSet: 12 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 22:26:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6883: [DOCS] Refactor impala authorization doc
Hello Fredy Wijaya, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10786 to look at the new patch set (#4). Change subject: IMPALA-6883: [DOCS] Refactor impala_authorization doc .. IMPALA-6883: [DOCS] Refactor impala_authorization doc Change-Id: I3df72adb25dcdcbc286934b048645f47d876b33d --- M docs/shared/impala_common.xml M docs/topics/impala_authorization.xml M docs/topics/impala_grant.xml 3 files changed, 491 insertions(+), 741 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/10786/4 -- To view, visit http://gerrit.cloudera.org:8080/10786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3df72adb25dcdcbc286934b048645f47d876b33d Gerrit-Change-Number: 10786 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-6352: Dump backtrace on failure of TestTableSample
Hello Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10851 to look at the new patch set (#2). Change subject: IMPALA-6352: Dump backtrace on failure of TestTableSample .. IMPALA-6352: Dump backtrace on failure of TestTableSample TestTableSample is a flaky test which has been failing very rarely due to a possible hung thread. Therefore this patch adds a timeout to the test and logs the backtrace of all impalads if timeout occurs, so we can get more information on the state of those threads. Change-Id: I73fcdd30863cee105584c947bb0c48cf872809c1 --- M tests/beeswax/impala_beeswax.py M tests/common/impala_connection.py M tests/query_test/test_tablesample.py 3 files changed, 49 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/10851/2 -- To view, visit http://gerrit.cloudera.org:8080/10851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I73fcdd30863cee105584c947bb0c48cf872809c1 Gerrit-Change-Number: 10851 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6352: Dump backtrace on failure of TestTableSample
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/10851 ) Change subject: IMPALA-6352: Dump backtrace on failure of TestTableSample .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/10851/1/tests/beeswax/impala_beeswax.py File tests/beeswax/impala_beeswax.py: http://gerrit.cloudera.org:8080/#/c/10851/1/tests/beeswax/impala_beeswax.py@370 PS1, Line 370: wait_for_completion_timeout > Maybe this should be wait_for_finished, since the query isn't really "compl Done http://gerrit.cloudera.org:8080/#/c/10851/1/tests/common/impala_connection.py File tests/common/impala_connection.py: http://gerrit.cloudera.org:8080/#/c/10851/1/tests/common/impala_connection.py@182 PS1, Line 182: def wait_for_completion_timeout(self, operation_handle, timeout): > Same comment Done http://gerrit.cloudera.org:8080/#/c/10851/1/tests/query_test/test_tablesample.py File tests/query_test/test_tablesample.py: http://gerrit.cloudera.org:8080/#/c/10851/1/tests/query_test/test_tablesample.py@63 PS1, Line 63: timed_out > Isn't this the opposite of timed out? I.e. True means success, False means Done -- To view, visit http://gerrit.cloudera.org:8080/10851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73fcdd30863cee105584c947bb0c48cf872809c1 Gerrit-Change-Number: 10851 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 22:11:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7185: low statestore custom cluster interval
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10845 ) Change subject: IMPALA-7185: low statestore custom cluster interval .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia5d1612283ff420d95b0dd0ca5a2a67f56765f79 Gerrit-Change-Number: 10845 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 22:06:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7185: low statestore custom cluster interval
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10845 ) Change subject: IMPALA-7185: low statestore custom cluster interval .. IMPALA-7185: low statestore custom cluster interval This changes the default statestore interval for the custom cluster tests. This can reduce the time taken for the cluster to start and metadata to load. On some tests this resulted in saving 5+ seconds per test. Overall it shaved around a minute off the custom cluster tests. Testing: Ran 10 iterations of the tests. Change-Id: Ia5d1612283ff420d95b0dd0ca5a2a67f56765f79 Reviewed-on: http://gerrit.cloudera.org:8080/10845 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/authorization/test_grant_revoke.py M tests/common/custom_cluster_test_suite.py 2 files changed, 12 insertions(+), 7 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia5d1612283ff420d95b0dd0ca5a2a67f56765f79 Gerrit-Change-Number: 10845 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-4784: Remove InProcessStatestore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10843 ) Change subject: IMPALA-4784: Remove InProcessStatestore .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2766/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2621873e593b36c9612a6402ac6c5d8e3b49cde9 Gerrit-Change-Number: 10843 Gerrit-PatchSet: 5 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 21:48:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4784: Remove InProcessStatestore
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10843 ) Change subject: IMPALA-4784: Remove InProcessStatestore .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2621873e593b36c9612a6402ac6c5d8e3b49cde9 Gerrit-Change-Number: 10843 Gerrit-PatchSet: 5 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 21:48:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4784: Remove InProcessStatestore
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10843 ) Change subject: IMPALA-4784: Remove InProcessStatestore .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2621873e593b36c9612a6402ac6c5d8e3b49cde9 Gerrit-Change-Number: 10843 Gerrit-PatchSet: 4 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 21:44:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6352: Dump backtrace on failure of TestTableSample
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10851 ) Change subject: IMPALA-6352: Dump backtrace on failure of TestTableSample .. Patch Set 1: (3 comments) The approach seems good, just a few comments on naming http://gerrit.cloudera.org:8080/#/c/10851/1/tests/beeswax/impala_beeswax.py File tests/beeswax/impala_beeswax.py: http://gerrit.cloudera.org:8080/#/c/10851/1/tests/beeswax/impala_beeswax.py@370 PS1, Line 370: wait_for_completion_timeout Maybe this should be wait_for_finished, since the query isn't really "complete" in any meaningful sense. Same applies to the above method. http://gerrit.cloudera.org:8080/#/c/10851/1/tests/common/impala_connection.py File tests/common/impala_connection.py: http://gerrit.cloudera.org:8080/#/c/10851/1/tests/common/impala_connection.py@182 PS1, Line 182: def wait_for_completion_timeout(self, operation_handle, timeout): Same comment http://gerrit.cloudera.org:8080/#/c/10851/1/tests/query_test/test_tablesample.py File tests/query_test/test_tablesample.py: http://gerrit.cloudera.org:8080/#/c/10851/1/tests/query_test/test_tablesample.py@63 PS1, Line 63: timed_out Isn't this the opposite of timed out? I.e. True means success, False means it timed out. -- To view, visit http://gerrit.cloudera.org:8080/10851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73fcdd30863cee105584c947bb0c48cf872809c1 Gerrit-Change-Number: 10851 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 21:42:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6352: Dump backtrace on failure of TestTableSample
Bikramjeet Vig has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10851 Change subject: IMPALA-6352: Dump backtrace on failure of TestTableSample .. IMPALA-6352: Dump backtrace on failure of TestTableSample TestTableSample is a flaky test which has been failing very rarely due to a possible hung thread. Therefore this patch adds a timeout to the test and logs the backtrace of all impalads if timeout occurs, so we can get more information on the state of those threads. Change-Id: I73fcdd30863cee105584c947bb0c48cf872809c1 --- M tests/beeswax/impala_beeswax.py M tests/common/impala_connection.py M tests/query_test/test_tablesample.py 3 files changed, 45 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/10851/1 -- To view, visit http://gerrit.cloudera.org:8080/10851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I73fcdd30863cee105584c947bb0c48cf872809c1 Gerrit-Change-Number: 10851 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig
[Impala-ASF-CR] IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10704 ) Change subject: IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2765/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/10704 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d6c26d42db090c8a15c602f6419ad6399c329e7 Gerrit-Change-Number: 10704 Gerrit-PatchSet: 11 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 21:29:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans
Pooja Nilangekar has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/10704 ) Change subject: IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans .. IMPALA-6625: Skip computing parquet conjuncts for non-Parquet scans This change ensures that the planner computes parquet conjuncts only when for scans containing parquet files. Additionally, it also handles PARQUET_DICTIONARY_FILTERING and PARQUET_READ_STATISTICS query options in the planner. Testing was carried out independently on parquet and non-parquet scans: 1. Parquet scans were tested via the existing parquet-filtering planner test. Additionally, a new test [parquet-filtering-disabled] was added to ensure that the explain plan generated skips parquet predicates based on the query options. 2. Non-parquet scans were tested manually to ensure that the functions to compute parquet conjucts were not invoked. Additional test cases were added to the parquet-filtering planner test to scan non parquet tables and ensure that the plans do not contain conjuncts based on parquet statistics. 3. A parquet partition was added to the alltypesmixedformat table in the functional database. Planner tests were added to ensure that Parquet conjuncts are constructed only when the Parquet partition is included in the query. Change-Id: I9d6c26d42db090c8a15c602f6419ad6399c329e7 --- M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M testdata/bin/create-load-data.sh M testdata/bin/load-dependent-tables.sql M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test A testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering-disabled.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/tablesample.test M testdata/workloads/functional-query/queries/QueryTest/mixed-format.test M testdata/workloads/functional-query/queries/QueryTest/show-stats.test M tests/query_test/test_rows_availability.py 14 files changed, 488 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/10704/11 -- To view, visit http://gerrit.cloudera.org:8080/10704 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9d6c26d42db090c8a15c602f6419ad6399c329e7 Gerrit-Change-Number: 10704 Gerrit-PatchSet: 11 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-4784: Remove InProcessStatestore
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/10843 ) Change subject: IMPALA-4784: Remove InProcessStatestore .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/10843/3/be/src/exprs/expr-test.cc File be/src/exprs/expr-test.cc: http://gerrit.cloudera.org:8080/#/c/10843/3/be/src/exprs/expr-test.cc@8738 PS3, Line 8738: Statestore* statestore = new Statestore(metrics.get()); > Would it make sense to make this a global scoped_ptr or similar? Seems triv I made this global, but making it a scoped pointer has problems due to no clean shutdown semantics for the statestore. See IMPALA-7235. http://gerrit.cloudera.org:8080/#/c/10843/3/be/src/service/session-expiry-test.cc File be/src/service/session-expiry-test.cc: http://gerrit.cloudera.org:8080/#/c/10843/3/be/src/service/session-expiry-test.cc@53 PS3, Line 53: Statestore* statestore = new Statestore(metrics.get()); > Same question Same as other comments: IMPALA-7235 http://gerrit.cloudera.org:8080/#/c/10843/3/be/src/statestore/statestore-test.cc File be/src/statestore/statestore-test.cc: http://gerrit.cloudera.org:8080/#/c/10843/3/be/src/statestore/statestore-test.cc@40 PS3, Line 40: Statestore* statestore = new Statestore(metrics.get()); > Same question here and below. Could also use the IGNORE_LEAKING_OBJECT macr Unfortunately, the Statestore does not have clean shut down semantics since it's expected that it will run for the life of the cluster. Due to that having a scoped_ptr for the Statestore will cause crashes since some of the threads it spawns run functions that have no exit conditions. I think this is why we leak the InProcessStatestore object in pretty much every BE test we use it in. It's possible to have the Statestore shut down cleanly, but I'd rather do that as a separate patch, since it could have edge cases of its own. For now, I'll use IGNORE_LEAKING_OBJECT in the tests. -- To view, visit http://gerrit.cloudera.org:8080/10843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2621873e593b36c9612a6402ac6c5d8e3b49cde9 Gerrit-Change-Number: 10843 Gerrit-PatchSet: 3 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 21:21:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4784: Remove InProcessStatestore
Hello Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10843 to look at the new patch set (#4). Change subject: IMPALA-4784: Remove InProcessStatestore .. IMPALA-4784: Remove InProcessStatestore InProcessStatestore was only used by statestore-test, expr-test and session-expiry-test. With a slight refactor of the Statestore class, InProcessStatestore becomes obsolete. This patch moves the ownership of the ThriftServer into the Statestore class and Statestore::Init() now takes a 'port' parameter instead of using the FLAGS_state_store_port directly. We also remove the InProcessStatestore completely. A follow on patch will get rid of the InProcessImpalaServer too (IMPALA-6013) Testing: Ran 'core' tests. Change-Id: I2621873e593b36c9612a6402ac6c5d8e3b49cde9 --- M be/src/exprs/expr-test.cc M be/src/service/session-expiry-test.cc M be/src/statestore/statestore-test.cc M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M be/src/statestore/statestored-main.cc M be/src/testutil/in-process-servers.cc M be/src/testutil/in-process-servers.h 8 files changed, 95 insertions(+), 125 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/10843/4 -- To view, visit http://gerrit.cloudera.org:8080/10843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2621873e593b36c9612a6402ac6c5d8e3b49cde9 Gerrit-Change-Number: 10843 Gerrit-PatchSet: 4 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6086: Use of permanent function should require SELECT privilege on DB
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/10850 )
Change subject: IMPALA-6086: Use of permanent function should require SELECT
privilege on DB
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10850/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java:
http://gerrit.cloudera.org:8080/#/c/10850/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@2071
PS1, Line 2071: authorize("create function to_lower(string) returns string
location " +
As mentioned in the previous review, this code doesn't create a function in the
catalog.
You can create a helper function like below.
private ScalarFunction addFunction(String db, String fnName, ArrayList
argTypes, Type retType, String uriPath, String symbolName) {
ScalarFunction fn = ScalarFunction.createForTesting(db, fnName,
argTypes, retType, uriPath, symbolName, null, null,
TFunctionBinaryType.NATIVE);
authzCatalog_.addFunction(fn);
return fn;
}
You can then use the new addFunction to create a function in the catalog.
fn = addFunction("functional", "to_lower", );
try {
TQueryOptions options = new TQueryOptions();
options.setEnable_expr_rewrites(true);
for (AuthzTest test: new AuthzTest[]{
authorize("select to_lower('ABCDEF')"),
// SQL rewrite enabled.
authorize(createAnalysisContext(options), "select to_lower('ABCDEF')")}) {
test.ok(...)
.error(...);
}
} finally {
removeFunction(fn);
}
You can chain bunch of ok() and error() methods to test on different level of
hierarchies, i.e. server, database and table. See other test cases as a
reference.
--
To view, visit http://gerrit.cloudera.org:8080/10850
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee70f15e4c04f7daaed9cac2400ec626e1fb0e57
Gerrit-Change-Number: 10850
Gerrit-PatchSet: 1
Gerrit-Owner: Zoram Thanga
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Mon, 02 Jul 2018 21:16:24 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-110 (part 2): Refactor PartitionedAggregationNode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10394 ) Change subject: IMPALA-110 (part 2): Refactor PartitionedAggregationNode .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10394 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e7bb583f54aa4add3738bde7f57cf3511ac567e Gerrit-Change-Number: 10394 Gerrit-PatchSet: 12 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 02 Jul 2018 21:07:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-110 (part 2): Refactor PartitionedAggregationNode
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10394 ) Change subject: IMPALA-110 (part 2): Refactor PartitionedAggregationNode .. IMPALA-110 (part 2): Refactor PartitionedAggregationNode This patch refactors PartitionedAggregationNode in preparation for supporting multiple distinct operators in a query. The primary goal of the refactor is to separate out the core aggregation functionality into a new type of object called an Aggregator. For now, each aggregation ExecNode will contain a single Aggregator. Then, future patches will extend the aggregation ExecNode to support taking a single input and processing it with multiple Aggregators, allowing us to support more exotic combinations of aggregate functions and groupings. Specifically, this patch splits PartitionedAggregationNode into five new classes: - Aggregator: a superclass containing the functionality that's shared between GroupingAggregator and NonGroupingAggregator. - GroupingAggregator: this class contains the bulk of the interesting aggregation code, including everything related to creating and updating partitions and hash tables, spilling, etc. - NonGroupingAggregator: this class handles the case of aggregations that don't have grouping exprs. Since these aggregations always result in just a single output row, the functionality here is relatively simple (eg. no spilling or streaming). - StreamingAggregationNode: this node performs a streaming preaggregation, where the input is retrieved from the child during GetNext() and passed to the GroupingAggregator (non-grouping do not support streaming) Eventually, we'll support a list of GroupingAggregators. - AggregationNode: this node performs a final aggregation, where the input is retrieved from the child during Open() and passed to the Aggregator. Currently the Aggregator can be either grouping or non-grouping. Eventually we'll support a list of GroupingAggregator and/or a single NonGroupingAggregator. Testing: - Passed a full exhaustive run. Change-Id: I9e7bb583f54aa4add3738bde7f57cf3511ac567e Reviewed-on: http://gerrit.cloudera.org:8080/10394 Reviewed-by: Thomas Marshall Tested-by: Impala Public Jenkins --- M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/impala-ir.cc M be/src/exec/CMakeLists.txt A be/src/exec/aggregation-node.cc A be/src/exec/aggregation-node.h A be/src/exec/aggregator.cc A be/src/exec/aggregator.h M be/src/exec/exec-node.cc M be/src/exec/exec-node.h R be/src/exec/grouping-aggregator-ir.cc A be/src/exec/grouping-aggregator-partition.cc A be/src/exec/grouping-aggregator.cc R be/src/exec/grouping-aggregator.h A be/src/exec/non-grouping-aggregator-ir.cc A be/src/exec/non-grouping-aggregator.cc A be/src/exec/non-grouping-aggregator.h D be/src/exec/partitioned-aggregation-node.cc A be/src/exec/streaming-aggregation-node.cc A be/src/exec/streaming-aggregation-node.h 19 files changed, 3,054 insertions(+), 2,239 deletions(-) Approvals: Thomas Marshall: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10394 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9e7bb583f54aa4add3738bde7f57cf3511ac567e Gerrit-Change-Number: 10394 Gerrit-PatchSet: 13 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-6086: Use of permanent function should require SELECT privilege on DB
Zoram Thanga has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10850 Change subject: IMPALA-6086: Use of permanent function should require SELECT privilege on DB .. IMPALA-6086: Use of permanent function should require SELECT privilege on DB To use a permanent UDF should require at leat SELECT privilege on the database. Functions that have constant arguments get constant-folded into string literals, losing their privilege requests in the process. This patch saves the privilege requests found during the first phase of query analysis, where all the objects and the privileges required to access them are identified. The requests are added back to the new analyzer created for re-analysis post expression rewrite. New FE test cases have been added to AuthorizationStmtTest. Manual tests were also done to identify the bug, as well as to test the fix. Ran private parameterized Jenkins job, exhaustive exploration and covering all tests. Change-Id: Iee70f15e4c04f7daaed9cac2400ec626e1fb0e57 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java 2 files changed, 30 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/10850/1 -- To view, visit http://gerrit.cloudera.org:8080/10850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iee70f15e4c04f7daaed9cac2400ec626e1fb0e57 Gerrit-Change-Number: 10850 Gerrit-PatchSet: 1 Gerrit-Owner: Zoram Thanga
[Impala-ASF-CR] IMPALA-6086: Use of permanent function should require SELECT privilege on DB
Zoram Thanga has abandoned this change. ( http://gerrit.cloudera.org:8080/10842 ) Change subject: IMPALA-6086: Use of permanent function should require SELECT privilege on DB .. Abandoned Raising a new review. -- To view, visit http://gerrit.cloudera.org:8080/10842 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I45f5b51363afbe46653b131fad9bf68e3e3ce71f Gerrit-Change-Number: 10842 Gerrit-PatchSet: 1 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-6086: Use of permanent function should require SELECT privilege on DB
Zoram Thanga has posted comments on this change. (
http://gerrit.cloudera.org:8080/10842 )
Change subject: IMPALA-6086: Use of permanent function should require SELECT
privilege on DB
..
Patch Set 1:
(3 comments)
Abandoning this patch, and raising a new one instead.
http://gerrit.cloudera.org:8080/#/c/10842/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java:
http://gerrit.cloudera.org:8080/#/c/10842/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@2576
PS1, Line 2576: AuthzOk(ctx, "create function default.to_lower(string)
returns string " +
> This doesn't actually create a function in the catalog. You need to do the
Thanks for the suggestion. I did not intent for the function to be reused
across test cases, and hence adding to the catalog is probably not required.
I've eliminated this step altogether in the new patch.
http://gerrit.cloudera.org:8080/#/c/10842/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@2578
PS1, Line 2578: ToUpper
> nit: it's probably better to call the function to_upper since the actual fu
Oops. Thanks for catching that. I meant to use ToLower and c/p'd the wrong line.
http://gerrit.cloudera.org:8080/#/c/10842/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@2596
PS1, Line 2596: AuthzError(ctx1, "select default.to_lower('ABCDEF')",
> Due to the way authorization works to prefer AuthorizationException over An
Please see above response. I've botched the rebase on top of
https://gerrit.cloudera.org/#/c/10841/, so I am going to abandon this change
and raise a new one instead.
--
To view, visit http://gerrit.cloudera.org:8080/10842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I45f5b51363afbe46653b131fad9bf68e3e3ce71f
Gerrit-Change-Number: 10842
Gerrit-PatchSet: 1
Gerrit-Owner: Zoram Thanga
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Zoram Thanga
Gerrit-Comment-Date: Mon, 02 Jul 2018 20:50:37 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7190: Remove unsupported format writer support
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/10823 ) Change subject: IMPALA-7190: Remove unsupported format writer support .. Patch Set 2: Code-Review+1 Carrying over +1. Minor changes, but can someone look at it again and +2 it. Thanks -- To view, visit http://gerrit.cloudera.org:8080/10823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I821dc7495a901f1658daa500daf3791b386c7185 Gerrit-Change-Number: 10823 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 20:13:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7190: Remove unsupported format writer support
Hello Lars Volker, Tim Armstrong, Csaba Ringhofer, Dan Hecht, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10823 to look at the new patch set (#2). Change subject: IMPALA-7190: Remove unsupported format writer support .. IMPALA-7190: Remove unsupported format writer support This patch removes write support for unsupported formats like Sequence, Avro and compressed text. Also, the related query options ALLOW_UNSUPPORTED_FORMATS and SEQ_COMPRESSION_MODE have been migrated to the REMOVED query options type. Testing: Ran exhaustive build. Change-Id: I821dc7495a901f1658daa500daf3791b386c7185 --- M be/src/exec/CMakeLists.txt D be/src/exec/hdfs-avro-table-writer.cc D be/src/exec/hdfs-avro-table-writer.h D be/src/exec/hdfs-sequence-table-writer.cc D be/src/exec/hdfs-sequence-table-writer.h M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-text-table-writer.cc M be/src/exec/hdfs-text-table-writer.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java M testdata/bad_avro_snap/README D testdata/workloads/functional-query/queries/QueryTest/avro-writer.test D testdata/workloads/functional-query/queries/QueryTest/seq-writer.test M testdata/workloads/functional-query/queries/QueryTest/set.test D testdata/workloads/functional-query/queries/QueryTest/text-writer.test A testdata/workloads/functional-query/queries/QueryTest/unsupported-writers.test M tests/common/test_dimensions.py M tests/hs2/test_hs2.py M tests/metadata/test_partition_metadata.py M tests/query_test/test_compressed_formats.py M tests/shell/test_shell_interactive.py 25 files changed, 130 insertions(+), 1,607 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/10823/2 -- To view, visit http://gerrit.cloudera.org:8080/10823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I821dc7495a901f1658daa500daf3791b386c7185 Gerrit-Change-Number: 10823 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7190: Remove unsupported format writer support
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/10823 ) Change subject: IMPALA-7190: Remove unsupported format writer support .. Patch Set 1: (4 comments) http://gerrit.cloudera.org:8080/#/c/10823/1/be/src/exec/hdfs-table-sink.cc File be/src/exec/hdfs-table-sink.cc: http://gerrit.cloudera.org:8080/#/c/10823/1/be/src/exec/hdfs-table-sink.cc@484 PS1, Line 484: } > presumably we do this error checking at runtime rather than analysis so tha Done. Added a test case to unsupported-writers.test http://gerrit.cloudera.org:8080/#/c/10823/1/tests/common/test_dimensions.py File tests/common/test_dimensions.py: http://gerrit.cloudera.org:8080/#/c/10823/1/tests/common/test_dimensions.py@111 PS1, Line 111: # Available Exec Options: > Let's just delete this whole comment, it's insanely stale. It's a terrible Done http://gerrit.cloudera.org:8080/#/c/10823/1/tests/query_test/test_compressed_formats.py File tests/query_test/test_compressed_formats.py: http://gerrit.cloudera.org:8080/#/c/10823/1/tests/query_test/test_compressed_formats.py@150 PS1, Line 150: def test_unsupported_writers(self, vector): > I would prefer to use unique_database, and remove the drops from unsupporte Done http://gerrit.cloudera.org:8080/#/c/10823/1/tests/shell/test_shell_interactive.py File tests/shell/test_shell_interactive.py: http://gerrit.cloudera.org:8080/#/c/10823/1/tests/shell/test_shell_interactive.py@a404 PS1, Line 404: > This line could be left here, but check that "... not in result.stduot". Th Done -- To view, visit http://gerrit.cloudera.org:8080/10823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I821dc7495a901f1658daa500daf3791b386c7185 Gerrit-Change-Number: 10823 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 20:11:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4784: Remove InProcessStatestore
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10843 ) Change subject: IMPALA-4784: Remove InProcessStatestore .. Patch Set 3: (3 comments) Seems good, thanks for the cleanup! http://gerrit.cloudera.org:8080/#/c/10843/3/be/src/exprs/expr-test.cc File be/src/exprs/expr-test.cc: http://gerrit.cloudera.org:8080/#/c/10843/3/be/src/exprs/expr-test.cc@8738 PS3, Line 8738: Statestore* statestore = new Statestore(metrics.get()); Would it make sense to make this a global scoped_ptr or similar? Seems trivial to avoid the memory leak now. http://gerrit.cloudera.org:8080/#/c/10843/3/be/src/service/session-expiry-test.cc File be/src/service/session-expiry-test.cc: http://gerrit.cloudera.org:8080/#/c/10843/3/be/src/service/session-expiry-test.cc@53 PS3, Line 53: Statestore* statestore = new Statestore(metrics.get()); Same question http://gerrit.cloudera.org:8080/#/c/10843/3/be/src/statestore/statestore-test.cc File be/src/statestore/statestore-test.cc: http://gerrit.cloudera.org:8080/#/c/10843/3/be/src/statestore/statestore-test.cc@40 PS3, Line 40: Statestore* statestore = new Statestore(metrics.get()); Same question here and below. Could also use the IGNORE_LEAKING_OBJECT macro. I figure it's easier to fix this now than have someone else with less context try to fix it later when enabling leak santizier. -- To view, visit http://gerrit.cloudera.org:8080/10843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2621873e593b36c9612a6402ac6c5d8e3b49cde9 Gerrit-Change-Number: 10843 Gerrit-PatchSet: 3 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 19:05:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1760: Implement shutdown command
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/10744 )
Change subject: IMPALA-1760: Implement shutdown command
..
Patch Set 7:
(6 comments)
Fixed a few minor things in the test code that I noticed when doing another
pass over it.
http://gerrit.cloudera.org:8080/#/c/10744/7/common/thrift/Frontend.thrift
File common/thrift/Frontend.thrift:
http://gerrit.cloudera.org:8080/#/c/10744/7/common/thrift/Frontend.thrift@610
PS7, Line 610: 13: optional TAdminRequest admin_request
> nit: 13 should be after 12: (L618)
I was thinking that it made sense to group the requests together rather than
listing things chronologically here, but I missed that there wasn't any logical
grouping already.
http://gerrit.cloudera.org:8080/#/c/10744/6/common/thrift/Types.thrift
File common/thrift/Types.thrift:
http://gerrit.cloudera.org:8080/#/c/10744/6/common/thrift/Types.thrift@104
PS6, Line 104: ADMIN_FN // Admin comment, e.g. ": shutdown()".
Typo
http://gerrit.cloudera.org:8080/#/c/10744/7/common/thrift/generate_error_codes.py
File common/thrift/generate_error_codes.py:
http://gerrit.cloudera.org:8080/#/c/10744/7/common/thrift/generate_error_codes.py@367
PS7, Line 367:
> nit: empty line
Done
http://gerrit.cloudera.org:8080/#/c/10744/7/fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java
File fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java:
http://gerrit.cloudera.org:8080/#/c/10744/7/fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java@104
PS7, Line 104:
> nit: indentation
Done
http://gerrit.cloudera.org:8080/#/c/10744/7/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java:
http://gerrit.cloudera.org:8080/#/c/10744/7/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@2075
PS7, Line 2075: .error(accessError("server"))
: .error(accessError("server"),
onServer(TPrivilegeLevel.REFRESH))
: .error(accessError("server"),
onServer(TPrivilegeLevel.SELECT))
: .error(accessError("server"), onDatabase("functional",
TPrivilegeLevel.ALL))
: .error(accessError("server"),
: onTable("functional", "alltypes",
TPrivilegeLevel.ALL));
> These errors can be simplified to .error(accessError("server"), allExcept(T
That's much nicer!
http://gerrit.cloudera.org:8080/#/c/10744/7/www/root.tmpl
File www/root.tmpl:
http://gerrit.cloudera.org:8080/#/c/10744/7/www/root.tmpl@30
PS7, Line 30:
> nit: whitespace
Done
--
To view, visit http://gerrit.cloudera.org:8080/10744
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d5606ccfec84db4482c1e7f0f198103aad141a0
Gerrit-Change-Number: 10744
Gerrit-PatchSet: 7
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Dan Hecht
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 02 Jul 2018 19:04:41 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1760: Implement shutdown command
Hello Fredy Wijaya, Dan Hecht,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10744
to look at the new patch set (#8).
Change subject: IMPALA-1760: Implement shutdown command
..
IMPALA-1760: Implement shutdown command
This allows graceful shutdown of executors and partially graceful
shutdown of coordinators (new operations fail, old operations can
continue).
Details:
* In order to allow future admin commands, this is implemented with
function-like syntax and does not add any reserved words.
* ALL privilege is required on the server
* The coordinator impalad that the client is connected to can be shut
down directly with ":shutdown()".
* Remote shutdown of another impalad is supported, e.g. with
":shutdown('hostname')", so that non-coordinators can be shut down
and for the convenience of the client, which does not have to
connect to the specific impalad. There is no assumption that the
other impalad is registered in the statestore; just that the
coordinator can connect to the other daemon's thrift endpoint.
This simplifies things and allows shutdown in various important
cases, e.g. statestore down.
* The shutdown time limit can be overridden to force a quicker or
slower shutdown.
* If shutting down, a banner is shown on the root debug page.
Workflow:
1. (if a coordinator) clients are prevented from submitting
queries to this coordinator via some out-of-band mechanism,
e.g. load balancer
2. the shutdown process is started via ":shutdown()"
3. a bit is set in the statestore and propagated to coordinators,
which stop scheduling fragment instances on this daemon
(if an executor).
4. the quiesce period (which is the AC queueing delay plus
some additional leeway) expires
5. once the daemon is drained (i.e. no fragments, no registered
queries), it shuts itself down.
6. If the daemon does not drain (e.g. rogue clients, long-running
queries), after a longer timeout it will shut down anyway.
What this does do:
* Executors can be shut down without causing a service-wide outage
* Shutting down an executor will not disrupt any short-running queries
and will wait for long-running queries up to a threshold.
* Coordinators can be shut down without query failures only if
there is an out-of-band mechanism to prevent submission of more
queries to the shut down coordinator.
* Long running queries or other issues (e.g. stuck fragments) will
slow down but not prevent eventual shutdown.
Limitations:
* The quiesce period needs to be configured to be greater than the
latency of statestore updates + scheduling + admission +
coordinator startup. Otherwise a coordinator may send a
fragment instance to the shutting down impalad. (We could
automate this configuration as a follow-on)
* The quiesce period means a minimum latency for shutdown,
even if the cluster is idle.
* We depend on the statestore detecting the process going down
if queries are still running on that backend when the timeout
expires. This may still be subject to existing problems,
e.g. IMPALA-2990.
Tests:
* Added parser, analysis and authorization tests.
* End-to-end test of shutting down impalads.
* End-to-end test of shutting down then restarting an executor while
queries are running.
* End-to-end test of shutting down a coordinator
- New queries cannot be started on coord, existing queries continue to run
- Exercises various Beeswax and HS2 operations.
Change-Id: I4d5606ccfec84db4482c1e7f0f198103aad141a0
---
M be/src/runtime/backend-client.h
M be/src/runtime/data-stream-test.cc
M be/src/scheduling/scheduler.cc
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-internal-service.cc
M be/src/service/impala-internal-service.h
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/testutil/fault-injection-util.h
M be/src/util/default-path-handlers.cc
M common/thrift/Frontend.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/StatestoreService.thrift
M common/thrift/Types.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/cup/sql-parser.cup
A fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/LimitElement.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M tests/common/impala_cluster.py
M tests/common/impala_service.py
M tests/custom_cluster/test_restart_services.py
M
[Impala-ASF-CR] IMPALA-7185: low statestore custom cluster interval
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10845 ) Change subject: IMPALA-7185: low statestore custom cluster interval .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2764/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia5d1612283ff420d95b0dd0ca5a2a67f56765f79 Gerrit-Change-Number: 10845 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 18:39:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7185: low statestore custom cluster interval
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10845 ) Change subject: IMPALA-7185: low statestore custom cluster interval .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia5d1612283ff420d95b0dd0ca5a2a67f56765f79 Gerrit-Change-Number: 10845 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 18:39:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1760: Implement shutdown command
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/10744 )
Change subject: IMPALA-1760: Implement shutdown command
..
Patch Set 7:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/10744/7/common/thrift/Frontend.thrift
File common/thrift/Frontend.thrift:
http://gerrit.cloudera.org:8080/#/c/10744/7/common/thrift/Frontend.thrift@610
PS7, Line 610: 13: optional TAdminRequest admin_request
nit: 13 should be after 12: (L618)
http://gerrit.cloudera.org:8080/#/c/10744/7/common/thrift/generate_error_codes.py
File common/thrift/generate_error_codes.py:
http://gerrit.cloudera.org:8080/#/c/10744/7/common/thrift/generate_error_codes.py@367
PS7, Line 367:
nit: empty line
http://gerrit.cloudera.org:8080/#/c/10744/7/fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java
File fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java:
http://gerrit.cloudera.org:8080/#/c/10744/7/fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java@104
PS7, Line 104:
nit: indentation
http://gerrit.cloudera.org:8080/#/c/10744/7/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java:
http://gerrit.cloudera.org:8080/#/c/10744/7/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@2075
PS7, Line 2075: .error(accessError("server"))
: .error(accessError("server"),
onServer(TPrivilegeLevel.REFRESH))
: .error(accessError("server"),
onServer(TPrivilegeLevel.SELECT))
: .error(accessError("server"), onDatabase("functional",
TPrivilegeLevel.ALL))
: .error(accessError("server"),
: onTable("functional", "alltypes",
TPrivilegeLevel.ALL));
These errors can be simplified to .error(accessError("server"),
allExcept(TPrivilegeLevel.ALL));
http://gerrit.cloudera.org:8080/#/c/10744/7/www/root.tmpl
File www/root.tmpl:
http://gerrit.cloudera.org:8080/#/c/10744/7/www/root.tmpl@30
PS7, Line 30:
nit: whitespace
--
To view, visit http://gerrit.cloudera.org:8080/10744
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d5606ccfec84db4482c1e7f0f198103aad141a0
Gerrit-Change-Number: 10744
Gerrit-PatchSet: 7
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Dan Hecht
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Comment-Date: Mon, 02 Jul 2018 18:26:46 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7185: low statestore custom cluster interval
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/10845 ) Change subject: IMPALA-7185: low statestore custom cluster interval .. Patch Set 3: Code-Review+2 This looks good to me. -- To view, visit http://gerrit.cloudera.org:8080/10845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia5d1612283ff420d95b0dd0ca5a2a67f56765f79 Gerrit-Change-Number: 10845 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Jul 2018 18:03:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-110 (part 2): Refactor PartitionedAggregationNode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10394 ) Change subject: IMPALA-110 (part 2): Refactor PartitionedAggregationNode .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2763/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10394 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e7bb583f54aa4add3738bde7f57cf3511ac567e Gerrit-Change-Number: 10394 Gerrit-PatchSet: 12 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 02 Jul 2018 17:37:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-110 (part 2): Refactor PartitionedAggregationNode
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/10394 ) Change subject: IMPALA-110 (part 2): Refactor PartitionedAggregationNode .. Patch Set 12: Code-Review+2 carrying forward -- To view, visit http://gerrit.cloudera.org:8080/10394 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e7bb583f54aa4add3738bde7f57cf3511ac567e Gerrit-Change-Number: 10394 Gerrit-PatchSet: 12 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 02 Jul 2018 17:37:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-110 (part 2): Refactor PartitionedAggregationNode
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/10394 ) Change subject: IMPALA-110 (part 2): Refactor PartitionedAggregationNode .. Patch Set 11: Code-Review+2 carrying forward -- To view, visit http://gerrit.cloudera.org:8080/10394 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e7bb583f54aa4add3738bde7f57cf3511ac567e Gerrit-Change-Number: 10394 Gerrit-PatchSet: 11 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 02 Jul 2018 17:36:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6883: [DOCS] Refactor impala authorization doc
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10786 ) Change subject: IMPALA-6883: [DOCS] Refactor impala_authorization doc .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/10786/3/docs/topics/impala_authorization.xml File docs/topics/impala_authorization.xml: http://gerrit.cloudera.org:8080/#/c/10786/3/docs/topics/impala_authorization.xml@178 PS3, Line 178: --sentry_config: Specifies the local path to the > nit: use -sentry_config for consistency with other flags Done -- To view, visit http://gerrit.cloudera.org:8080/10786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3df72adb25dcdcbc286934b048645f47d876b33d Gerrit-Change-Number: 10786 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 02 Jul 2018 17:24:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Pranay Singh has posted comments on this change. ( http://gerrit.cloudera.org:8080/10847 ) Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. Patch Set 1: > How does your change interact with the Breakpad integration? > SIGSEGV and SIGABRT are also handled by the Breakpad signal handler > (https://chromium.googlesource.com/breakpad/breakpad/+/master/src/client/linux/handler/exception_handler.cc#115). > > It should have a test, possibly in the custom-cluster-tests folder, > to validate that Impala does the expected for each signal it could > receive. > > An alternative approach could be to print the received signal into > the logs when writing a minidump. I'm resorting to default handlers after the message has been dumped but anyways I'll test it with breakpad and see if it affects the core/minidump creation -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 1 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Comment-Date: Mon, 02 Jul 2018 16:54:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/10847 ) Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. Patch Set 1: How does your change interact with the Breakpad integration? SIGSEGV and SIGABRT are also handled by the Breakpad signal handler (https://chromium.googlesource.com/breakpad/breakpad/+/master/src/client/linux/handler/exception_handler.cc#115). It should have a test, possibly in the custom-cluster-tests folder, to validate that Impala does the expected for each signal it could receive. An alternative approach could be to print the received signal into the logs when writing a minidump. -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 1 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 02 Jul 2018 16:48:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7231: group plan nodes into pipelines
Tim Armstrong has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/10848 ) Change subject: IMPALA-7231: group plan nodes into pipelines .. IMPALA-7231: group plan nodes into pipelines This adds some informational output to explain plans and sends the information to the backend. The idea is that this will make it easier to explain how Impala's pipelined execution works and also enable future work on profile analysis that can more intelligently group plan nodes. Tests: * Updated planner tests to include new output. Change-Id: I1d10eb14d997242f445e5c5fc5362d5410370721 --- M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/JoinNode.java A fe/src/main/java/org/apache/impala/planner/PipelineMembership.java M fe/src/main/java/org/apache/impala/planner/PlanFragment.java M fe/src/main/java/org/apache/impala/planner/PlanNode.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/SubplanNode.java M fe/src/main/java/org/apache/impala/planner/UnionNode.java M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test M testdata/workloads/functional-planner/queries/PlannerTest/kudu-selectivity.test M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test M testdata/workloads/functional-planner/queries/PlannerTest/min-max-runtime-filters.test M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test M testdata/workloads/functional-planner/queries/PlannerTest/partition-pruning.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test M testdata/workloads/functional-planner/queries/PlannerTest/tablesample.test M testdata/workloads/functional-planner/queries/PlannerTest/union.test M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test 24 files changed, 1,114 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/10848/2 -- To view, visit http://gerrit.cloudera.org:8080/10848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1d10eb14d997242f445e5c5fc5362d5410370721 Gerrit-Change-Number: 10848 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-1760: Implement shutdown command
Hello Dan Hecht,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10744
to look at the new patch set (#7).
Change subject: IMPALA-1760: Implement shutdown command
..
IMPALA-1760: Implement shutdown command
This allows graceful shutdown of executors and partially graceful
shutdown of coordinators (new operations fail, old operations can
continue).
Details:
* In order to allow future admin commands, this is implemented with
function-like syntax and does not add any reserved words.
* ALL privilege is required on the server
* The coordinator impalad that the client is connected to can be shut
down directly with ":shutdown()".
* Remote shutdown of another impalad is supported, e.g. with
":shutdown('hostname')", so that non-coordinators can be shut down
and for the convenience of the client, which does not have to
connect to the specific impalad. There is no assumption that the
other impalad is registered in the statestore; just that the
coordinator can connect to the other daemon's thrift endpoint.
This simplifies things and allows shutdown in various important
cases, e.g. statestore down.
* The shutdown time limit can be overridden to force a quicker or
slower shutdown.
* If shutting down, a banner is shown on the root debug page.
Workflow:
1. (if a coordinator) clients are prevented from submitting
queries to this coordinator via some out-of-band mechanism,
e.g. load balancer
2. the shutdown process is started via ":shutdown()"
3. a bit is set in the statestore and propagated to coordinators,
which stop scheduling fragment instances on this daemon
(if an executor).
4. the quiesce period (which is the AC queueing delay plus
some additional leeway) expires
5. once the daemon is drained (i.e. no fragments, no registered
queries), it shuts itself down.
6. If the daemon does not drain (e.g. rogue clients, long-running
queries), after a longer timeout it will shut down anyway.
What this does do:
* Executors can be shut down without causing a service-wide outage
* Shutting down an executor will not disrupt any short-running queries
and will wait for long-running queries up to a threshold.
* Coordinators can be shut down without query failures only if
there is an out-of-band mechanism to prevent submission of more
queries to the shut down coordinator.
* Long running queries or other issues (e.g. stuck fragments) will
slow down but not prevent eventual shutdown.
Limitations:
* The quiesce period needs to be configured to be greater than the
latency of statestore updates + scheduling + admission +
coordinator startup. Otherwise a coordinator may send a
fragment instance to the shutting down impalad. (We could
automate this configuration as a follow-on)
* The quiesce period means a minimum latency for shutdown,
even if the cluster is idle.
* We depend on the statestore detecting the process going down
if queries are still running on that backend when the timeout
expires. This may still be subject to existing problems,
e.g. IMPALA-2990.
Tests:
* Added parser, analysis and authorization tests.
* End-to-end test of shutting down impalads.
* End-to-end test of shutting down then restarting an executor while
queries are running.
* End-to-end test of shutting down a coordinator
- New queries cannot be started on coord, existing queries continue to run
- Exercises various Beeswax and HS2 operations.
Change-Id: I4d5606ccfec84db4482c1e7f0f198103aad141a0
---
M be/src/runtime/backend-client.h
M be/src/runtime/data-stream-test.cc
M be/src/scheduling/scheduler.cc
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-internal-service.cc
M be/src/service/impala-internal-service.h
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/testutil/fault-injection-util.h
M be/src/util/default-path-handlers.cc
M common/thrift/Frontend.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/StatestoreService.thrift
M common/thrift/Types.thrift
M common/thrift/generate_error_codes.py
M fe/src/main/cup/sql-parser.cup
A fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/LimitElement.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M tests/common/impala_cluster.py
M tests/common/impala_service.py
M tests/custom_cluster/test_restart_services.py
M tests/hs2/hs2_test_suite.py
M
[Impala-ASF-CR] IMPALA-6918: Implement COMMENT ON COLUMN
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10754 ) Change subject: IMPALA-6918: Implement COMMENT ON COLUMN .. Patch Set 6: Code-Review+1 Updated the commit message. Carry +1. -- To view, visit http://gerrit.cloudera.org:8080/10754 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8976705b27e98c1540941d0a6cba0e18bad48437 Gerrit-Change-Number: 10754 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 02 Jul 2018 15:37:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6918: Implement COMMENT ON COLUMN
Fredy Wijaya has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/10754 ) Change subject: IMPALA-6918: Implement COMMENT ON COLUMN .. IMPALA-6918: Implement COMMENT ON COLUMN This patch implements updating comment on column table or view. Syntax: COMMENT ON COLUMN db.table_or_view.col IS 'comment' Testing: - Added new front-end tests - Ran all front-end tests - Added new end-to-end tests - Ran end-to-end DDL tests Change-Id: I8976705b27e98c1540941d0a6cba0e18bad48437 --- M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup A fe/src/main/java/org/apache/impala/analysis/ColumnName.java A fe/src/main/java/org/apache/impala/analysis/CommentOnColumnStmt.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M tests/metadata/test_ddl.py M tests/metadata/test_ddl_base.py 11 files changed, 323 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/10754/6 -- To view, visit http://gerrit.cloudera.org:8080/10754 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8976705b27e98c1540941d0a6cba0e18bad48437 Gerrit-Change-Number: 10754 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-6918: Implement COMMENT ON COLUMN
Adam Holley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10754 ) Change subject: IMPALA-6918: Implement COMMENT ON COLUMN .. Patch Set 5: Code-Review+1 Only nit is in the commit message, the database name is not included in the syntax. -- To view, visit http://gerrit.cloudera.org:8080/10754 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8976705b27e98c1540941d0a6cba0e18bad48437 Gerrit-Change-Number: 10754 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 02 Jul 2018 14:33:13 + Gerrit-HasComments: No
