[Impala-ASF-CR] IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14592 ) Change subject: IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala. .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id03d8d4d41a2ac1b15e7060e2a013e334d044ee7 Gerrit-Change-Number: 14592 Gerrit-PatchSet: 11 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 09 Nov 2019 06:36:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14592 ) Change subject: IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala. .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5200/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id03d8d4d41a2ac1b15e7060e2a013e334d044ee7 Gerrit-Change-Number: 14592 Gerrit-PatchSet: 11 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 09 Nov 2019 02:10:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14592 ) Change subject: IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala. .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4995/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id03d8d4d41a2ac1b15e7060e2a013e334d044ee7 Gerrit-Change-Number: 14592 Gerrit-PatchSet: 11 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 09 Nov 2019 01:54:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7984: Port runtime filter from Thrift RPC to KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13882 ) Change subject: IMPALA-7984: Port runtime filter from Thrift RPC to KRPC .. Patch Set 31: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13882 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b394796d250286510e157ae326882bfc01d387a Gerrit-Change-Number: 13882 Gerrit-PatchSet: 31 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Sat, 09 Nov 2019 01:54:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7984: Port runtime filter from Thrift RPC to KRPC
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13882 ) Change subject: IMPALA-7984: Port runtime filter from Thrift RPC to KRPC .. IMPALA-7984: Port runtime filter from Thrift RPC to KRPC Previously the aggregation and propagation of a runtime filter in Impala is implemented using Thrift RPC, which suffers from a disadvantage that the number of connections in a cluster grows with both the number of queries and cluster size. This patch ports the functions that implement the aggregation and propagation of a runtime filter, i.e., UpdateFilter() and PublishFilter(), respctively, to KRPC, which requires only one connection per direction between every pair of hosts, thus reducing the number of connections in a cluster. In addition, this patch also incorporates KRPC sidecar when the runtime filter is a Bloom filter. KRPC sidecar eliminates the need for an extra copy of the Bloom filter contents when a Bloom filter is serialized to be transmitted and hence reduces the serialization overhead. Due to the incorporation of KRPC sidecar, a SpinLock is also added to prevent a BloomFilter from being deallocated before its associated KRPC call finishes. Two related BE tests bloom-filter-test.cc and bloom-filter-benchmark.cc are also modified accordingly because of the changes to the signatures of some functions in BloomFilter. Testing: This patch has passed the exhaustive tests. Change-Id: I6b394796d250286510e157ae326882bfc01d387a Reviewed-on: http://gerrit.cloudera.org:8080/13882 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/benchmarks/bloom-filter-benchmark.cc M be/src/runtime/backend-client.h M be/src/runtime/client-cache.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator-filter-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/data-stream-test.cc M be/src/runtime/decimal-value.h M be/src/runtime/decimal-value.inline.h M be/src/runtime/exec-env.cc M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/runtime-filter-bank.h M be/src/runtime/runtime-filter.h M be/src/runtime/timestamp-value.h M be/src/scheduling/request-pool-service.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/service/frontend.h M be/src/service/impala-internal-service.cc M be/src/service/impala-internal-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/bloom-filter-test.cc M be/src/util/bloom-filter.cc M be/src/util/bloom-filter.h M be/src/util/min-max-filter-test.cc M be/src/util/min-max-filter.cc M be/src/util/min-max-filter.h M common/protobuf/common.proto M common/protobuf/data_stream_service.proto M common/thrift/ImpalaInternalService.thrift 39 files changed, 1,094 insertions(+), 757 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13882 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6b394796d250286510e157ae326882bfc01d387a Gerrit-Change-Number: 13882 Gerrit-PatchSet: 32 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-9137, IMPALA-9138: Mark failed RPC as retryable, add dst node to blacklist
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14677 ) Change subject: IMPALA-9137, IMPALA-9138: Mark failed RPC as retryable, add dst node to blacklist .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4994/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I733cca13847fde43c8ea2ae574d3ae04bd06419c Gerrit-Change-Number: 14677 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 09 Nov 2019 01:47:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14592 ) Change subject: IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala. .. Patch Set 11: Gerrit UI said there was a merge conflict. I rebased it an uploaded a new PS. Could you merge it again? Thank you. -- To view, visit http://gerrit.cloudera.org:8080/14592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id03d8d4d41a2ac1b15e7060e2a013e334d044ee7 Gerrit-Change-Number: 14592 Gerrit-PatchSet: 11 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Sat, 09 Nov 2019 01:10:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala.
Anurag Mantripragada has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/14592 ) Change subject: IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala. .. IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala. This is the first of several changes to use informational, unenforced primary key(pk) and foreign key(fk) specifications in Impala. The parent JIRA for this effort is IMPALA-3531. This change adds support for adding pk/fk information during create table DDLs. There is only limited SQL syntax support as of now and will add various other SQL styles including ANSI syntax support in later changes. Currently the only supported way of adding fk/pk information is after the column definitions. Examples are: CREATE TABLE pk(col1 INT, col2 STRING, PRIMARY KEY(col1, col2)); CREATE TABLE fk(id INT, col1 INT, col2 STRING, PRIMARY KEY(id), FOREIGN KEY(col1, col2) REFERENCES pk(col1, col2)); In the current implementation, manual specification of constraint names is not supported. Internally we use UUIDs for contraint name generation. Additionally, three constraint states are supported to comply with Hive's implementation which were taken from Oracle. DISABLE (default true) NOVALIDATE (default true) RELY (default true) More info here: https://docs.oracle.com/database/121/DWHSG/schemas.htm#DWHSG9053 These constraints can be optionally specified after each PK/FK specification like: CREATE TABLE pk(id INT, PRIMARY KEY(id) DISABLE, NOVALIDATE, RELY); However, a specification like this will throw an analysis errror: CREATE TABLE pk(id INT, PRIMARY KEY(id) ENABLE, VALIDATE, RELY); Notes: - toSql support is not fully functional. Observability changes like showing PK/FK information in DESCRIBE output will be done separately. - Retrieval of primary keys and foreign keys is currently not supported in Local Catalog Mode. Tests: Added tests to: - AnalyzeDDLTest#TestCreateTable - ParserTest#TestCreateTable - ToSqlTest#TestCreateTable - Built against both Hive-2 and Hive-3 Change-Id: Id03d8d4d41a2ac1b15e7060e2a013e334d044ee7 --- M common/thrift/CatalogObjects.thrift M common/thrift/JniCatalog.thrift M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java 18 files changed, 749 insertions(+), 59 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/14592/11 -- To view, visit http://gerrit.cloudera.org:8080/14592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id03d8d4d41a2ac1b15e7060e2a013e334d044ee7 Gerrit-Change-Number: 14592 Gerrit-PatchSet: 11 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9137, IMPALA-9138: Mark failed RPC as retryable, add dst node to blacklist
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14677 Change subject: IMPALA-9137, IMPALA-9138: Mark failed RPC as retryable, add dst node to blacklist .. IMPALA-9137, IMPALA-9138: Mark failed RPC as retryable, add dst node to blacklist Introduces two optional fields to TStatus: TErrorType and TRPCErrorMessage. TErrorType introduces a notion of "types" to TStatus objects. For now there are only two types, GENERAL and RETRYABLE. TRPCErrorMessage is set if the TStatus was created as the result of a failed RPC call. It contains the TNetworkAddress of the destination node of the failed RPC. When a Status object is created, SetIsRetryable() can be used to mark the Status as retryable and SetRPCErrorMsg(RPCErrorMsg) can be used to add a TRPCErrorMessage to a Status object. When the Coordinator updates the states of (Coordinator::UpdateBackendExecStatus), if it receives a Status where Status::HasRPCErrorMsg() is true, it takes the RPCErrorMsg destination node, and adds it to the blacklist. Currently, if a Status is marked as retryable (Status::IsRetryable() == true), nothing happens. The change is simply meant to lay the groundwork for future changes. Only RPC failures in KrpcDataStreamSender are marked as retryable and have a RPCErrorMsg set. Re-factored the Thrift files a bit and added a Common.thrift file for all commonly used Thrift structures. Testing: * Ran core tests * Planning to add more tests after IMPALA-8138 is merged Change-Id: I733cca13847fde43c8ea2ae574d3ae04bd06419c --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/common/status.cc M be/src/common/status.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/krpc-data-stream-sender.cc M be/src/util/container-util.h M be/src/util/error-util.h M common/protobuf/common.proto M common/thrift/CMakeLists.txt M common/thrift/CatalogObjects.thrift M common/thrift/CatalogService.thrift A common/thrift/Common.thrift M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/StatestoreService.thrift M common/thrift/Status.thrift M common/thrift/Types.thrift 18 files changed, 206 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/14677/1 -- To view, visit http://gerrit.cloudera.org:8080/14677 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I733cca13847fde43c8ea2ae574d3ae04bd06419c Gerrit-Change-Number: 14677 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar
[Impala-ASF-CR] IMPALA-8138: Reintroduce rpc debugging options
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14641 ) Change subject: IMPALA-8138: Reintroduce rpc debugging options .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4993/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14641 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9c047ebce6d32c5ae461f70279391fa2df4c2029 Gerrit-Change-Number: 14641 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Sat, 09 Nov 2019 00:41:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8138: Reintroduce rpc debugging options
Thomas Tauber-Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/14641 )
Change subject: IMPALA-8138: Reintroduce rpc debugging options
..
Patch Set 3:
(12 comments)
http://gerrit.cloudera.org:8080/#/c/14641/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14641/2//COMMIT_MSG@25
PS2, Line 25: optional 'error
> do you mean IMPALA_SERVICE_POOL?
Done
http://gerrit.cloudera.org:8080/#/c/14641/2/be/src/rpc/impala-service-pool.h
File be/src/rpc/impala-service-pool.h:
http://gerrit.cloudera.org:8080/#/c/14641/2/be/src/rpc/impala-service-pool.h@115
PS2, Line 115: std::string hostname_;
: std::string p
> nit: use std::string
Done
http://gerrit.cloudera.org:8080/#/c/14641/2/be/src/rpc/impala-service-pool.cc
File be/src/rpc/impala-service-pool.cc:
http://gerrit.cloudera.org:8080/#/c/14641/2/be/src/rpc/impala-service-pool.cc@194
PS2, Line 194: mulating rpc errors. To use, specify:
: // --debug_actions=IMPALA_SERVICE_POOL:::
> would be good to document this some more, it is basically a DebugAction tha
Done
http://gerrit.cloudera.org:8080/#/c/14641/2/be/src/rpc/rpc-mgr-test.cc
File be/src/rpc/rpc-mgr-test.cc:
http://gerrit.cloudera.org:8080/#/c/14641/2/be/src/rpc/rpc-mgr-test.cc@322
PS2, Line 322: Ping
> what is Ping suppose to mean?
Its the name of a particular rpc that is part of the Ping service, which is a
toy service we use for testing, see above in this test.
http://gerrit.cloudera.org:8080/#/c/14641/2/be/src/util/debug-util.cc
File be/src/util/debug-util.cc:
http://gerrit.cloudera.org:8080/#/c/14641/2/be/src/util/debug-util.cc@395
PS2, Line 395: }
: string error_msg = tokens.size() == 3 ?
: tokens[2] :
> should returning tokens[2] only be done if the action is FAIL? might be goo
I'm not sure what you mean. Its a property of this particular debug action that
it takes the parameter 'error message' and returns it in the error, which is
documented here in the comment above and in ImpalaService.thrift. Other debug
actions return on OK status.
http://gerrit.cloudera.org:8080/#/c/14641/2/be/src/util/debug-util.cc@399
PS2, Line 399:
: if (ImpaladMetrics::DEBUG_ACTION_NUM_FAIL != nullptr) {
: ImpaladMetrics::DEBUG_ACTION_NUM_FAIL->Increment(1l);
: }
: return Status(TErrorCode::INTERNAL_ERROR, error_msg);
: } else {
: D
> is this thread safe?
Done
http://gerrit.cloudera.org:8080/#/c/14641/2/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/14641/2/common/thrift/ImpalaService.thrift@92
PS2, Line 92: ::...::@@@..."
> as these become more and more complex, it might be worth re-visiting the fo
Sure, since the debug action in this patch is being passed in as a command line
flag its a little awkward to make it JSON or Thrift but its worth thinking
about.
http://gerrit.cloudera.org:8080/#/c/14641/2/common/thrift/ImpalaService.thrift@100
PS2, Line 100: [@] returns
> this is optional right? should we document the behavior if it is omitted?
Done
http://gerrit.cloudera.org:8080/#/c/14641/2/tests/custom_cluster/test_rpc_exception.py
File tests/custom_cluster/test_rpc_exception.py:
http://gerrit.cloudera.org:8080/#/c/14641/2/tests/custom_cluster/test_rpc_exception.py@27
PS2, Line 27: # DataStreamService rpc names
: TRANSMIT_DATA_RPC = "TransmitData"
: END_DATA_STREAM_RPC = "EndDataStream"
:
: # Error to specify for ImpalaServicePool to reject rpcs with a
'server too busy' error.
: REJECT_TOO_BUSY_MSG = "REJECT_TOO_BUSY"
:
> can you document these all a bit?
Done
http://gerrit.cloudera.org:8080/#/c/14641/2/tests/custom_cluster/test_rpc_exception.py@76
PS2, Line 76: assert self._get_num_fails(impalad) > 0
:
: def _get_fail_action(rpc, error=None, port=KRPC_PORT, p=0.1):
> can you document this a bit more and explain what exactly this debug action
Done
http://gerrit.cloudera.org:8080/#/c/14641/2/tests/custom_cluster/test_rpc_exception.py@88
PS2, Line 88: @CustomCluster
> define this and END_ERROR at the top of the file, next to REJECT_TOO_BUSY_M
Done
http://gerrit.cloudera.org:8080/#/c/14641/2/tests/custom_cluster/test_rpc_exception.py@88
PS2, Line 88: Suite.with_args("--
> is this and END_DATA_STREAM_ERROR suppose to be defined in this patch?
Done
--
To view, visit http://gerrit.cloudera.org:8080/14641
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9c047ebce6d32c5ae461f70279391fa2df4c2029
Gerrit-Change-Number: 14641
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Tauber-Marshall
Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8138: Reintroduce rpc debugging options
Hello Michael Ho, Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14641 to look at the new patch set (#3). Change subject: IMPALA-8138: Reintroduce rpc debugging options .. IMPALA-8138: Reintroduce rpc debugging options In the past, Impala had a very simple 'fault injection' framework for simulating failed rpcs between impalads. With the move to KRPC, that framework was not carried over, and we lost the ability to test certain failure scenarios. This patch reintroduces this functionality. It removes the prior fault injection framework in favor of the existing debug action framework, which is more flexible. To facilitate this, a few modifications are made to the debug action framework: - In addition to matching on a label, debug actions may now match on optional arguments. In this patch, the debug action IMPALA_SERVICE_POOL takes the arguments 'host', 'port', and 'rpc name' to allow simulating the failure of specific rpcs to specific impalads. - The FAIL action now takes an optional 'error message' parameter. In this patch, the debug action RPC_SERVICE_POOL uses this to simulate different types of rpc errors, eg. 'service too busy'. - The FAIL action increments a metric, 'impala.debug_action.fail', so that tests can check that it has actually been hit. Prior to this patch the tests in test_rpc_exception.py where all passing spuriously as the faults they were supposed to be testing were no longer being injected. This patch uses these new mechanisms to introduce tests that simulate failures in DataStreamService rpcs. Follow up patches will add test cases for ControlService rpcs. Change-Id: I9c047ebce6d32c5ae461f70279391fa2df4c2029 --- M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr-kerberized-test.cc M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr-test.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/runtime/backend-client.h M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/test-env.cc M be/src/service/control-service.cc M be/src/service/data-stream-service.cc M be/src/service/impala-internal-service.cc M be/src/testutil/CMakeLists.txt D be/src/testutil/fault-injection-util.cc D be/src/testutil/fault-injection-util.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h M common/thrift/ImpalaService.thrift M common/thrift/metrics.json M tests/common/impala_cluster.py M tests/common/impala_service.py M tests/custom_cluster/test_rpc_exception.py 26 files changed, 225 insertions(+), 283 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/14641/3 -- To view, visit http://gerrit.cloudera.org:8080/14641 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9c047ebce6d32c5ae461f70279391fa2df4c2029 Gerrit-Change-Number: 14641 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-9082: make WebserverTest error checking stricter
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14672 ) Change subject: IMPALA-9082: make WebserverTest error checking stricter .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4992/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I820336271cf25130538ceae2eed10a72a73d2adc Gerrit-Change-Number: 14672 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Nov 2019 23:37:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9082: make WebserverTest error checking stricter
Thomas Tauber-Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14672 Change subject: IMPALA-9082: make WebserverTest error checking stricter .. IMPALA-9082: make WebserverTest error checking stricter WebserverTest::TestWithSpnego has been flaky lately, but I have been unable to repro it. This patch is an attempt to make it easier to debug the issue the next time it shows up in automated builds. The test performs a GET that is expected to fail and then checks that the metrics show the failed GET. The flaky failures occur when the metrics do not show a failed attempt. It appears that what's happening is the GET is failing before actually reaching the webserver. However, because the GET is expected to fail and because we only verify that it did fail by checking that HttpGet() returned some error status, whatever unexpected error is occuring is getting lost. This patch instead checks the actual text of the error that is returned by HttpGet() to make sure it is correct and logs the error if it isn't. Change-Id: I820336271cf25130538ceae2eed10a72a73d2adc --- M be/src/testutil/gtest-util.h M be/src/util/webserver-test.cc 2 files changed, 13 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/14672/1 -- To view, visit http://gerrit.cloudera.org:8080/14672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I820336271cf25130538ceae2eed10a72a73d2adc Gerrit-Change-Number: 14672 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall
[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14307 ) Change subject: IMPALA-7506: support global INVALIDATE METADATA in local catalog mode .. Patch Set 13: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 Gerrit-Change-Number: 14307 Gerrit-PatchSet: 13 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 08 Nov 2019 22:35:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14307 ) Change subject: IMPALA-7506: support global INVALIDATE METADATA in local catalog mode .. IMPALA-7506: support global INVALIDATE METADATA in local catalog mode The minimal catalog object version of valid catalog objects is used to implement global invalidate metadata in legacy catalog mode. Coordinator sends DDL RPC to catalogd for global invalidate metadata and gets the expected min catalog version in the response. It's the version when catalogd starts to reset the entire catalog, which means when the reset is done, all valid catalog objects should be associated with a catalog version larger than it. Coordinator will wait until its min catalog version exceeds this value, which means it has processed all the updates of the reset propagated from the catalogd via statestored. If SYNC_DDL is set, the coordinator will also wait until other coordinators reach the same statestore topic version with it, so they have also processed the same updates and had the latest catalog after reset. In local catalog mode, the coordinator does not cache all the metadata. Instead, it caches them on-demand (based on query requests), and removes them based on the Guava cache configurations (size or TTL) or explicit invalidation from the catalog topic updates. So it's hard to track the minimal catalog object version correctly. This patch adds a new field (lastResetCatalogVersion) in TCatalog to propagate the catalog version when catalogd starts to reset the entire metadata. Each time when catalogd generates a new topic update, it will generate a TCatalogObject of CATALOG type containing the state of the catalog which includes this new field. When coordinator receives a new value of lastResetCatalogVersion in a topic update, it means catalogd has reset the entire catalog. Coordinator will then clear its cache to remove all stale catalog objects. It's possible that some fresh items being removed too. They will be refetched on demand. After the invalidation, there are no catalog object cached with catalog version <= lastResetCatalogVersion. Because stale cache has been cleared and all metadata from catalogd is newer than lastResetCatalogVersion. So lastResetCatalogVersion + 1 is the lower bound (included) of min catalog object version of a coordinator. This patch also exposes the lower bound of catalog object version of via a new metric "catalog.catalog-object-version-lower-bound" to ease debugging. IMPALA-9136 is also fixed in this patch. Tests: - Recover all existing tests that have been disabled due to this missing feature - Add custom cluster test for concurrent DDLs with INVALIDATE METADATA - Run CORE tests Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 Reviewed-on: http://gerrit.cloudera.org:8080/14307 Reviewed-by: Vihang Karajgaonkar Tested-by: Impala Public Jenkins --- M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/impalad-metrics.cc M be/src/util/impalad-metrics.h M common/thrift/CatalogObjects.thrift M common/thrift/Frontend.thrift M common/thrift/metrics.json M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java M fe/src/main/java/org/apache/impala/catalog/CatalogObjectImpl.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M tests/authorization/test_grant_revoke.py M tests/authorization/test_ranger.py M tests/common/skip.py A tests/custom_cluster/test_concurrent_ddls.py M tests/custom_cluster/test_local_catalog.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py 19 files changed, 383 insertions(+), 115 deletions(-) Approvals: Vihang Karajgaonkar: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 Gerrit-Change-Number: 14307 Gerrit-PatchSet: 14 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14592 ) Change subject: IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala. .. Patch Set 10: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id03d8d4d41a2ac1b15e7060e2a013e334d044ee7 Gerrit-Change-Number: 14592 Gerrit-PatchSet: 10 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 08 Nov 2019 22:28:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9109: Add top-k metadata loading ranking on catalogd UI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14600 ) Change subject: IMPALA-9109: Add top-k metadata loading ranking on catalogd UI .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14600 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9305a867d7053cde9acc42dae6e47ee440f1a8bf Gerrit-Change-Number: 14600 Gerrit-PatchSet: 12 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Fri, 08 Nov 2019 22:23:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9109: Add top-k metadata loading ranking on catalogd UI
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14600 ) Change subject: IMPALA-9109: Add top-k metadata loading ranking on catalogd UI .. IMPALA-9109: Add top-k metadata loading ranking on catalogd UI Add functions in CatalogUsageMonitor to monitor and report the catalog usage of the tables have the longest metadata loading time(Including maximum, median, 75th-ile, 95th-ile, 99th-ile time). Set default tables loading metrics capacity to 100. However, there might be a problem here because we only keep the capacity size to 100. For example, there might be case like a table has higher median loading time but has lower Maximum loading time which cannot make itself to the Top-100. For now, we will ignore case like that because we are aiming to find the tables with maximum longest loading time. Add the sorted table in Catalog server web-ui. The loading time is sorted by the maximum from load_duration metrics. But users can sort by other metrics in catalogd debug UI. Testing: - Add end-to-end test for webpage to verify the label and text exist in catalog debug page. Verify all fields are in JSON response - Launch Impala and activate some tables to see the table loading time shown successfully on the catalog debug UI page. Change-Id: I9305a867d7053cde9acc42dae6e47ee440f1a8bf Reviewed-on: http://gerrit.cloudera.org:8080/14600 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M common/thrift/JniCatalog.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/CatalogUsageMonitor.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M tests/webserver/test_web_pages.py M www/catalog.tmpl M www/scripts/util.js 10 files changed, 307 insertions(+), 14 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14600 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9305a867d7053cde9acc42dae6e47ee440f1a8bf Gerrit-Change-Number: 14600 Gerrit-PatchSet: 13 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-7984: Port runtime filter from Thrift RPC to KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13882 ) Change subject: IMPALA-7984: Port runtime filter from Thrift RPC to KRPC .. Patch Set 31: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5199/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13882 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b394796d250286510e157ae326882bfc01d387a Gerrit-Change-Number: 13882 Gerrit-PatchSet: 31 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 08 Nov 2019 21:32:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7984: Port runtime filter from Thrift RPC to KRPC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13882 ) Change subject: IMPALA-7984: Port runtime filter from Thrift RPC to KRPC .. Patch Set 31: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13882 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b394796d250286510e157ae326882bfc01d387a Gerrit-Change-Number: 13882 Gerrit-PatchSet: 31 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Fri, 08 Nov 2019 21:32:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9128: part 1: log on slow data stream RPCs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 ) Change subject: IMPALA-9128: part 1: log on slow data stream RPCs .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4991/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd Gerrit-Change-Number: 14662 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 08 Nov 2019 21:31:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9092 : Disable show create table tests on Kudu
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14664 ) Change subject: IMPALA-9092 : Disable show create table tests on Kudu .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4989/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I37c0b6d82372bc6380285afcd94f0c1e123f2eda Gerrit-Change-Number: 14664 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 08 Nov 2019 21:26:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9128: part 1: log on slow data stream RPCs
Hello Thomas Tauber-Marshall, Lars Volker, Todd Lipcon, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14662
to look at the new patch set (#4).
Change subject: IMPALA-9128: part 1: log on slow data stream RPCs
..
IMPALA-9128: part 1: log on slow data stream RPCs
Allows modifying the threshold for KRPC's server-side slow
RPC logging (which is enabled for all KRPCs).
Added additional logging for data stream RPCs TransmitData
and EndDataStream, and for slow waits that delay the query.
Adds statistics for RPC time to provide some clues if there
are slow data stream RPCs.
I tested this with a low threshold and delays added:
start-impala-cluster.py \
--impalad_args=--impala_slow_rpc_threshold_ms=1 \
--impalad_args=--debug_actions=END_DATA_STREAM_DELAY:JITTER@3000@1.0
Example Profile output:
- NetworkThroughput: (Avg: 102.98 MB/sec ; Min: 5.58 MB/sec ; Max: 171.79
MB/sec ; Number of samples: 296)
- RpcNetworkTime: (Avg: 13.468ms ; Min: 91.309us ; Max: 2s395ms ; Number of
samples: 299)
- RpcRecvrTime: (Avg: 13.406ms ; Min: 83.160us ; Max: 2s395ms ; Number of
samples: 299)
Example log output (with log threshold of 1ms):
I1107 14:33:50.487251 24933 krpc-data-stream-sender.cc:363]
ad4fa70619170ace:b58b2eba0006] Long delay waiting for RPC to
127.0.1.1:27000 (fragment_instance_id=ad4fa70619170ace:b58b2eba): took
451.036ms
I1107 14:33:51.295518 21361 rpcz_store.cc:265] Call
impala.DataStreamService.EndDataStream from 127.0.0.1:43952 (request call id
82) took 1259ms. Request Metrics: {}
I1107 14:33:44.843204 21332 krpc-data-stream-sender.cc:342] Slow TransmitData
RPC to 127.0.1.1:27000
(fragment_instance_id=ad4fa70619170ace:b58b2eba0006): took 2.194ms.
Receiver time: 457.902us Network time: 1.736ms
I1107 14:33:45.139068 21333 krpc-data-stream-sender.cc:342] Slow EndDataStream
RPC to 127.0.1.1:27001
(fragment_instance_id=ad4fa70619170ace:b58b2eba0004): took 61.340ms.
Receiver time: 81.908us Network time: 61.259ms
Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
---
M be/src/common/global-flags.cc
M be/src/rpc/rpc-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
4 files changed, 87 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/14662/4
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-9128: part 2: dump traces for slow RPCs
Tim Armstrong has abandoned this change. ( http://gerrit.cloudera.org:8080/14669 ) Change subject: IMPALA-9128: part 2: dump traces for slow RPCs .. Abandoned Didn't mean to publish -- To view, visit http://gerrit.cloudera.org:8080/14669 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Ia53da1ffe2d07906d44cef6c7d0a80ef130c7724 Gerrit-Change-Number: 14669 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9128: part 1: log on slow data stream RPCs
Hello Thomas Tauber-Marshall, Lars Volker, Todd Lipcon, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14662
to look at the new patch set (#3).
Change subject: IMPALA-9128: part 1: log on slow data stream RPCs
..
IMPALA-9128: part 1: log on slow data stream RPCs
Allows modifying the threshold for KRPC's server-side slow
RPC logging (which is enabled for all KRPCs).
Added additional logging for data stream RPCs TransmitData
and EndDataStream, and for slow waits that delay the query.
Adds statistics for RPC time to provide some clues if there
are slow data stream RPCs.
I tested this with a low threshold and delays added:
start-impala-cluster.py \
--impalad_args=--impala_slow_rpc_threshold_ms=1 \
--impalad_args=--debug_actions=END_DATA_STREAM_DELAY:JITTER@3000@1.0
Example Profile output:
- NetworkThroughput: (Avg: 102.98 MB/sec ; Min: 5.58 MB/sec ; Max: 171.79
MB/sec ; Number of samples: 296)
- RpcNetworkTime: (Avg: 13.468ms ; Min: 91.309us ; Max: 2s395ms ; Number of
samples: 299)
- RpcRecvrTime: (Avg: 13.406ms ; Min: 83.160us ; Max: 2s395ms ; Number of
samples: 299)
Example log output (with log threshold of 1ms):
I1107 14:33:50.487251 24933 krpc-data-stream-sender.cc:363]
ad4fa70619170ace:b58b2eba0006] Long delay waiting for RPC to
127.0.1.1:27000 (fragment_instance_id=ad4fa70619170ace:b58b2eba): took
451.036ms
I1107 14:33:51.295518 21361 rpcz_store.cc:265] Call
impala.DataStreamService.EndDataStream from 127.0.0.1:43952 (request call id
82) took 1259ms. Request Metrics: {}
I1107 14:33:44.843204 21332 krpc-data-stream-sender.cc:342] Slow TransmitData
RPC to 127.0.1.1:27000
(fragment_instance_id=ad4fa70619170ace:b58b2eba0006): took 2.194ms.
Receiver time: 457.902us Network time: 1.736ms
I1107 14:33:45.139068 21333 krpc-data-stream-sender.cc:342] Slow EndDataStream
RPC to 127.0.1.1:27001
(fragment_instance_id=ad4fa70619170ace:b58b2eba0004): took 61.340ms.
Receiver time: 81.908us Network time: 61.259ms
Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
---
M be/src/common/global-flags.cc
M be/src/rpc/rpc-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
4 files changed, 87 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/14662/3
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-9128: improved diagnostics for slow data stream RPCs
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 ) Change subject: IMPALA-9128: improved diagnostics for slow data stream RPCs .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/14662/2/be/src/runtime/krpc-data-stream-sender.cc File be/src/runtime/krpc-data-stream-sender.cc: http://gerrit.cloudera.org:8080/#/c/14662/2/be/src/runtime/krpc-data-stream-sender.cc@374 PS2, Line 374: LOG(INFO) << "Long delay waiting for RPC to " << TNetworkAddressToString(address_) > Doing this inside the loop does result in quite a lot of log spam, I should Done -- To view, visit http://gerrit.cloudera.org:8080/14662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd Gerrit-Change-Number: 14662 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 08 Nov 2019 20:45:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9128: part 2: dump traces for slow RPCs
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14669 Change subject: IMPALA-9128: part 2: dump traces for slow RPCs .. IMPALA-9128: part 2: dump traces for slow RPCs This adds trace events for data stream RPCs and dumps them when they take longer than --impala_slow_rpc_threshold_ms. I needed to modify the KRPC code to do this because it currently only dumps traces for RPCs with deadlines. I plan to add some version of this upstream in Kudu so that we don't diverge our KRPC implementation. Change-Id: Ia53da1ffe2d07906d44cef6c7d0a80ef130c7724 --- M be/src/kudu/rpc/rpcz_store.cc M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-recvr.cc M be/src/runtime/krpc-data-stream-sender.cc 4 files changed, 36 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/14669/1 -- To view, visit http://gerrit.cloudera.org:8080/14669 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia53da1ffe2d07906d44cef6c7d0a80ef130c7724 Gerrit-Change-Number: 14669 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-9092 : Disable show create table tests on Kudu
Vihang Karajgaonkar has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/14664 ) Change subject: IMPALA-9092 : Disable show create table tests on Kudu .. IMPALA-9092 : Disable show create table tests on Kudu This patch temporarily disables the Kudu tests which fail when we try to bump up the CDP build number due the HMS translation. The tests will be re-enabled back again when the fix for IMPALA-9092 is submitted. Testing Done: 1. Bumped up the CDP build number to 1507246 which has the HMS translation in it. 2. Ran core tests and found the tests which are failing for Kudu due to this issue. 3. Ran the failing tests again and confirm they are not failing anymore. Change-Id: I37c0b6d82372bc6380285afcd94f0c1e123f2eda --- M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java M tests/common/skip.py M tests/metadata/test_ddl.py M tests/metadata/test_show_create_table.py M tests/query_test/test_kudu.py 6 files changed, 25 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/14664/3 -- To view, visit http://gerrit.cloudera.org:8080/14664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I37c0b6d82372bc6380285afcd94f0c1e123f2eda Gerrit-Change-Number: 14664 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-9128: improved diagnostics for slow data stream RPCs
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 ) Change subject: IMPALA-9128: improved diagnostics for slow data stream RPCs .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/14662/2/be/src/runtime/krpc-data-stream-sender.cc File be/src/runtime/krpc-data-stream-sender.cc: http://gerrit.cloudera.org:8080/#/c/14662/2/be/src/runtime/krpc-data-stream-sender.cc@374 PS2, Line 374: LOG(INFO) << "Long delay waiting for RPC to " << TNetworkAddressToString(address_) Doing this inside the loop does result in quite a lot of log spam, I should probably move it outside the loop or rate-limit it. -- To view, visit http://gerrit.cloudera.org:8080/14662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd Gerrit-Change-Number: 14662 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 08 Nov 2019 20:28:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9085: [DOCS] Refactored impala s3.xml
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/14627 ) Change subject: IMPALA-9085: [DOCS] Refactored impala_s3.xml .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/14627/3/docs/topics/impala_s3.xml File docs/topics/impala_s3.xml: http://gerrit.cloudera.org:8080/#/c/14627/3/docs/topics/impala_s3.xml@a406 PS3, Line 406: > why delete this example? I don't see the example adding values. It is repeating what's already said or describing things very straightforard. Let me know if you feel this whole section should be added back in. -- To view, visit http://gerrit.cloudera.org:8080/14627 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib274968a0412b4b8757f31ab674d4b82311de70a Gerrit-Change-Number: 14627 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Fri, 08 Nov 2019 19:31:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9110: Add table loading time break-down metrics for HdfsTable
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14611 ) Change subject: IMPALA-9110: Add table loading time break-down metrics for HdfsTable .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/14611/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14611/2//COMMIT_MSG@17 PS2, Line 17: We added "hms-load-tbl-schema", "load-duration-all-column-stats". : Also, we logged the loadValidWriteIdList() time Are you planning to add metrics for partition load time separately? I think it will useful to do it in one patch if possible. In my opinion just having a tbl load time without partitions may not be very interesting. Do you think we should have a metric for loading all the partitions of a along with the table? Its also tricky because the time value depends on the number of partitions which are being loaded. So the metric may not give any insights (for example, a large table with many partitions may show small loading times, if we are only loading one partition compared to a small table which loads all the partitions). http://gerrit.cloudera.org:8080/#/c/14611/2/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/14611/2/fe/src/main/java/org/apache/impala/catalog/Table.java@38 PS2, Line 38: import org.apache.impala.common.*; I think the convention is to not use * and import each individually for improved readability and not importing unnecessary classes when they are added to the package in the future. http://gerrit.cloudera.org:8080/#/c/14611/2/fe/src/main/java/org/apache/impala/catalog/TableLoader.java File fe/src/main/java/org/apache/impala/catalog/TableLoader.java: http://gerrit.cloudera.org:8080/#/c/14611/2/fe/src/main/java/org/apache/impala/catalog/TableLoader.java@71 PS2, Line 71: hmsLoadSW Shouldn't this line be after line number 86? http://gerrit.cloudera.org:8080/#/c/14611/2/fe/src/main/java/org/apache/impala/catalog/TableLoadingMgr.java File fe/src/main/java/org/apache/impala/catalog/TableLoadingMgr.java: http://gerrit.cloudera.org:8080/#/c/14611/2/fe/src/main/java/org/apache/impala/catalog/TableLoadingMgr.java@264 PS2, Line 264: startTableLoadingSubmitterThreads Indeed, this is very confusing. Looks like we have pool of threads to submit the loadTask. All these threads are really doing is to taking out the table from the deque, making sure its not being already loaded and then if its not being loaded already, submit it to the next pool tblLoadingPool_ which actually does the loading of the table. This path is used for loading the tables which are requested by coordinators (prioritized loading) when they are analyzing the queries or the background loading which is enabled with the flag loadInBackground_ is set. I found a bug in the reset() method which adds the tables to the backgroundload unnecessarily (see IMPALA-9139) I think we can get rid of the pool for submitting the tasks. and we can simply this by making any load request would either be a background load (offerLast in the queue) and prioritized loading (offerFirst in the queue). Since this code is time-consuming, we don't need a pool of threads for doing this. The loadAsync() could be implemented using a prioritizedLoad code-path so that its same as what we have today. Created IMPALA-9140 for this. -- To view, visit http://gerrit.cloudera.org:8080/14611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5381f9316df588b2004876c6cd9fb7e674085b10 Gerrit-Change-Number: 14611 Gerrit-PatchSet: 2 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 08 Nov 2019 19:15:04 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9045: Filter base directories of open/aborted compactions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14547 ) Change subject: IMPALA-9045: Filter base directories of open/aborted compactions .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14547 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb895df38bc075e4767e44a6887dbe3000a19ea6 Gerrit-Change-Number: 14547 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 08 Nov 2019 18:37:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9045: Filter base directories of open/aborted compactions
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14547 ) Change subject: IMPALA-9045: Filter base directories of open/aborted compactions .. IMPALA-9045: Filter base directories of open/aborted compactions Base directories are in the format of base__. The part helps to decide whether a base directory is fully written, or is it still being written by a compaction job. Compaction jobs don't increase the write id of a table, hence the part cannot be used for that. Before this commit Impala didn't check the validity of , therefore it might read the contents of a half-written base directory. With this change Impala retrieves the valid transaction list from HMS and checks if is committed. Testing * Added an e2e test that simulates in-progress compactions * Added frontend test that filters based on custom valid txn list Change-Id: Idb895df38bc075e4767e44a6887dbe3000a19ea6 Reviewed-on: http://gerrit.cloudera.org:8080/14547 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/catalog/HdfsPartitionTest.java M fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java M tests/query_test/test_acid.py 12 files changed, 247 insertions(+), 44 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14547 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Idb895df38bc075e4767e44a6887dbe3000a19ea6 Gerrit-Change-Number: 14547 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14592 ) Change subject: IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala. .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5198/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id03d8d4d41a2ac1b15e7060e2a013e334d044ee7 Gerrit-Change-Number: 14592 Gerrit-PatchSet: 10 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 08 Nov 2019 18:04:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14592 ) Change subject: IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala. .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id03d8d4d41a2ac1b15e7060e2a013e334d044ee7 Gerrit-Change-Number: 14592 Gerrit-PatchSet: 10 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 08 Nov 2019 18:04:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14592 ) Change subject: IMPALA-2112: Support primary key/foreign key constraints as part of create table in Impala. .. Patch Set 9: Code-Review+2 Patch looks good to me. Quanlong has also given a +1 so I am promoting this to a +2 -- To view, visit http://gerrit.cloudera.org:8080/14592 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id03d8d4d41a2ac1b15e7060e2a013e334d044ee7 Gerrit-Change-Number: 14592 Gerrit-PatchSet: 9 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 08 Nov 2019 18:04:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14307 ) Change subject: IMPALA-7506: support global INVALIDATE METADATA in local catalog mode .. Patch Set 13: Code-Review+2 Thanks for making the changes. This would help fill an important gap in catalog-v2 for the users. -- To view, visit http://gerrit.cloudera.org:8080/14307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 Gerrit-Change-Number: 14307 Gerrit-PatchSet: 13 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 08 Nov 2019 18:02:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7506: support global INVALIDATE METADATA in local catalog mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14307 ) Change subject: IMPALA-7506: support global INVALIDATE METADATA in local catalog mode .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5197/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14307 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib61a7ab1ffa062620ffbc2dadc34bd7a8ca9e549 Gerrit-Change-Number: 14307 Gerrit-PatchSet: 13 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 08 Nov 2019 18:02:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9109: Add top-k metadata loading ranking on catalogd UI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14600 ) Change subject: IMPALA-9109: Add top-k metadata loading ranking on catalogd UI .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5196/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14600 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9305a867d7053cde9acc42dae6e47ee440f1a8bf Gerrit-Change-Number: 14600 Gerrit-PatchSet: 12 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Fri, 08 Nov 2019 17:58:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9109: Add top-k metadata loading ranking on catalogd UI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14600 ) Change subject: IMPALA-9109: Add top-k metadata loading ranking on catalogd UI .. Patch Set 12: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14600 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9305a867d7053cde9acc42dae6e47ee440f1a8bf Gerrit-Change-Number: 14600 Gerrit-PatchSet: 12 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Fri, 08 Nov 2019 17:58:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9109: Add top-k metadata loading ranking on catalogd UI
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14600 ) Change subject: IMPALA-9109: Add top-k metadata loading ranking on catalogd UI .. Patch Set 11: > Patch Set 11: Code-Review+2 There are 2 other +1s as well. So I am good to give a +2 and promote this patch to be committed. -- To view, visit http://gerrit.cloudera.org:8080/14600 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9305a867d7053cde9acc42dae6e47ee440f1a8bf Gerrit-Change-Number: 14600 Gerrit-PatchSet: 11 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Fri, 08 Nov 2019 17:57:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9109: Add top-k metadata loading ranking on catalogd UI
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/14600 )
Change subject: IMPALA-9109: Add top-k metadata loading ranking on catalogd UI
..
Patch Set 10:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14600/10/www/scripts/util.js
File www/scripts/util.js:
http://gerrit.cloudera.org:8080/#/c/14600/10/www/scripts/util.js@52
PS10, Line 52: if (!hour && value >= 1000) {
: re += (Math.floor(value / 1000) + "s");
: value = value % 1000;
: second = true;
: }
> The output for this will be 10h. I am using the same logic as https://githu
ah, okay. Thanks for the explanation. Although, I think it definitely makes it
clearer if we add a comment saying that. Not a blocker though.
--
To view, visit http://gerrit.cloudera.org:8080/14600
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9305a867d7053cde9acc42dae6e47ee440f1a8bf
Gerrit-Change-Number: 14600
Gerrit-PatchSet: 10
Gerrit-Owner: Jiawei Wang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jiawei Wang
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Xiaomeng Zhang
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Comment-Date: Fri, 08 Nov 2019 17:56:56 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9109: Add top-k metadata loading ranking on catalogd UI
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14600 ) Change subject: IMPALA-9109: Add top-k metadata loading ranking on catalogd UI .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14600 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9305a867d7053cde9acc42dae6e47ee440f1a8bf Gerrit-Change-Number: 14600 Gerrit-PatchSet: 11 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Xiaomeng Zhang Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Fri, 08 Nov 2019 17:57:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8525: preads should use hdfsPreadFully rather than hdfsPread
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14635 ) Change subject: IMPALA-8525: preads should use hdfsPreadFully rather than hdfsPread .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4988/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14635 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29ea34897096bc790abdeb98073a47f1c4c10feb Gerrit-Change-Number: 14635 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Nov 2019 17:20:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8525: preads should use hdfsPreadFully rather than hdfsPread
Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/14635 )
Change subject: IMPALA-8525: preads should use hdfsPreadFully rather than
hdfsPread
..
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/14635/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14635/2//COMMIT_MSG@9
PS2, Line 9: Modifies HdfsFileReader so that it calls hdfsPreadFully instead of
> Is hdfsPreadFully supported for all filesystems?
It's supported by all filesystems supported by Impala - HDFS, S3A, ADLS, ABFS
as well as Google Cloud Store (GCS).
http://gerrit.cloudera.org:8080/#/c/14635/2/be/src/runtime/io/hdfs-file-reader.cc
File be/src/runtime/io/hdfs-file-reader.cc:
http://gerrit.cloudera.org:8080/#/c/14635/2/be/src/runtime/io/hdfs-file-reader.cc@137
PS2, Line 137: int chunk_size = bytes_to_read - *bytes_read;
> Maybe rename to bytes_remaining or similar, given that we're not trying to
Done
http://gerrit.cloudera.org:8080/#/c/14635/2/be/src/runtime/io/hdfs-file-reader.cc@224
PS2, Line 224: if (FLAGS_use_hdfs_pread ||
IsS3APath(scan_range_->file_string()->c_str())) {
> Should we switch ABFS too? Since we disabled the chunking fix for that as w
Oddly enough, none of this makes a significant difference for ABFS. I plan to
investigate why in a separate JIRA. I don't think the chunk-size optimization
was added for ABFS with much experimentation, so I don't think we lose much,
especially with the fix in HDFS-14285.
--
To view, visit http://gerrit.cloudera.org:8080/14635
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I29ea34897096bc790abdeb98073a47f1c4c10feb
Gerrit-Change-Number: 14635
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 08 Nov 2019 16:36:44 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8525: preads should use hdfsPreadFully rather than hdfsPread
Hello Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14635 to look at the new patch set (#3). Change subject: IMPALA-8525: preads should use hdfsPreadFully rather than hdfsPread .. IMPALA-8525: preads should use hdfsPreadFully rather than hdfsPread Modifies HdfsFileReader so that it calls hdfsPreadFully instead of hdfsPread. hdfsPreadFully is a new libhdfs API introduced by HDFS-14564 (Add libhdfs APIs for readFully; add readFully to ByteBufferPositionedReadable). hdfsPreadFully improves performance of preads, especially when reading data from S3. The major difference between hdfsPread and hdfsPreadFully is that hdfsPreadFully is guaranteed to read all the requested bytes, whereas hdfsPread is only guaranteed to read up to the number of requested bytes. hdfsPreadFully reduces the amount of JNI array allocations necessary when reading data from S3. When any read method in libhdfs is called, the method allocates an array whose size is equal to the amount of data requested. The issue is that Java's InputStream#read only guarantees that it will read up to the amount of data requested. This can lead to issues where a libhdfs read request allocates a large Java array, even though the read request only partially fills it up. PositionedReadable#readFully on the other hand, guarantees that all requested data will be read, thus preventing any unnecessary JNI array allocations. hdfsPreadFully improves the effectiveness of fs.s3a.experimental.input.fadvise=RANDOM (HADOOP-13203). S3A recommends setting fadvise=RANDOM when doing random reads, which is common in Impala when reading Parquet or ORC files. fadvise=RANDOM causes the HTTP GET request that reads the S3 data to simply request the data bounded by the parameters of the current read request (e.g. for 'read(long position, ..., int length)' it requests 'length' bytes). The chunk-size optimization in HdfsFileReader hurts performance when fadvise=RANDOM because each HTTP GET request will only request 'chunk-size' amount of bytes at a time. Which is why this patch removes the chunk-size optimization as well. hdfsPreadFully helps here because all the data in the scan range will be requested by a single HTTP GET request. Since hdfsPreadFully improves S3 read performance, this patch enables preads for S3A files by default. Even if fadvise=SEQUENTIAL, hdfsPreadFully still improves performance since it avoids unnecessary JNI allocation overhead. The chunk-size optimization (added in https://gerrit.cloudera.org/#/c/63/) is no longer necessary after this patch. hdfsPreadFully prevents any unnecessary array allocations. Furthermore, it is likely the chunk-size optimization was added due to overhead fixed by HDFS-14285. Fixes a bug in IMPALA-8884 where the 'impala-server.io-mgr.queue-$i.read-size' statistics were being updated with the chunk-size passed to HdfsFileReader::ReadFromPosInternal, which is not necessarily equivalent to the amount of data actually read. Testing: * Ran core tests * Ran core tests on S3 Change-Id: I29ea34897096bc790abdeb98073a47f1c4c10feb --- M be/src/common/global-flags.cc M be/src/runtime/io/hdfs-file-reader.cc M be/src/runtime/io/hdfs-file-reader.h M be/src/runtime/io/local-file-reader.cc M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc 6 files changed, 20 insertions(+), 52 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/14635/3 -- To view, visit http://gerrit.cloudera.org:8080/14635 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I29ea34897096bc790abdeb98073a47f1c4c10feb Gerrit-Change-Number: 14635 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9126: part 1: hash join build partition cleanup
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14632 ) Change subject: IMPALA-9126: part 1: hash join build partition cleanup .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4987/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14632 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife8d0fa5dd14c7d3f3d726dd38c07d8cbceabadb Gerrit-Change-Number: 14632 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Nov 2019 15:14:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9128: improved diagnostics for slow data stream RPCs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14662 ) Change subject: IMPALA-9128: improved diagnostics for slow data stream RPCs .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4986/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd Gerrit-Change-Number: 14662 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 08 Nov 2019 15:13:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9131: Use single quotes around FORMAT clause in CAST
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/14665 ) Change subject: IMPALA-9131: Use single quotes around FORMAT clause in CAST .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/14665/1/fe/src/main/java/org/apache/impala/analysis/CastExpr.java File fe/src/main/java/org/apache/impala/analysis/CastExpr.java: http://gerrit.cloudera.org:8080/#/c/14665/1/fe/src/main/java/org/apache/impala/analysis/CastExpr.java@209 PS1, Line 209: castFormat_ Are single quotes allowed inside 'castFormat_' ? If yes, I assume the printout should contain escaped single quotes instead. -- To view, visit http://gerrit.cloudera.org:8080/14665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3310abfa6f3ccbbe4c437846c6dd05791153e6f7 Gerrit-Change-Number: 14665 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Anonymous Coward (536) Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Comment-Date: Fri, 08 Nov 2019 14:56:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9126: part 1: hash join build partition cleanup
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14632 ) Change subject: IMPALA-9126: part 1: hash join build partition cleanup .. Patch Set 4: This is a very incremental bit of cleanup that will help with separating the build side for parallel plans, but I think is self contained and ready so I pushed it out for review. -- To view, visit http://gerrit.cloudera.org:8080/14632 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ife8d0fa5dd14c7d3f3d726dd38c07d8cbceabadb Gerrit-Change-Number: 14632 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Nov 2019 14:32:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9126: part 1: hash join build partition cleanup
Tim Armstrong has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/14632 ) Change subject: IMPALA-9126: part 1: hash join build partition cleanup .. IMPALA-9126: part 1: hash join build partition cleanup Clarify some invariants (e.g. which join modes actually require build partition data to be attached to row batches). Move some logic for maintenance of the hash partitions to the builder. Testing: Ran exhaustive tests. We should already have adequate coverage for spilling and non-spilling hash joins. Change-Id: Ife8d0fa5dd14c7d3f3d726dd38c07d8cbceabadb --- M be/src/exec/blocking-join-node.cc M be/src/exec/blocking-join-node.h A be/src/exec/join-op.h M be/src/exec/nested-loop-join-node.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h 8 files changed, 146 insertions(+), 92 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/14632/4 -- To view, visit http://gerrit.cloudera.org:8080/14632 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ife8d0fa5dd14c7d3f3d726dd38c07d8cbceabadb Gerrit-Change-Number: 14632 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-9126: part 1: hash join build partition cleanup
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/14632 )
Change subject: IMPALA-9126: part 1: hash join build partition cleanup
..
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/14632/4/be/src/exec/partitioned-hash-join-node.cc
File be/src/exec/partitioned-hash-join-node.cc:
http://gerrit.cloudera.org:8080/#/c/14632/4/be/src/exec/partitioned-hash-join-node.cc@339
PS4, Line 339: if (state_ == PROBING_SPILLED_PARTITION &&
NeedToProcessUnmatchedBuildRows(join_op_)) {
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/14632/4/be/src/exec/partitioned-hash-join-node.cc@1154
PS4, Line 1154: hash_tbl_iterator_ =
output_build_partitions_.front()->hash_tbl()->FirstUnmatched(ht_ctx_.get());
line too long (101 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/14632
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ife8d0fa5dd14c7d3f3d726dd38c07d8cbceabadb
Gerrit-Change-Number: 14632
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Fri, 08 Nov 2019 14:31:37 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9128: improved diagnostics for slow data stream RPCs
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/14662 )
Change subject: IMPALA-9128: improved diagnostics for slow data stream RPCs
..
Patch Set 1:
(7 comments)
Address most of the comments, just had another question about the receiver
timing.
http://gerrit.cloudera.org:8080/#/c/14662/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/14662/1//COMMIT_MSG@16
PS1, Line 16: a
> typo
Done
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/common/global-flags.cc
File be/src/common/global-flags.cc:
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/common/global-flags.cc@291
PS1, Line 291: DEFINE_int64(impala_slow_rpc_threshold_ms, 2 * 60 * 1000,
> (Advanced)?
Done. Did both.
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.h
File be/src/runtime/krpc-data-stream-sender.h:
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.h@225
PS1, Line 225: RuntimeProfile::SummaryStatsCounter* recvr_time_stats_ =
nullptr;
> Does this provide us with enough info to determine if slow rpcs from the re
I think you're right on that one - it looks like the timer starts when the
incoming RPC is queued, not when Impala's handler is called. It's asymmetrical
though in the the timer stops before the output RPC is queued, so it includes
the KRPC overhead in one direction only (at least that's what it looks like
based on my limited understanding).
I think ideally we'd use tracing to get more granular breakdown of where the
time went (I'm also looking into that). I'm a little reluctant to add another
timer that's very similar to an existing one, just cause of the confusion
factor.
I wonder if it would make more sense to tweak receiver_time_ns so that it only
includes the Impala RPC handling time. I can't really tell if the current
definition of receiver time is that way for a good reason or just arbitrarily.
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.cc
File be/src/runtime/krpc-data-stream-sender.cc:
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.cc@342
PS1, Line 342: LOG(INFO) << "slow " << rpc_name << " RPC to " <<
TNetworkAddressToString(address_)
> nit: Slow
Done
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.cc@361
PS1, Line 361: int64_t elapsed_time_ms = elapsed_time_ns / NANOS_PER_MICRO /
MICROS_PER_MILLI;
: if (elapsed_time_ms > FLAGS_impala_slow_rpc_threshold_ms) {
> IsSlowRpc
Done
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.cc@453
PS1, Line 453: DoRpcFn rpc_fn =
> Do we want to collect these timing stats for failed rpcs too?
I added the slow RPC logging for failed RPCs. The stats seemed a bit confusing
because we don't have the receiver time, so can't calculate the same values. I
decided not to tackle that problem
http://gerrit.cloudera.org:8080/#/c/14662/1/be/src/runtime/krpc-data-stream-sender.cc@565
PS1, Line 565: DoRpcFn rpc_fn =
> Same as above: do we want these stats for failed rpcs?
See above - I added the logging only.
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Fri, 08 Nov 2019 14:28:40 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9128: improved diagnostics for slow data stream RPCs
Hello Thomas Tauber-Marshall, Lars Volker, Todd Lipcon, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14662
to look at the new patch set (#2).
Change subject: IMPALA-9128: improved diagnostics for slow data stream RPCs
..
IMPALA-9128: improved diagnostics for slow data stream RPCs
Allows modifying the threshold for KRPC's server-side slow
RPC logging (which is enabled for all KRPCs).
Added additional logging for data stream RPCs TransmitData
and EndDataStream, and for slow waits that delay the query.
Adds statistics for RPC time to provide some clues if there
are slow data stream RPCs.
I tested this with a low threshold and delays added:
start-impala-cluster.py \
--impalad_args=--impala_slow_rpc_threshold_ms=1 \
--impalad_args=--debug_actions=END_DATA_STREAM_DELAY:JITTER@3000@1.0
Example Profile output:
- NetworkThroughput: (Avg: 102.98 MB/sec ; Min: 5.58 MB/sec ; Max: 171.79
MB/sec ; Number of samples: 296)
- RpcNetworkTime: (Avg: 13.468ms ; Min: 91.309us ; Max: 2s395ms ; Number of
samples: 299)
- RpcRecvrTime: (Avg: 13.406ms ; Min: 83.160us ; Max: 2s395ms ; Number of
samples: 299)
Example log output (with log threshold of 1ms):
I1107 14:33:50.487251 24933 krpc-data-stream-sender.cc:363]
ad4fa70619170ace:b58b2eba0006] Long delay waiting for RPC to
127.0.1.1:27000 (fragment_instance_id=ad4fa70619170ace:b58b2eba): took
451.036ms
I1107 14:33:51.295518 21361 rpcz_store.cc:265] Call
impala.DataStreamService.EndDataStream from 127.0.0.1:43952 (request call id
82) took 1259ms. Request Metrics: {}
I1107 14:33:44.843204 21332 krpc-data-stream-sender.cc:342] Slow TransmitData
RPC to 127.0.1.1:27000
(fragment_instance_id=ad4fa70619170ace:b58b2eba0006): took 2.194ms.
Receiver time: 457.902us Network time: 1.736ms
I1107 14:33:45.139068 21333 krpc-data-stream-sender.cc:342] Slow EndDataStream
RPC to 127.0.1.1:27001
(fragment_instance_id=ad4fa70619170ace:b58b2eba0004): took 61.340ms.
Receiver time: 81.908us Network time: 61.259ms
Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
---
M be/src/common/global-flags.cc
M be/src/rpc/rpc-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/krpc-data-stream-sender.h
4 files changed, 87 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/14662/2
--
To view, visit http://gerrit.cloudera.org:8080/14662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I258ac91b9fbbdbc86d0e8091c34f511f8957c4cd
Gerrit-Change-Number: 14662
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Thomas Tauber-Marshall
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-9045: Filter base directories of open/aborted compactions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14547 ) Change subject: IMPALA-9045: Filter base directories of open/aborted compactions .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5195/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14547 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb895df38bc075e4767e44a6887dbe3000a19ea6 Gerrit-Change-Number: 14547 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 08 Nov 2019 14:09:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9045: Filter base directories of open/aborted compactions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14547 ) Change subject: IMPALA-9045: Filter base directories of open/aborted compactions .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14547 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb895df38bc075e4767e44a6887dbe3000a19ea6 Gerrit-Change-Number: 14547 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 08 Nov 2019 14:09:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9131: Use single quotes around FORMAT clause in CAST
Norbert Luksa has posted comments on this change. ( http://gerrit.cloudera.org:8080/14665 ) Change subject: IMPALA-9131: Use single quotes around FORMAT clause in CAST .. Patch Set 1: Should not the queries in QueryTest/date.test and QueryTest/cast_format_table.test be changed to use single quotes instead? It's also used in quer_test/test_cast_with_format.py. -- To view, visit http://gerrit.cloudera.org:8080/14665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3310abfa6f3ccbbe4c437846c6dd05791153e6f7 Gerrit-Change-Number: 14665 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Anonymous Coward (536) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Comment-Date: Fri, 08 Nov 2019 13:27:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9045: Filter base directories of open/aborted compactions
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/14547 ) Change subject: IMPALA-9045: Filter base directories of open/aborted compactions .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14547 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb895df38bc075e4767e44a6887dbe3000a19ea6 Gerrit-Change-Number: 14547 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 08 Nov 2019 13:01:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9131: Use single quotes around FORMAT clause in CAST
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14665 ) Change subject: IMPALA-9131: Use single quotes around FORMAT clause in CAST .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4985/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3310abfa6f3ccbbe4c437846c6dd05791153e6f7 Gerrit-Change-Number: 14665 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Anonymous Coward (536) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 08 Nov 2019 09:49:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9131: Use single quotes around FORMAT clause in CAST
Anonymous Coward (536) has posted comments on this change. ( http://gerrit.cloudera.org:8080/14665 ) Change subject: IMPALA-9131: Use single quotes around FORMAT clause in CAST .. Patch Set 1: Code-Review+1 Thanks for this fix Gabor. -- To view, visit http://gerrit.cloudera.org:8080/14665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3310abfa6f3ccbbe4c437846c6dd05791153e6f7 Gerrit-Change-Number: 14665 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Anonymous Coward (536) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 08 Nov 2019 09:28:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9131: Use single quotes around FORMAT clause in CAST
Gabor Kaszab has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14665 Change subject: IMPALA-9131: Use single quotes around FORMAT clause in CAST .. IMPALA-9131: Use single quotes around FORMAT clause in CAST When running a CAST(..FORMAT..) query then the header of the output shows the value of the FORMAT clause surrounded by double quotes. However, the SQL way is to use single quotes for strings so this patch changes the printout from using double quotes to use single quotes instead. Change-Id: I3310abfa6f3ccbbe4c437846c6dd05791153e6f7 --- M fe/src/main/java/org/apache/impala/analysis/CastExpr.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java 2 files changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/14665/1 -- To view, visit http://gerrit.cloudera.org:8080/14665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3310abfa6f3ccbbe4c437846c6dd05791153e6f7 Gerrit-Change-Number: 14665 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab
