[Impala-ASF-CR] IMPALA-8316. Update re2 to the latest version
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12778 ) Change subject: IMPALA-8316. Update re2 to the latest version .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12778 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id41ca642f5f48fd6237e13f7cab0445e0a402816 Gerrit-Change-Number: 12778 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 19 Mar 2019 04:17:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7800: Reject new connections after --fe service threads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12579 ) Change subject: IMPALA-7800: Reject new connections after --fe_service_threads .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2461/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12579 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 Gerrit-Change-Number: 12579 Gerrit-PatchSet: 7 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Tue, 19 Mar 2019 03:17:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7800: Reject new connections after --fe service threads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12579 ) Change subject: IMPALA-7800: Reject new connections after --fe_service_threads .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2462/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12579 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 Gerrit-Change-Number: 12579 Gerrit-PatchSet: 8 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Tue, 19 Mar 2019 03:21:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7800: Reject new connections after --fe service threads
Hello Michael Ho, Thomas Marshall, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12579 to look at the new patch set (#8). Change subject: IMPALA-7800: Reject new connections after --fe_service_threads .. IMPALA-7800: Reject new connections after --fe_service_threads The current implementation of the FE thrift server waits indefinitely to open the new session, if the maximum number of FE service threads specified by --fe_service_threads has been allocated. This patch introduces a startup flag to control how the server should treat new connection requests if we have run out of the configured number of server threads. If --accepted_cnxn_timeout > 0, new connection requests are rejected by the server if we can't get a server thread within the specified timeout. We set the default timeout to be 5 minutes. The old behavior can be restored by setting --accepted_cnxn_timeout=0, i.e., no timeout. The timeout applies only to client facing thrift servers, i.e., HS2 and Beeswax servers. Testing: Added a new custom cluster test suite to exercise the new code. Ran core and exhaustive tests. Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 --- M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/TAcceptQueueServer.h M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/service/impala-server.cc M common/thrift/metrics.json A tests/custom_cluster/test_frontend_connection_limit.py 7 files changed, 245 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/12579/8 -- To view, visit http://gerrit.cloudera.org:8080/12579 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 Gerrit-Change-Number: 12579 Gerrit-PatchSet: 8 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-7800: Reject new connections after --fe service threads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12579 ) Change subject: IMPALA-7800: Reject new connections after --fe_service_threads .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/12579/7/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/12579/7/be/src/service/impala-server.cc@240 PS7, Line 240: "(Advanced) The amount of time in milliseconds an accepted connection will wait in the " line too long (92 > 90) -- To view, visit http://gerrit.cloudera.org:8080/12579 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 Gerrit-Change-Number: 12579 Gerrit-PatchSet: 7 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Tue, 19 Mar 2019 02:37:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7800: Reject new connections after --fe service threads
Zoram Thanga has posted comments on this change. ( http://gerrit.cloudera.org:8080/12579 ) Change subject: IMPALA-7800: Reject new connections after --fe_service_threads .. Patch Set 7: (1 comment) Fixed http://gerrit.cloudera.org:8080/#/c/12579/7/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/12579/7/be/src/service/impala-server.cc@240 PS7, Line 240: "(Advanced) The amount of time in milliseconds an accepted connection will wait in the " > line too long (92 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/12579 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 Gerrit-Change-Number: 12579 Gerrit-PatchSet: 7 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Tue, 19 Mar 2019 02:39:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7800: Reject new connections after --fe service threads
Hello Michael Ho, Thomas Marshall, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12579 to look at the new patch set (#7). Change subject: IMPALA-7800: Reject new connections after --fe_service_threads .. IMPALA-7800: Reject new connections after --fe_service_threads The current implementation of the FE thrift server waits indefinitely to open the new session, if the maximum number of FE service threads specified by --fe_service_threads has been allocated. This patch introduces a startup flag to control how the server should treat new connection requests if we have run out of the configured number of server threads. If --accepted_cnxn_timeout > 0, new connection requests are rejected by the server if we can't get a server thread within the specified timeout. We set the default timeout to be 5 minutes. The old behavior can be restored by setting --accepted_cnxn_timeout=0, i.e., no timeout. The timeout applies only to client facing thrift servers, i.e., HS2 and Beeswax servers. Testing: Added a new custom cluster test suite to exercise the new code. Ran core and exhaustive tests. Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 --- M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/TAcceptQueueServer.h M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/service/impala-server.cc M common/thrift/metrics.json A tests/custom_cluster/test_frontend_connection_limit.py 7 files changed, 245 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/12579/7 -- To view, visit http://gerrit.cloudera.org:8080/12579 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 Gerrit-Change-Number: 12579 Gerrit-PatchSet: 7 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-7800: Reject new connections after --fe service threads
Zoram Thanga has posted comments on this change. ( http://gerrit.cloudera.org:8080/12579 ) Change subject: IMPALA-7800: Reject new connections after --fe_service_threads .. Patch Set 6: (6 comments) Addressed your comments, Michael. Please have another look. Thanks. http://gerrit.cloudera.org:8080/#/c/12579/6/be/src/rpc/TAcceptQueueServer.h File be/src/rpc/TAcceptQueueServer.h: http://gerrit.cloudera.org:8080/#/c/12579/6/be/src/rpc/TAcceptQueueServer.h@88 PS6, Line 88: void CleanupAndClose(const std::string& error, > Please add a brief comment about what this function does and what those par Done http://gerrit.cloudera.org:8080/#/c/12579/6/be/src/rpc/TAcceptQueueServer.cpp File be/src/rpc/TAcceptQueueServer.cpp: http://gerrit.cloudera.org:8080/#/c/12579/6/be/src/rpc/TAcceptQueueServer.cpp@29 PS6, Line 29: DEFINE_int64(accepted_cnxn_timeout, 300, : "(Advanced) The amount of time in seconds an accepted connection will wait in the " : "queue before we time it out and reject the connection request. A value of 0 " : "means there is no timeout."); > Sorry but now that I realize this may affect all existing Thrift based serv Changed the implementation to pass the timeout value via the constructor chain from ImpalaServer. Now only HS2 and Beeswax servers have the timeout. http://gerrit.cloudera.org:8080/#/c/12579/6/be/src/rpc/TAcceptQueueServer.cpp@195 PS6, Line 195: wait_time > nit: coding convention in Impala usually keeps the unit as the suffix of th Done http://gerrit.cloudera.org:8080/#/c/12579/6/be/src/rpc/TAcceptQueueServer.cpp@205 PS6, Line 205: msecs > nit: we either use ms or milliseconds. Done http://gerrit.cloudera.org:8080/#/c/12579/6/be/src/rpc/TAcceptQueueServer.cpp@207 PS6, Line 207: (wait_result == THRIFT_ETIMEDOUT) > Do we care about any potential non-zero result value ? Should we bail out i It does not seem to return any other non-zero value besides THRIFT_TIMEDOUT. https://github.com/apache/thrift/blob/master/lib/cpp/src/thrift/concurrency/Monitor.cpp#L78 http://gerrit.cloudera.org:8080/#/c/12579/6/be/src/rpc/TAcceptQueueServer.cpp@265 PS6, Line 265: if (FLAGS_accepted_cnxn_timeout != 0) : entry->expiration_time_ = : MonotonicMillis() + FLAGS_accepted_cnxn_timeout * MILLIS_PER_SEC; > nit styling: Done -- To view, visit http://gerrit.cloudera.org:8080/12579 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64 Gerrit-Change-Number: 12579 Gerrit-PatchSet: 6 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Tue, 19 Mar 2019 02:36:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8312 : Alter database operations have race condition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12789 ) Change subject: IMPALA-8312 : Alter database operations have race condition .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2460/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12789 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I32c8c96a6029bf9d9db37ea8315f6c9603b5a2fc Gerrit-Change-Number: 12789 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 19 Mar 2019 01:42:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8097: mt dop for all queries via hidden flag
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12257 ) Change subject: IMPALA-8097: mt_dop for all queries via hidden flag .. IMPALA-8097: mt_dop for all queries via hidden flag --unlock_mt_dop=true unlocks mt_dop for all queries including joins and inserts. This disables the parallel plans with separate join builds when running standalone, because these are not executable until IMPALA-4224 is implemented. Inserts work without modification - they were disabled because of lack of testing and the possibility for generating many small files with unpartitioned inserts - see IMPALA-8125. Testing: Add custom cluster test that exercise joins, runtime filters and inserts as a sanity check for the flag. Ran exhaustive build. Manually ran TPC-H and TPC-DS tests against a minicluster with mt_dop = 4. Change-Id: I72f0b02a005e8bf22fd17b8fb5aabf8c0d9b6b15 Reviewed-on: http://gerrit.cloudera.org:8080/12257 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/common/global-flags.cc M be/src/exec/blocking-join-node.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test A testdata/workloads/functional-query/queries/QueryTest/joins_mt_dop.test A tests/custom_cluster/test_mt_dop.py 13 files changed, 200 insertions(+), 20 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I72f0b02a005e8bf22fd17b8fb5aabf8c0d9b6b15 Gerrit-Change-Number: 12257 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8312 : Alter database operations have race condition
Vihang Karajgaonkar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12789 Change subject: IMPALA-8312 : Alter database operations have race condition .. IMPALA-8312 : Alter database operations have race condition This patch fixes a race condition in the alter database implementation in the catalogOpExecutor. The original implementation did a in-place modification of the metastore database object in the Db. This can lead to partially updated database object becoming visible to a reading thread causing potential problems. In order to fix the race, the patch makes a copy of the existing database object, modifies the copy and then atomically switches the actual database object with the modified copy. This is done while holding the metastoreddlLock, and then taking the write lock on the catalog version object which makes it consistent with the other catalog write operations. Added a test which consistently reproduces the race. The test creating many reader threads and a writer thread which continuously keeps changing the owner name and its type by issuing a alter database operation. The test fails without the patch. After the patch the test passes. The race also applies to the alter database set comment operation, although its hard to write a test for that code-path. Change-Id: I32c8c96a6029bf9d9db37ea8315f6c9603b5a2fc --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A fe/src/test/java/org/apache/impala/catalog/AlterDatabaseTest.java 4 files changed, 288 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/12789/4 -- To view, visit http://gerrit.cloudera.org:8080/12789 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I32c8c96a6029bf9d9db37ea8315f6c9603b5a2fc Gerrit-Change-Number: 12789 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8097: mt dop for all queries via hidden flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12257 ) Change subject: IMPALA-8097: mt_dop for all queries via hidden flag .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I72f0b02a005e8bf22fd17b8fb5aabf8c0d9b6b15 Gerrit-Change-Number: 12257 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 19 Mar 2019 01:20:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8316. Update re2 to the latest version
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12778 ) Change subject: IMPALA-8316. Update re2 to the latest version .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3922/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12778 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id41ca642f5f48fd6237e13f7cab0445e0a402816 Gerrit-Change-Number: 12778 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 18 Mar 2019 23:56:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8317: Add support for list type flags in Impala shell config file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12781 ) Change subject: IMPALA-8317: Add support for list type flags in Impala shell config file .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2459/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12781 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I824ca15b4e1064a391b13deef9cecd34c928ef73 Gerrit-Change-Number: 12781 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (395) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 18 Mar 2019 23:50:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/12694 ) Change subject: IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored .. Patch Set 2: Code-Review+2 Forwarding +2 -- To view, visit http://gerrit.cloudera.org:8080/12694 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0499cee7c532db19cddac3906198d965b27ea604 Gerrit-Change-Number: 12694 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 18 Mar 2019 23:53:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5393. Use THREAD LOCAL state for regexp
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/12772 ) Change subject: IMPALA-5393. Use THREAD_LOCAL state for regexp .. Patch Set 2: Code-Review+2 Forwarding +2 -- To view, visit http://gerrit.cloudera.org:8080/12772 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc331151a302e755701cb08adb3e6f289d54c3a6 Gerrit-Change-Number: 12772 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 18 Mar 2019 23:55:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5393. Use THREAD LOCAL state for regexp
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12772 ) Change subject: IMPALA-5393. Use THREAD_LOCAL state for regexp .. IMPALA-5393. Use THREAD_LOCAL state for regexp This changes the built-in regexp-related UDFs to use THREAD_LOCAL re2::RE instances instead of FRAGMENT_LOCAL. Although re2::RE is thread-safe, it achieves that thread safety through a certain amount of locking. Using thread-local regexps improves performance substantially. I ran a simple test query: select sum(l_linenumber) from item_20x where length(regexp_extract(l_shipinstruct, '.*', 0)) > 0 on a table with three underlying parquet files (thus getting 3 scanner threads). Prior to this change, the query took ~60 seconds and burned 2m16sec CPU time. With this change, it took ~19sec and 43s CPU time. For a query with more scanner threads, the improvement should be even more dramatic. The only potential downside of this change is slightly increased memory consumption by having one RE instance per thread, but the REs themselves should be small relative to all of the other per-scanner-thread memory. Change-Id: Ibc331151a302e755701cb08adb3e6f289d54c3a6 Reviewed-on: http://gerrit.cloudera.org:8080/12772 Tested-by: Impala Public Jenkins Reviewed-by: Todd Lipcon --- M be/src/exprs/string-functions-ir.cc 1 file changed, 6 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Verified Todd Lipcon: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/12772 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ibc331151a302e755701cb08adb3e6f289d54c3a6 Gerrit-Change-Number: 12772 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12694 ) Change subject: IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored .. IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored When creating a Kudu table, we would use the 'PRIMARY KEY (...)' clause to determine which columns made up the primary key, but the order of those columns would be ignored. Thus a statement like: CREATE TABLE (x int, y int, PRIMARY KEY (y, x)) STORED AS KUDU; would silently create a table with an (x,y) primary key instead of a (y,x) key. This can have substantial performance implications. This fixes the frontend to correctly throw an error in this case. This might be incompatible if someone was previously relying on the bug, but I think it's worth fixing because it was clearly doing the wrong thing. Change-Id: I0499cee7c532db19cddac3906198d965b27ea604 Reviewed-on: http://gerrit.cloudera.org:8080/12694 Tested-by: Impala Public Jenkins Reviewed-by: Todd Lipcon --- M fe/src/main/java/org/apache/impala/common/PrintUtils.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/test/java/org/apache/impala/util/PrintUtilsTest.java M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test 4 files changed, 51 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Verified Todd Lipcon: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/12694 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I0499cee7c532db19cddac3906198d965b27ea604 Gerrit-Change-Number: 12694 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8317: Add support for list type flags in Impala shell config file
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12781 ) Change subject: IMPALA-8317: Add support for list type flags in Impala shell config file .. Patch Set 4: Code-Review+1 (1 comment) Carry +1. http://gerrit.cloudera.org:8080/#/c/12781/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12781/3//COMMIT_MSG@16 PS3, Line 16: Testing: > Can you add a test to verify that the inline '--var=..' will work compatib Done -- To view, visit http://gerrit.cloudera.org:8080/12781 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I824ca15b4e1064a391b13deef9cecd34c928ef73 Gerrit-Change-Number: 12781 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (395) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 18 Mar 2019 23:05:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8317: Add support for list type flags in Impala shell config file
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/12781 ) Change subject: IMPALA-8317: Add support for list type flags in Impala shell config file .. IMPALA-8317: Add support for list type flags in Impala shell config file This patch adds support for list type flags in Impala shell config file, i.e. those that use action="append", such as --var and --query_option. To make it less error-prone, this patch also updates the logic for bool flags in the config file to also look at the correct type from the argument parser instead of relying on whether or not the default values are set in impala_shell_config_defaults.py. Testing: - Added a new test for list type flags - Ran all shell E2E tests Change-Id: I824ca15b4e1064a391b13deef9cecd34c928ef73 --- M shell/option_parser.py M tests/shell/good_impalarc M tests/shell/test_shell_commandline.py 3 files changed, 28 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/12781/4 -- To view, visit http://gerrit.cloudera.org:8080/12781 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I824ca15b4e1064a391b13deef9cecd34c928ef73 Gerrit-Change-Number: 12781 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (395) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8303: [DOCS] Take 2 Fixed the rev and id for 3.2
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12780 ) Change subject: IMPALA-8303: [DOCS] Take 2 Fixed the rev and id for 3.2 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12780 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I55913d79f603036625ff5f047c10670666d4e4e0 Gerrit-Change-Number: 12780 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 18 Mar 2019 22:15:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8303: [DOCS] Take 2 Fixed the rev and id for 3.2
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12780 ) Change subject: IMPALA-8303: [DOCS] Take 2 Fixed the rev and id for 3.2 .. IMPALA-8303: [DOCS] Take 2 Fixed the rev and id for 3.2 Change-Id: I55913d79f603036625ff5f047c10670666d4e4e0 Reviewed-on: http://gerrit.cloudera.org:8080/12780 Tested-by: Impala Public Jenkins Reviewed-by: Alex Rodoni --- M docs/topics/impala_new_features.xml 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Impala Public Jenkins: Verified Alex Rodoni: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/12780 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I55913d79f603036625ff5f047c10670666d4e4e0 Gerrit-Change-Number: 12780 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12694 ) Change subject: IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12694 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0499cee7c532db19cddac3906198d965b27ea604 Gerrit-Change-Number: 12694 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 18 Mar 2019 21:44:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8317: Add support for list type flags in Impala shell config file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12781 ) Change subject: IMPALA-8317: Add support for list type flags in Impala shell config file .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2458/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12781 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I824ca15b4e1064a391b13deef9cecd34c928ef73 Gerrit-Change-Number: 12781 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (395) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 18 Mar 2019 21:40:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8317: Add support for list type flags in Impala shell config file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12781 ) Change subject: IMPALA-8317: Add support for list type flags in Impala shell config file .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2457/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12781 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I824ca15b4e1064a391b13deef9cecd34c928ef73 Gerrit-Change-Number: 12781 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (395) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 18 Mar 2019 21:31:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8097: mt dop for all queries via hidden flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12257 ) Change subject: IMPALA-8097: mt_dop for all queries via hidden flag .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2456/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I72f0b02a005e8bf22fd17b8fb5aabf8c0d9b6b15 Gerrit-Change-Number: 12257 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 18 Mar 2019 21:29:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5393. Use THREAD LOCAL state for regexp
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12772 ) Change subject: IMPALA-5393. Use THREAD_LOCAL state for regexp .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12772 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc331151a302e755701cb08adb3e6f289d54c3a6 Gerrit-Change-Number: 12772 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 18 Mar 2019 21:12:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8317: Add support for list type flags in Impala shell config file
Anonymous Coward (395) has posted comments on this change. ( http://gerrit.cloudera.org:8080/12781 ) Change subject: IMPALA-8317: Add support for list type flags in Impala shell config file .. Patch Set 3: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/12781/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12781/3//COMMIT_MSG@16 PS3, Line 16: Testing: Can you add a test to verify that the inline '--var=..' will work compatibly with the default 'keyval' in the config_file as well? LGTM otherwise. -- To view, visit http://gerrit.cloudera.org:8080/12781 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I824ca15b4e1064a391b13deef9cecd34c928ef73 Gerrit-Change-Number: 12781 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Anonymous Coward (395) Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 18 Mar 2019 21:03:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8317: AAdd support for list type flags in Impala shell config file
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12781 Change subject: IMPALA-8317: AAdd support for list type flags in Impala shell config file .. IMPALA-8317: AAdd support for list type flags in Impala shell config file This patch adds support for list type flags in Impala shell config file, i.e. those that use action="append", such as --var and --query_option. To make it less error-prone, this patch also updates the logic for bool flags in the config file to also look at the correct type from the argument parser instead of relying on whether or not the default values are set in impala_shell_config_defaults.py. Testing: - Added a new test for list type flags - Ran all shell E2E tests Change-Id: I824ca15b4e1064a391b13deef9cecd34c928ef73 --- M shell/option_parser.py M tests/shell/good_impalarc M tests/shell/test_shell_commandline.py 3 files changed, 20 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/12781/2 -- To view, visit http://gerrit.cloudera.org:8080/12781 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I824ca15b4e1064a391b13deef9cecd34c928ef73 Gerrit-Change-Number: 12781 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8317: Add support for list type flags in Impala shell config file
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12781 ) Change subject: IMPALA-8317: Add support for list type flags in Impala shell config file .. IMPALA-8317: Add support for list type flags in Impala shell config file This patch adds support for list type flags in Impala shell config file, i.e. those that use action="append", such as --var and --query_option. To make it less error-prone, this patch also updates the logic for bool flags in the config file to also look at the correct type from the argument parser instead of relying on whether or not the default values are set in impala_shell_config_defaults.py. Testing: - Added a new test for list type flags - Ran all shell E2E tests Change-Id: I824ca15b4e1064a391b13deef9cecd34c928ef73 --- M shell/option_parser.py M tests/shell/good_impalarc M tests/shell/test_shell_commandline.py 3 files changed, 20 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/12781/3 -- To view, visit http://gerrit.cloudera.org:8080/12781 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I824ca15b4e1064a391b13deef9cecd34c928ef73 Gerrit-Change-Number: 12781 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8097: mt dop for all queries via hidden flag
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12257 ) Change subject: IMPALA-8097: mt_dop for all queries via hidden flag .. Patch Set 10: Code-Review+2 Missed updating a planner test -- To view, visit http://gerrit.cloudera.org:8080/12257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I72f0b02a005e8bf22fd17b8fb5aabf8c0d9b6b15 Gerrit-Change-Number: 12257 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 18 Mar 2019 20:45:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8097: mt dop for all queries via hidden flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12257 ) Change subject: IMPALA-8097: mt_dop for all queries via hidden flag .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3921/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I72f0b02a005e8bf22fd17b8fb5aabf8c0d9b6b15 Gerrit-Change-Number: 12257 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 18 Mar 2019 20:45:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8097: mt dop for all queries via hidden flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12257 ) Change subject: IMPALA-8097: mt_dop for all queries via hidden flag .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I72f0b02a005e8bf22fd17b8fb5aabf8c0d9b6b15 Gerrit-Change-Number: 12257 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 18 Mar 2019 20:45:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8097: mt dop for all queries via hidden flag
Hello Michael Ho, Bharath Vissapragada, Paul Rogers, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12257 to look at the new patch set (#10). Change subject: IMPALA-8097: mt_dop for all queries via hidden flag .. IMPALA-8097: mt_dop for all queries via hidden flag --unlock_mt_dop=true unlocks mt_dop for all queries including joins and inserts. This disables the parallel plans with separate join builds when running standalone, because these are not executable until IMPALA-4224 is implemented. Inserts work without modification - they were disabled because of lack of testing and the possibility for generating many small files with unpartitioned inserts - see IMPALA-8125. Testing: Add custom cluster test that exercise joins, runtime filters and inserts as a sanity check for the flag. Ran exhaustive build. Manually ran TPC-H and TPC-DS tests against a minicluster with mt_dop = 4. Change-Id: I72f0b02a005e8bf22fd17b8fb5aabf8c0d9b6b15 --- M be/src/common/global-flags.cc M be/src/exec/blocking-join-node.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test A testdata/workloads/functional-query/queries/QueryTest/joins_mt_dop.test A tests/custom_cluster/test_mt_dop.py 13 files changed, 200 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/12257/10 -- To view, visit http://gerrit.cloudera.org:8080/12257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I72f0b02a005e8bf22fd17b8fb5aabf8c0d9b6b15 Gerrit-Change-Number: 12257 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8097: mt dop for all queries via hidden flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12257 ) Change subject: IMPALA-8097: mt_dop for all queries via hidden flag .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3918/ -- To view, visit http://gerrit.cloudera.org:8080/12257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I72f0b02a005e8bf22fd17b8fb5aabf8c0d9b6b15 Gerrit-Change-Number: 12257 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 18 Mar 2019 20:25:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8316. Update re2 to the latest version
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12778 ) Change subject: IMPALA-8316. Update re2 to the latest version .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12778 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id41ca642f5f48fd6237e13f7cab0445e0a402816 Gerrit-Change-Number: 12778 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 18 Mar 2019 20:17:41 + Gerrit-HasComments: No
[Impala-ASF-CR] Use `wget http://169.254.169.254/` to determine if we're running in aws
Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/12727 ) Change subject: Use `wget http://169.254.169.254/` to determine if we're running in aws .. Patch Set 2: Code-Review+1 (1 comment) The check looks OK; see my suggestion about adding more explanation to where wget is called. http://gerrit.cloudera.org:8080/#/c/12727/2/bin/bootstrap_system.sh File bin/bootstrap_system.sh: http://gerrit.cloudera.org:8080/#/c/12727/2/bin/bootstrap_system.sh@239 PS2, Line 239: If on EC2, use Amazon's ntp servers I'd suggest expanding the comment text here with a brief description about the URL and the purpose of the connection (verifying that the instance is running on EC2). Since this script is one of the first things a new Impala contributor encounters, they might get suspicious when they see it trying to connect to an unusual URL containing a numerical IP address -- not everyone is familiar with Amazon's metadata services. Repeating what's in the commit message is probably OK, it's just that the comment here stays associated with the wget line it describes. -- To view, visit http://gerrit.cloudera.org:8080/12727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Gerrit-Change-Number: 12727 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Mon, 18 Mar 2019 19:37:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR](asf-site) Remove company affiliations: concerns of ASF trademarks@
Jim Apple has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12779 ) Change subject: Remove company affiliations: concerns of ASF trademarks@ .. Remove company affiliations: concerns of ASF trademarks@ Change-Id: I3164eb9d8bededc1c298e9c6853cb7a7dee02eb8 Reviewed-on: http://gerrit.cloudera.org:8080/12779 Reviewed-by: Thomas Marshall Tested-by: Jim Apple --- M index.html 1 file changed, 1 insertion(+), 8 deletions(-) Approvals: Thomas Marshall: Looks good to me, approved Jim Apple: Verified -- To view, visit http://gerrit.cloudera.org:8080/12779 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: merged Gerrit-Change-Id: I3164eb9d8bededc1c298e9c6853cb7a7dee02eb8 Gerrit-Change-Number: 12779 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR](asf-site) Remove company affiliations: concerns of ASF trademarks@
Jim Apple has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12779 Change subject: Remove company affiliations: concerns of ASF trademarks@ .. Remove company affiliations: concerns of ASF trademarks@ Change-Id: I3164eb9d8bededc1c298e9c6853cb7a7dee02eb8 --- M index.html 1 file changed, 1 insertion(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/12779/1 -- To view, visit http://gerrit.cloudera.org:8080/12779 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: newchange Gerrit-Change-Id: I3164eb9d8bededc1c298e9c6853cb7a7dee02eb8 Gerrit-Change-Number: 12779 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR](asf-site) Remove company affiliations: concerns of ASF trademarks@
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/12779 ) Change subject: Remove company affiliations: concerns of ASF trademarks@ .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12779 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: asf-site Gerrit-MessageType: comment Gerrit-Change-Id: I3164eb9d8bededc1c298e9c6853cb7a7dee02eb8 Gerrit-Change-Number: 12779 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Mon, 18 Mar 2019 18:37:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8303: [DOCS] Take 2 Fixed the rev and id for 3.2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12780 ) Change subject: IMPALA-8303: [DOCS] Take 2 Fixed the rev and id for 3.2 .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/279/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12780 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I55913d79f603036625ff5f047c10670666d4e4e0 Gerrit-Change-Number: 12780 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 18 Mar 2019 18:09:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8303: [DOCS] Impala 3.2 release notes
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12752 ) Change subject: IMPALA-8303: [DOCS] Impala 3.2 release notes .. Patch Set 2: > (1 comment) Fixed in a new patch. Added you as a reviewer. -- To view, visit http://gerrit.cloudera.org:8080/12752 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id50811c68ec29c218358a87288f08ad2bae16b55 Gerrit-Change-Number: 12752 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 18 Mar 2019 18:06:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8303: [DOCS] Take 2 Fixed the rev and id for 3.2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12780 ) Change subject: IMPALA-8303: [DOCS] Take 2 Fixed the rev and id for 3.2 .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/279/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12780 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I55913d79f603036625ff5f047c10670666d4e4e0 Gerrit-Change-Number: 12780 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 18 Mar 2019 18:06:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8303: [DOCS] Take 2 Fixed the rev and id for 3.2
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12780 Change subject: IMPALA-8303: [DOCS] Take 2 Fixed the rev and id for 3.2 .. IMPALA-8303: [DOCS] Take 2 Fixed the rev and id for 3.2 Change-Id: I55913d79f603036625ff5f047c10670666d4e4e0 --- M docs/topics/impala_new_features.xml 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/12780/1 -- To view, visit http://gerrit.cloudera.org:8080/12780 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I55913d79f603036625ff5f047c10670666d4e4e0 Gerrit-Change-Number: 12780 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12694 ) Change subject: IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2455/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12694 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0499cee7c532db19cddac3906198d965b27ea604 Gerrit-Change-Number: 12694 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 18 Mar 2019 18:03:58 + Gerrit-HasComments: No
[native-toolchain-CR] Add ccache support
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12574 ) Change subject: Add ccache support .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144 Gerrit-Change-Number: 12574 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 18 Mar 2019 17:59:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8316. Update re2 to the latest version
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12778 ) Change subject: IMPALA-8316. Update re2 to the latest version .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2454/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12778 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id41ca642f5f48fd6237e13f7cab0445e0a402816 Gerrit-Change-Number: 12778 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 18 Mar 2019 17:59:19 + Gerrit-HasComments: No
[native-toolchain-CR] Add ccache support
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12574 ) Change subject: Add ccache support .. Add ccache support Here we add ccache support, which speeds up building the entire toolchain. This almost works out of the box, except for the following complications: * We bootstrap our own gcc, so CCACHE_COMPILERCHECK must be set to 'content', this is because by default ccache includes the modification time of the compiler, which would prevent us to reuse the cache between runs. * We specifically set CC and CXX variables, the initial approach of setting them to 'ccache $CC' breaks boost (and possibly others) because they try to run `ccache $CC`. A second approach that i tried was relying on PATH ordering to make ccache execute the right thing. When using the system compiler, we want: PATH=$CCACHE_PATH:/usr/bin:... CC=$CCACHE_PATH/gcc After bootstrapping we want: PATH=$CCACHE_PATH:build/gcc-4.9.2:$PATH CC=$CCACHE_PATH/gcc I think complex modification of PATH is too britle and would cause ccache to execute the system compiler when $CCACHE_PATH == /usr/bin. Instead we create a wrapper around ccache that executes CC and place it at the beginning of the path, we set CC to this wrapper script. I ran `make -j DISTROS="debian8 redhat6 redhat7 sles12 ubuntu1604 ubuntu1804"` on the following configs: disabled ccache: 3.77user 1.80system 2:08:37elapsed 0%CPU (0avgtext+0avgdata 52680maxresident)k empty ccache: 3.97user 1.83system 2:15:51elapsed 0%CPU (0avgtext+0avgdata 49956maxresident)k empty ccache (compressed): 3.86user 2.05system 2:15:12elapsed 0%CPU (0avgtext+0avgdata 53824maxresident)k full ccache: 2.25user 1.79system 49:14.07elapsed 0%CPU (0avgtext+0avgdata 47520maxresident)k full ccache: (compressed): 2.40user 1.80system 49:45.01elapsed 0%CPU (0avgtext+0avgdata 51484maxresident)k Here's ccache -s after the full ccache build (statistics were cleared before strting the build). + ccache -s cache directory build_docker/ccache primary config build_docker/ccache/ccache.conf secondary config (readonly)/etc/ccache.conf cache hit (direct)138821 cache hit (preprocessed) 17437 cache miss 225 cache hit rate 99.86 % called for link15551 called for preprocessing9567 multiple source files 42 compile failed 7705 preprocessor error 2647 bad compiler arguments 6047 unsupported source language 42 autoconf compile/link 18651 unsupported compiler option 342 unsupported code directive24 no input file 4323 cleanups performed 0 files in cache289759 cache size 30.5 GB max cache size 50.0 GB Statistics using CCACHE_COMPRESS=1 are similar but cache size is ~7GB. Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144 Reviewed-on: http://gerrit.cloudera.org:8080/12574 Reviewed-by: Philip Zeyliger Tested-by: Tim Armstrong --- M .gitignore M docker/all/assert-dependencies-present.py M docker/all/postinstall.sh M docker/debian7.df M docker/debian8.df M docker/redhat6.df M docker/redhat7.df M docker/sles12.df M docker/ubuntu1204.df M docker/ubuntu1404.df M docker/ubuntu1604.df M docker/ubuntu1804.df M functions.sh M in-docker.py M init-compiler.sh M init.sh 16 files changed, 154 insertions(+), 113 deletions(-) Approvals: Philip Zeyliger: Looks good to me, approved Tim Armstrong: Verified -- To view, visit http://gerrit.cloudera.org:8080/12574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144 Gerrit-Change-Number: 12574 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/12694 ) Change subject: IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored .. Patch Set 2: (1 comment) fixed the missing javadoc and rebased http://gerrit.cloudera.org:8080/#/c/12694/1/fe/src/main/java/org/apache/impala/common/PrintUtils.java File fe/src/main/java/org/apache/impala/common/PrintUtils.java: http://gerrit.cloudera.org:8080/#/c/12694/1/fe/src/main/java/org/apache/impala/common/PrintUtils.java@205 PS1, Line 205: /** > brief comment Done -- To view, visit http://gerrit.cloudera.org:8080/12694 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0499cee7c532db19cddac3906198d965b27ea604 Gerrit-Change-Number: 12694 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 18 Mar 2019 17:23:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8316. Update re2 to the latest version
Hello Lars Volker, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/12778 to review the following change. Change subject: IMPALA-8316. Update re2 to the latest version .. IMPALA-8316. Update re2 to the latest version This updates re2 to the latest tagged release from github. I benchmarked this with a simple query: select sum(l_linenumber) from item_20x where length(regexp_extract(l_shipinstruct, '.*', 0)) > 0 Prior to the change: - TotalCpuTime: 42s848ms - wall time: ~19sec With the change: - TotalCpuTime: 33s634ms - wall time: 14-15sec Change-Id: Id41ca642f5f48fd6237e13f7cab0445e0a402816 --- M bin/impala-config.sh 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/12778/1 -- To view, visit http://gerrit.cloudera.org:8080/12778 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id41ca642f5f48fd6237e13f7cab0445e0a402816 Gerrit-Change-Number: 12778 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Lars Volker
[native-toolchain-CR] Add ccache support
Hector Acosta has posted comments on this change. ( http://gerrit.cloudera.org:8080/12574 ) Change subject: Add ccache support .. Patch Set 2: > Hector, are you blocked on something? Or just waiting for someone > to merge? Just waiting on someone to merge. -- To view, visit http://gerrit.cloudera.org:8080/12574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144 Gerrit-Change-Number: 12574 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 18 Mar 2019 17:14:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12694 ) Change subject: IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3920/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12694 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0499cee7c532db19cddac3906198d965b27ea604 Gerrit-Change-Number: 12694 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 18 Mar 2019 17:23:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored
Hello Thomas Marshall, Jean-Daniel Cryans, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12694 to look at the new patch set (#2). Change subject: IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored .. IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored When creating a Kudu table, we would use the 'PRIMARY KEY (...)' clause to determine which columns made up the primary key, but the order of those columns would be ignored. Thus a statement like: CREATE TABLE (x int, y int, PRIMARY KEY (y, x)) STORED AS KUDU; would silently create a table with an (x,y) primary key instead of a (y,x) key. This can have substantial performance implications. This fixes the frontend to correctly throw an error in this case. This might be incompatible if someone was previously relying on the bug, but I think it's worth fixing because it was clearly doing the wrong thing. Change-Id: I0499cee7c532db19cddac3906198d965b27ea604 --- M fe/src/main/java/org/apache/impala/common/PrintUtils.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/test/java/org/apache/impala/util/PrintUtilsTest.java M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test 4 files changed, 51 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/12694/2 -- To view, visit http://gerrit.cloudera.org:8080/12694 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0499cee7c532db19cddac3906198d965b27ea604 Gerrit-Change-Number: 12694 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jean-Daniel Cryans Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-5393. Use THREAD LOCAL state for regexp
Hello Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12772 to look at the new patch set (#2). Change subject: IMPALA-5393. Use THREAD_LOCAL state for regexp .. IMPALA-5393. Use THREAD_LOCAL state for regexp This changes the built-in regexp-related UDFs to use THREAD_LOCAL re2::RE instances instead of FRAGMENT_LOCAL. Although re2::RE is thread-safe, it achieves that thread safety through a certain amount of locking. Using thread-local regexps improves performance substantially. I ran a simple test query: select sum(l_linenumber) from item_20x where length(regexp_extract(l_shipinstruct, '.*', 0)) > 0 on a table with three underlying parquet files (thus getting 3 scanner threads). Prior to this change, the query took ~60 seconds and burned 2m16sec CPU time. With this change, it took ~19sec and 43s CPU time. For a query with more scanner threads, the improvement should be even more dramatic. The only potential downside of this change is slightly increased memory consumption by having one RE instance per thread, but the REs themselves should be small relative to all of the other per-scanner-thread memory. Change-Id: Ibc331151a302e755701cb08adb3e6f289d54c3a6 --- M be/src/exprs/string-functions-ir.cc 1 file changed, 6 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/12772/2 -- To view, visit http://gerrit.cloudera.org:8080/12772 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibc331151a302e755701cb08adb3e6f289d54c3a6 Gerrit-Change-Number: 12772 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[native-toolchain-CR] IMPALA-8316. Update re2 to latest version
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12771 ) Change subject: IMPALA-8316. Update re2 to latest version .. IMPALA-8316. Update re2 to latest version Change-Id: I41f749500fb5e86d3c46621177d2ed256b8a4130 Reviewed-on: http://gerrit.cloudera.org:8080/12771 Reviewed-by: Lars Volker Tested-by: Todd Lipcon --- M buildall.sh 1 file changed, 2 insertions(+), 1 deletion(-) Approvals: Lars Volker: Looks good to me, approved Todd Lipcon: Verified -- To view, visit http://gerrit.cloudera.org:8080/12771 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I41f749500fb5e86d3c46621177d2ed256b8a4130 Gerrit-Change-Number: 12771 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Todd Lipcon
[native-toolchain-CR] IMPALA-8316. Update re2 to latest version
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/12771 ) Change subject: IMPALA-8316. Update re2 to latest version .. Patch Set 1: Verified+1 Build ran and artifacts are now on S3 with id 255-58192fb6bc -- To view, visit http://gerrit.cloudera.org:8080/12771 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I41f749500fb5e86d3c46621177d2ed256b8a4130 Gerrit-Change-Number: 12771 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 18 Mar 2019 17:07:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5843: Use page index in Parquet files to skip pages
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12065 ) Change subject: IMPALA-5843: Use page index in Parquet files to skip pages .. Patch Set 7: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2453/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a Gerrit-Change-Number: 12065 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 18 Mar 2019 17:04:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5393. Use THREAD LOCAL state for regexp
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12772 ) Change subject: IMPALA-5393. Use THREAD_LOCAL state for regexp .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3919/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12772 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc331151a302e755701cb08adb3e6f289d54c3a6 Gerrit-Change-Number: 12772 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 18 Mar 2019 17:03:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5393. Use THREAD LOCAL state for regexp
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/12772 ) Change subject: IMPALA-5393. Use THREAD_LOCAL state for regexp .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/12772/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12772/1//COMMIT_MSG@10 PS1, Line 10: instea > nit: typo Done -- To view, visit http://gerrit.cloudera.org:8080/12772 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc331151a302e755701cb08adb3e6f289d54c3a6 Gerrit-Change-Number: 12772 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 18 Mar 2019 17:03:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8305: Generate JUnitXML for DCHECK failures
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12751 ) Change subject: IMPALA-8305: Generate JUnitXML for DCHECK failures .. IMPALA-8305: Generate JUnitXML for DCHECK failures DCHECKs log at the FATAL level and produce a statement starting with "Check failed:". This generates a JUnitXML when it sees a FATAL log file with "Check failed:". Some backend death tests specifically test hitting certain DCHECKs, so the be_tests directory is ignored. Something that hits a DCHECK would also generate a minidump, so this is just additional information beyond the existing JUnitXML for the minidump. Testing: - Tested on logs from a test that hit a DCHECK in impalad - Ran a normal error-free run Change-Id: Ic45d01b6a87b1d81967f1e10028af96a7cc3071a Reviewed-on: http://gerrit.cloudera.org:8080/12751 Reviewed-by: David Knupp Tested-by: Impala Public Jenkins --- M bin/jenkins/finalize.sh 1 file changed, 15 insertions(+), 0 deletions(-) Approvals: David Knupp: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12751 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic45d01b6a87b1d81967f1e10028af96a7cc3071a Gerrit-Change-Number: 12751 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/6521 ) Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. Patch Set 6: This is still relevant if someone has patience to read through all the bash. -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 18 Mar 2019 16:44:56 + Gerrit-HasComments: No
[native-toolchain-CR] Add ccache support
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12574 ) Change subject: Add ccache support .. Patch Set 2: Hector, are you blocked on something? Or just waiting for someone to merge? -- To view, visit http://gerrit.cloudera.org:8080/12574 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ieef291d1294a204b0b8da7e7aa4fa642cdd5e144 Gerrit-Change-Number: 12574 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 18 Mar 2019 16:44:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/12481 ) Change subject: IMPALA-7368: Add initial support for DATE type .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/12481/11/be/src/exprs/aggregate-functions-ir.cc File be/src/exprs/aggregate-functions-ir.cc: http://gerrit.cloudera.org:8080/#/c/12481/11/be/src/exprs/aggregate-functions-ir.cc@381 PS11, Line 381: val_struct->sum / val_struct->count > I agree avg(date) is likely not important. Kind-of nice to have for consist Yea, my feeling was just that one fewer feature to test and bug-fix and support is good if it's not an important use case. -- To view, visit http://gerrit.cloudera.org:8080/12481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f Gerrit-Change-Number: 12481 Gerrit-PatchSet: 11 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 18 Mar 2019 16:36:31 + Gerrit-HasComments: Yes
[native-toolchain-CR] IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Hello Tianyi Wang, Philip Zeyliger, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/6521 to look at the new patch set (#6). Change subject: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so C++ packages depend on these shared objects. Our toolchain packages should always be run using the toolchain version of the shared objects that it ships. Previously the toolchain artifacts were often (but not always) linked against system version of libgcc.so and libstdc++.so, which can cause compatibility problems. This is a major problem on recent Linux distros like Ubuntu 16.04 that default to the new C++11 ABI because the system libstdc++.so is not compatible with gcc-4.9.2-generated binaries. It also means behaviour of toolchain artifacts may be less consistent across systems. This patch does two things to ensure that toolchain artifacts link to the correct libraries: 1. fixes the RPATHs added to executables and shared objects so that they always point to the ../lib/ and ../lib64/ directories. 2. adds symlinks from the lib/ directory to libstd++.so and libgcc.so where required by executables and shared objects. Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 --- M functions.sh M init-compiler.sh M source/flatbuffers/build.sh M source/kudu/build.sh 4 files changed, 122 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/21/6521/6 -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8097: mt dop for all queries via hidden flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12257 ) Change subject: IMPALA-8097: mt_dop for all queries via hidden flag .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2452/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I72f0b02a005e8bf22fd17b8fb5aabf8c0d9b6b15 Gerrit-Change-Number: 12257 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 18 Mar 2019 16:42:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5843: Use page index in Parquet files to skip pages
Hello Michael Ho, Lars Volker, Pooja Nilangekar, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12065 to look at the new patch set (#7). Change subject: IMPALA-5843: Use page index in Parquet files to skip pages .. IMPALA-5843: Use page index in Parquet files to skip pages This commit implements page filtering based on the Parquet page index. The read and evaluation of the page index is done by the HdfsParquetScanner. At first, we determine the row ranges we are interested in, and based on the row ranges we determine the candidate pages for each column that we are reading. We still issue one ScanRange per column chunk, but we specify sub-ranges that store the candidate pages, i.e. we don't read the whole column chunk, but only fractions of it. Pages are not aligned across column chunks, i.e. page #2 of column A might store completely different rows than page #2 of column B. It means we need to implement some kind of row-skipping logic when we read the data pages. This logic is implemented in BaseScalarColumnReader and ScalarColumnReader. Collection column readers know nothing about page filtering. Page filtering can be turned off by setting the query option 'read_parquet_page_index' to false. Testing: * added added some unit tests for the row range and page selection logic * generated various Parquet files with Parquet-MR * enabled Page index writing and wrote selective queries against tables written by Impala. Current tests are likely to use page filtering transparently. Performance: * measured locally, observed 3x to 10x speedup for selective queries TODO: * run standard benchmarks * measure performance for remote reads Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a --- M be/src/common/global-flags.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/parquet/CMakeLists.txt M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h M be/src/exec/parquet/parquet-column-readers.cc M be/src/exec/parquet/parquet-column-readers.h M be/src/exec/parquet/parquet-column-stats.cc M be/src/exec/parquet/parquet-column-stats.h A be/src/exec/parquet/parquet-common-test.cc M be/src/exec/parquet/parquet-common.cc M be/src/exec/parquet/parquet-common.h M be/src/exec/parquet/parquet-level-decoder.h A be/src/exec/parquet/parquet-page-index.cc A be/src/exec/parquet/parquet-page-index.h M be/src/exprs/literal.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/dict-encoding.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M testdata/data/README A testdata/data/alltypes_tiny_pages.parquet A testdata/data/decimals_1_10.parquet A testdata/data/double_nested_decimals.parquet A testdata/data/nested_decimals.parquet A testdata/workloads/functional-query/queries/QueryTest/nested-types-parquet-page-index.test A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index-alltypes-tiny-pages.test A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index-large.test A testdata/workloads/functional-query/queries/QueryTest/parquet-page-index.test M testdata/workloads/functional-query/queries/QueryTest/stats-extrapolation.test M tests/query_test/test_parquet_stats.py 33 files changed, 2,662 insertions(+), 80 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/12065/7 -- To view, visit http://gerrit.cloudera.org:8080/12065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0cc99f129f2048dbafbe7f5a51d1ea3a5005731a Gerrit-Change-Number: 12065 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12481 ) Change subject: IMPALA-7368: Add initial support for DATE type .. Patch Set 11: (3 comments) http://gerrit.cloudera.org:8080/#/c/12481/11//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12481/11//COMMIT_MSG@70 PS11, Line 70: complete DATE type implementation > should we consider turning it off by default and only enabling it once we'r It seems easier to leave it enabled so long as the current support is correct. I think we should convince ourselves that the testing is adequate before merging this though. http://gerrit.cloudera.org:8080/#/c/12481/11//COMMIT_MSG@72 PS11, Line 72: - Add date support to the random query generator. > should we transition TPC-DS test tables over to 'date' instead of 'string' Seems reasonable to do once the support is complete, I don't think it's a requirement to call DATE support "complete" though. http://gerrit.cloudera.org:8080/#/c/12481/11/be/src/exprs/aggregate-functions-ir.cc File be/src/exprs/aggregate-functions-ir.cc: http://gerrit.cloudera.org:8080/#/c/12481/11/be/src/exprs/aggregate-functions-ir.cc@381 PS11, Line 381: val_struct->sum / val_struct->count > Is this correct behavior when the dates are negative? Dividing as signed in I agree avg(date) is likely not important. Kind-of nice to have for consistency but I guess if there's any concerns about the semantics we can just avoid the issues by not having it. -- To view, visit http://gerrit.cloudera.org:8080/12481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f Gerrit-Change-Number: 12481 Gerrit-PatchSet: 11 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 18 Mar 2019 16:13:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12481 ) Change subject: IMPALA-7368: Add initial support for DATE type .. Patch Set 10: (1 comment) http://gerrit.cloudera.org:8080/#/c/12481/10/tests/query_test/test_date_queries.py File tests/query_test/test_date_queries.py: http://gerrit.cloudera.org:8080/#/c/12481/10/tests/query_test/test_date_queries.py@107 PS10, Line 107: def test_impala_shell(self): We don't need to test this for multiple file formats, right? -- To view, visit http://gerrit.cloudera.org:8080/12481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f Gerrit-Change-Number: 12481 Gerrit-PatchSet: 10 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 18 Mar 2019 16:08:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12481 ) Change subject: IMPALA-7368: Add initial support for DATE type .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/12481/11//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12481/11//COMMIT_MSG@29 PS11, Line 29: - from STRING to DATE if the source string value is used in a : context where a DATE value is expected. > it's interesting that this differs from the behavior of TIMESTAMP today, at The interval implementation in Impala is kind-of weird because it's implemented as a special operator instead of as an INTERVAL data type. I don't think this is related to the (lack of) implicit casting but it does relate to the standard support. -- To view, visit http://gerrit.cloudera.org:8080/12481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f Gerrit-Change-Number: 12481 Gerrit-PatchSet: 11 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 18 Mar 2019 16:02:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8097: mt dop for all queries via hidden flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12257 ) Change subject: IMPALA-8097: mt_dop for all queries via hidden flag .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3918/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I72f0b02a005e8bf22fd17b8fb5aabf8c0d9b6b15 Gerrit-Change-Number: 12257 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 18 Mar 2019 15:57:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8097: mt dop for all queries via hidden flag
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12257 ) Change subject: IMPALA-8097: mt_dop for all queries via hidden flag .. Patch Set 9: Code-Review+2 (2 comments) I'm going to promote the 3 +1s to a +2. Scream if you disagree. http://gerrit.cloudera.org:8080/#/c/12257/8//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12257/8//COMMIT_MSG@14 PS8, Line 14: until IMPALA-4224 is implemented. Inserts work without > Also, as the way it's implemented now, the memory consumption for broadcast Yes http://gerrit.cloudera.org:8080/#/c/12257/8/be/src/exec/partitioned-hash-join-builder.cc File be/src/exec/partitioned-hash-join-builder.cc: http://gerrit.cloudera.org:8080/#/c/12257/8/be/src/exec/partitioned-hash-join-builder.cc@108 PS8, Line 108: // TODO: IMPALA-4400 - implement local aggregation of runtime filters. > May want to add a reference to IMPALA-4400 which is abput aggregating the f Done -- To view, visit http://gerrit.cloudera.org:8080/12257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I72f0b02a005e8bf22fd17b8fb5aabf8c0d9b6b15 Gerrit-Change-Number: 12257 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 18 Mar 2019 15:57:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8097: mt dop for all queries via hidden flag
Hello Michael Ho, Bharath Vissapragada, Paul Rogers, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12257 to look at the new patch set (#9). Change subject: IMPALA-8097: mt_dop for all queries via hidden flag .. IMPALA-8097: mt_dop for all queries via hidden flag --unlock_mt_dop=true unlocks mt_dop for all queries including joins and inserts. This disables the parallel plans with separate join builds when running standalone, because these are not executable until IMPALA-4224 is implemented. Inserts work without modification - they were disabled because of lack of testing and the possibility for generating many small files with unpartitioned inserts - see IMPALA-8125. Testing: Add custom cluster test that exercise joins, runtime filters and inserts as a sanity check for the flag. Ran exhaustive build. Manually ran TPC-H and TPC-DS tests against a minicluster with mt_dop = 4. Change-Id: I72f0b02a005e8bf22fd17b8fb5aabf8c0d9b6b15 --- M be/src/common/global-flags.cc M be/src/exec/blocking-join-node.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java A testdata/workloads/functional-query/queries/QueryTest/joins_mt_dop.test A tests/custom_cluster/test_mt_dop.py 12 files changed, 122 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/12257/9 -- To view, visit http://gerrit.cloudera.org:8080/12257 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I72f0b02a005e8bf22fd17b8fb5aabf8c0d9b6b15 Gerrit-Change-Number: 12257 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8303: [DOCS] Impala 3.2 release notes
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/12752 ) Change subject: IMPALA-8303: [DOCS] Impala 3.2 release notes .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/12752/2/docs/topics/impala_new_features.xml File docs/topics/impala_new_features.xml: http://gerrit.cloudera.org:8080/#/c/12752/2/docs/topics/impala_new_features.xml@49 PS2, Line 49: 3.1.0 Sorry for bothering you after I gave a +2 but I think this should be 3.2.0 here. -- To view, visit http://gerrit.cloudera.org:8080/12752 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id50811c68ec29c218358a87288f08ad2bae16b55 Gerrit-Change-Number: 12752 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 18 Mar 2019 09:19:45 + Gerrit-HasComments: Yes
