[Impala-ASF-CR] IMPALA-6395: Add a flag for data stream sender's buffer size
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9026 ) Change subject: IMPALA-6395: Add a flag for data stream sender's buffer size .. IMPALA-6395: Add a flag for data stream sender's buffer size This change adds a flag to control the maximum size in bytes a row batch in a data stream sender's channel can accumulate before the row batch is sent over the wire. Increasing this value will better amortize the cost of compression and RPC per row batch. The default value is 16KB per channel. Change-Id: I385f4b7a0671bb2d7872bee60d476c375680b5c2 Reviewed-on: http://gerrit.cloudera.org:8080/9026 Reviewed-by: Michael Ho Tested-by: Impala Public Jenkins --- M be/src/exec/data-sink.cc 1 file changed, 7 insertions(+), 3 deletions(-) Approvals: Michael Ho: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/9026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I385f4b7a0671bb2d7872bee60d476c375680b5c2 Gerrit-Change-Number: 9026 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6395: Add a flag for data stream sender's buffer size
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9026 ) Change subject: IMPALA-6395: Add a flag for data stream sender's buffer size .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/9026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I385f4b7a0671bb2d7872bee60d476c375680b5c2 Gerrit-Change-Number: 9026 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jan 2018 06:56:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4924: Enable Decimal V2 by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9062 ) Change subject: IMPALA-4924: Enable Decimal V2 by default .. Patch Set 6: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1794/ -- To view, visit http://gerrit.cloudera.org:8080/9062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibbdd05bf986b7947f106b396017faa3a0bd87fd7 Gerrit-Change-Number: 9062 Gerrit-PatchSet: 6 Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zach Amsden Gerrit-Comment-Date: Wed, 24 Jan 2018 05:26:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6435: Disable codegen for CHAR literals.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9102 ) Change subject: IMPALA-6435: Disable codegen for CHAR literals. .. IMPALA-6435: Disable codegen for CHAR literals. Currently we do not codegen CHAR types. This change checks for CHAR literals in a expr and disables codegen. Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 Reviewed-on: http://gerrit.cloudera.org:8080/9102 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M be/src/exprs/literal.cc M testdata/workloads/functional-query/queries/QueryTest/disable-codegen.test 2 files changed, 20 insertions(+), 1 deletion(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/9102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 Gerrit-Change-Number: 9102 Gerrit-PatchSet: 5 Gerrit-Owner: anujphadke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-ASF-CR] IMPALA-6435: Disable codegen for CHAR literals.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9102 ) Change subject: IMPALA-6435: Disable codegen for CHAR literals. .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/9102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 Gerrit-Change-Number: 9102 Gerrit-PatchSet: 4 Gerrit-Owner: anujphadke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Wed, 24 Jan 2018 05:00:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5191: Standardize column alias behavior
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8801 ) Change subject: IMPALA-5191: Standardize column alias behavior .. Patch Set 14: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1792/ -- To view, visit http://gerrit.cloudera.org:8080/8801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0f82483b486acf6953876cfa672b0d034f3709a8 Gerrit-Change-Number: 8801 Gerrit-PatchSet: 14 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 24 Jan 2018 04:11:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6410: Tool to cherrypick changes across branches.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9045 ) Change subject: IMPALA-6410: Tool to cherrypick changes across branches. .. IMPALA-6410: Tool to cherrypick changes across branches. This script compares two branches and optionally cherry-picks changes across. It uses the Gerrit Change-Id as the key, and it supports a configuration file and a string to ignore commits. Change-Id: I6120ec2d6e914a1e5fda568178b32aafda8722a9 Reviewed-on: http://gerrit.cloudera.org:8080/9045 Reviewed-by: Jim Apple Tested-by: Impala Public Jenkins --- A bin/compare_branches.py A bin/ignored_commits.json 2 files changed, 274 insertions(+), 0 deletions(-) Approvals: Jim Apple: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/9045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6120ec2d6e914a1e5fda568178b32aafda8722a9 Gerrit-Change-Number: 9045 Gerrit-PatchSet: 6 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Taras Bobrovytsky
[Impala-ASF-CR] IMPALA-6410: Tool to cherrypick changes across branches.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9045 ) Change subject: IMPALA-6410: Tool to cherrypick changes across branches. .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/9045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6120ec2d6e914a1e5fda568178b32aafda8722a9 Gerrit-Change-Number: 9045 Gerrit-PatchSet: 5 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Comment-Date: Wed, 24 Jan 2018 03:34:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption
Xianda Ke has posted comments on this change. ( http://gerrit.cloudera.org:8080/9032 ) Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption .. Patch Set 9: Thank Tim a lot for the review. @Sailesh, would you please help review IsModeSupported() in openssl-util.cc. -- To view, visit http://gerrit.cloudera.org:8080/9032 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24 Gerrit-Change-Number: 9032 Gerrit-PatchSet: 9 Gerrit-Owner: Xianda Ke Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Xianda Ke Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 24 Jan 2018 03:21:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6395: Add a flag for data stream sender's buffer size
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9026 ) Change subject: IMPALA-6395: Add a flag for data stream sender's buffer size .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1795/ -- To view, visit http://gerrit.cloudera.org:8080/9026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I385f4b7a0671bb2d7872bee60d476c375680b5c2 Gerrit-Change-Number: 9026 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jan 2018 03:14:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6395: Add a flag for data stream sender's buffer size
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/9026 ) Change subject: IMPALA-6395: Add a flag for data stream sender's buffer size .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/9026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I385f4b7a0671bb2d7872bee60d476c375680b5c2 Gerrit-Change-Number: 9026 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jan 2018 03:10:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6395: Add a flag for data stream sender's buffer size
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/9026 ) Change subject: IMPALA-6395: Add a flag for data stream sender's buffer size .. Patch Set 2: GVO failed with IMPALA-6399 -- To view, visit http://gerrit.cloudera.org:8080/9026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I385f4b7a0671bb2d7872bee60d476c375680b5c2 Gerrit-Change-Number: 9026 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jan 2018 03:10:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5237: Support a quoted string in date/time format
Kim Jin Chul has posted comments on this change. ( http://gerrit.cloudera.org:8080/8508 ) Change subject: IMPALA-5237: Support a quoted string in date/time format .. Patch Set 6: The problem at IMPALA-3942 has been resolved. Hence, let me continue this task and provide a new patch set soon. -- To view, visit http://gerrit.cloudera.org:8080/8508 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie34055ac695748bcfb110bfa6ed5308f469ea178 Gerrit-Change-Number: 8508 Gerrit-PatchSet: 6 Gerrit-Owner: Kim Jin Chul Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Kim Jin Chul Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jan 2018 03:02:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3193: Show table's comment on show tables
Kim Jin Chul has posted comments on this change. ( http://gerrit.cloudera.org:8080/8851 ) Change subject: IMPALA-3193: Show table's comment on show tables .. Patch Set 7: (12 comments) Applied update. http://gerrit.cloudera.org:8080/#/c/8851/7/be/src/catalog/catalog.h File be/src/catalog/catalog.h: http://gerrit.cloudera.org:8080/#/c/8851/7/be/src/catalog/catalog.h@79 PS7, Line 79: Returns all matching table names and table comments, per Hive's : /// "SHOW TABLES ". Each table name of TTableInfo returned is unqualified. : /// Comment cannot be set when table is not loaded. If pattern is NULL, match all : /// tables otherwise match only those tables that match the pattern string. Patterns : /// are "p1|p2|p3" where | denotes choice, and each pN may contain wildcards denoted : /// by '*' which match all strings. Table comments can be empty if table is not loaded. > Let's fix this comment a bit: Done http://gerrit.cloudera.org:8080/#/c/8851/7/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/8851/7/be/src/service/client-request-state.cc@280 PS7, Line 280: if (!tbl_info.__isset.comment) DCHECK(tbl_info.comment.empty()); : comments.push_back(tbl_info.comment); > This check is a bit confusing. Ideally, you want to ensure that names and c Sorry, actually I didn't catch your intention for adding DCHECK in the previous comment. I still don't understand your last sentence. Why should we consider this(i.e. check)? As far as I understand, comment should be either empty(table is not loaded case or a given empty string as a comment) or not empty. Hence, we just put it to comments. Anyway, we should guarantee that the number of names should be equal to the number of comments. Let me remove the DCHECK code line which seems to be meaningless now. http://gerrit.cloudera.org:8080/#/c/8851/7/be/src/service/frontend.h File be/src/service/frontend.h: http://gerrit.cloudera.org:8080/#/c/8851/7/be/src/service/frontend.h@57 PS7, Line 57: /// Returns all matching table names and table comments. : /// If pattern is NULL, match all tables otherwise match only those tables that : /// match the pattern string. Patterns are "p1|p2|p3" where | denotes choice, : /// and each pN may contain wildcards denoted by '*' which match all strings. : /// The TSessionState parameter is used to filter results of metadata operations when : /// authorization is enabled. If this is a user initiated request, it should : /// be set to the user's current session. If this is an Impala internal request, : /// the session should be set to NULL which will skip privilege checks returning all : /// results. Table comments can be empty if table is not loaded. > See previous comment about this comment :). Done http://gerrit.cloudera.org:8080/#/c/8851/7/be/src/service/frontend.cc File be/src/service/frontend.cc: http://gerrit.cloudera.org:8080/#/c/8851/7/be/src/service/frontend.cc@145 PS7, Line 145: return JniUtil::CallJniMethod(fe_, get_tables_info_id_, params, : tables_info_result); > nit: check if it fits in a single line Done http://gerrit.cloudera.org:8080/#/c/8851/7/common/thrift/Frontend.thrift File common/thrift/Frontend.thrift: http://gerrit.cloudera.org:8080/#/c/8851/7/common/thrift/Frontend.thrift@91 PS7, Line 91: Comment needs to indicate when and if this is set. For instance, : // it's not clear if this is set if the table is loaded but there is no comment. > I believe this was my comment. The point was to explicitly specify the beha Done. Sorry, it's my mistake. I was confusing table 'comment' and your 'comment'. http://gerrit.cloudera.org:8080/#/c/8851/7/common/thrift/Frontend.thrift@96 PS7, Line 96: // getTablesInfo returns a list of table names and a list of table comment. > comments is not consistent with the struct definition, plz update Done http://gerrit.cloudera.org:8080/#/c/8851/7/common/thrift/Frontend.thrift@98 PS7, Line 98: tableinfos > tables_info Done http://gerrit.cloudera.org:8080/#/c/8851/6/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/8851/6/fe/src/main/java/org/apache/impala/catalog/Table.java@407 PS6, Line 407: blic > My thinking is that we're not particularly consistent with using final in o Thanks for sharing your idea. It's just my curiosity. I want to know the reason why you thought. Actually I didn't find the use of final keyword elsewhere except for class member variable. I would like to follow other authors' preference. http://gerrit.cloudera.org:8080/#/c/8851/7/fe/src/main/java/org/apache/impala/service
[Impala-ASF-CR] IMPALA-3193: Show table's comment on show tables
Hello Dimitris Tsirogiannis, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8851 to look at the new patch set (#8). Change subject: IMPALA-3193: Show table's comment on show tables .. IMPALA-3193: Show table's comment on show tables Currently SHOW TABLES command does not display given comment when creating a table. DESC shows column's comment and SHOW DATABASES also shows database's comment. On this change, table's comment is displayed if the table is loaded. If the table is not loaded, an empty comment is returned. Testing: Adds E2E test: TestShowTables.testTableCommentVisibility Change-Id: I44f814af05db6f3c027718ade9f474f8b8153bcd --- M be/src/catalog/catalog-server.cc M be/src/catalog/catalog.cc M be/src/catalog/catalog.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java M testdata/workloads/functional-query/queries/QueryTest/show.test M tests/metadata/test_metadata_query_statements.py 18 files changed, 304 insertions(+), 197 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/8851/8 -- To view, visit http://gerrit.cloudera.org:8080/8851 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I44f814af05db6f3c027718ade9f474f8b8153bcd Gerrit-Change-Number: 8851 Gerrit-PatchSet: 8 Gerrit-Owner: Kim Jin Chul Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Kim Jin Chul
[Impala-ASF-CR] IMPALA-6215: Removes race when using LibCache.
Hello Tim Armstrong, Bikramjeet Vig, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9089 to look at the new patch set (#3). Change subject: IMPALA-6215: Removes race when using LibCache. .. IMPALA-6215: Removes race when using LibCache. LibCache's api to provide access to locally cached files has a race. Currently, the client of the cache accesses the locally cached path as a string, but nothing guarantees that the associated file is not removed before the client is done using it. This race is suspected as the root cause for the flakiness seen in IMPALA-6092. These tests fail once in a while with classloader errors unable to load java udf classes. In these tests, the lib cache makes no guarantee that the path to the jar will remain valid from the time the path is acquired through the time needed to fetch the jar and resolve the needed classes. LibCache offers liveness guarantees for shared objects via reference counting. The fix in this patch extends this API to also cover paths to locally cached files. Testing: - added a test to test_udfs.py that does many concurrent udf uses and removals. By increasing the concurrent operations to 100, the issue in IMPALA-6092 is locally reproducible on every run. With this fix, the problem is no longer reproducible with this test. Change-Id: I9175085201fe8b11424ab8f88d7b992cb7b0daea --- M be/src/codegen/llvm-codegen.cc M be/src/exec/external-data-source-executor.cc M be/src/exprs/hive-udf-call.cc M be/src/exprs/hive-udf-call.h M be/src/runtime/lib-cache.cc M be/src/runtime/lib-cache.h M be/src/service/fe-support.cc M tests/query_test/test_udfs.py 8 files changed, 167 insertions(+), 52 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/89/9089/3 -- To view, visit http://gerrit.cloudera.org:8080/9089 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9175085201fe8b11424ab8f88d7b992cb7b0daea Gerrit-Change-Number: 9089 Gerrit-PatchSet: 3 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-6215: Removes race when using LibCache.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/9089 ) Change subject: IMPALA-6215: Removes race when using LibCache. .. Patch Set 2: (2 comments) Absolutely agree that lib-cache api is bit clunky. Main thing with this patch is the fix so that we can run the tests as before (I've reverted my previous change in this patch). I've kept the api the same as GetSoFunctionPtr for uniformity but agree that this should be simplified. I'll file a JIRA-- what's your favorite current handle api in the code base for reference? http://gerrit.cloudera.org:8080/#/c/9089/2/be/src/codegen/llvm-codegen.cc File be/src/codegen/llvm-codegen.cc: http://gerrit.cloudera.org:8080/#/c/9089/2/be/src/codegen/llvm-codegen.cc@333 PS2, Line 333: MakeScopeExitTrigger([entry]() { : if (entry != nullptr) LibCache::instance()->DecrementUseCount(entry);} : ); > Isn't LibCacheEntry shared between all of the callers? yes, it's shared between callers (same deal with the GetSoFunctionPtr) http://gerrit.cloudera.org:8080/#/c/9089/2/tests/query_test/test_udfs.py File tests/query_test/test_udfs.py: http://gerrit.cloudera.org:8080/#/c/9089/2/tests/query_test/test_udfs.py@443 PS2, Line 443: create_fn_to_drop = "create function {0}.foo_{1}() returns string LOCATION '{2}' SYMBOL='org.apache.impala.TestUpdateUdf'" : create_fn_to_use = "create function {0}.use_it(string) returns string LOCATION '{1}' SYMBOL='org.apache.impala.TestUdf'" : drop_fn = "drop function if exists {0}.foo_{1}()" : use_fn = "select * from (select max(int_col) from functional.alltypesagg where {0}.use_it(string_col) = 'blah' union all (select max(int_col) from functional.alltypesagg where {0}.use_it(String_col) > '1' union all (select max(int_col) from functional.alltypesagg where {0}.use_it(string_col) > '1'))) v" : > nit: long lines Done -- To view, visit http://gerrit.cloudera.org:8080/9089 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9175085201fe8b11424ab8f88d7b992cb7b0daea Gerrit-Change-Number: 9089 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 24 Jan 2018 02:56:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6190/6246: Add instances tab and event sequence
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8758 ) Change subject: IMPALA-6190/6246: Add instances tab and event sequence .. IMPALA-6190/6246: Add instances tab and event sequence This change adds tracking of the current state during the execution of a fragment instance. The current state is then reported back to the coordinator and exposed to users via a new tab in the query detail debug webpage. This change also adds an event timeline to fragment instances in the query profile. The timeline measures the time since backend-local query start at which particular events complete. Events are derived from the current state of the execution of a fragment instance. For example: - Prepare Finished: 13.436ms (13.436ms) - First Batch Produced: 1s022ms (1s008ms) - First Batch Sent: 1s022ms (455.558us) - ExecInternal Finished: 2s783ms (1s760ms) I added automated tests for both extensions and additionally verified the change by manual inspection. Here are the TPCH performance comparison results between this change and the previous commit on a 16 node cluster. ++---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | ++---+-++++ | TPCH(_300) | parquet / none / none | 18.47 | -0.94% | 9.72 | -1.08% | ++---+-++++ ++--+---++-++++-+---+ | Workload | Query| File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Num Clients | Iters | ++--+---++-++++-+---+ | TPCH(_300) | TPCH-Q5 | parquet / none / none | 48.88 | 46.93 | +4.15% | 0.14%| 3.61%| 1 | 3 | | TPCH(_300) | TPCH-Q13 | parquet / none / none | 21.64 | 21.15 | +2.29% | 2.06%| 1.84%| 1 | 3 | | TPCH(_300) | TPCH-Q11 | parquet / none / none | 1.71 | 1.70| +1.12% | 0.54%| 2.51%| 1 | 3 | | TPCH(_300) | TPCH-Q18 | parquet / none / none | 33.15 | 32.79 | +1.09% | 0.13%| 2.03%| 1 | 3 | | TPCH(_300) | TPCH-Q14 | parquet / none / none | 5.95 | 5.90| +0.82% | 2.19%| 0.49%| 1 | 3 | | TPCH(_300) | TPCH-Q1 | parquet / none / none | 13.99 | 13.90 | +0.63% | 0.25%| 1.39%| 1 | 3 | | TPCH(_300) | TPCH-Q2 | parquet / none / none | 3.44 | 3.44| +0.00% | * 20.29% * | * 20.76% * | 1 | 3 | | TPCH(_300) | TPCH-Q6 | parquet / none / none | 1.21 | 1.22| -0.01% | 0.06%| 0.06%| 1 | 3 | | TPCH(_300) | TPCH-Q20 | parquet / none / none | 3.51 | 3.51| -0.11% | 7.15%| 7.30%| 1 | 3 | | TPCH(_300) | TPCH-Q16 | parquet / none / none | 6.89 | 6.91| -0.21% | 0.65%| 0.55%| 1 | 3 | | TPCH(_300) | TPCH-Q4 | parquet / none / none | 4.78 | 4.80| -0.38% | 0.06%| 0.59%| 1 | 3 | | TPCH(_300) | TPCH-Q19 | parquet / none / none | 30.78 | 31.04 | -0.83% | 0.45%| 1.03%| 1 | 3 | | TPCH(_300) | TPCH-Q22 | parquet / none / none | 6.06 | 6.12| -1.02% | 1.51%| 2.12%| 1 | 3 | | TPCH(_300) | TPCH-Q10 | parquet / none / none | 9.43 | 9.58| -1.54% | 0.69%| 3.30%| 1 | 3 | | TPCH(_300) | TPCH-Q21 | parquet / none / none | 93.41 | 95.18 | -1.86% | 0.08%| 0.81%| 1 | 3 | | TPCH(_300) | TPCH-Q15 | parquet / none / none | 3.40 | 3.47| -1.99% | 0.72%| 1.27%| 1 | 3 | | TPCH(_300) | TPCH-Q7 | parquet / none / none | 44.98 | 46.24 | -2.71% | 1.83%| 1.27%| 1 | 3 | | TPCH(_300) | TPCH-Q3 | parquet / none / none | 28.06 | 29.11 | -3.61% | 1.62%| 1.23%| 1 | 3 | | TPCH(_300) | TPCH-Q12 | parquet / none / none | 3.15 | 3.28| -3.80% | 0.96%| 1.32%| 1 | 3 | | TPCH(_300) | TPCH-Q9 | parquet / none / none | 29.47 | 30.80 | -4.30% | 0.29%| 0.34%| 1 | 3 | | TPCH(_300) | TPCH-Q17 | parquet / none / none | 4.37 | 4.62| -5.33% | 0.63%| 0.54%| 1
[Impala-ASF-CR] IMPALA-6190/6246: Add instances tab and event sequence
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8758 ) Change subject: IMPALA-6190/6246: Add instances tab and event sequence .. Patch Set 18: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/8758 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I626456b6afa9101ffd5cda10c4096d63d7f9 Gerrit-Change-Number: 8758 Gerrit-PatchSet: 18 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jan 2018 02:41:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5440 Add planner tests with extreme statistics values
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/9065 ) Change subject: IMPALA-5440 Add planner tests with extreme statistics values .. Patch Set 3: (7 comments) http://gerrit.cloudera.org:8080/#/c/9065/3/fe/src/test/java/org/apache/impala/planner/PlannerTest.java File fe/src/test/java/org/apache/impala/planner/PlannerTest.java: http://gerrit.cloudera.org:8080/#/c/9065/3/fe/src/test/java/org/apache/impala/planner/PlannerTest.java@522 PS3, Line 522: from tpch.lineitem a, tpch.lineitem b, tpch.lineitem c"; : checkCardinalityBounds(query, "t what is this testing? http://gerrit.cloudera.org:8080/#/c/9065/3/fe/src/test/java/org/apache/impala/planner/PlannerTest.java@526 PS3, Line 526: nion select * from tpch.lineitem b " : + "union select * from tpch.lineitem is this expected to overflow? perhaps feed in large number of rows. http://gerrit.cloudera.org:8080/#/c/9065/3/fe/src/test/java/org/apache/impala/planner/PlannerTest.java@532 PS3, Line 532: h.lineitem.l_o is this supposed to overflow? http://gerrit.cloudera.org:8080/#/c/9065/3/fe/src/test/java/org/apache/impala/planner/PlannerTest.java@536 PS3, Line 536: tpch.lineitem" is this supposed to overflow? perhaps mention the operator that this is testing (exchange?) http://gerrit.cloudera.org:8080/#/c/9065/3/fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java File fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java: http://gerrit.cloudera.org:8080/#/c/9065/3/fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java@788 PS3, Line 788: pls add comments. http://gerrit.cloudera.org:8080/#/c/9065/3/fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java@792 PS3, Line 792: execRequest == null) : return; single line http://gerrit.cloudera.org:8080/#/c/9065/3/fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java@802 PS3, Line 802: e.estimated_stats.ca is this guaranteed to be non-null? -- To view, visit http://gerrit.cloudera.org:8080/9065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I86dec47cf1438882cafaec53e97864ccfcdff3cb Gerrit-Change-Number: 9065 Gerrit-PatchSet: 3 Gerrit-Owner: xyutin...@cloudera.com Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: xyutin...@cloudera.com Gerrit-Comment-Date: Wed, 24 Jan 2018 02:22:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6215: Removes race when using LibCache.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9089 ) Change subject: IMPALA-6215: Removes race when using LibCache. .. Patch Set 2: (1 comment) I think the usage LibCacheEntry* interface is analogous to the existing GetSoFunctionPtr() interface. I think in a lot of other places we've exposed handles that automatically close or DCHECK wrapping the internal data structure. I agree that the alternative approach is less error-prone so is worth considering. I don't feel strongly about whether to do it in this patch or defer the work (the use ScopeExitTrigger pattern solves the problem but is a bit clunky). http://gerrit.cloudera.org:8080/#/c/9089/2/be/src/codegen/llvm-codegen.cc File be/src/codegen/llvm-codegen.cc: http://gerrit.cloudera.org:8080/#/c/9089/2/be/src/codegen/llvm-codegen.cc@333 PS2, Line 333: MakeScopeExitTrigger([entry]() { : if (entry != nullptr) LibCache::instance()->DecrementUseCount(entry);} : ); > maybe do this in LibCacheEntry's destructor and use a unique_ptr here Isn't LibCacheEntry shared between all of the callers? -- To view, visit http://gerrit.cloudera.org:8080/9089 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9175085201fe8b11424ab8f88d7b992cb7b0daea Gerrit-Change-Number: 9089 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jan 2018 01:56:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6430: Log relevant debug pages if wait for metric value times out
Hello Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9098 to look at the new patch set (#2). Change subject: IMPALA-6430: Log relevant debug pages if wait_for_metric_value times out .. IMPALA-6430: Log relevant debug pages if wait_for_metric_value times out Log the memz, metrics and query page if the method wait_for_metric_value times out. This would help us understand the state of the defaulting impalad when the time out happens. Change-Id: I069dad48ede709c4114f4d7175861f98321be6cf --- M tests/common/impala_service.py 1 file changed, 8 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/9098/2 -- To view, visit http://gerrit.cloudera.org:8080/9098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I069dad48ede709c4114f4d7175861f98321be6cf Gerrit-Change-Number: 9098 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6395: Add a flag for data stream sender's buffer size
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9026 ) Change subject: IMPALA-6395: Add a flag for data stream sender's buffer size .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1789/ -- To view, visit http://gerrit.cloudera.org:8080/9026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I385f4b7a0671bb2d7872bee60d476c375680b5c2 Gerrit-Change-Number: 9026 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jan 2018 01:50:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6430: Log relevant debug pages if wait for metric value times out
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/9098 ) Change subject: IMPALA-6430: Log relevant debug pages if wait_for_metric_value times out .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/9098/1/tests/common/impala_service.py File tests/common/impala_service.py: http://gerrit.cloudera.org:8080/#/c/9098/1/tests/common/impala_service.py@123 PS1, Line 123: json.dumps(self.read_debug_webpage('queries?json'))) > how about threadz and rpcz? In case there's a stuck thread or RPC. Done -- To view, visit http://gerrit.cloudera.org:8080/9098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I069dad48ede709c4114f4d7175861f98321be6cf Gerrit-Change-Number: 9098 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 24 Jan 2018 01:49:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4924: Enable Decimal V2 by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9062 ) Change subject: IMPALA-4924: Enable Decimal V2 by default .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1794/ -- To view, visit http://gerrit.cloudera.org:8080/9062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibbdd05bf986b7947f106b396017faa3a0bd87fd7 Gerrit-Change-Number: 9062 Gerrit-PatchSet: 6 Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zach Amsden Gerrit-Comment-Date: Wed, 24 Jan 2018 01:44:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6435: Disable codegen for CHAR literals.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9102 ) Change subject: IMPALA-6435: Disable codegen for CHAR literals. .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1793/ -- To view, visit http://gerrit.cloudera.org:8080/9102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 Gerrit-Change-Number: 9102 Gerrit-PatchSet: 4 Gerrit-Owner: anujphadke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Wed, 24 Jan 2018 01:43:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6435: Disable codegen for CHAR literals.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9102 ) Change subject: IMPALA-6435: Disable codegen for CHAR literals. .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/9102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 Gerrit-Change-Number: 9102 Gerrit-PatchSet: 4 Gerrit-Owner: anujphadke Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Wed, 24 Jan 2018 01:43:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6435: Disable codegen for CHAR literals.
anujphadke has posted comments on this change. ( http://gerrit.cloudera.org:8080/9102 ) Change subject: IMPALA-6435: Disable codegen for CHAR literals. .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/9102/3/testdata/workloads/functional-query/queries/QueryTest/disable-codegen.test File testdata/workloads/functional-query/queries/QueryTest/disable-codegen.test: http://gerrit.cloudera.org:8080/#/c/9102/3/testdata/workloads/functional-query/queries/QueryTest/disable-codegen.test@38 PS3, Line 38: select cast('a' as char(4)) as s from functional.alltypestiny > Can you add a similar test for cast(NULL as char(4))? Or add it to this que Done -- To view, visit http://gerrit.cloudera.org:8080/9102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 Gerrit-Change-Number: 9102 Gerrit-PatchSet: 4 Gerrit-Owner: anujphadke Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Wed, 24 Jan 2018 01:40:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6215: Removes race when using LibCache.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/9089 ) Change subject: IMPALA-6215: Removes race when using LibCache. .. Patch Set 2: (2 comments) look good to me. But the interface for lib-cache seems conceptually wrong to me. The client to a cache should not be responsible for a cache object's lifetime (keeping track of ref counts, since relying on the client for this will make it prone to bugs). A more cleaner implementation of libcache should just return a LibCacheEntry that keeps track of its refcount internally based on its lifetime and also exposes whether that entry is alive, and if not, then gives the option to refresh it. Not sure if we should fix this in this patch or file a separate JIRA. http://gerrit.cloudera.org:8080/#/c/9089/2/be/src/codegen/llvm-codegen.cc File be/src/codegen/llvm-codegen.cc: http://gerrit.cloudera.org:8080/#/c/9089/2/be/src/codegen/llvm-codegen.cc@333 PS2, Line 333: MakeScopeExitTrigger([entry]() { : if (entry != nullptr) LibCache::instance()->DecrementUseCount(entry);} : ); maybe do this in LibCacheEntry's destructor and use a unique_ptr here http://gerrit.cloudera.org:8080/#/c/9089/2/tests/query_test/test_udfs.py File tests/query_test/test_udfs.py: http://gerrit.cloudera.org:8080/#/c/9089/2/tests/query_test/test_udfs.py@443 PS2, Line 443: create_fn_to_drop = "create function {0}.foo_{1}() returns string LOCATION '{2}' SYMBOL='org.apache.impala.TestUpdateUdf'" : create_fn_to_use = "create function {0}.use_it(string) returns string LOCATION '{1}' SYMBOL='org.apache.impala.TestUdf'" : drop_fn = "drop function if exists {0}.foo_{1}()" : use_fn = "select * from (select max(int_col) from functional.alltypesagg where {0}.use_it(string_col) = 'blah' union all (select max(int_col) from functional.alltypesagg where {0}.use_it(String_col) > '1' union all (select max(int_col) from functional.alltypesagg where {0}.use_it(string_col) > '1'))) v" : nit: long lines -- To view, visit http://gerrit.cloudera.org:8080/9089 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9175085201fe8b11424ab8f88d7b992cb7b0daea Gerrit-Change-Number: 9089 Gerrit-PatchSet: 2 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Bikramjeet Vig Gerrit-Comment-Date: Wed, 24 Jan 2018 01:39:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6435: Disable codegen for CHAR literals.
Hello Philip Zeyliger, Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9102 to look at the new patch set (#4). Change subject: IMPALA-6435: Disable codegen for CHAR literals. .. IMPALA-6435: Disable codegen for CHAR literals. Currently we do not codegen CHAR types. This change checks for CHAR literals in a expr and disables codegen. Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 --- M be/src/exprs/literal.cc M testdata/workloads/functional-query/queries/QueryTest/disable-codegen.test 2 files changed, 20 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/9102/4 -- To view, visit http://gerrit.cloudera.org:8080/9102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 Gerrit-Change-Number: 9102 Gerrit-PatchSet: 4 Gerrit-Owner: anujphadke Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke
[Impala-ASF-CR] IMPALA-4924: Enable Decimal V2 by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9062 ) Change subject: IMPALA-4924: Enable Decimal V2 by default .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1788/ -- To view, visit http://gerrit.cloudera.org:8080/9062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibbdd05bf986b7947f106b396017faa3a0bd87fd7 Gerrit-Change-Number: 9062 Gerrit-PatchSet: 6 Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zach Amsden Gerrit-Comment-Date: Wed, 24 Jan 2018 01:36:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6429: Fix decimal division
Taras Bobrovytsky has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9114 Change subject: IMPALA-6429: Fix decimal division .. IMPALA-6429: Fix decimal division Before this patch, it was possible for an overflow to not be detected when doing a decimal division. When scaling up the dividend before doing the division, we do not check for overflow. This is ok if the we are scaling up by 10^38 or less because the result is guaranteed to fit into 256 bits. However, when we are scaling up by more than 38, the result may not fit into 256 bits and overflow. This overflow may not be detected. The problem is fixed by checking for overflow when scaling up the dividend. Testing: - Added some BE tests Change-Id: Ibd1075d9c78986cd975dd29c1125d71ba6560c23 --- M be/src/exprs/expr-test.cc M be/src/runtime/decimal-value.inline.h M be/src/util/decimal-util.h 3 files changed, 46 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/9114/1 -- To view, visit http://gerrit.cloudera.org:8080/9114 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ibd1075d9c78986cd975dd29c1125d71ba6560c23 Gerrit-Change-Number: 9114 Gerrit-PatchSet: 1 Gerrit-Owner: Taras Bobrovytsky
[Impala-ASF-CR] IMPALA-5191: Standardize column alias behavior
Alex Behm has uploaded a new patch set (#14) to the change originally created by Zoltan Borok-Nagy. ( http://gerrit.cloudera.org:8080/8801 ) Change subject: IMPALA-5191: Standardize column alias behavior .. IMPALA-5191: Standardize column alias behavior We should not perform alias substitution in the subexpressions of GROUP BY, HAVING, and ORDER BY to be more standard conformant. === Allowed === SELECT int_col / 2 AS x FROM functional.alltypes GROUP BY x; SELECT int_col / 2 AS x FROM functional.alltypes ORDER BY x; SELECT NOT bool_col AS nb FROM functional.alltypes GROUP BY nb HAVING nb; === Not allowed === SELECT int_col / 2 AS x FROM functional.alltypes GROUP BY x / 2; SELECT int_col / 2 AS x FROM functional.alltypes ORDER BY -x; SELECT int_col / 2 AS x FROM functional.alltypes GROUP BY x HAVING x > 3; Some extra checks were added to AnalyzeExprsTest.java. I had to update other tests to make them pass since the new behavior is more restrictive. I added alias.test to the end-to-end tests. Cherry-picks: not for 2.x. Change-Id: I0f82483b486acf6953876cfa672b0d034f3709a8 --- M fe/src/main/java/org/apache/impala/analysis/QueryStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test A testdata/workloads/functional-query/queries/QueryTest/alias.test M tests/query_test/test_queries.py 8 files changed, 240 insertions(+), 49 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/8801/14 -- To view, visit http://gerrit.cloudera.org:8080/8801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0f82483b486acf6953876cfa672b0d034f3709a8 Gerrit-Change-Number: 8801 Gerrit-PatchSet: 14 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-5191: Standardize column alias behavior
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8801 ) Change subject: IMPALA-5191: Standardize column alias behavior .. Patch Set 14: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1792/ -- To view, visit http://gerrit.cloudera.org:8080/8801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0f82483b486acf6953876cfa672b0d034f3709a8 Gerrit-Change-Number: 8801 Gerrit-PatchSet: 14 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 24 Jan 2018 00:35:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5191: Standardize column alias behavior
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/8801 ) Change subject: IMPALA-5191: Standardize column alias behavior .. Patch Set 14: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0f82483b486acf6953876cfa672b0d034f3709a8 Gerrit-Change-Number: 8801 Gerrit-PatchSet: 14 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 24 Jan 2018 00:34:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6024: Min sample bytes for COMPUTE STATS TABLESAMPLE
Alex Behm has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9113 Change subject: IMPALA-6024: Min sample bytes for COMPUTE STATS TABLESAMPLE .. IMPALA-6024: Min sample bytes for COMPUTE STATS TABLESAMPLE Adds a new query option COMPUTE_STATS_MIN_SAMPLE_SIZE which is the minimum number of bytes that will be scanned in COMPUTE STATS TABLESAMPLE, regardless of the user-supplied sampling percent. The motivation is to prevent sampling for very small tables where accurate stats can be obtained cheaply without sampling. This patch changes COMPUTE STATS TABLESAMPLE to run the regular COMPUTE STATS if the effective sampling percent is 0% or 100%. For a 100% sampling rate, the sampling-based stats queries are more expensive and produce less accurate stats than the regular COMPUTE STATS. Default: COMPUTE_STATS_MIN_SAMPLE_SIZE=1GB Testing: - added new unit tests and ran them locally Change-Id: I2cb91a40bec50b599875109c2f7c5bf6f41c2400 --- M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M tests/custom_cluster/test_stats_extrapolation.py 10 files changed, 125 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/9113/1 -- To view, visit http://gerrit.cloudera.org:8080/9113 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2cb91a40bec50b599875109c2f7c5bf6f41c2400 Gerrit-Change-Number: 9113 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Behm
[Impala-ASF-CR] IMPALA-6410: Tool to cherrypick changes across branches.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9045 ) Change subject: IMPALA-6410: Tool to cherrypick changes across branches. .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1791/ -- To view, visit http://gerrit.cloudera.org:8080/9045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6120ec2d6e914a1e5fda568178b32aafda8722a9 Gerrit-Change-Number: 9045 Gerrit-PatchSet: 5 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Comment-Date: Wed, 24 Jan 2018 00:03:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6059: Enhance ltrim()/rtrim() functions to trim any set of characters.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8349 ) Change subject: IMPALA-6059: Enhance ltrim()/rtrim() functions to trim any set of characters. .. Patch Set 12: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/8349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a5ae3f59762e70c3268a01e14ed57a9e36b8d79 Gerrit-Change-Number: 8349 Gerrit-PatchSet: 12 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Tue, 23 Jan 2018 23:44:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6059: Enhance ltrim()/rtrim() functions to trim any set of characters.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8349 ) Change subject: IMPALA-6059: Enhance ltrim()/rtrim() functions to trim any set of characters. .. IMPALA-6059: Enhance ltrim()/rtrim() functions to trim any set of characters. This patch generalizes ltrim()/rtrim() functions to accept a second argument that specifies the set of characters to be removed from the leading/trailing end of the target string: ltrim(string text[, characters text]) rtrim(string text[, characters text]) A common string trimming method has been added to StringFunctions, which is called from the general ltrim/rtrim/btrim functions. The functions also share prepare and close operations. New StringFunctions tests have been added to ExprTest for the new forms of ltrim() and rtrim(). New tests to cover handling of special characters have also been added. Note that our string handling functions only work with the ASCII character set. Handling of other character sets lies outside the scope of this patch. The existing ltrim()/rtrim()/trim() functions that take only one argument have been updated to use the more general methods. Testing: Queries like the following were run on a 1.5-billion row tpch_parquet.lineitem table, with the old and new implementations to ensure there is no performance regression: 1. select count(trim(l_shipinstruct)), count(trim(l_returnflag)), ... 2. select count(*) from t where trim(l_shipinstruct) = '' and ... Change-Id: I8a5ae3f59762e70c3268a01e14ed57a9e36b8d79 Reviewed-on: http://gerrit.cloudera.org:8080/8349 Reviewed-by: Michael Ho Tested-by: Impala Public Jenkins --- M be/src/exprs/expr-test.cc M be/src/exprs/string-functions-ir.cc M be/src/exprs/string-functions.h M common/function-registry/impala_functions.py 4 files changed, 178 insertions(+), 88 deletions(-) Approvals: Michael Ho: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/8349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8a5ae3f59762e70c3268a01e14ed57a9e36b8d79 Gerrit-Change-Number: 8349 Gerrit-PatchSet: 13 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9032 ) Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption .. Patch Set 9: Code-Review+1 LGTM. Might be good if Sailesh had a quick look to see if there are any potential misuses of the OpenSSL APIs. -- To view, visit http://gerrit.cloudera.org:8080/9032 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24 Gerrit-Change-Number: 9032 Gerrit-PatchSet: 9 Gerrit-Owner: Xianda Ke Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Xianda Ke Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 23 Jan 2018 23:37:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4835: prerequisite buffer pool changes
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/9097 ) Change subject: IMPALA-4835: prerequisite buffer pool changes .. Patch Set 2: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/9097 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8f2196722df59f2d367787c0550058022e296e24 Gerrit-Change-Number: 9097 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 23:07:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6190/6246: Add instances tab and event sequence
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8758 ) Change subject: IMPALA-6190/6246: Add instances tab and event sequence .. Patch Set 18: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1790/ -- To view, visit http://gerrit.cloudera.org:8080/8758 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I626456b6afa9101ffd5cda10c4096d63d7f9 Gerrit-Change-Number: 8758 Gerrit-PatchSet: 18 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 23:05:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6190/6246: Add instances tab and event sequence
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/8758 ) Change subject: IMPALA-6190/6246: Add instances tab and event sequence .. Patch Set 18: Code-Review+2 Re-added status.h include. Carrying Tim's +2. -- To view, visit http://gerrit.cloudera.org:8080/8758 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I626456b6afa9101ffd5cda10c4096d63d7f9 Gerrit-Change-Number: 8758 Gerrit-PatchSet: 18 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 23:04:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6190/6246: Add instances tab and event sequence
Hello Michael Ho, Joe McDonnell, Tim Armstrong, Dan Hecht, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8758 to look at the new patch set (#18). Change subject: IMPALA-6190/6246: Add instances tab and event sequence .. IMPALA-6190/6246: Add instances tab and event sequence This change adds tracking of the current state during the execution of a fragment instance. The current state is then reported back to the coordinator and exposed to users via a new tab in the query detail debug webpage. This change also adds an event timeline to fragment instances in the query profile. The timeline measures the time since backend-local query start at which particular events complete. Events are derived from the current state of the execution of a fragment instance. For example: - Prepare Finished: 13.436ms (13.436ms) - First Batch Produced: 1s022ms (1s008ms) - First Batch Sent: 1s022ms (455.558us) - ExecInternal Finished: 2s783ms (1s760ms) I added automated tests for both extensions and additionally verified the change by manual inspection. Here are the TPCH performance comparison results between this change and the previous commit on a 16 node cluster. ++---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | ++---+-++++ | TPCH(_300) | parquet / none / none | 18.47 | -0.94% | 9.72 | -1.08% | ++---+-++++ ++--+---++-++++-+---+ | Workload | Query| File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Num Clients | Iters | ++--+---++-++++-+---+ | TPCH(_300) | TPCH-Q5 | parquet / none / none | 48.88 | 46.93 | +4.15% | 0.14%| 3.61%| 1 | 3 | | TPCH(_300) | TPCH-Q13 | parquet / none / none | 21.64 | 21.15 | +2.29% | 2.06%| 1.84%| 1 | 3 | | TPCH(_300) | TPCH-Q11 | parquet / none / none | 1.71 | 1.70| +1.12% | 0.54%| 2.51%| 1 | 3 | | TPCH(_300) | TPCH-Q18 | parquet / none / none | 33.15 | 32.79 | +1.09% | 0.13%| 2.03%| 1 | 3 | | TPCH(_300) | TPCH-Q14 | parquet / none / none | 5.95 | 5.90| +0.82% | 2.19%| 0.49%| 1 | 3 | | TPCH(_300) | TPCH-Q1 | parquet / none / none | 13.99 | 13.90 | +0.63% | 0.25%| 1.39%| 1 | 3 | | TPCH(_300) | TPCH-Q2 | parquet / none / none | 3.44 | 3.44| +0.00% | * 20.29% * | * 20.76% * | 1 | 3 | | TPCH(_300) | TPCH-Q6 | parquet / none / none | 1.21 | 1.22| -0.01% | 0.06%| 0.06%| 1 | 3 | | TPCH(_300) | TPCH-Q20 | parquet / none / none | 3.51 | 3.51| -0.11% | 7.15%| 7.30%| 1 | 3 | | TPCH(_300) | TPCH-Q16 | parquet / none / none | 6.89 | 6.91| -0.21% | 0.65%| 0.55%| 1 | 3 | | TPCH(_300) | TPCH-Q4 | parquet / none / none | 4.78 | 4.80| -0.38% | 0.06%| 0.59%| 1 | 3 | | TPCH(_300) | TPCH-Q19 | parquet / none / none | 30.78 | 31.04 | -0.83% | 0.45%| 1.03%| 1 | 3 | | TPCH(_300) | TPCH-Q22 | parquet / none / none | 6.06 | 6.12| -1.02% | 1.51%| 2.12%| 1 | 3 | | TPCH(_300) | TPCH-Q10 | parquet / none / none | 9.43 | 9.58| -1.54% | 0.69%| 3.30%| 1 | 3 | | TPCH(_300) | TPCH-Q21 | parquet / none / none | 93.41 | 95.18 | -1.86% | 0.08%| 0.81%| 1 | 3 | | TPCH(_300) | TPCH-Q15 | parquet / none / none | 3.40 | 3.47| -1.99% | 0.72%| 1.27%| 1 | 3 | | TPCH(_300) | TPCH-Q7 | parquet / none / none | 44.98 | 46.24 | -2.71% | 1.83%| 1.27%| 1 | 3 | | TPCH(_300) | TPCH-Q3 | parquet / none / none | 28.06 | 29.11 | -3.61% | 1.62%| 1.23%| 1 | 3 | | TPCH(_300) | TPCH-Q12 | parquet / none / none | 3.15 | 3.28| -3.80% | 0.96%| 1.32%| 1 | 3 | | TPCH(_300) | TPCH-Q9 | parquet / none / none | 29.47 | 30.80 | -4.30% | 0.29%| 0.34%| 1 | 3 | | TPCH(_300) | TPCH-Q17
[native-toolchain-CR] Patch thrift-0.9.3 for python 2.6.
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9109 Change subject: Patch thrift-0.9.3 for python 2.6. .. Patch thrift-0.9.3 for python 2.6. Thrift 0.9.3 doesn't work with python 2.6. The compatibility is broken by THRIFT-2846 and is fixed as a part of THRIFT-3505. This patch adds patch 3 to thrift 0.9.3, which is the upstream patch excluding tests and makefiles. The impala test test_client_ssl broken by thrift-0.9.3-p2 and python 2.6 now works. Change-Id: I9a7951e8b99c1d4b866e1d331860e3b82aa3a3e9 --- M buildall.sh A source/thrift/thrift-0.9.3-patches/0003-THRIFT-3505.patch 2 files changed, 434 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/09/9109/1 -- To view, visit http://gerrit.cloudera.org:8080/9109 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9a7951e8b99c1d4b866e1d331860e3b82aa3a3e9 Gerrit-Change-Number: 9109 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang
[Impala-ASF-CR] IMPALA-5440 Add planner tests with extreme statistics values
xyutin...@cloudera.com has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/9065 ) Change subject: IMPALA-5440 Add planner tests with extreme statistics values .. IMPALA-5440 Add planner tests with extreme statistics values Change-Id: I86dec47cf1438882cafaec53e97864ccfcdff3cb --- M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 2 files changed, 102 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/9065/3 -- To view, visit http://gerrit.cloudera.org:8080/9065 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I86dec47cf1438882cafaec53e97864ccfcdff3cb Gerrit-Change-Number: 9065 Gerrit-PatchSet: 3 Gerrit-Owner: xyutin...@cloudera.com Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: xyutin...@cloudera.com
[Impala-ASF-CR] IMPALA-6190/6246: Add instances tab and event sequence
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8758 ) Change subject: IMPALA-6190/6246: Add instances tab and event sequence .. Patch Set 17: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1786/ -- To view, visit http://gerrit.cloudera.org:8080/8758 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I626456b6afa9101ffd5cda10c4096d63d7f9 Gerrit-Change-Number: 8758 Gerrit-PatchSet: 17 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 22:26:35 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] IMPALA-6105 unix timestamp returns a number of seconds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9084 ) Change subject: [DOCS] IMPALA-6105 unix_timestamp returns a number of seconds .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/9084 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie502322368c0052d4767191cf8f7ecb7ac5e7a16 Gerrit-Change-Number: 9084 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: John Russell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 22:26:31 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] IMPALA-6105 unix timestamp returns a number of seconds
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9084 ) Change subject: [DOCS] IMPALA-6105 unix_timestamp returns a number of seconds .. [DOCS] IMPALA-6105 unix_timestamp returns a number of seconds Added a note that the unix_timestamp function returns a number of seconds. Change-Id: Ie502322368c0052d4767191cf8f7ecb7ac5e7a16 Reviewed-on: http://gerrit.cloudera.org:8080/9084 Reviewed-by: John Russell Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M docs/topics/impala_datetime_functions.xml 1 file changed, 6 insertions(+), 3 deletions(-) Approvals: John Russell: Looks good to me, but someone else must approve Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/9084 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie502322368c0052d4767191cf8f7ecb7ac5e7a16 Gerrit-Change-Number: 9084 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: John Russell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6395: Add a flag for data stream sender's buffer size
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9026 ) Change subject: IMPALA-6395: Add a flag for data stream sender's buffer size .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1789/ -- To view, visit http://gerrit.cloudera.org:8080/9026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I385f4b7a0671bb2d7872bee60d476c375680b5c2 Gerrit-Change-Number: 9026 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 22:20:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4132: Use -fno-omit-frame-pointer
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8612 ) Change subject: IMPALA-4132: Use -fno-omit-frame-pointer .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/8612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7d0d88ba015a847356ed0274fe91017b98cb941 Gerrit-Change-Number: 8612 Gerrit-PatchSet: 3 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 22:19:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4132: Use -fno-omit-frame-pointer
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8612 ) Change subject: IMPALA-4132: Use -fno-omit-frame-pointer .. IMPALA-4132: Use -fno-omit-frame-pointer Using -fno-omit-frame-pointer would keep the frame pointers for functions in the register. As a result we expect more useful stack traces to be produced. For testing performance benchmark was executed on TPC-H and TPC-DS without any significant discrepancy from the baseline results. (For the specific measures check the attachments in the Jira.) Change-Id: Ib7d0d88ba015a847356ed0274fe91017b98cb941 Reviewed-on: http://gerrit.cloudera.org:8080/8612 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M be/CMakeLists.txt 1 file changed, 2 insertions(+), 1 deletion(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/8612 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib7d0d88ba015a847356ed0274fe91017b98cb941 Gerrit-Change-Number: 8612 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] [DOCS] IMPALA-6105 unix timestamp returns a number of seconds
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9084 ) Change subject: [DOCS] IMPALA-6105 unix_timestamp returns a number of seconds .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/192/ -- To view, visit http://gerrit.cloudera.org:8080/9084 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie502322368c0052d4767191cf8f7ecb7ac5e7a16 Gerrit-Change-Number: 9084 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: John Russell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 22:13:48 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] IMPALA-6105 unix timestamp returns a number of seconds
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9084 ) Change subject: [DOCS] IMPALA-6105 unix_timestamp returns a number of seconds .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/9084 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie502322368c0052d4767191cf8f7ecb7ac5e7a16 Gerrit-Change-Number: 9084 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: John Russell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 22:13:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6435: Disable codegen for CHAR literals.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9102 ) Change subject: IMPALA-6435: Disable codegen for CHAR literals. .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/9102/3/testdata/workloads/functional-query/queries/QueryTest/disable-codegen.test File testdata/workloads/functional-query/queries/QueryTest/disable-codegen.test: http://gerrit.cloudera.org:8080/#/c/9102/3/testdata/workloads/functional-query/queries/QueryTest/disable-codegen.test@38 PS3, Line 38: select cast('a' as char(4)) as s from functional.alltypestiny Can you add a similar test for cast(NULL as char(4))? Or add it to this query? It seems to work ok now but it seems like we're missing test coverage. -- To view, visit http://gerrit.cloudera.org:8080/9102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 Gerrit-Change-Number: 9102 Gerrit-PatchSet: 3 Gerrit-Owner: anujphadke Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Tue, 23 Jan 2018 22:10:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] IMPALA-6105 unix timestamp returns a number of seconds
John Russell has posted comments on this change. ( http://gerrit.cloudera.org:8080/9084 ) Change subject: [DOCS] IMPALA-6105 unix_timestamp returns a number of seconds .. Patch Set 2: Code-Review+1 LGTM. I'll let Tim give the +2. -- To view, visit http://gerrit.cloudera.org:8080/9084 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie502322368c0052d4767191cf8f7ecb7ac5e7a16 Gerrit-Change-Number: 9084 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: John Russell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 21:58:26 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] IMPALA-6105 unix timestamp returns a number of seconds
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/9084 ) Change subject: [DOCS] IMPALA-6105 unix_timestamp returns a number of seconds .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/9084/1/docs/topics/impala_datetime_functions.xml File docs/topics/impala_datetime_functions.xml: http://gerrit.cloudera.org:8080/#/c/9084/1/docs/topics/impala_datetime_functions.xml@2531 PS1, Line 2531: date time > date and time Done http://gerrit.cloudera.org:8080/#/c/9084/1/docs/topics/impala_datetime_functions.xml@2532 PS1, Line 2532: called with arguments, the first argument represented as a > the Done http://gerrit.cloudera.org:8080/#/c/9084/1/docs/topics/impala_datetime_functions.xml@2532 PS1, Line 2532: > Remove trailing space. Done -- To view, visit http://gerrit.cloudera.org:8080/9084 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie502322368c0052d4767191cf8f7ecb7ac5e7a16 Gerrit-Change-Number: 9084 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: John Russell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 21:57:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] IMPALA-6105 unix timestamp returns a number of seconds
Hello John Russell, Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9084 to look at the new patch set (#2). Change subject: [DOCS] IMPALA-6105 unix_timestamp returns a number of seconds .. [DOCS] IMPALA-6105 unix_timestamp returns a number of seconds Added a note that the unix_timestamp function returns a number of seconds. Change-Id: Ie502322368c0052d4767191cf8f7ecb7ac5e7a16 --- M docs/topics/impala_datetime_functions.xml 1 file changed, 6 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/9084/2 -- To view, visit http://gerrit.cloudera.org:8080/9084 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie502322368c0052d4767191cf8f7ecb7ac5e7a16 Gerrit-Change-Number: 9084 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: John Russell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-4924: Enable Decimal V2 by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9062 ) Change subject: IMPALA-4924: Enable Decimal V2 by default .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1788/ -- To view, visit http://gerrit.cloudera.org:8080/9062 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibbdd05bf986b7947f106b396017faa3a0bd87fd7 Gerrit-Change-Number: 9062 Gerrit-PatchSet: 6 Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zach Amsden Gerrit-Comment-Date: Tue, 23 Jan 2018 21:53:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6435: Disable codegen for CHAR literals.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/9102 ) Change subject: IMPALA-6435: Disable codegen for CHAR literals. .. Patch Set 3: Code-Review+1 Looks better to me. Not an area of my expertise, though. -- To view, visit http://gerrit.cloudera.org:8080/9102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 Gerrit-Change-Number: 9102 Gerrit-PatchSet: 3 Gerrit-Owner: anujphadke Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Tue, 23 Jan 2018 21:53:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption
Hello Zoltan Borok-Nagy, Sailesh Mukil, Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9032 to look at the new patch set (#9). Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption .. IMPALA-6219: Use AES-GCM for spill-to-disk encryption AES-GCM can be very fast(~10 times faster than CFB+SHA256), but it requires an instruction that Impala can currently run without (CLMUL). In order to be fast, we dispatch to GCM mode at run-time based on the CPU and OpenSSL version. Testing: run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and buffered-tuple-stream-test. add two cases GcmIntegrity & EncryptoArbitraryLength for openssl-util-test Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24 --- M be/src/runtime/tmp-file-mgr.cc M be/src/util/cpu-info.cc M be/src/util/cpu-info.h M be/src/util/openssl-util-test.cc M be/src/util/openssl-util.cc M be/src/util/openssl-util.h 6 files changed, 232 insertions(+), 78 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/9032/9 -- To view, visit http://gerrit.cloudera.org:8080/9032 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24 Gerrit-Change-Number: 9032 Gerrit-PatchSet: 9 Gerrit-Owner: Xianda Ke Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Xianda Ke Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-6338: Fix flaky test profile fragment instances
Hello Michael Ho, Sailesh Mukil, Joe McDonnell, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8997 to look at the new patch set (#3). Change subject: IMPALA-6338: Fix flaky test_profile_fragment_instances .. IMPALA-6338: Fix flaky test_profile_fragment_instances test_profile_fragment_instances checks that, once all the results have been returned, every fragment instance appears in the query profile for a query that internally cancels fragment instances that are still executing when the results have been fully returned. Every fis is guaranteed to send a profile to the coordinator in Finalize(), but previously fragment profiles were not applied by the coordinator if the backend was 'done', defined as either all instances having completed or one has entered an error state (including cancelled). So, the test could fail by the following sequence: - Some fragment for a particular backend sends an update to the coordinator. 'returned_all_results_' is true, so the coordinator responds indicating the the backend should cancel its remaining fragments. - Another fragment from that backend executes Finalize() and reports that it was cancelled. This causes the coordinator to consider the entire backend to be 'done'. - A third fragment, which had not previously sent a report from the reporting thread, from the same backend executes Finalize(). This report will not be applied by the coordinator as the backend is considered 'done', so this fragment will not appear in the final profile. The solution is to change the definition of 'done' to not include a backend that has been cancelled but still has fragments that haven't completed. This guarantees that for queries that complete successfully and are cancelled internally, all fis will send a report and have it applied by the coordinator before all results have been returned, since if eos is true Coordinator::GetNext() calls WaitForBackendCompletion(), which in this situation will now wait for all fis to Finalize(). Returning results for queries that are cancelled by the user is unaffected as the manual cancel path causes WaitForBackendCompletion(). Testing: - Ran test_profile_fragment_instances in a loop with no failures. I can reliably repro the original problem with a few carefully placed sleeps. Change-Id: I3a1e3c62952003f37f88fe2b662bb11889ed --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M tests/query_test/test_observability.py 5 files changed, 25 insertions(+), 18 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/8997/3 -- To view, visit http://gerrit.cloudera.org:8080/8997 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3a1e3c62952003f37f88fe2b662bb11889ed Gerrit-Change-Number: 8997 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil
[Impala-ASF-CR] IMPALA-6338: Fix flaky test profile fragment instances
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/8997 ) Change subject: IMPALA-6338: Fix flaky test_profile_fragment_instances .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/8997/2/be/src/runtime/coordinator-backend-state.h File be/src/runtime/coordinator-backend-state.h: http://gerrit.cloudera.org:8080/#/c/8997/2/be/src/runtime/coordinator-backend-state.h@206 PS2, Line 206: either because the client called Cancel() or because the coordinator : /// initiated cancel because all the results were already returned > It could also be because another backend hit an error. Done http://gerrit.cloudera.org:8080/#/c/8997/2/be/src/runtime/coordinator-backend-state.h@243 PS2, Line 243: other than cancel > We're making a base assumption that a CANCELLED status is only obtained as I don't think that there are any situations in which a fragment instance would try to cancel the query without it being the result of some error, and of course if the coordinator sees an error it will cancel the query. I did some grepping around to confirm this, but I'll get someone with more authority to chime in. http://gerrit.cloudera.org:8080/#/c/8997/2/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/8997/2/be/src/runtime/coordinator.cc@926 PS2, Line 926: // If the query was cancelled by the user, don't process the update. : if (query_status_.IsCancelled()) return Status::OK(); > Shouldn't we tell the backend to cancel itself if the query was cancelled b Yes, we're relying on Cancel() having already sent cancels to all of the backends. -- To view, visit http://gerrit.cloudera.org:8080/8997 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3a1e3c62952003f37f88fe2b662bb11889ed Gerrit-Change-Number: 8997 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Tue, 23 Jan 2018 21:44:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] IMPALA-6105 unix timestamp returns a number of seconds
John Russell has posted comments on this change. ( http://gerrit.cloudera.org:8080/9084 ) Change subject: [DOCS] IMPALA-6105 unix_timestamp returns a number of seconds .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/9084/1/docs/topics/impala_datetime_functions.xml File docs/topics/impala_datetime_functions.xml: http://gerrit.cloudera.org:8080/#/c/9084/1/docs/topics/impala_datetime_functions.xml@2531 PS1, Line 2531: date time date and time http://gerrit.cloudera.org:8080/#/c/9084/1/docs/topics/impala_datetime_functions.xml@2532 PS1, Line 2532: Remove trailing space. http://gerrit.cloudera.org:8080/#/c/9084/1/docs/topics/impala_datetime_functions.xml@2532 PS1, Line 2532: called with arguments, the first argument represented as a the -- To view, visit http://gerrit.cloudera.org:8080/9084 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie502322368c0052d4767191cf8f7ecb7ac5e7a16 Gerrit-Change-Number: 9084 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: John Russell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 21:41:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6193: Track memory of incoming data streams
Hello Michael Ho, Tim Armstrong, Bikramjeet Vig, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8914 to look at the new patch set (#8). Change subject: IMPALA-6193: Track memory of incoming data streams .. IMPALA-6193: Track memory of incoming data streams This change adds memory tracking to incoming transmit data RPCs when using KRPC. We track memory against a global tracker called "Data Stream Service" until it is handed over to the stream manager. There we track it in a global tracker called "Data Stream Queued RPC Calls" until a receiver registers and takes over the early sender RPCs. Inside the receiver, memory for deferred RPCs is tracked against the fragment instance's memtracker until we unpack the batches and add them to the row batch queue. The DCHECK in MemTracker::Close() covers that all memory consumed by a tracker gets release eventually. In addition to that, this change adds a custom cluster test that makes sure that queued memory gets tracked by inspecting the peak consumption of the new memtrackers. Change-Id: I2df1204d2483313a8a18e5e3be6cec9e402614c4 --- M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/runtime/exec-env.cc M be/src/runtime/krpc-data-stream-mgr.cc M be/src/runtime/krpc-data-stream-mgr.h M be/src/runtime/krpc-data-stream-recvr.cc M be/src/runtime/mem-tracker.h M be/src/util/memory-metrics.h M common/protobuf/data_stream_service.proto A tests/custom_cluster/test_krpc_mem_usage.py A tests/verifiers/mem_usage_verifier.py 13 files changed, 283 insertions(+), 39 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/8914/8 -- To view, visit http://gerrit.cloudera.org:8080/8914 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2df1204d2483313a8a18e5e3be6cec9e402614c4 Gerrit-Change-Number: 8914 Gerrit-PatchSet: 8 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6435: Disable codegen for CHAR literals.
anujphadke has posted comments on this change. ( http://gerrit.cloudera.org:8080/9102 ) Change subject: IMPALA-6435: Disable codegen for CHAR literals. .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/9102/1/be/src/exprs/literal.cc File be/src/exprs/literal.cc: http://gerrit.cloudera.org:8080/#/c/9102/1/be/src/exprs/literal.cc@a392 PS1, Line 392: > This seems wrong; you deleted the handlig for TYPE_STRING. Ooops! Sorry I had removed some of the code to debug some test failure. Forgot to git add it back again. Removed the TYPE_CHAR case here and I return earlier. -- To view, visit http://gerrit.cloudera.org:8080/9102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 Gerrit-Change-Number: 9102 Gerrit-PatchSet: 3 Gerrit-Owner: anujphadke Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: anujphadke Gerrit-Comment-Date: Tue, 23 Jan 2018 21:20:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6435: Disable codegen for CHAR literals.
Hello Philip Zeyliger, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9102 to look at the new patch set (#3). Change subject: IMPALA-6435: Disable codegen for CHAR literals. .. IMPALA-6435: Disable codegen for CHAR literals. Currently we do not codegen CHAR types. This change checks for CHAR literals in a expr and disables codegen. Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 --- M be/src/exprs/literal.cc M testdata/workloads/functional-query/queries/QueryTest/disable-codegen.test 2 files changed, 18 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/9102/3 -- To view, visit http://gerrit.cloudera.org:8080/9102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 Gerrit-Change-Number: 9102 Gerrit-PatchSet: 3 Gerrit-Owner: anujphadke Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-6435: Disable codegen for CHAR literals.
Hello Philip Zeyliger, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9102 to look at the new patch set (#2). Change subject: IMPALA-6435: Disable codegen for CHAR literals. .. IMPALA-6435: Disable codegen for CHAR literals. Currently we do not codegen CHAR types. This change checks for CHAR literals in a expr and disables codegen. Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 --- M be/src/exprs/literal.cc M testdata/workloads/functional-query/queries/QueryTest/disable-codegen.test 2 files changed, 22 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/9102/2 -- To view, visit http://gerrit.cloudera.org:8080/9102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 Gerrit-Change-Number: 9102 Gerrit-PatchSet: 2 Gerrit-Owner: anujphadke Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR](refs/meta/config) Fixing typo: removing "heads" from "refs/sandbox"
Lars Volker has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9106 ) Change subject: Fixing typo: removing "heads" from "refs/sandbox" .. Fixing typo: removing "heads" from "refs/sandbox" Change-Id: I66637c16de543dcc056dd7c3983846822ae7d315 Reviewed-on: http://gerrit.cloudera.org:8080/9106 Reviewed-by: Lars Volker Tested-by: Lars Volker --- M project.config 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Lars Volker: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/9106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: refs/meta/config Gerrit-MessageType: merged Gerrit-Change-Id: I66637c16de543dcc056dd7c3983846822ae7d315 Gerrit-Change-Number: 9106 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR](refs/meta/config) Fixing typo: removing "heads" from "refs/sandbox"
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/9106 ) Change subject: Fixing typo: removing "heads" from "refs/sandbox" .. Patch Set 2: Verified+1 Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/9106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: refs/meta/config Gerrit-MessageType: comment Gerrit-Change-Id: I66637c16de543dcc056dd7c3983846822ae7d315 Gerrit-Change-Number: 9106 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 23 Jan 2018 21:00:10 + Gerrit-HasComments: No
[Impala-ASF-CR](refs/meta/config) Fixing typo: removing "heads" from "refs/sandbox"
Philip Zeyliger has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9106 Change subject: Fixing typo: removing "heads" from "refs/sandbox" .. Fixing typo: removing "heads" from "refs/sandbox" Change-Id: I66637c16de543dcc056dd7c3983846822ae7d315 --- M project.config 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/9106/2 -- To view, visit http://gerrit.cloudera.org:8080/9106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: refs/meta/config Gerrit-MessageType: newchange Gerrit-Change-Id: I66637c16de543dcc056dd7c3983846822ae7d315 Gerrit-Change-Number: 9106 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger
[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end
Kim Jin Chul has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 ) Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end .. Patch Set 11: Thanks you all for the reviews! -- To view, visit http://gerrit.cloudera.org:8080/8818 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec Gerrit-Change-Number: 8818 Gerrit-PatchSet: 11 Gerrit-Owner: Kim Jin Chul Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Kim Jin Chul Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 20:53:22 + Gerrit-HasComments: No
[Impala-ASF-CR](refs/meta/config) Expose refs/heads/2.x and refs/sandbox//*.
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/9101 ) Change subject: Expose refs/heads/2.x and refs/sandbox//*. .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/9101 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: refs/meta/config Gerrit-MessageType: comment Gerrit-Change-Id: I21f37eedc2a17275ed9112f0cd478d13f5c55953 Gerrit-Change-Number: 9101 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 23 Jan 2018 20:52:55 + Gerrit-HasComments: No
[Impala-ASF-CR](refs/meta/config) Expose refs/heads/2.x and refs/sandbox//*.
Lars Volker has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9101 ) Change subject: Expose refs/heads/2.x and refs/sandbox//*. .. Expose refs/heads/2.x and refs/sandbox//*. refs/heads/2.x is exposed to impala-public-jenkins. refs/sandbox//* is exposed to impala-public-jenkins as well as committers. Change-Id: I21f37eedc2a17275ed9112f0cd478d13f5c55953 Reviewed-on: http://gerrit.cloudera.org:8080/9101 Reviewed-by: Lars Volker Tested-by: Lars Volker --- M project.config 1 file changed, 6 insertions(+), 1 deletion(-) Approvals: Lars Volker: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/9101 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: refs/meta/config Gerrit-MessageType: merged Gerrit-Change-Id: I21f37eedc2a17275ed9112f0cd478d13f5c55953 Gerrit-Change-Number: 9101 Gerrit-PatchSet: 4 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR](refs/meta/config) Expose refs/heads/2.x and refs/sandbox//*.
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/9101 ) Change subject: Expose refs/heads/2.x and refs/sandbox//*. .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/9101 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: refs/meta/config Gerrit-MessageType: comment Gerrit-Change-Id: I21f37eedc2a17275ed9112f0cd478d13f5c55953 Gerrit-Change-Number: 9101 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 23 Jan 2018 20:42:22 + Gerrit-HasComments: No
[Impala-ASF-CR](refs/meta/config) Expose refs/heads/2.x and refs/sandbox//*.
Hello Lars Volker, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9101 to look at the new patch set (#3). Change subject: Expose refs/heads/2.x and refs/sandbox//*. .. Expose refs/heads/2.x and refs/sandbox//*. refs/heads/2.x is exposed to impala-public-jenkins. refs/sandbox//* is exposed to impala-public-jenkins as well as committers. Change-Id: I21f37eedc2a17275ed9112f0cd478d13f5c55953 --- M project.config 1 file changed, 6 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/9101/3 -- To view, visit http://gerrit.cloudera.org:8080/9101 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: refs/meta/config Gerrit-MessageType: newpatchset Gerrit-Change-Id: I21f37eedc2a17275ed9112f0cd478d13f5c55953 Gerrit-Change-Number: 9101 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR](refs/meta/config) Modify access rules
Philip Zeyliger has abandoned this change. ( http://gerrit.cloudera.org:8080/9104 ) Change subject: Modify access rules .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/9104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: refs/meta/config Gerrit-MessageType: abandon Gerrit-Change-Id: I0ad3cec33990043480decb3b1bb37893d5f64b6e Gerrit-Change-Number: 9104 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger
[Impala-ASF-CR](refs/meta/config) Expose refs/heads/2.x and refs/sandbox//*.
Hello Lars Volker, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9101 to look at the new patch set (#2). Change subject: Expose refs/heads/2.x and refs/sandbox//*. .. Expose refs/heads/2.x and refs/sandbox//*. refs/heads/2.x is exposed to impala-public-jenkins. refs/sandbox//* is exposed to impala-public-jenkins as well as committers. Change-Id: I21f37eedc2a17275ed9112f0cd478d13f5c55953 --- M project.config 1 file changed, 6 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/9101/2 -- To view, visit http://gerrit.cloudera.org:8080/9101 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: refs/meta/config Gerrit-MessageType: newpatchset Gerrit-Change-Id: I21f37eedc2a17275ed9112f0cd478d13f5c55953 Gerrit-Change-Number: 9101 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR](refs/meta/config) Modify access rules
Philip Zeyliger has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9104 Change subject: Modify access rules .. Modify access rules Change-Id: I0ad3cec33990043480decb3b1bb37893d5f64b6e --- M project.config 1 file changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/9104/1 -- To view, visit http://gerrit.cloudera.org:8080/9104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: refs/meta/config Gerrit-MessageType: newchange Gerrit-Change-Id: I0ad3cec33990043480decb3b1bb37893d5f64b6e Gerrit-Change-Number: 9104 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger
[Impala-ASF-CR] IMPALA-3193: Show table's comment on show tables
Dimitris Tsirogiannis has posted comments on this change. ( http://gerrit.cloudera.org:8080/8851 ) Change subject: IMPALA-3193: Show table's comment on show tables .. Patch Set 7: (12 comments) http://gerrit.cloudera.org:8080/#/c/8851/7/be/src/catalog/catalog.h File be/src/catalog/catalog.h: http://gerrit.cloudera.org:8080/#/c/8851/7/be/src/catalog/catalog.h@79 PS7, Line 79: Returns all matching table names and table comments, per Hive's : /// "SHOW TABLES ". Each table name of TTableInfo returned is unqualified. : /// Comment cannot be set when table is not loaded. If pattern is NULL, match all : /// tables otherwise match only those tables that match the pattern string. Patterns : /// are "p1|p2|p3" where | denotes choice, and each pN may contain wildcards denoted : /// by '*' which match all strings. Table comments can be empty if table is not loaded. Let's fix this comment a bit: "Returns information of all matching tables in 'db' based on the input 'pattern'. If pattern is NULL, return information of all tables in 'db' otherwise return information of only those tables whose names match the pattern string. Patterns are "p1|p2|p3" where | denotes choice, and each pN may contain wildcards denoted by '*' which match all strings. Table information includes the fully qualified table name and, optionally, the table's comment, if set. Table comment will be empty if a table is not loaded." http://gerrit.cloudera.org:8080/#/c/8851/7/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/8851/7/be/src/service/client-request-state.cc@280 PS7, Line 280: if (!tbl_info.__isset.comment) DCHECK(tbl_info.comment.empty()); : comments.push_back(tbl_info.comment); This check is a bit confusing. Ideally, you want to ensure that names and comments will have the same number of elements. So, either check that comment is always set or set it yourself to an empty string and put it to 'comments'. http://gerrit.cloudera.org:8080/#/c/8851/7/be/src/service/frontend.h File be/src/service/frontend.h: http://gerrit.cloudera.org:8080/#/c/8851/7/be/src/service/frontend.h@57 PS7, Line 57: /// Returns all matching table names and table comments. : /// If pattern is NULL, match all tables otherwise match only those tables that : /// match the pattern string. Patterns are "p1|p2|p3" where | denotes choice, : /// and each pN may contain wildcards denoted by '*' which match all strings. : /// The TSessionState parameter is used to filter results of metadata operations when : /// authorization is enabled. If this is a user initiated request, it should : /// be set to the user's current session. If this is an Impala internal request, : /// the session should be set to NULL which will skip privilege checks returning all : /// results. Table comments can be empty if table is not loaded. See previous comment about this comment :). http://gerrit.cloudera.org:8080/#/c/8851/7/be/src/service/frontend.cc File be/src/service/frontend.cc: http://gerrit.cloudera.org:8080/#/c/8851/7/be/src/service/frontend.cc@145 PS7, Line 145: return JniUtil::CallJniMethod(fe_, get_tables_info_id_, params, : tables_info_result); nit: check if it fits in a single line http://gerrit.cloudera.org:8080/#/c/8851/7/common/thrift/Frontend.thrift File common/thrift/Frontend.thrift: http://gerrit.cloudera.org:8080/#/c/8851/7/common/thrift/Frontend.thrift@91 PS7, Line 91: Comment needs to indicate when and if this is set. For instance, : // it's not clear if this is set if the table is loaded but there is no comment. I believe this was my comment. The point was to explicitly specify the behavior re setting the comment field. http://gerrit.cloudera.org:8080/#/c/8851/7/common/thrift/Frontend.thrift@96 PS7, Line 96: // getTablesInfo returns a list of table names and a list of table comment. comments is not consistent with the struct definition, plz update http://gerrit.cloudera.org:8080/#/c/8851/7/common/thrift/Frontend.thrift@98 PS7, Line 98: tableinfos tables_info http://gerrit.cloudera.org:8080/#/c/8851/6/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/8851/6/fe/src/main/java/org/apache/impala/catalog/Table.java@407 PS6, Line 407: blic > Done. By the way, may I know why you would not prefer final keyword here? I My thinking is that we're not particularly consistent with using final in our java-based codebase and we typically don't use it unless we absolutely have to. I see how it can help in some cases. So, you may keep it if you want. http://gerrit.cloudera.org:8080/
[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8818 ) Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end .. IMPALA-3942: Fix wrongly escaped string literal in front-end String literal can be wrapped by either single or double quotes. There are some holes in escaping the string literal: - toSql() always returns strings that are single quoted, resulting in improper escaping in the output if the original string was actually double quoted. - It is not always possible to determine if a string "should" be single or double quoted. e.g. concat('a', "b") The solution is to normalize any string which comes from user's given string or a generated string (e.g. constant fold by the rewritter rule) to single quotes. Testing: Added unit tests into expr-test, ToSqlTest Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec Reviewed-on: http://gerrit.cloudera.org:8080/8818 Reviewed-by: Alex Behm Tested-by: Impala Public Jenkins --- M be/src/exprs/expr-test.cc M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java 3 files changed, 93 insertions(+), 4 deletions(-) Approvals: Alex Behm: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/8818 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec Gerrit-Change-Number: 8818 Gerrit-PatchSet: 11 Gerrit-Owner: Kim Jin Chul Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Kim Jin Chul Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-3942: Fix wrongly escaped string literal in front-end
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8818 ) Change subject: IMPALA-3942: Fix wrongly escaped string literal in front-end .. Patch Set 10: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/8818 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibc4b5f5d8ffaa8feb96a466959427a04b3b06fec Gerrit-Change-Number: 8818 Gerrit-PatchSet: 10 Gerrit-Owner: Kim Jin Chul Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Kim Jin Chul Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 20:35:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6435: Disable codegen for CHAR literals.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/9102 ) Change subject: IMPALA-6435: Disable codegen for CHAR literals. .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/9102/1/be/src/exprs/literal.cc File be/src/exprs/literal.cc: http://gerrit.cloudera.org:8080/#/c/9102/1/be/src/exprs/literal.cc@a392 PS1, Line 392: This seems wrong; you deleted the handlig for TYPE_STRING. -- To view, visit http://gerrit.cloudera.org:8080/9102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 Gerrit-Change-Number: 9102 Gerrit-PatchSet: 1 Gerrit-Owner: anujphadke Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 23 Jan 2018 20:32:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR](2.x) IMPALA-6318: Revert "Adjustment for hanging query cancellation test"
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/9100 ) Change subject: IMPALA-6318: Revert "Adjustment for hanging query cancellation test" .. Patch Set 1: GVO failed, but we know this test is flaky. (There's a fix for that.) "[gw12] FAILED query_test/test_observability.py::TestObservability::test_query_profile_thrift_timestamps" -- To view, visit http://gerrit.cloudera.org:8080/9100 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I51acee49b5a17c4852410b7568fd1d092b114a6d Gerrit-Change-Number: 9100 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 20:30:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6435: Disable codegen for CHAR literals.
anujphadke has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9102 Change subject: IMPALA-6435: Disable codegen for CHAR literals. .. IMPALA-6435: Disable codegen for CHAR literals. Currently we do not codegen CHAR types. This change checks for CHAR literals in a expr and disables codegen. Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 --- M be/src/exprs/literal.cc M testdata/workloads/functional-query/queries/QueryTest/disable-codegen.test 2 files changed, 18 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/9102/1 -- To view, visit http://gerrit.cloudera.org:8080/9102 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7e4e27350c53bc69ce412a004e392e7480214f73 Gerrit-Change-Number: 9102 Gerrit-PatchSet: 1 Gerrit-Owner: anujphadke
[Impala-ASF-CR](refs/meta/config) Expose refs/heads/2.x and refs/scratch/2.x-staging to impala-public-jenkins.
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/9101 ) Change subject: Expose refs/heads/2.x and refs/scratch/2.x-staging to impala-public-jenkins. .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/9101 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: refs/meta/config Gerrit-MessageType: comment Gerrit-Change-Id: I21f37eedc2a17275ed9112f0cd478d13f5c55953 Gerrit-Change-Number: 9101 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 23 Jan 2018 20:25:58 + Gerrit-HasComments: No
[Impala-ASF-CR](refs/meta/config) Expose refs/heads/2.x and refs/scratch/2.x-staging to impala-public-jenkins.
Philip Zeyliger has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9101 Change subject: Expose refs/heads/2.x and refs/scratch/2.x-staging to impala-public-jenkins. .. Expose refs/heads/2.x and refs/scratch/2.x-staging to impala-public-jenkins. Change-Id: I21f37eedc2a17275ed9112f0cd478d13f5c55953 --- M project.config 1 file changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/9101/1 -- To view, visit http://gerrit.cloudera.org:8080/9101 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: refs/meta/config Gerrit-MessageType: newchange Gerrit-Change-Id: I21f37eedc2a17275ed9112f0cd478d13f5c55953 Gerrit-Change-Number: 9101 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger
[Impala-ASF-CR](2.x) IMPALA-6318: Revert "Adjustment for hanging query cancellation test"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9100 ) Change subject: IMPALA-6318: Revert "Adjustment for hanging query cancellation test" .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1783/ -- To view, visit http://gerrit.cloudera.org:8080/9100 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I51acee49b5a17c4852410b7568fd1d092b114a6d Gerrit-Change-Number: 9100 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 20:24:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6059: Enhance ltrim()/rtrim() functions to trim any set of characters.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8349 ) Change subject: IMPALA-6059: Enhance ltrim()/rtrim() functions to trim any set of characters. .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1787/ -- To view, visit http://gerrit.cloudera.org:8080/8349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a5ae3f59762e70c3268a01e14ed57a9e36b8d79 Gerrit-Change-Number: 8349 Gerrit-PatchSet: 12 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Tue, 23 Jan 2018 20:21:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6059: Enhance ltrim()/rtrim() functions to trim any set of characters.
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/8349 ) Change subject: IMPALA-6059: Enhance ltrim()/rtrim() functions to trim any set of characters. .. Patch Set 12: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8349 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8a5ae3f59762e70c3268a01e14ed57a9e36b8d79 Gerrit-Change-Number: 8349 Gerrit-PatchSet: 12 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Tue, 23 Jan 2018 20:20:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption
Hello Zoltan Borok-Nagy, Sailesh Mukil, Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9032 to look at the new patch set (#8). Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption .. IMPALA-6219: Use AES-GCM for spill-to-disk encryption AES-GCM can be very fast(~10 times faster than CFB+SHA256), but it requires an instruction that Impala can currently run without (CLMUL). In order to be fast, we dispatch to GCM mode at run-time based on the CPU and OpenSSL version. Testing: run runtime tmp-file-mgr-test, openssl-util-test, buffer-pool-test and buffered-tuple-stream-test. add two cases GcmIntegrity & EncryptoArbitraryLength for openssl-util-test Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24 --- M be/src/runtime/tmp-file-mgr.cc M be/src/util/cpu-info.cc M be/src/util/cpu-info.h M be/src/util/openssl-util-test.cc M be/src/util/openssl-util.cc M be/src/util/openssl-util.h 6 files changed, 232 insertions(+), 78 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/9032/8 -- To view, visit http://gerrit.cloudera.org:8080/9032 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1ea87b82a8897ee8bfa187715ac1c52883790d24 Gerrit-Change-Number: 9032 Gerrit-PatchSet: 8 Gerrit-Owner: Xianda Ke Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Xianda Ke Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-3916: Reserve SQL:2016 reserved words
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/9096 ) Change subject: IMPALA-3916: Reserve SQL:2016 reserved words .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/9096/1/be/src/common/global-flags.cc File be/src/common/global-flags.cc: http://gerrit.cloudera.org:8080/#/c/9096/1/be/src/common/global-flags.cc@193 PS1, Line 193: DEFINE_bool(reserve_impala3_words, true, "Reserve impala 3 reserved words."); I think we'll want to change this list again, so we may want to leave ourselves some wiggle room. Perhaps: --reserved_words_list=[IMPALA2.11,IMPALA3.0], with default=IMPALA3.0? By making it string-valued (or, really, enum-valued), we have the potential to do IMPALA4.0 or IMPALA3.3 in the future. I don't think this will complicate your parsing code significantly. http://gerrit.cloudera.org:8080/#/c/9096/1/fe/src/main/cup/sql-parser.cup File fe/src/main/cup/sql-parser.cup: http://gerrit.cloudera.org:8080/#/c/9096/1/fe/src/main/cup/sql-parser.cup@301 PS1, Line 301: // A word is an arbitrary token composed of digits and at least one letter. Is there documentation here you might want to add about how all the keyword stuff is handled? Thanks! http://gerrit.cloudera.org:8080/#/c/9096/1/fe/src/main/jflex/sql-scanner.flex File fe/src/main/jflex/sql-scanner.flex: http://gerrit.cloudera.org:8080/#/c/9096/1/fe/src/main/jflex/sql-scanner.flex@62 PS1, Line 62: static final HashSet sql16ReservedWords = new HashSet<>(Arrays.asList( Are our lists of reserved words now effectively split between sql-parser.cup (in its KW_* list, lines 251-279) and here? Should we have them all in one place, one way or another? -- To view, visit http://gerrit.cloudera.org:8080/9096 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If1b295e6a77e840cf1b794c2eb73e1b9d2b8ddd6 Gerrit-Change-Number: 9096 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Tue, 23 Jan 2018 19:49:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6383: free memory after skipping parquet row groups
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/9059 ) Change subject: IMPALA-6383: free memory after skipping parquet row groups .. Patch Set 4: I'm happy to +2 this one. Alex, do you have any remaining questions? -- To view, visit http://gerrit.cloudera.org:8080/9059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95713675455f7635fa3f72616b166f35e2a46c1a Gerrit-Change-Number: 9059 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 19:34:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3282: Adds regexp escape built-in function
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/8900 ) Change subject: IMPALA-3282: Adds regexp_escape built-in function .. Patch Set 7: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/8900 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I84c3e0ded26f6eb20794c38b75be9b25cd111e4b Gerrit-Change-Number: 8900 Gerrit-PatchSet: 7 Gerrit-Owner: Kim Jin Chul Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Kim Jin Chul Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 19:16:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6383: free memory after skipping parquet row groups
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/9059 ) Change subject: IMPALA-6383: free memory after skipping parquet row groups .. Patch Set 4: Anyone want to do a +2 review? -- To view, visit http://gerrit.cloudera.org:8080/9059 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I95713675455f7635fa3f72616b166f35e2a46c1a Gerrit-Change-Number: 9059 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 19:13:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4835: Part 3: switch I/O buffers to buffer pool
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8966 ) Change subject: IMPALA-4835: Part 3: switch I/O buffers to buffer pool .. Patch Set 11: Fixed the remaining core tests. -- To view, visit http://gerrit.cloudera.org:8080/8966 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic09c6196b31e55b301df45cc56d0b72cfece6786 Gerrit-Change-Number: 8966 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 19:12:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4835: Part 3: switch I/O buffers to buffer pool
Hello Bikramjeet Vig, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8966 to look at the new patch set (#11). Change subject: IMPALA-4835: Part 3: switch I/O buffers to buffer pool .. IMPALA-4835: Part 3: switch I/O buffers to buffer pool This is the final patch to switch the Disk I/O manager to allocate all buffer from the buffer pool and to reserve the buffers required for a query upfront. * The planner reserves enough memory to run a single scanner per scan node. * The multi-threaded scan node must increase reservation before spinning up more threads. * The scanner implementations must be careful to stay within their assigned reservation. The row-oriented scanners were most straightforward, since they only have a single scan range active at a time. A single I/O buffer is sufficient to scan the whole file but more I/O buffers can improve I/O throughput. Parquet is more complex because it issues a scan range per column and the sizes of the columns on disk are not known during planning. To deal with this, the reservation in the frontend is based on a heuristic involving the file size and # columns. The Parquet scanner can then divvy up reservation to columns based on the size of column data on disk. I adjusted how the 'mem_limit' is divided between buffer pool and non buffer pool memory for low mem_limits to account for the increase in buffer pool memory. Testing: * Added more planner tests to cover reservation calcs for scan node. * Test scanners for all file formats with the reservation denial debug action, to test behaviour when the scanners hit reservation limits. * Updated memory and buffer pool limits for tests. Perf: I ran TPC-H and targeted perf locally comparing with master. Both showed small improvements of a few percent and no regressions of note. I still need to run cluster perf tests. Change-Id: Ic09c6196b31e55b301df45cc56d0b72cfece6786 --- M be/src/exec/hdfs-parquet-scanner.cc M be/src/exec/hdfs-parquet-scanner.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scan-node-mt.cc M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scan-node.h M be/src/exec/parquet-column-readers.cc M be/src/exec/parquet-column-readers.h M be/src/exec/scanner-context.cc M be/src/exec/scanner-context.h M be/src/runtime/bufferpool/reservation-tracker-test.cc M be/src/runtime/bufferpool/reservation-util.cc M be/src/runtime/io/disk-io-mgr-stress-test.cc M be/src/runtime/io/disk-io-mgr-stress.cc M be/src/runtime/io/disk-io-mgr-stress.h M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/request-context.cc M be/src/runtime/io/request-context.h M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc M be/src/runtime/tmp-file-mgr.cc M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/util/BitUtil.java M fe/src/test/java/org/apache/impala/util/BitUtilTest.java M testdata/workloads/functional-planner/queries/PlannerTest/constant-folding.test M testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test M testdata/workloads/functional-planner/queries/PlannerTest/fk-pk-join-detection.test M testdata/workloads/functional-planner/queries/PlannerTest/max-row-size.test M testdata/workloads/functional-planner/queries/PlannerTest/min-max-runtime-filters.test M testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test M testdata/workloads/functional-planner/queries/PlannerTest/partition-pruning.test M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test M testdata/workloads/functional-planner/queries/PlannerTest/sort-expr-materialization.test M testdata/workloads/functional-planner/queries/PlannerTest/spillable-buffer-sizing.test M testdata/workloads/functional-planner/queries/PlannerTest/tablesample.test M testdata/workloads/functional-planner/queries/PlannerTest/union.test M testdata/workloads/functional-query/queries/QueryTest/admission-reject-min-reservation.test M testdata/workloads/functional-query/queries/QueryTest/analytic-fns.test M testdata/workloads/functional-query/queries/QueryTest/codegen-mem-limit.test M testdata/workloads/functional-query/queries/QueryTest/disk-spill-encryption.test M testdata/workloads/functional-query/queries/QueryTest/explain-level0.test M testdata/workloads/functional-query/queries/QueryTest/explain-level1.test M testdata/workloads/functional-query/queries/QueryTest/explain-level2.test M testdata/workloads/functional-query/queries/QueryTest/explain-level3.test M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch.test M testdata/workloads/functional-query/queries/QueryTest/ru
[Impala-ASF-CR] IMPALA-6190/6246: Add instances tab and event sequence
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8758 ) Change subject: IMPALA-6190/6246: Add instances tab and event sequence .. Patch Set 17: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1786/ -- To view, visit http://gerrit.cloudera.org:8080/8758 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I626456b6afa9101ffd5cda10c4096d63d7f9 Gerrit-Change-Number: 8758 Gerrit-PatchSet: 17 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 18:58:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6190/6246: Add instances tab and event sequence
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/8758 ) Change subject: IMPALA-6190/6246: Add instances tab and event sequence .. Patch Set 17: Code-Review+2 Fixed a rogue DCHECK, ran expr-test and test_data_errors locally. Carrying Tim's +2. -- To view, visit http://gerrit.cloudera.org:8080/8758 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I626456b6afa9101ffd5cda10c4096d63d7f9 Gerrit-Change-Number: 8758 Gerrit-PatchSet: 17 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 23 Jan 2018 18:58:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6190/6246: Add instances tab and event sequence
Hello Michael Ho, Joe McDonnell, Tim Armstrong, Dan Hecht, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8758 to look at the new patch set (#16). Change subject: IMPALA-6190/6246: Add instances tab and event sequence .. IMPALA-6190/6246: Add instances tab and event sequence This change adds tracking of the current state during the execution of a fragment instance. The current state is then reported back to the coordinator and exposed to users via a new tab in the query detail debug webpage. This change also adds an event timeline to fragment instances in the query profile. The timeline measures the time since backend-local query start at which particular events complete. Events are derived from the current state of the execution of a fragment instance. For example: - Prepare Finished: 13.436ms (13.436ms) - First Batch Produced: 1s022ms (1s008ms) - First Batch Sent: 1s022ms (455.558us) - ExecInternal Finished: 2s783ms (1s760ms) I added automated tests for both extensions and additionally verified the change by manual inspection. Here are the TPCH performance comparison results between this change and the previous commit on a 16 node cluster. ++---+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | ++---+-++++ | TPCH(_300) | parquet / none / none | 18.47 | -0.94% | 9.72 | -1.08% | ++---+-++++ ++--+---++-++++-+---+ | Workload | Query| File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Num Clients | Iters | ++--+---++-++++-+---+ | TPCH(_300) | TPCH-Q5 | parquet / none / none | 48.88 | 46.93 | +4.15% | 0.14%| 3.61%| 1 | 3 | | TPCH(_300) | TPCH-Q13 | parquet / none / none | 21.64 | 21.15 | +2.29% | 2.06%| 1.84%| 1 | 3 | | TPCH(_300) | TPCH-Q11 | parquet / none / none | 1.71 | 1.70| +1.12% | 0.54%| 2.51%| 1 | 3 | | TPCH(_300) | TPCH-Q18 | parquet / none / none | 33.15 | 32.79 | +1.09% | 0.13%| 2.03%| 1 | 3 | | TPCH(_300) | TPCH-Q14 | parquet / none / none | 5.95 | 5.90| +0.82% | 2.19%| 0.49%| 1 | 3 | | TPCH(_300) | TPCH-Q1 | parquet / none / none | 13.99 | 13.90 | +0.63% | 0.25%| 1.39%| 1 | 3 | | TPCH(_300) | TPCH-Q2 | parquet / none / none | 3.44 | 3.44| +0.00% | * 20.29% * | * 20.76% * | 1 | 3 | | TPCH(_300) | TPCH-Q6 | parquet / none / none | 1.21 | 1.22| -0.01% | 0.06%| 0.06%| 1 | 3 | | TPCH(_300) | TPCH-Q20 | parquet / none / none | 3.51 | 3.51| -0.11% | 7.15%| 7.30%| 1 | 3 | | TPCH(_300) | TPCH-Q16 | parquet / none / none | 6.89 | 6.91| -0.21% | 0.65%| 0.55%| 1 | 3 | | TPCH(_300) | TPCH-Q4 | parquet / none / none | 4.78 | 4.80| -0.38% | 0.06%| 0.59%| 1 | 3 | | TPCH(_300) | TPCH-Q19 | parquet / none / none | 30.78 | 31.04 | -0.83% | 0.45%| 1.03%| 1 | 3 | | TPCH(_300) | TPCH-Q22 | parquet / none / none | 6.06 | 6.12| -1.02% | 1.51%| 2.12%| 1 | 3 | | TPCH(_300) | TPCH-Q10 | parquet / none / none | 9.43 | 9.58| -1.54% | 0.69%| 3.30%| 1 | 3 | | TPCH(_300) | TPCH-Q21 | parquet / none / none | 93.41 | 95.18 | -1.86% | 0.08%| 0.81%| 1 | 3 | | TPCH(_300) | TPCH-Q15 | parquet / none / none | 3.40 | 3.47| -1.99% | 0.72%| 1.27%| 1 | 3 | | TPCH(_300) | TPCH-Q7 | parquet / none / none | 44.98 | 46.24 | -2.71% | 1.83%| 1.27%| 1 | 3 | | TPCH(_300) | TPCH-Q3 | parquet / none / none | 28.06 | 29.11 | -3.61% | 1.62%| 1.23%| 1 | 3 | | TPCH(_300) | TPCH-Q12 | parquet / none / none | 3.15 | 3.28| -3.80% | 0.96%| 1.32%| 1 | 3 | | TPCH(_300) | TPCH-Q9 | parquet / none / none | 29.47 | 30.80 | -4.30% | 0.29%| 0.34%| 1 | 3 | | TPCH(_300) | TPCH-Q17