[Impala-ASF-CR](2.x) IMPALA-7095: clean up scan node profiles
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12848 ) Change subject: IMPALA-7095: clean up scan node profiles .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I77286282d42e7764bfdf94c7ec47cec9d743f787 Gerrit-Change-Number: 12848 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 26 Mar 2019 05:46:27 + Gerrit-HasComments: No
[Impala-ASF-CR] Re-land IMPALA-5393. Use THREAD LOCAL state for regexp
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12845 ) Change subject: Re-land IMPALA-5393. Use THREAD_LOCAL state for regexp .. Re-land IMPALA-5393. Use THREAD_LOCAL state for regexp This re-lands commit 6e8c330f40da087ca0d8ba844cd9d97a8e60ff67 which was reverted in d3428a58d8f54d1a64d5aeb1af3f76b7ffcb53d0. The revert was due to an assumption that this commit depended on the new version of re2 (which was correctly reverted due to a toolchain issue). In fact this commit does not depend on any toolchain changes. Original commit message follows This changes the built-in regexp-related UDFs to use THREAD_LOCAL re2::RE instances instead of FRAGMENT_LOCAL. Although re2::RE is thread-safe, it achieves that thread safety through a certain amount of locking. Using thread-local regexps improves performance substantially. I ran a simple test query: select sum(l_linenumber) from item_20x where length(regexp_extract(l_shipinstruct, '.*', 0)) > 0 on a table with three underlying parquet files (thus getting 3 scanner threads). Prior to this change, the query took ~60 seconds and burned 2m16sec CPU time. With this change, it took ~19sec and 43s CPU time. For a query with more scanner threads, the improvement should be even more dramatic. The only potential downside of this change is slightly increased memory consumption by having one RE instance per thread, but the REs themselves should be small relative to all of the other per-scanner-thread memory. Change-Id: I9ae0703efeb2429813b2a712f1accf1b0a4a409e Reviewed-on: http://gerrit.cloudera.org:8080/12845 Reviewed-by: Lars Volker Tested-by: Impala Public Jenkins --- M be/src/exprs/string-functions-ir.cc 1 file changed, 6 insertions(+), 6 deletions(-) Approvals: Lars Volker: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9ae0703efeb2429813b2a712f1accf1b0a4a409e Gerrit-Change-Number: 12845 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12844 ) Change subject: IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 .. IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 Fixes in IMPALA-8317 and IMPALA-8337 introduced third-party dependencies in Impala shell which is problematic in multi-Python environment. This patch rewrites the fixes using an alternative solution when dealing with duplicate options without any third-party dependencies. For example: [impala] keyval=msg1=hello,keyval=msg2=world Testing: - Ran all shell tests on Python 2.6 and 2.7. - Ran make_shell_tarball.sh and ran Impala shell from the tarball without any issue. Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 Reviewed-on: http://gerrit.cloudera.org:8080/12844 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M infra/python/deps/requirements.txt M shell/option_parser.py M tests/shell/good_impalarc 3 files changed, 8 insertions(+), 28 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12844 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 Gerrit-Change-Number: 12844 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12844 ) Change subject: IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12844 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 Gerrit-Change-Number: 12844 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 26 Mar 2019 04:03:45 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7095: clean up scan node profiles
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12848 ) Change subject: IMPALA-7095: clean up scan node profiles .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3955/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I77286282d42e7764bfdf94c7ec47cec9d743f787 Gerrit-Change-Number: 12848 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 26 Mar 2019 01:55:04 + Gerrit-HasComments: No
[Impala-ASF-CR] Re-land IMPALA-5393. Use THREAD LOCAL state for regexp
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12845 ) Change subject: Re-land IMPALA-5393. Use THREAD_LOCAL state for regexp .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ae0703efeb2429813b2a712f1accf1b0a4a409e Gerrit-Change-Number: 12845 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 26 Mar 2019 00:52:20 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7095: clean up scan node profiles
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12848 ) Change subject: IMPALA-7095: clean up scan node profiles .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2538/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I77286282d42e7764bfdf94c7ec47cec9d743f787 Gerrit-Change-Number: 12848 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 26 Mar 2019 00:41:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/12481 )
Change subject: IMPALA-7368: Add initial support for DATE type
..
Patch Set 14:
(5 comments)
Looking pretty close to good. I just have some questions digging into the
function resolution stuff. I don't have a really specific example of something
I think is wrong, but I want to make sure we're pretty clear on what our
function resolution policy is and that we think it won't have any unexpected
user behavior. I'll spend some more time on that tomorrow
http://gerrit.cloudera.org:8080/#/c/12481/14/fe/src/main/java/org/apache/impala/analysis/PartitionSpec.java
File fe/src/main/java/org/apache/impala/analysis/PartitionSpec.java:
http://gerrit.cloudera.org:8080/#/c/12481/14/fe/src/main/java/org/apache/impala/analysis/PartitionSpec.java@52
PS14, Line 52: public List getPartitionSpecKeyValues() {
return partitionSpec_; }
Do we ever expect this to get called prior to analysis?
Maybe it makes sense to add a bool analyzed_ and
Preconditions.checkState(analyzed_); here to ensure that we don't ever return
the pre-analysis KVs? This makes sure that we don't expose out the
partitionSpec_ before it becomes immutable.
http://gerrit.cloudera.org:8080/#/c/12481/14/fe/src/main/java/org/apache/impala/analysis/PartitionSpec.java@156
PS14, Line 156: public List toThrift() {
same (Precondition on analysis if possible)
http://gerrit.cloudera.org:8080/#/c/12481/11/fe/src/main/java/org/apache/impala/catalog/Function.java
File fe/src/main/java/org/apache/impala/catalog/Function.java:
http://gerrit.cloudera.org:8080/#/c/12481/11/fe/src/main/java/org/apache/impala/catalog/Function.java@191
PS11, Line 191: public boolean compare(Function other, CompareMode mode) {
> I took a quick look at the SQL ANSI standard, but couldn't find anything ab
hm, I spent some time researching this across multiple databases:
DB2's logic is very clearly described:
https://www.ibm.com/support/knowledgecenter/en/SSEPEK_11.0.0/sqlref/src/tpc/db2z_determinbestfitfunction.html
(it seems to go left-to-right)
Postgres's is also pretty well described and seems relatively similar to what's
done here:
https://www.postgresql.org/docs/10/typeconv-func.html
Hive's also seems to bail earlier with an "ambiguous" error when doing method
resolution for UDFs: "Closest match is defined as the one that requires the
least number of arguments to be converted. In case more than one matches are
found, the method throws an ambiguous method exception"
Comparing Postgres to what's implemented here, it seems like our two gaps are:
1) they have a well-defined concept of "preferred type". I checked the pg
catalog and for the 'date/time' category, timestamptz is the 'preferred type':
=> select oid,typname, typispreferred from pg_type where typcategory = 'D';
oid | typname | typispreferred
---+-+
702 | abstime | f
1082 | date| f
1083 | time| f
1114 | timestamp | f
1184 | timestamptz | t
1266 | timetz | f
12401 | time_stamp | f
(7 rows)
It seems that, because it only allows implicit conversion to "preferred" types,
it's somewhat more strict than what we do today, though.
2) they have a concept of 'unknown' type to represent string literals in
queries as being distinct from an expression with a defined type (eg any other
expression than a string literal). I wonder if we should be treating
StringLiteral casts differently than other exprs?
http://gerrit.cloudera.org:8080/#/c/12481/11/fe/src/main/java/org/apache/impala/catalog/Function.java@236
PS11, Line 236: // Check trailing varargs.
I think we could simplify this code a lot by adding some utility function like:
Type[] tryExtendArgsToLength(int numArgs) {
if (!hasVarArgs_ || argTypes_.length <= numArgs) return argTypes_;
Type[] ret = Arrays.copyOf(argTypes_, numArgs);
for (int i = argTypes_.length; i < numArgs; i++) {
ret[i] = getVarArgsType();
}
return ret;
}
then in these comparison functions, we can extend the varargs to match the
user-provided call types and do the rest of the logic on fixed-length argument
lists, without having to duplicate code between L227-234 and L239-247.
http://gerrit.cloudera.org:8080/#/c/12481/11/fe/src/main/java/org/apache/impala/util/FunctionUtils.java
File fe/src/main/java/org/apache/impala/util/FunctionUtils.java:
http://gerrit.cloudera.org:8080/#/c/12481/11/fe/src/main/java/org/apache/impala/util/FunctionUtils.java@243
PS11, Line 243: return max_func;
it's interesting that our function resolution returns the first fit, from the
order returned by the catalog. Best I can tell, we never will fail with an
"ambiguous resolution" error. Now that we have both DATE and TIMESTAMP (and
assumedly we'll add DATETIME at some point?) is there really no case that we
want to force an explicit match by
[Impala-ASF-CR](2.x) IMPALA-7095: clean up scan node profiles
Quanlong Huang has posted comments on this change. (
http://gerrit.cloudera.org:8080/12848 )
Change subject: IMPALA-7095: clean up scan node profiles
..
Patch Set 1:
Conflicts: be/src/exec/data-source-scan-node.cc
334 while (true) {
335 {
336 SCOPED_TIMER(materialize_tuple_timer());
337 // copy rows until we hit the limit/capacity or until we exhaust
input_batch_
338 while (!ReachedLimit() && !row_batch->AtCapacity() &&
InputBatchHasNext()) {
339 <<< HEAD
340 RETURN_IF_ERROR(MaterializeNextRow(tuple_pool, tuple));
341 ===
342 RETURN_IF_ERROR(MaterializeNextRow(state->local_time_zone(),
tuple_pool, tuple));
343 ++rows_read;
344 >>> 5d67245... IMPALA-7095: clean up scan node profiles
345 int row_idx = row_batch->AddRow();
346 TupleRow* tuple_row = row_batch->GetRow(row_idx);
347 tuple_row->SetTuple(tuple_idx_, tuple);
348
349 if (ExecNode::EvalConjuncts(evals, num_conjuncts, tuple_row)) {
350 row_batch->CommitLastRow();
351 tuple = reinterpret_cast(
352 reinterpret_cast(tuple) + tuple_desc_->byte_size());
353 ++num_rows_returned_;
354 }
355 ++next_row_idx_;
356 }
357 if (ReachedLimit() || row_batch->AtCapacity() || input_batch_->eos) {
358 *eos = ReachedLimit() || input_batch_->eos;
359 COUNTER_SET(rows_returned_counter_, num_rows_returned_);
360 COUNTER_ADD(rows_read_counter_, rows_read);
361 return Status::OK();
362 }
363 }
How to fix:
Use the old usage of "MaterializeNextRow(tuple_pool, tuple))":
MaterializeNextRow(tuple_pool, tuple);
++rows_read;
--
To view, visit http://gerrit.cloudera.org:8080/12848
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I77286282d42e7764bfdf94c7ec47cec9d743f787
Gerrit-Change-Number: 12848
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 26 Mar 2019 00:02:23 +
Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7095: clean up scan node profiles
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/12848 to review the following change. Change subject: IMPALA-7095: clean up scan node profiles .. IMPALA-7095: clean up scan node profiles Add counters to scan node implementations where they make sense but were missing (e.g. row batch queue counters for multithread Kudu scans) and remove them where they don't make sense (e.g. scanner thread counters for non-multithreaded scans). Refactors the multithreaded Kudu and HDFS scans to share logic via composition (single inheritance doesn't work for this case), which enables the same set of counters to be maintained with shared code. The row batch queueing and thread tracking is now shared. I looked at combining the logic around 'status_', 'lock_' and 'done_' between the two but the details were different enough that it didn't seem worth abstracting. Adds a PeakScannerThreadConcurrency counter - this answers a common question. Fixes RowsRead for data source scans. Fix some of the comments to be more accurate/useful. Testing: Ran exhaustive tests. Ran various types of scans (HDFS, Kudu, HBase, Data source) and inspected the profile output manually. Change-Id: I77286282d42e7764bfdf94c7ec47cec9d743f787 Reviewed-on: http://gerrit.cloudera.org:8080/10810 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M be/src/exec/data-source-scan-node.cc M be/src/exec/data-source-scan-node.h M be/src/exec/hbase-scan-node.cc M be/src/exec/hbase-scan-node.h M be/src/exec/hbase-table-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scan-node-mt.h M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scan-node.h M be/src/exec/hdfs-text-scanner.cc M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-scan-node-mt.cc M be/src/exec/kudu-scan-node.cc M be/src/exec/kudu-scan-node.h M be/src/exec/scan-node.cc M be/src/exec/scan-node.h M be/src/runtime/fragment-instance-state.cc M be/src/util/blocking-queue.h M be/src/util/thread.h 20 files changed, 496 insertions(+), 357 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/12848/1 -- To view, visit http://gerrit.cloudera.org:8080/12848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: I77286282d42e7764bfdf94c7ec47cec9d743f787 Gerrit-Change-Number: 12848 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-2990: timeout unresponsive queries in coordinator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12299 ) Change subject: IMPALA-2990: timeout unresponsive queries in coordinator .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2537/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987 Gerrit-Change-Number: 12299 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 25 Mar 2019 23:42:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12846 ) Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/12846/4/bin/impala-config.sh File bin/impala-config.sh: http://gerrit.cloudera.org:8080/#/c/12846/4/bin/impala-config.sh@37 PS4, Line 37: # parse command line options > It seems like options to impala-config.sh are currently passed by environme Yeah, I think it would be best to avoid making this a special option that behaves differently to everything else. A lot of scripts source impala-config.sh without arguments. This works today for the two valid ways to set options - via environment variables or by setting them in impala-config-local.sh/impala-config-branch.sh -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 25 Mar 2019 23:34:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12844 ) Change subject: IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 .. Patch Set 4: > Patch Set 4: Code-Review+2 > > Hmm, my understanding was that we would not need duplicate keys if var would > get its own config group. > impala-shell --var=msg1=hello --var=msg2=hello > could be expressed by: > [impala.var] > msg1=hello > msg2=world Ah, I see what you mean. -- To view, visit http://gerrit.cloudera.org:8080/12844 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 Gerrit-Change-Number: 12844 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 25 Mar 2019 23:11:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12844 ) Change subject: IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12844 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 Gerrit-Change-Number: 12844 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 25 Mar 2019 23:11:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12844 ) Change subject: IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3954/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12844 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 Gerrit-Change-Number: 12844 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 25 Mar 2019 23:11:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2990: timeout unresponsive queries in coordinator
Thomas Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/12299 )
Change subject: IMPALA-2990: timeout unresponsive queries in coordinator
..
Patch Set 5:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/12299/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/12299/4//COMMIT_MSG@30
PS4, Line 30: - Ran the stress test on tpch 500 on a 10 node cluster for 1000
> not too concerned about this -- average query concurrency is going to be in
Done
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/runtime/coordinator.h
File be/src/runtime/coordinator.h:
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/runtime/coordinator.h@140
PS4, Line 140: if
> nit: lower case 'i'
Done
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/runtime/coordinator.cc@750
PS4, Line 750: return 0;
> this is racy since exec_rpcs_complete_barrier_ is changed from nullptr to a
Done
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/service/impala-server.cc@215
PS4, Line 215: DEFINE_int32(status_report_interval_ms, 5000,
> instead of introducing a new flag, can we just set this based on the config
I think you're saying to set this to 1.1 * GetMaxReportRetryMs()?
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/service/impala-server.cc@398
PS4, Line 398:
> unrelated to this patch, but why is this even an option? shouldn't the back
Of course, even in this configuration, backends will report their final status.
This allows users to significantly reduce the load on the coordinator at the
cost of not being able to monitor the progress of in-flight queries.
I don't have strong feelings about whether or not this should be deprecated.
I'm not aware of any users who rely on this. I spent a little time in 'git
blame' and its been an option for years but there doesn't seem to be a
documented reason for it.
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/service/impala-server.cc@2226
PS4, Line 2226: }
> Instead of forcing that the coordinator and executors all agree on this con
Sure, that's pretty easy.
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/service/impala-server.cc@2227
PS4, Line 2227: }
:
> don't we already log all the flags at startup? I think it'd be better to av
Done
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/service/impala-server.cc@2237
PS4, Line 2237: ate->
> I noticed that 'Offer' seems to call to BlockingPut under the covers, which
Done
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/service/impala-server.cc@2288
PS4, Line 2288: return Status::OK();
: }
:
: void ImpalaServer::ExpireQuery(ClientRequestState* crs, const
Status& status) {
: DCHECK(!status.ok());
: cancellation_thread_pool_->Offer(
> are these flags documented and user-settable? Is it too late to change the
Yes, all of these flags are documented and settable.
The flag 'status_report_max_retries' shipped in 3.2, though it could of course
be deprecated if we feel a max retry time is a significant improvement.
http://gerrit.cloudera.org:8080/#/c/12299/4/common/thrift/generate_error_codes.py
File common/thrift/generate_error_codes.py:
http://gerrit.cloudera.org:8080/#/c/12299/4/common/thrift/generate_error_codes.py@403
PS4, Line 403: 3m
> ms?
Done
--
To view, visit http://gerrit.cloudera.org:8080/12299
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987
Gerrit-Change-Number: 12299
Gerrit-PatchSet: 5
Gerrit-Owner: Thomas Marshall
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: Todd Lipcon
Gerrit-Comment-Date: Mon, 25 Mar 2019 23:00:27 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-2990: timeout unresponsive queries in coordinator
Hello Michael Ho, Philip Zeyliger, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12299 to look at the new patch set (#5). Change subject: IMPALA-2990: timeout unresponsive queries in coordinator .. IMPALA-2990: timeout unresponsive queries in coordinator The coordinator currently waits indefinitely if it does not receive a status report from a backend. This could cause a query to hang indefinitely in certain situations, for example if the backend decides to cancel itself as a result of failed status report rpcs. This patch adds a thread to ImpalaServer which periodically iterates over all queries for which that server is the coordinator and cancels any that haven't had a report from a backend in a certain amount of time. The timeout is calculated as the longest a backend will attempt to retry sending status reports before giving up and cancelling itself. With the default flags, this timeout is about 15 minutes. The thread wakes up at an interval of the calculated timeout + 10% TODO: - Write functional tests once the appropriate mechanisms are in place to simulate errors (IMPALA-8138) Testing: - Ran the stress test on tpch 500 on a 10 node cluster for 1000 queries with the debug actions: 'REPORT_EXEC_STATUS_SEND:FAIL@0.1|REPORT_EXEC_STATUS_RECV:FAIL@0.1' Prior to this patch, this setup results in hanging queries. With this patch, no hangs were observed. Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/ImpalaInternalService.thrift M common/thrift/generate_error_codes.py 10 files changed, 155 insertions(+), 41 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/12299/5 -- To view, visit http://gerrit.cloudera.org:8080/12299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I196c8c6a5633b1960e2c3a3884777be9b3824987 Gerrit-Change-Number: 12299 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/12846 ) Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/12846/4/bin/impala-config.sh File bin/impala-config.sh: http://gerrit.cloudera.org:8080/#/c/12846/4/bin/impala-config.sh@37 PS4, Line 37: # parse command line options It seems like options to impala-config.sh are currently passed by environment variable, for example USE_KUDU_DEBUG_BUILD can be set before sourcing impala-config.sh. Is there a particular reason you chose to add arguments to impala-config.sh rather than using the existing mechanism? -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 25 Mar 2019 22:52:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Bump toolchain version to 8430101a-b65f-4e4d-921e-767771649ae4-713afc77a8
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12824 ) Change subject: Bump toolchain version to 8430101a-b65f-4e4d-921e-767771649ae4-713afc77a8 .. Bump toolchain version to 8430101a-b65f-4e4d-921e-767771649ae4-713afc77a8 This bumps the version of two toolchain components: - IMPALA-8316: Update re2 to the latest version This updates re2 to the latest tagged release from github. Todd Lipcon benchmarked this with a simple query: select sum(l_linenumber) from item_20x where length(regexp_extract(l_shipinstruct, '.*', 0)) > 0 Prior to the change: - TotalCpuTime: 42s848ms - wall time: ~19sec With the change: - TotalCpuTime: 33s634ms - wall time: 14-15sec - This patch also bumps the toolchain Kudu version to 1.9.0 Change-Id: Ie9298709c534beeccfe36e212e0064d575cec9ac Reviewed-on: http://gerrit.cloudera.org:8080/12824 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/impala-config.sh 1 file changed, 3 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12824 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie9298709c534beeccfe36e212e0064d575cec9ac Gerrit-Change-Number: 12824 Gerrit-PatchSet: 4 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-7800: Reject new connections after --fe service threads
Zoram Thanga has posted comments on this change. (
http://gerrit.cloudera.org:8080/12579 )
Change subject: IMPALA-7800: Reject new connections after --fe_service_threads
..
Patch Set 8:
(5 comments)
Thanks for your comments, Andrew. Please see PS 9.
http://gerrit.cloudera.org:8080/#/c/12579/8/be/src/rpc/TAcceptQueueServer.h
File be/src/rpc/TAcceptQueueServer.h:
http://gerrit.cloudera.org:8080/#/c/12579/8/be/src/rpc/TAcceptQueueServer.h@97
PS8, Line 97: /// Name of the thrift server
> Add terminating period (nit).
Done
http://gerrit.cloudera.org:8080/#/c/12579/8/be/src/rpc/TAcceptQueueServer.h@113
PS8, Line 113: /// Number of connections rejected due to timeout
> Add terminating period (nit).
Done
http://gerrit.cloudera.org:8080/#/c/12579/8/be/src/rpc/TAcceptQueueServer.cpp
File be/src/rpc/TAcceptQueueServer.cpp:
http://gerrit.cloudera.org:8080/#/c/12579/8/be/src/rpc/TAcceptQueueServer.cpp@220
PS8, Line 220: } catch (string s) {
> I know this isn't part of this change, but can we really ever get string ex
Good question. I am not certain about string exceptions, but the intent here
seems to be to catch non-thrift exceptions.
http://gerrit.cloudera.org:8080/#/c/12579/8/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/12579/8/be/src/service/impala-server.cc@241
PS8, Line 241: "the queue before we time it out and reject the connection
request. A value of 0 "
> In the flags that configure the accept queue we call it "the post-accept, p
Changed the wording. Please let me know if this looks better.
http://gerrit.cloudera.org:8080/#/c/12579/8/tests/custom_cluster/test_frontend_connection_limit.py
File tests/custom_cluster/test_frontend_connection_limit.py:
http://gerrit.cloudera.org:8080/#/c/12579/8/tests/custom_cluster/test_frontend_connection_limit.py@89
PS8, Line 89: raise ImpalaBeeswaxException(e.message, e)
> I am confused about this. Why do we raise an exception here? I see that th
We need to catch the exception to close() the session so that the query gets
unregistered, and re-raise the exception. Otherwise, the async query hangs
around and makes the test fail. The comment in the above line tries to allude
to that. Please let me know if I should expand the comment.
--
To view, visit http://gerrit.cloudera.org:8080/12579
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64
Gerrit-Change-Number: 12579
Gerrit-PatchSet: 8
Gerrit-Owner: Zoram Thanga
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: Zoram Thanga
Gerrit-Comment-Date: Mon, 25 Mar 2019 21:34:14 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/12846 )
Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3
..
Patch Set 4:
(17 comments)
http://gerrit.cloudera.org:8080/#/c/12846/4/bin/bootstrap_toolchain.py
File bin/bootstrap_toolchain.py:
http://gerrit.cloudera.org:8080/#/c/12846/4/bin/bootstrap_toolchain.py@434
PS4, Line 434: os.getenv("USE_CDP
do we have any utility code anywhere that's more permission than this? I can
see someone setting it to '1' and being very confused why it's not working.
http://gerrit.cloudera.org:8080/#/c/12846/4/bin/bootstrap_toolchain.py@450
PS4, Line 450: present
maybe say 'set' here since it doesn't actually need to be present? (it will be
makedirred below)
http://gerrit.cloudera.org:8080/#/c/12846/4/bin/bootstrap_toolchain.py@466
PS4, Line 466: # TODO the tar file name in the cdp build don't match with the
version number. Hard
: # coding the name here currently
: file_name = "{0}.tar.gz".format(dir_name)
is this TODO inaccurate? it looks like from the code here it does match.
http://gerrit.cloudera.org:8080/#/c/12846/4/bin/create-test-configuration.sh
File bin/create-test-configuration.sh:
http://gerrit.cloudera.org:8080/#/c/12846/4/bin/create-test-configuration.sh@146
PS4, Line 146: # Hive schema SQL scripts include other scripts using \i, which
expects absolute paths.
: # Switch to the scripts directory to make this work.
: pushd ${HIVE_HOME}/bin
this pushd/popd is no longer relevant now that you're using schematool, right?
http://gerrit.cloudera.org:8080/#/c/12846/4/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/12846/4/bin/impala-config.sh@38
PS4, Line 38: for ARG in $*
nit: usually 'do' is on the same line
http://gerrit.cloudera.org:8080/#/c/12846/4/bin/impala-config.sh@41
PS4, Line 41: -use-hive3)
I think '--' instead of '-' is more common for long arg names
http://gerrit.cloudera.org:8080/#/c/12846/4/bin/impala-config.sh@44
PS4, Line 44: -help)
same
http://gerrit.cloudera.org:8080/#/c/12846/4/bin/impala-config.sh@49
PS4, Line 49: esac
do you want a default case here that prints usage info? otherwise a typo in the
args would just be silently ignored.
http://gerrit.cloudera.org:8080/#/c/12846/4/bin/impala-config.sh@310
PS4, Line 310: export METASTORE_DB=${METASTORE_DB-"$(cut -c-63 <<<
HMS$ESCAPED_IMPALA_HOME)_cdp"}
I'm assuming the 63-character 'cut' here is because of a 63-character limit in
db names in postgres or something. Given that, I guess we need to cut to 59
instead of 63?
http://gerrit.cloudera.org:8080/#/c/12846/4/bin/impala-config.sh@767
PS4, Line 767: echo "IMPALA_HIVE_VERSION = $IMPALA_HIVE_VERSION"
nit: indentation off
http://gerrit.cloudera.org:8080/#/c/12846/4/fe/src/test/resources/postgresql-hive-site.xml.cdp.template
File fe/src/test/resources/postgresql-hive-site.xml.cdp.template:
http://gerrit.cloudera.org:8080/#/c/12846/4/fe/src/test/resources/postgresql-hive-site.xml.cdp.template@99
PS4, Line 99:
[Impala-ASF-CR] IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12844 ) Change subject: IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12844 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 Gerrit-Change-Number: 12844 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 25 Mar 2019 21:12:42 + Gerrit-HasComments: No
[Impala-ASF-CR] Use 'wget http://169.254.169.254/' to determine if we're running in aws
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12727 ) Change subject: Use 'wget http://169.254.169.254/' to determine if we're running in aws .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Gerrit-Change-Number: 12727 Gerrit-PatchSet: 6 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Mon, 25 Mar 2019 21:01:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12846 ) Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2536/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 25 Mar 2019 20:59:34 + Gerrit-HasComments: No
[Impala-ASF-CR] Re-land IMPALA-5393. Use THREAD LOCAL state for regexp
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/12845 ) Change subject: Re-land IMPALA-5393. Use THREAD_LOCAL state for regexp .. Patch Set 1: > Patch Set 1: > > Yeah FWIW I think Lars did the right thing in erring on the side of reverting > in this case. yep, when in doubt, revert to keep builds green for sure -- To view, visit http://gerrit.cloudera.org:8080/12845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ae0703efeb2429813b2a712f1accf1b0a4a409e Gerrit-Change-Number: 12845 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 25 Mar 2019 20:49:50 + Gerrit-HasComments: No
[Impala-ASF-CR] Re-land IMPALA-5393. Use THREAD LOCAL state for regexp
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12845 ) Change subject: Re-land IMPALA-5393. Use THREAD_LOCAL state for regexp .. Patch Set 1: Yeah FWIW I think Lars did the right thing in erring on the side of reverting in this case. -- To view, visit http://gerrit.cloudera.org:8080/12845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ae0703efeb2429813b2a712f1accf1b0a4a409e Gerrit-Change-Number: 12845 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 25 Mar 2019 20:39:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7918: Remove support for authorization policy file
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12637 ) Change subject: IMPALA-7918: Remove support for authorization policy file .. IMPALA-7918: Remove support for authorization policy file This patch removes support for the authorization_policy_file. When the flag is passed, the backend will issue a warning message that the flag is being ignored. Tests relying on the authorization_policy_file flag have been updated to rely on sentry server instead. Testing: - Ran all FE tests - Ran all E2E tests Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd Reviewed-on: http://gerrit.cloudera.org:8080/12637 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/common/global-flags.cc M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/analysis/AuthorizationStmt.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthProvider.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryPolicyService.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java M fe/src/test/java/org/apache/impala/service/CustomClusterGroupMapper.java D fe/src/test/resources/authz-policy.ini.template M testdata/bin/create-load-data.sh M tests/authorization/test_authorization.py M tests/conftest.py 22 files changed, 427 insertions(+), 1,005 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12637 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd Gerrit-Change-Number: 12637 Gerrit-PatchSet: 12 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7918: Remove support for authorization policy file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12637 ) Change subject: IMPALA-7918: Remove support for authorization policy file .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12637 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd Gerrit-Change-Number: 12637 Gerrit-PatchSet: 11 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 25 Mar 2019 20:23:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Vihang Karajgaonkar has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/12846 ) Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 .. IMPALA-8345 : Add option to set up minicluster to use Hive 3 As a first step to integrate Impala with Hive 3.1.0 this patch modifies the minicluster scripts to use Hive 3.1.0 instead of CDH Hive 2.1.1. In order to make sure that existing setups don't break this option is enabled via a command line argument to bin/impala-config.sh. This command line argument (-use-hive3) sets up certain environment variables such that Hive 3.1.0 based binaries can be used to instantiate Hive service (Hiveserver2 and metastore). The default is still Hive 2.1.1 Also, since Hive 3.1.1 uses a upgraded metastore schema, this patch makes use of a different database name so that it is easy to switch from working from one environment which uses Hive 2.1.1 metastore to another which usese Hive 3.1.0 metastore. In order to do so users should follow the below steps: 1. Open a new terminal 2. Run bin/bootstrap_toolchain.py 2. source bin/impala-config.sh -use-hive3 3. source bin/create-test-configuration.sh -create-metastore The above step should provide "-create-metastore" only the first time so that a new metastore db is created and the Hive 3.1.0 schema is initialized. For all subsequent invocations, the "-create-metastore" argument can be skipped. We should still source this script since the hive-site.xml of Hive 3.1.0 is slightly different than Hive 2.1.0 and needs to be regenerated. 4. Start services using the testdata/bin/run-all.sh Note that the testing was performed locally by downloading the Hive 3.1 binaries into toolchain/cdp_components-976603/apache-hive-3.1.0.6.0.99.0-9-bin. Once the binaries are available in S3 bucket, the bootstrap_toolchain script should automatically do this for you. Testing Done: 1. Made sure that the cluster comes up with Hive 3.1 when the steps above are performed. 2. Made sure that existing scripts work as they do currently when argument is not provided. 3. Impala cluster comes and connects to HMS 3.1.0 (Note that Impala still uses Hive 2.1.1 client. Upgrading client libraries in Impala will be done as a separate change) Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 --- M bin/bootstrap_toolchain.py M bin/create-test-configuration.sh M bin/impala-config.sh A fe/src/test/resources/postgresql-hive-site.xml.cdp.template M testdata/bin/run-hive-server.sh 5 files changed, 372 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/12846/4 -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 4 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] Re-land IMPALA-5393. Use THREAD LOCAL state for regexp
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12845 ) Change subject: Re-land IMPALA-5393. Use THREAD_LOCAL state for regexp .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3953/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ae0703efeb2429813b2a712f1accf1b0a4a409e Gerrit-Change-Number: 12845 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 25 Mar 2019 20:02:16 + Gerrit-HasComments: No
[Impala-ASF-CR] Re-land IMPALA-5393. Use THREAD LOCAL state for regexp
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/12845 ) Change subject: Re-land IMPALA-5393. Use THREAD_LOCAL state for regexp .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/12845/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12845/1//COMMIT_MSG@9 PS1, Line 9: This re-lands commit 6e8c330f40da087ca0d8ba844cd9d97a8e60ff67 which was > Thanks for re-submitting this, and apologies for the unnecessary revert. As k, just wanted to give context on why it was re-landed. In the past I've seen reverts and resubmissions and been confused about the context when looking at the git history a year later. -- To view, visit http://gerrit.cloudera.org:8080/12845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ae0703efeb2429813b2a712f1accf1b0a4a409e Gerrit-Change-Number: 12845 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 25 Mar 2019 20:02:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12846 ) Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2535/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 25 Mar 2019 19:57:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12846 ) Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 .. Patch Set 3: (4 comments) http://gerrit.cloudera.org:8080/#/c/12846/3/bin/bootstrap_toolchain.py File bin/bootstrap_toolchain.py: http://gerrit.cloudera.org:8080/#/c/12846/3/bin/bootstrap_toolchain.py@465 PS3, Line 465: p flake8: F841 local variable 'platform_label' is assigned to but never used http://gerrit.cloudera.org:8080/#/c/12846/3/bin/create-test-configuration.sh File bin/create-test-configuration.sh: http://gerrit.cloudera.org:8080/#/c/12846/3/bin/create-test-configuration.sh@132 PS3, Line 132: # Certain configurations (like SentrySyncHMSNotificationsPostListener) does not work with HMS 3.1.0 line too long (101 > 90) http://gerrit.cloudera.org:8080/#/c/12846/3/testdata/bin/run-hive-server.sh File testdata/bin/run-hive-server.sh: http://gerrit.cloudera.org:8080/#/c/12846/3/testdata/bin/run-hive-server.sh@66 PS3, Line 66: export HIVE_METASTORE_HADOOP_OPTS="-verbose:class -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=30010" line too long (121 > 90) http://gerrit.cloudera.org:8080/#/c/12846/3/testdata/bin/run-hive-server.sh@69 PS3, Line 69: # CDH Hive metastore scripts do not do so. This is currently to make sure that we can run all the tests line too long (103 > 90) -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 25 Mar 2019 19:41:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8345 : Add option to set up minicluster to use Hive 3
Vihang Karajgaonkar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12846 Change subject: IMPALA-8345 : Add option to set up minicluster to use Hive 3 .. IMPALA-8345 : Add option to set up minicluster to use Hive 3 As a first step to integrate Impala with Hive 3.1.0 this patch modifies the minicluster scripts to use Hive 3.1.0 instead of CDH Hive 2.1.1. In order to make sure that existing setups don't break this option is enabled via a command line argument to bin/impala-config.sh. This command line argument (-use-hive3) sets up certain environment variables such that Hive 3.1.0 based binaries can be used to instantiate Hive service (Hiveserver2 and metastore). The default is still Hive 2.1.1 Also, since Hive 3.1.1 uses a upgraded metastore schema, this patch makes use of a different database name so that it is easy to switch from working from one environment which uses Hive 2.1.1 metastore to another which usese Hive 3.1.0 metastore. In order to do so users should follow the below steps: 1. Open a new terminal 2. Run bin/bootstrap_toolchain.py 2. source bin/impala-config.sh -use-hive3 3. source bin/create-test-configuration.sh -create-metastore The above step should provide "-create-metastore" only the first time so that a new metastore db is created and the Hive 3.1.0 schema is initialized. For all subsequent invocations, the "-create-metastore" argument can be skipped. We should still source this script since the hive-site.xml of Hive 3.1.0 is slightly different than Hive 2.1.0 and needs to be regenerated. 4. Start services using the testdata/bin/run-all.sh Note that the testing was performed locally by downloading the Hive 3.1 binaries into toolchain/cdp_components-976603/apache-hive-3.1.0.6.0.99.0-9-bin. Once the binaries are available in S3 bucket, the bootstrap_toolchain script should automatically do this for you. Testing Done: 1. Made sure that the cluster comes up with Hive 3.1 when the steps above are performed. 2. Made sure that existing scripts work as they do currently when argument is not provided. 3. Impala cluster comes and connects to HMS 3.1.0 (Note that Impala still uses Hive 2.1.1 client. Upgrading client libraries in Impala will be done as a separate change) Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 --- M bin/bootstrap_toolchain.py M bin/create-test-configuration.sh M bin/impala-config.sh A fe/src/test/resources/postgresql-hive-site.xml.cdp.template M testdata/bin/run-hive-server.sh 5 files changed, 374 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/12846/3 -- To view, visit http://gerrit.cloudera.org:8080/12846 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icfed856c1f5429ed45fd3d9cb08a5d1bb96a9605 Gerrit-Change-Number: 12846 Gerrit-PatchSet: 3 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8330: Impala shell config file should use flag names
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/12823 ) Change subject: IMPALA-8330: Impala shell config file should use flag names .. Patch Set 2: (2 comments) Looks good http://gerrit.cloudera.org:8080/#/c/12823/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12823/2//COMMIT_MSG@10 PS2, Line 10: both short and long flag names instead of dest names because dest names My life would be easier if you explained what a 'dest name' is here. http://gerrit.cloudera.org:8080/#/c/12823/2//COMMIT_MSG@24 PS2, Line 24: ; Flags can be repeated. Flags can be repeated. So if I have query_options=DEFAULT_FILE_FORMAT=parquet Q=DEFAULT_FILE_FORMAT=orc will it all work OK? -- To view, visit http://gerrit.cloudera.org:8080/12823 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic43603c1b538af08fddcab1b2c1f6ad1af1a6cb9 Gerrit-Change-Number: 12823 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 25 Mar 2019 18:51:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8308: [DOCS] Impala 3.2 Release
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12838 ) Change subject: IMPALA-8308: [DOCS] Impala 3.2 Release .. Patch Set 4: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/288/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12838 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I939e391d9546bbe868fd3045af81882fcaee3b03 Gerrit-Change-Number: 12838 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 25 Mar 2019 18:21:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8308: [DOCS] Impala 3.2 Release
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12838 ) Change subject: IMPALA-8308: [DOCS] Impala 3.2 Release .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/12838/3/docs/topics/impala_new_features.xml File docs/topics/impala_new_features.xml: http://gerrit.cloudera.org:8080/#/c/12838/3/docs/topics/impala_new_features.xml@197 PS3, Line 197: cept rev="3.1.0" id="new_features_31"> : New Features in : :For the full li > I think for 3.2.0 we should also include a reference like this one for the Done -- To view, visit http://gerrit.cloudera.org:8080/12838 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I939e391d9546bbe868fd3045af81882fcaee3b03 Gerrit-Change-Number: 12838 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 25 Mar 2019 18:18:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12844 ) Change subject: IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/12844/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12844/4//COMMIT_MSG@14 PS4, Line 14: [impala] : keyval=msg1=hello,keyval=msg2=world > An alternative approach to options with multiple key/value pairs is to trea I don't think the implementation in [impala.query_options] supports duplicate keys. For example: [impala.query_options] EXPLAIN_LEVEL=1 explain_LEVEL=2 The code will take the second explain_level and not both. It gets trickier too if we ever support a new flag that has "action=append", then we will always have to support a new section in the config file. The reason why we supported duplicate keys in the earlier patch is for this particular use case. impala-shell --var=msg1=hello --var=msg2=hello [impala] var=msg1=hello var=msg2=world It would be much nicer for the user experience, however it comes with its own set of problems too :( -- To view, visit http://gerrit.cloudera.org:8080/12844 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 Gerrit-Change-Number: 12844 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 25 Mar 2019 18:18:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8308: [DOCS] Impala 3.2 Release
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12838 ) Change subject: IMPALA-8308: [DOCS] Impala 3.2 Release .. Patch Set 4: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/288/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12838 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I939e391d9546bbe868fd3045af81882fcaee3b03 Gerrit-Change-Number: 12838 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 25 Mar 2019 18:18:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8308: [DOCS] Impala 3.2 Release
Hello Gabor Kaszab, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12838 to look at the new patch set (#4). Change subject: IMPALA-8308: [DOCS] Impala 3.2 Release .. IMPALA-8308: [DOCS] Impala 3.2 Release The following notes were added: - New features - Fixed issues - Incompatible changes - Upgrade notes Change-Id: I939e391d9546bbe868fd3045af81882fcaee3b03 --- M docs/topics/impala_fixed_issues.xml M docs/topics/impala_incompatible_changes.xml M docs/topics/impala_new_features.xml M docs/topics/impala_upgrading.xml 4 files changed, 249 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/12838/4 -- To view, visit http://gerrit.cloudera.org:8080/12838 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I939e391d9546bbe868fd3045af81882fcaee3b03 Gerrit-Change-Number: 12838 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7800: Reject new connections after --fe service threads
Andrew Sherman has posted comments on this change. (
http://gerrit.cloudera.org:8080/12579 )
Change subject: IMPALA-7800: Reject new connections after --fe_service_threads
..
Patch Set 8:
(5 comments)
All I have are nits and questions
http://gerrit.cloudera.org:8080/#/c/12579/8/be/src/rpc/TAcceptQueueServer.h
File be/src/rpc/TAcceptQueueServer.h:
http://gerrit.cloudera.org:8080/#/c/12579/8/be/src/rpc/TAcceptQueueServer.h@97
PS8, Line 97: /// Name of the thrift server
Add terminating period (nit).
http://gerrit.cloudera.org:8080/#/c/12579/8/be/src/rpc/TAcceptQueueServer.h@113
PS8, Line 113: /// Number of connections rejected due to timeout
Add terminating period (nit).
http://gerrit.cloudera.org:8080/#/c/12579/8/be/src/rpc/TAcceptQueueServer.cpp
File be/src/rpc/TAcceptQueueServer.cpp:
http://gerrit.cloudera.org:8080/#/c/12579/8/be/src/rpc/TAcceptQueueServer.cpp@220
PS8, Line 220: } catch (string s) {
I know this isn't part of this change, but can we really ever get string
exceptions?
http://gerrit.cloudera.org:8080/#/c/12579/8/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/12579/8/be/src/service/impala-server.cc@241
PS8, Line 241: "the queue before we time it out and reject the connection
request. A value of 0 "
In the flags that configure the accept queue we call it "the post-accept,
pre-setup connection queue", which is rather verbose, but do we want to use the
same wording for consistency? Also the description might be clearer if it was
rewritten to not use "we" (this is a definite nit)
http://gerrit.cloudera.org:8080/#/c/12579/8/tests/custom_cluster/test_frontend_connection_limit.py
File tests/custom_cluster/test_frontend_connection_limit.py:
http://gerrit.cloudera.org:8080/#/c/12579/8/tests/custom_cluster/test_frontend_connection_limit.py@89
PS8, Line 89: raise ImpalaBeeswaxException(e.message, e)
I am confused about this. Why do we raise an exception here? I see that the
test is marked xfail(raises=ImpalaBeeswaxException)but doesn't make the test
fragile in the case where some other bit of code raises a
ImpalaBeeswaxException? Maybe this is a python testing pattern I am unfamiliar
with.
--
To view, visit http://gerrit.cloudera.org:8080/12579
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb345c1d84cc2f691f54ded467f253e758f87e64
Gerrit-Change-Number: 12579
Gerrit-PatchSet: 8
Gerrit-Owner: Zoram Thanga
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: Zoram Thanga
Gerrit-Comment-Date: Mon, 25 Mar 2019 18:00:02 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12844 ) Change subject: IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2534/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12844 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 Gerrit-Change-Number: 12844 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 25 Mar 2019 17:31:42 + Gerrit-HasComments: No
[native-toolchain-CR] Add documentation for docker-based toolchain builds
Thomas Marshall has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12810 ) Change subject: Add documentation for docker-based toolchain builds .. Add documentation for docker-based toolchain builds This commit adds basic documentation with regard to docker-based toolchain builds. It briefly outlines how to build images and how to use them to build the toolchain. I also added the ability to fetch images from a docker registry. And fixed an 'undefined' variable problem with docker-based builds. Change-Id: Id971bda58d537fa15ec63004f744d3f730bad170 Reviewed-on: http://gerrit.cloudera.org:8080/12810 Reviewed-by: Thomas Marshall Tested-by: Thomas Marshall --- M Makefile A docker/README.md M functions.sh 3 files changed, 52 insertions(+), 5 deletions(-) Approvals: Thomas Marshall: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12810 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id971bda58d537fa15ec63004f744d3f730bad170 Gerrit-Change-Number: 12810 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] Re-land IMPALA-5393. Use THREAD LOCAL state for regexp
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12845 ) Change subject: Re-land IMPALA-5393. Use THREAD_LOCAL state for regexp .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2533/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ae0703efeb2429813b2a712f1accf1b0a4a409e Gerrit-Change-Number: 12845 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 25 Mar 2019 17:28:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12844 ) Change subject: IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2532/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12844 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 Gerrit-Change-Number: 12844 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 25 Mar 2019 17:22:57 + Gerrit-HasComments: No
[native-toolchain-CR] Add documentation for docker-based toolchain builds
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/12810 ) Change subject: Add documentation for docker-based toolchain builds .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12810 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id971bda58d537fa15ec63004f744d3f730bad170 Gerrit-Change-Number: 12810 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Mon, 25 Mar 2019 17:22:36 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump toolchain version to 8430101a-b65f-4e4d-921e-767771649ae4-713afc77a8
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12824 ) Change subject: Bump toolchain version to 8430101a-b65f-4e4d-921e-767771649ae4-713afc77a8 .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12824 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9298709c534beeccfe36e212e0064d575cec9ac Gerrit-Change-Number: 12824 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 25 Mar 2019 17:21:40 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump toolchain version to 8430101a-b65f-4e4d-921e-767771649ae4-713afc77a8
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12824 ) Change subject: Bump toolchain version to 8430101a-b65f-4e4d-921e-767771649ae4-713afc77a8 .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3952/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12824 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9298709c534beeccfe36e212e0064d575cec9ac Gerrit-Change-Number: 12824 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 25 Mar 2019 17:21:41 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump toolchain version to 8430101a-b65f-4e4d-921e-767771649ae4-713afc77a8
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/12824 ) Change subject: Bump toolchain version to 8430101a-b65f-4e4d-921e-767771649ae4-713afc77a8 .. Patch Set 2: gvo failed due to IMPALA-8336 -- To view, visit http://gerrit.cloudera.org:8080/12824 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9298709c534beeccfe36e212e0064d575cec9ac Gerrit-Change-Number: 12824 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Mon, 25 Mar 2019 17:21:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/12844 ) Change subject: IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 .. Patch Set 4: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/12844/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12844/4//COMMIT_MSG@14 PS4, Line 14: [impala] : keyval=msg1=hello,keyval=msg2=world An alternative approach to options with multiple key/value pairs is to treat them as separate config group like query_options, see https://gerrit.cloudera.org/#/c/8038/ I do not want to force it on you, but I think that it results in easier to read impala.rc files and would be consistent with the way query options works. Its drawback is that its implementation is more complex, so it may be a bad direction if we want to add several list type options. (I should have mentioned this in the original review, https://gerrit.cloudera.org/#/c/12781, but I did not have time to look at it. ) -- To view, visit http://gerrit.cloudera.org:8080/12844 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 Gerrit-Change-Number: 12844 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 25 Mar 2019 17:20:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Use 'wget http://169.254.169.254/' to determine if we're running in aws
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12727 ) Change subject: Use 'wget http://169.254.169.254/' to determine if we're running in aws .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2531/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Gerrit-Change-Number: 12727 Gerrit-PatchSet: 6 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Mon, 25 Mar 2019 17:19:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12844 ) Change subject: IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2530/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12844 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 Gerrit-Change-Number: 12844 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 25 Mar 2019 17:12:03 + Gerrit-HasComments: No
[Impala-ASF-CR] Re-land IMPALA-5393. Use THREAD LOCAL state for regexp
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12845 ) Change subject: Re-land IMPALA-5393. Use THREAD_LOCAL state for regexp .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/12845/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12845/1//COMMIT_MSG@9 PS1, Line 9: This re-lands commit 6e8c330f40da087ca0d8ba844cd9d97a8e60ff67 which was Thanks for re-submitting this, and apologies for the unnecessary revert. As a minor note, we generally seem ok with just reverting the revert and submit that, including the "Revert: Revert: IMPALA-..." commit message. -- To view, visit http://gerrit.cloudera.org:8080/12845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ae0703efeb2429813b2a712f1accf1b0a4a409e Gerrit-Change-Number: 12845 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 25 Mar 2019 17:02:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/12844 ) Change subject: IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 .. IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 Fixes in IMPALA-8317 and IMPALA-8337 introduced third-party dependencies in Impala shell which is problematic in multi-Python environment. This patch rewrites the fixes using an alternative solution when dealing with duplicate options without any third-party dependencies. For example: [impala] keyval=msg1=hello,keyval=msg2=world Testing: - Ran all shell tests on Python 2.6 and 2.7. - Ran make_shell_tarball.sh and ran Impala shell from the tarball without any issue. Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 --- M infra/python/deps/requirements.txt M shell/option_parser.py M tests/shell/good_impalarc 3 files changed, 8 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/12844/4 -- To view, visit http://gerrit.cloudera.org:8080/12844 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 Gerrit-Change-Number: 12844 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] Re-land IMPALA-5393. Use THREAD LOCAL state for regexp
Hello Lars Volker, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/12845 to review the following change. Change subject: Re-land IMPALA-5393. Use THREAD_LOCAL state for regexp .. Re-land IMPALA-5393. Use THREAD_LOCAL state for regexp This re-lands commit 6e8c330f40da087ca0d8ba844cd9d97a8e60ff67 which was reverted in d3428a58d8f54d1a64d5aeb1af3f76b7ffcb53d0. The revert was due to an assumption that this commit depended on the new version of re2 (which was correctly reverted due to a toolchain issue). In fact this commit does not depend on any toolchain changes. Original commit message follows This changes the built-in regexp-related UDFs to use THREAD_LOCAL re2::RE instances instead of FRAGMENT_LOCAL. Although re2::RE is thread-safe, it achieves that thread safety through a certain amount of locking. Using thread-local regexps improves performance substantially. I ran a simple test query: select sum(l_linenumber) from item_20x where length(regexp_extract(l_shipinstruct, '.*', 0)) > 0 on a table with three underlying parquet files (thus getting 3 scanner threads). Prior to this change, the query took ~60 seconds and burned 2m16sec CPU time. With this change, it took ~19sec and 43s CPU time. For a query with more scanner threads, the improvement should be even more dramatic. The only potential downside of this change is slightly increased memory consumption by having one RE instance per thread, but the REs themselves should be small relative to all of the other per-scanner-thread memory. Change-Id: I9ae0703efeb2429813b2a712f1accf1b0a4a409e --- M be/src/exprs/string-functions-ir.cc 1 file changed, 6 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/12845/1 -- To view, visit http://gerrit.cloudera.org:8080/12845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9ae0703efeb2429813b2a712f1accf1b0a4a409e Gerrit-Change-Number: 12845 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] Use 'wget http://169.254.169.254/' to determine if we're running in aws
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12727 ) Change subject: Use 'wget http://169.254.169.254/' to determine if we're running in aws .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3951/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Gerrit-Change-Number: 12727 Gerrit-PatchSet: 6 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Mon, 25 Mar 2019 16:43:06 + Gerrit-HasComments: No
[Impala-ASF-CR] Use 'wget http://169.254.169.254/' to determine if we're running in aws
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/12727 ) Change subject: Use 'wget http://169.254.169.254/' to determine if we're running in aws .. Patch Set 6: Code-Review+2 Thanks for this! -- To view, visit http://gerrit.cloudera.org:8080/12727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Gerrit-Change-Number: 12727 Gerrit-PatchSet: 6 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Mon, 25 Mar 2019 16:41:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12844 ) Change subject: IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 .. IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 Fixes in IMPALA-8317 and IMPALA-8337 introduced third-party dependencies in Impala shell which is problematic in multi-Python environment. This patch rewrites the fixes using an alternative solution when dealing with duplicate options without any third-party dependencies. For example: [impala] keyval=msg1=hello,keyval=msg2=world Testing: - Ran all shell tests on Python 2.6 and 2.7. - Ran make_shell_tarball.sh and ran Impala shell from the tarball without any issue. Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 --- M infra/python/deps/requirements.txt M shell/option_parser.py M tests/shell/good_impalarc 3 files changed, 8 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/12844/3 -- To view, visit http://gerrit.cloudera.org:8080/12844 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 Gerrit-Change-Number: 12844 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] Use 'wget http://169.254.169.254/' to determine if we're running in aws
Hello Lars Volker, Laszlo Gaal, Jim Apple, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12727 to look at the new patch set (#6). Change subject: Use 'wget http://169.254.169.254/' to determine if we're running in aws .. Use 'wget http://169.254.169.254/' to determine if we're running in aws https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html lists the above endpoint as the first choice. In my running instance 'dmidecode -s bios-version' prints out: 1.0 dmidecode --string system-uuid|grep ^ec2 seems like a valid alternative here. I removed the 'grep' assertions since it's possible that images may have already configured ntp here. Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 --- M bin/bootstrap_system.sh 1 file changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/12727/6 -- To view, visit http://gerrit.cloudera.org:8080/12727 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iddb2574dbcb3f97cf697095d1777e51ce463b205 Gerrit-Change-Number: 12727 Gerrit-PatchSet: 6 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12844 Change subject: IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 .. IMPALA-8340: Rewrite fixes in IMPALA-8317 and IMPALA-8337 Fixes in IMPALA-8317 and IMPALA-8337 introduced third-party dependencies in Impala shell which is problematic in multi-Python environment. This patch rewrites the fixes using an alternative solution when dealing with duplicate options without any third-party dependencies. For example: [impala] keyval=msg1=hello,keyval=msg2=world Testing: - Ran all shell tests on Python 2.6 and 2.7. Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 --- M infra/python/deps/requirements.txt M shell/option_parser.py M tests/shell/good_impalarc 3 files changed, 8 insertions(+), 28 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/12844/2 -- To view, visit http://gerrit.cloudera.org:8080/12844 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifc0bf391ba26cf5a34f622a4157d7287453cc539 Gerrit-Change-Number: 12844 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-7918: Remove support for authorization policy file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12637 ) Change subject: IMPALA-7918: Remove support for authorization policy file .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2529/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12637 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd Gerrit-Change-Number: 12637 Gerrit-PatchSet: 10 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 25 Mar 2019 16:02:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7918: Remove support for authorization policy file
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12637 ) Change subject: IMPALA-7918: Remove support for authorization policy file .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12637 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd Gerrit-Change-Number: 12637 Gerrit-PatchSet: 10 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 25 Mar 2019 15:46:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7918: Remove support for authorization policy file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12637 ) Change subject: IMPALA-7918: Remove support for authorization policy file .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3950/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12637 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd Gerrit-Change-Number: 12637 Gerrit-PatchSet: 11 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 25 Mar 2019 15:46:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7918: Remove support for authorization policy file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12637 ) Change subject: IMPALA-7918: Remove support for authorization policy file .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12637 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd Gerrit-Change-Number: 12637 Gerrit-PatchSet: 11 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 25 Mar 2019 15:46:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7918: Remove support for authorization policy file
Austin Nobis has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/12637 ) Change subject: IMPALA-7918: Remove support for authorization policy file .. IMPALA-7918: Remove support for authorization policy file This patch removes support for the authorization_policy_file. When the flag is passed, the backend will issue a warning message that the flag is being ignored. Tests relying on the authorization_policy_file flag have been updated to rely on sentry server instead. Testing: - Ran all FE tests - Ran all E2E tests Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd --- M be/src/common/global-flags.cc M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/analysis/AuthorizationStmt.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthProvider.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryPolicyService.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java M fe/src/test/java/org/apache/impala/service/CustomClusterGroupMapper.java D fe/src/test/resources/authz-policy.ini.template M testdata/bin/create-load-data.sh M tests/authorization/test_authorization.py M tests/conftest.py 22 files changed, 427 insertions(+), 1,005 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/12637/10 -- To view, visit http://gerrit.cloudera.org:8080/12637 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic2a52c2d5d35f58fbff8c088fb0bf30169625ebd Gerrit-Change-Number: 12637 Gerrit-PatchSet: 10 Gerrit-Owner: Austin Nobis Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR](2.x) Ignore revert and recover of "IMPALA-6988: Implement ALTER TABLE/VIEW SET OWNER"
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12843 ) Change subject: Ignore revert and recover of "IMPALA-6988: Implement ALTER TABLE/VIEW SET OWNER" .. Ignore revert and recover of "IMPALA-6988: Implement ALTER TABLE/VIEW SET OWNER" Change-Id: I260afacd01d07459a2c413b35161f048e1a884de Reviewed-on: http://gerrit.cloudera.org:8080/12843 Reviewed-by: Fredy Wijaya Tested-by: Quanlong Huang --- M bin/ignored_commits.json 1 file changed, 5 insertions(+), 1 deletion(-) Approvals: Fredy Wijaya: Looks good to me, approved Quanlong Huang: Verified -- To view, visit http://gerrit.cloudera.org:8080/12843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: merged Gerrit-Change-Id: I260afacd01d07459a2c413b35161f048e1a884de Gerrit-Change-Number: 12843 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR](2.x) Ignore revert and recover of "IMPALA-6988: Implement ALTER TABLE/VIEW SET OWNER"
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/12843 ) Change subject: Ignore revert and recover of "IMPALA-6988: Implement ALTER TABLE/VIEW SET OWNER" .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I260afacd01d07459a2c413b35161f048e1a884de Gerrit-Change-Number: 12843 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 25 Mar 2019 15:27:38 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Ignore revert and recover of "IMPALA-6988: Implement ALTER TABLE/VIEW SET OWNER"
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12843 ) Change subject: Ignore revert and recover of "IMPALA-6988: Implement ALTER TABLE/VIEW SET OWNER" .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I260afacd01d07459a2c413b35161f048e1a884de Gerrit-Change-Number: 12843 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 25 Mar 2019 15:17:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8308: [DOCS] Impala 3.2 Release
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/12838 ) Change subject: IMPALA-8308: [DOCS] Impala 3.2 Release .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/12838/3/docs/topics/impala_new_features.xml File docs/topics/impala_new_features.xml: http://gerrit.cloudera.org:8080/#/c/12838/3/docs/topics/impala_new_features.xml@197 PS3, Line 197: For the full list of issues closed in this release, including the : issues marked as new features or improvements, see the : changelog for . I think for 3.2.0 we should also include a reference like this one for the full list of changes besides the shortened list of noteworthy changes. -- To view, visit http://gerrit.cloudera.org:8080/12838 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I939e391d9546bbe868fd3045af81882fcaee3b03 Gerrit-Change-Number: 12838 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 25 Mar 2019 10:23:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR](2.x) Ignore revert and recover of "IMPALA-6988: Implement ALTER TABLE/VIEW SET OWNER"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12843 ) Change subject: Ignore revert and recover of "IMPALA-6988: Implement ALTER TABLE/VIEW SET OWNER" .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2528/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I260afacd01d07459a2c413b35161f048e1a884de Gerrit-Change-Number: 12843 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 25 Mar 2019 08:59:47 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Ignore revert and recover of "IMPALA-6988: Implement ALTER TABLE/VIEW SET OWNER"
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12843 Change subject: Ignore revert and recover of "IMPALA-6988: Implement ALTER TABLE/VIEW SET OWNER" .. Ignore revert and recover of "IMPALA-6988: Implement ALTER TABLE/VIEW SET OWNER" Change-Id: I260afacd01d07459a2c413b35161f048e1a884de --- M bin/ignored_commits.json 1 file changed, 5 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/12843/1 -- To view, visit http://gerrit.cloudera.org:8080/12843 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: I260afacd01d07459a2c413b35161f048e1a884de Gerrit-Change-Number: 12843 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
