[Impala-ASF-CR] IMPALA-6503: Support reading complex types from ORC
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/12168 )
Change subject: IMPALA-6503: Support reading complex types from ORC
..
Patch Set 25:
(33 comments)
Ah, I missed that this was almost committed. I had started reviewing but hadn't
finished so hadn't posted the comments yet. I'll post my comments here for now.
Quanlong, do you mind if I build a patch on this to take care of my suggested
cleanups?
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/hdfs-orc-scanner.h
File be/src/exec/hdfs-orc-scanner.h:
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/hdfs-orc-scanner.h@29
PS18, Line 29: #include "exec/orc-metadata-utils.h"
nit: do you need to include this here or could you get by with forward
declarations?
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/hdfs-orc-scanner.h@155
PS18, Line 155: int64_t stripe_rows_read_ = 0;
nit: nullptr is the default initializer already for unique_ptr
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/hdfs-orc-scanner.h@174
PS18, Line 174:
nit: "owned by"
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/hdfs-orc-scanner.h@182
PS18, Line 182: tch batch updated
I noticed that the ORC docs are not very consistent on terminology here. These
are equivalent to "column" IDs as referenced above, right?
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/hdfs-orc-scanner.h@235
PS18, Line 235: /// Advances 'stripe_idx_' to the next non-empty stripe and
initializes
can you update the doc to explain the 'column_reader' parameter here? The docs
say this transfers tuples from orc_root_batch_, but in fact maybe it is
transferring from the provided column reader?
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/hdfs-orc-scanner.h@243
PS18, Line 243:
'selected_indices' doesn't seem to exist here. Should it be "selected nodes"?
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/hdfs-orc-scanner.h@244
PS18, Line 244: /// Materialize collection(list/map) tuples belong to the
'row_idx'-th row of
I'm having trouble understanding this documentation. Maybe worth giving an
example here?
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/hdfs-orc-scanner.h@249
PS18, Line 249:
row_reader_options_
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.h
File be/src/exec/orc-column-readers.h:
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.h@33
PS18, Line 33: types
nit: typo: "type"
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.h@41
PS18, Line 41: /// while ( /* has more rows to read */ ) {
Performance-wise, would it be possible to actually materialize a column at a
time? Or do you think doing it a tuple at a time will be fast once we later
introduce codegen?
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.h@57
PS18, Line 57: guaranteed
typo: guarantee
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.h@72
PS18, Line 72: OrcColumnReader(const orc::Type* node, const SlotDescriptor*
slot_desc,
: HdfsOrcScanner* scanner);
instead of having both a public constructor and a public factory function
above, could this be another factory function, with an appropriate name? Or
could this be made private if it's only meant to be used by the factory
function itself?
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.h@79
PS18, Line 79: virtual bool MaterializeTuple() const { return true; }
Can we rename this to sound more like a boolean? As is, it sounds like a
function which materializes a tuple.
I'm also a little confused on the description -- I would think that as
described, this function returns whether or not the reader is responsible for
materializing a specific slot?
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.h@88
PS18, Line 88: orc::ColumnVectorBatch
can this be const?
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.h@122
PS18, Line 122: class OrcBoolColumnReader : public OrcColumnReader {
do we need to define all the subtypes of readers in the .h file or could they
be moved to the .cc file?
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.cc
File be/src/exec/orc-column-readers.cc:
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.cc@46
PS18, Line 46: switch (slot_desc->type().type) {
I think it's worth a comment explaining why we switched above on
node->getKind() but from here downward we're switching on the Impala slot type.
Have we already verified the compatibility elsewhere? Or will we verify
compatibility inside the constructor?
http://gerrit.cloudera.org:8080/#/c/12168/18/be/src/exec/orc-column-readers.cc@102
PS18, Line 102: DCHECK(false) <<
[Impala-ASF-CR] IMPALA-8284. KuduTableSink spends too much CPU in KuduSchema::Column()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12692 ) Change subject: IMPALA-8284. KuduTableSink spends too much CPU in KuduSchema::Column() .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3893/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12692 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b4d14d20252bdb190f50ebaaf6179a46eafb932 Gerrit-Change-Number: 12692 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Fri, 08 Mar 2019 05:55:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6503: Support reading complex types from ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12168 ) Change subject: IMPALA-6503: Support reading complex types from ORC .. Patch Set 24: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I244dc9d2b3e425393f90e45632cb8cdbea6cf790 Gerrit-Change-Number: 12168 Gerrit-PatchSet: 24 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 08 Mar 2019 04:39:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6503: Support reading complex types from ORC
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12168 ) Change subject: IMPALA-6503: Support reading complex types from ORC .. IMPALA-6503: Support reading complex types from ORC We've supported reading primitive types from ORC files (IMPALA-5717). In this patch we add support for complex types (struct/array/map). In IMPALA-5717, we leverage the ORC lib to parse ORC binaries (data in io buffer read from DiskIoMgr). The ORC lib can materialize ORC column binaries into its representation (orc::ColumnVectorBatch). Then we transform values in orc::ColumnVectorBatch into impala::Tuples in hdfs-orc-scanner. We don't need to do anything about decoding/decompression since they are handled by the ORC lib. Fortunately, the ORC lib already supports complex types, we can still leverage it to support complex types. What we need to add in IMPALA-6503 are two things: 1. Specify which nested columns we need in the form required by the ORC lib (Get list of ORC type ids from tuple descriptors) 2. Transform outputs of ORC lib (nested orc::ColumnVectorBatch) into Impala's representation (Slots/Tuples/RowBatches) To format the materialization, we implement several ORC column readers in hdfs-orc-scanner. Each kind of reader treats a column type and transforms outputs of the ORC lib into tuple/slot values. Tests: * Enable existing tests for complex types (test_nested_types.py, test_tpch_nested_queries.py) for ORC. * Run exhaustive tests in DEBUG and RELEASE builds. Change-Id: I244dc9d2b3e425393f90e45632cb8cdbea6cf790 Reviewed-on: http://gerrit.cloudera.org:8080/12168 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/CMakeLists.txt M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-orc-scanner.h A be/src/exec/orc-column-readers.cc A be/src/exec/orc-column-readers.h A be/src/exec/orc-metadata-utils.cc A be/src/exec/orc-metadata-utils.h M be/src/runtime/descriptors.cc M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java A testdata/ComplexTypesTbl/README A testdata/ComplexTypesTbl/nonnullable.orc A testdata/ComplexTypesTbl/nullable.orc M testdata/bin/create-load-data.sh M testdata/bin/load_nested.py M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-planner/queries/PlannerTest/complex-types-file-formats.test M testdata/workloads/functional-query/queries/DataErrorsTest/orc-type-checks.test M testdata/workloads/functional-query/queries/QueryTest/max-nesting-depth.test M testdata/workloads/functional-query/queries/QueryTest/nested-types-subplan.test M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch-limit.test M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch-mem-limit-single-node.test M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch-mem-limit.test M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch.test M testdata/workloads/tpch_nested/tpch_nested_core.csv M testdata/workloads/tpch_nested/tpch_nested_dimensions.csv M testdata/workloads/tpch_nested/tpch_nested_exhaustive.csv M testdata/workloads/tpch_nested/tpch_nested_pairwise.csv M tests/query_test/test_nested_types.py M tests/query_test/test_tpch_nested_queries.py 30 files changed, 1,943 insertions(+), 466 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I244dc9d2b3e425393f90e45632cb8cdbea6cf790 Gerrit-Change-Number: 12168 Gerrit-PatchSet: 25 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8250: Clean up JNI warnings.
Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/12660 )
Change subject: IMPALA-8250: Clean up JNI warnings.
..
IMPALA-8250: Clean up JNI warnings.
Using LIBHDFS_OPTS+="-Xcheck:jni" revealed a handful of warnings related to
(a) checking for exceptions and (b) leaking local references.
Checking for exceptions required sprinkling RETURN_ERROR_IF_EXC
left and right. I chose not to expand the JniCall infrastructure
to handle this more generally at the moment.
The leaky local references are a bit harder. In the logs, they show up
as "WARNING: JNI local refs: 2597, exceeds capacity: 35" or similar. A
few of these errors seem to be not in our code. The ones that I've
found in our code stemmed from HBaseTableScanner::GetRowKey(): this
method uses local references and wasn't returning them. Using a
JniLocalFrame seems to have taken care of the warnings.
I have added code to skip test_large_strings when JNI checking is
enabled. This test takes forever (presumably because JNI is checking
bounds on strings very aggressively), and times out. The time out also
causes some metric-related checks to fail (since a query is still in
flight).
Debugging this required customizing my JDK to give stack traces
when these warnings occurred. The following diff facilitated
this.
diff -r 76a9c9cf14f1 src/share/vm/prims/jniCheck.cpp
--- a/src/share/vm/prims/jniCheck.cpp Tue Jan 15 10:43:31 2019 +
+++ b/src/share/vm/prims/jniCheck.cpp Wed Feb 27 11:57:13 2019 -0800
@@ -143,11 +143,30 @@
static const char * fatal_instance_field_mismatch = "Field type (instance)
mismatch in JNI get/set field operations";
static const char * fatal_non_string = "JNI string operation received a
non-string";
+// thisone: whether to print every time, or maybe, depending on future
+// how many future stacks we want printed (totally racy); helps catch
+// missing exception handling if there's a way to tickle that code
+// reliably.
+static inline void dump_native_stack(JavaThread* thr, bool thisone, int
future) {
+ static int fut_stacks = 0; // racy!
+ if (fut_stacks > 0) {
+thisone = true;
+fut_stacks--;
+ }
+ if (future > 0) fut_stacks = future;
+ if (thisone) {
+frame fr = os::current_frame();
+char buf[6000];
+tty->print_cr("Thread: %s %d", thr->get_thread_name(),
thr->osthread()->thread_id());
+print_native_stack(tty, fr, thr, buf, sizeof(buf));
+ }
+}
// When in VM state:
static void ReportJNIWarning(JavaThread* thr, const char *msg) {
tty->print_cr("WARNING in native method: %s", msg);
thr->print_stack();
+ dump_native_stack(thr, true, 0);
}
// When in NATIVE state:
@@ -199,11 +218,14 @@
tty->print_cr("WARNING in native method: JNI call made without
checking exceptions when required to from %s",
thr->get_pending_jni_exception_check());
thr->print_stack();
+ dump_native_stack(thr, true, 10);
)
thr->clear_pending_jni_exception_check(); // Just complain once
}
}
+
+
/**
* Add to the planned number of handles. I.e. plus current live & warning
threshold
*/
@@ -254,9 +276,12 @@
tty->print_cr("WARNING: JNI local refs: %zu, exceeds capacity: %zu",
live_handles, planned_capacity);
thr->print_stack();
+ dump_native_stack(thr, true, 0);
)
// Complain just the once, reset to current + warn threshold
add_planned_handle_capacity(handles, 0);
+ } else {
+dump_native_stack(thr, false, 0);
}
}
Change-Id: Idd1709f749a764c1d947704bc64306493863b45f
Reviewed-on: http://gerrit.cloudera.org:8080/12660
Reviewed-by: Joe McDonnell
Tested-by: Impala Public Jenkins
---
M be/src/catalog/catalog.cc
M be/src/exec/hbase-table-scanner.cc
M be/src/exprs/hive-udf-call.cc
M be/src/runtime/hbase-table-factory.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M tests/query_test/test_insert.py
7 files changed, 55 insertions(+), 10 deletions(-)
Approvals:
Joe McDonnell: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/12660
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Idd1709f749a764c1d947704bc64306493863b45f
Gerrit-Change-Number: 12660
Gerrit-PatchSet: 5
Gerrit-Owner: Philip Zeyliger
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8250: Clean up JNI warnings.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12660 ) Change subject: IMPALA-8250: Clean up JNI warnings. .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd1709f749a764c1d947704bc64306493863b45f Gerrit-Change-Number: 12660 Gerrit-PatchSet: 4 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 08 Mar 2019 03:35:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7974: [DOCS] Document automatic invalidation via HMS notification
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12700 ) Change subject: IMPALA-7974: [DOCS] Document automatic invalidation via HMS notification .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/255/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12700 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c8a4ec77152c839564b1d69c76dfa041a0ebd7f Gerrit-Change-Number: 12700 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 08 Mar 2019 03:21:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7974: [DOCS] Document automatic invalidation via HMS notification
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12700 ) Change subject: IMPALA-7974: [DOCS] Document automatic invalidation via HMS notification .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/256/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12700 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c8a4ec77152c839564b1d69c76dfa041a0ebd7f Gerrit-Change-Number: 12700 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 08 Mar 2019 03:21:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7974: [DOCS] Document automatic invalidation via HMS notification
Hello Vihang Karajgaonkar, Adrian Ng, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12700 to look at the new patch set (#2). Change subject: IMPALA-7974: [DOCS] Document automatic invalidation via HMS notification .. IMPALA-7974: [DOCS] Document automatic invalidation via HMS notification - Moved all Metadata V2 related topics to a new doc. - Added the new doc under Impala Scalability. - Will track the other 2 new features in separate PRs. Change-Id: I6c8a4ec77152c839564b1d69c76dfa041a0ebd7f --- M docs/impala.ditamap M docs/topics/impala_config_options.xml A docs/topics/impala_metadata.xml M docs/topics/impala_perf_stats.xml 4 files changed, 175 insertions(+), 54 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/12700/2 -- To view, visit http://gerrit.cloudera.org:8080/12700 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6c8a4ec77152c839564b1d69c76dfa041a0ebd7f Gerrit-Change-Number: 12700 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-7974: [DOCS] Document automatic invalidation via HMS notification
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12700 ) Change subject: IMPALA-7974: [DOCS] Document automatic invalidation via HMS notification .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/256/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12700 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c8a4ec77152c839564b1d69c76dfa041a0ebd7f Gerrit-Change-Number: 12700 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Adrian Ng (389) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 08 Mar 2019 03:03:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7974: [DOCS] Document automatic invalidation via HMS notification
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12700 ) Change subject: IMPALA-7974: [DOCS] Document automatic invalidation via HMS notification .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/255/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12700 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6c8a4ec77152c839564b1d69c76dfa041a0ebd7f Gerrit-Change-Number: 12700 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 08 Mar 2019 02:59:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7974: [DOCS] Document automatic invalidation via HMS notification
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12700 Change subject: IMPALA-7974: [DOCS] Document automatic invalidation via HMS notification .. IMPALA-7974: [DOCS] Document automatic invalidation via HMS notification - Moved all Metadata V2 related topics to a new doc. - Added the new doc under Impala Scalability. - Will track the other 2 new features in separate PRs. Change-Id: I6c8a4ec77152c839564b1d69c76dfa041a0ebd7f --- M docs/impala.ditamap M docs/topics/impala_config_options.xml A docs/topics/impala_metadata.xml M docs/topics/impala_perf_stats.xml 4 files changed, 290 insertions(+), 54 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/12700/1 -- To view, visit http://gerrit.cloudera.org:8080/12700 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6c8a4ec77152c839564b1d69c76dfa041a0ebd7f Gerrit-Change-Number: 12700 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR](2.x) Ignore "IMPALA-7236: Fix the parsing of ALLOW ERASURE CODED FILES"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12697 ) Change subject: Ignore "IMPALA-7236: Fix the parsing of ALLOW_ERASURE_CODED_FILES" .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2391/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I68677c3bd8cc70f0e6e6880f514502d787a39efc Gerrit-Change-Number: 12697 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Mar 2019 00:25:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6503: Support reading complex types from ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12168 ) Change subject: IMPALA-6503: Support reading complex types from ORC .. Patch Set 24: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I244dc9d2b3e425393f90e45632cb8cdbea6cf790 Gerrit-Change-Number: 12168 Gerrit-PatchSet: 24 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 08 Mar 2019 00:24:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6503: Support reading complex types from ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12168 ) Change subject: IMPALA-6503: Support reading complex types from ORC .. Patch Set 24: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3892/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I244dc9d2b3e425393f90e45632cb8cdbea6cf790 Gerrit-Change-Number: 12168 Gerrit-PatchSet: 24 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 08 Mar 2019 00:24:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8147: part 2: Remove make impala.sh
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12556 ) Change subject: IMPALA-8147: part 2: Remove make_impala.sh .. IMPALA-8147: part 2: Remove make_impala.sh The previous commit rendered this scripts unused from within the Impala repository. This removes make_impala.sh and related scripts entirely. Testing: Did a trial run with the -debug_and_release flag to make sure that it could build all artifacts as expected. Change-Id: I8688ef9fe651475907cb319eb0b336fd6bef8dfe Reviewed-on: http://gerrit.cloudera.org:8080/12556 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- D bin/make_asan.sh D bin/make_debug.sh D bin/make_impala.sh D bin/make_release.sh 4 files changed, 0 insertions(+), 246 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12556 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8688ef9fe651475907cb319eb0b336fd6bef8dfe Gerrit-Change-Number: 12556 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6503: Support reading complex types from ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12168 ) Change subject: IMPALA-6503: Support reading complex types from ORC .. Patch Set 23: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I244dc9d2b3e425393f90e45632cb8cdbea6cf790 Gerrit-Change-Number: 12168 Gerrit-PatchSet: 23 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 08 Mar 2019 00:20:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8147: part 2: Remove make impala.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12556 ) Change subject: IMPALA-8147: part 2: Remove make_impala.sh .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12556 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8688ef9fe651475907cb319eb0b336fd6bef8dfe Gerrit-Change-Number: 12556 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Mar 2019 00:15:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8138: Reintroduce rpc debugging options
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/12297 ) Change subject: IMPALA-8138: Reintroduce rpc debugging options .. Patch Set 1: (1 comment) I will try to get https://gerrit.cloudera.org/#/c/12672/ into submittable shape, http://gerrit.cloudera.org:8080/#/c/12297/1/be/src/runtime/krpc-backend-client.h File be/src/runtime/krpc-backend-client.h: http://gerrit.cloudera.org:8080/#/c/12297/1/be/src/runtime/krpc-backend-client.h@28 PS1, Line 28: class KrpcBackendClient : public ControlServiceProxy > > Should this class show that it is a decorated form of Thanks this all seems fine. -- To view, visit http://gerrit.cloudera.org:8080/12297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2046cb9dadf846ea90c04e95781b2bbde3325941 Gerrit-Change-Number: 12297 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 07 Mar 2019 23:51:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR](2.x) Ignore "IMPALA-7236: Fix the parsing of ALLOW ERASURE CODED FILES"
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12697 ) Change subject: Ignore "IMPALA-7236: Fix the parsing of ALLOW_ERASURE_CODED_FILES" .. Ignore "IMPALA-7236: Fix the parsing of ALLOW_ERASURE_CODED_FILES" Change-Id: I68677c3bd8cc70f0e6e6880f514502d787a39efc Reviewed-on: http://gerrit.cloudera.org:8080/12697 Reviewed-by: Tim Armstrong Tested-by: Tim Armstrong --- M bin/ignored_commits.json 1 file changed, 3 insertions(+), 1 deletion(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: merged Gerrit-Change-Id: I68677c3bd8cc70f0e6e6880f514502d787a39efc Gerrit-Change-Number: 12697 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR](2.x) Ignore "IMPALA-7236: Fix the parsing of ALLOW ERASURE CODED FILES"
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12697 ) Change subject: Ignore "IMPALA-7236: Fix the parsing of ALLOW_ERASURE_CODED_FILES" .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I68677c3bd8cc70f0e6e6880f514502d787a39efc Gerrit-Change-Number: 12697 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Mar 2019 23:48:47 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Ignore "IMPALA-7236: Fix the parsing of ALLOW ERASURE CODED FILES"
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12697 ) Change subject: Ignore "IMPALA-7236: Fix the parsing of ALLOW_ERASURE_CODED_FILES" .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I68677c3bd8cc70f0e6e6880f514502d787a39efc Gerrit-Change-Number: 12697 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Mar 2019 23:47:37 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Ignore "IMPALA-7236: Fix the parsing of ALLOW ERASURE CODED FILES"
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12697 Change subject: Ignore "IMPALA-7236: Fix the parsing of ALLOW_ERASURE_CODED_FILES" .. Ignore "IMPALA-7236: Fix the parsing of ALLOW_ERASURE_CODED_FILES" Change-Id: I68677c3bd8cc70f0e6e6880f514502d787a39efc --- M bin/ignored_commits.json 1 file changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/12697/1 -- To view, visit http://gerrit.cloudera.org:8080/12697 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: I68677c3bd8cc70f0e6e6880f514502d787a39efc Gerrit-Change-Number: 12697 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] IMPALA-6503: Support reading complex types from ORC
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/12168 ) Change subject: IMPALA-6503: Support reading complex types from ORC .. Patch Set 23: Thanks for all your review! Really glad that we reach another milestone! -- To view, visit http://gerrit.cloudera.org:8080/12168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I244dc9d2b3e425393f90e45632cb8cdbea6cf790 Gerrit-Change-Number: 12168 Gerrit-PatchSet: 23 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 07 Mar 2019 23:41:47 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-6883: [DOCS] Refactor impala authorization doc
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12688 ) Change subject: IMPALA-6883: [DOCS] Refactor impala_authorization doc .. IMPALA-6883: [DOCS] Refactor impala_authorization doc Change-Id: I3df72adb25dcdcbc286934b048645f47d876b33d Reviewed-on: http://gerrit.cloudera.org:8080/10786 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins Reviewed-on: http://gerrit.cloudera.org:8080/12688 Reviewed-by: Fredy Wijaya --- M docs/impala_keydefs.ditamap M docs/shared/impala_common.xml M docs/topics/impala_authorization.xml M docs/topics/impala_grant.xml 4 files changed, 565 insertions(+), 614 deletions(-) Approvals: Impala Public Jenkins: Verified Fredy Wijaya: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/12688 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: merged Gerrit-Change-Id: I3df72adb25dcdcbc286934b048645f47d876b33d Gerrit-Change-Number: 12688 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12696 ) Change subject: [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12696 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadf183a1579a3a2480d6ee546714251b05620246 Gerrit-Change-Number: 12696 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Mar 2019 23:39:05 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12696 ) Change subject: [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP .. [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP Change-Id: Iadf183a1579a3a2480d6ee546714251b05620246 Reviewed-on: http://gerrit.cloudera.org:8080/12696 Tested-by: Impala Public Jenkins Reviewed-by: Alex Rodoni --- M docs/shared/impala_common.xml M docs/topics/impala_literals.xml 2 files changed, 16 insertions(+), 17 deletions(-) Approvals: Impala Public Jenkins: Verified Alex Rodoni: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/12696 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iadf183a1579a3a2480d6ee546714251b05620246 Gerrit-Change-Number: 12696 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12696 ) Change subject: [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP .. Patch Set 3: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/254/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12696 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadf183a1579a3a2480d6ee546714251b05620246 Gerrit-Change-Number: 12696 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Mar 2019 23:27:59 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12696 ) Change subject: [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/253/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12696 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadf183a1579a3a2480d6ee546714251b05620246 Gerrit-Change-Number: 12696 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Mar 2019 23:27:35 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12696 ) Change subject: [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/252/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12696 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadf183a1579a3a2480d6ee546714251b05620246 Gerrit-Change-Number: 12696 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Mar 2019 23:27:06 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12696 ) Change subject: [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP .. Patch Set 3: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/254/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12696 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadf183a1579a3a2480d6ee546714251b05620246 Gerrit-Change-Number: 12696 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Mar 2019 23:19:38 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12696 to look at the new patch set (#3). Change subject: [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP .. [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP Change-Id: Iadf183a1579a3a2480d6ee546714251b05620246 --- M docs/shared/impala_common.xml M docs/topics/impala_literals.xml 2 files changed, 16 insertions(+), 17 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/12696/3 -- To view, visit http://gerrit.cloudera.org:8080/12696 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iadf183a1579a3a2480d6ee546714251b05620246 Gerrit-Change-Number: 12696 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8250: Clean up JNI warnings.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/12660 ) Change subject: IMPALA-8250: Clean up JNI warnings. .. Patch Set 4: Code-Review+2 Thanks! -- To view, visit http://gerrit.cloudera.org:8080/12660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd1709f749a764c1d947704bc64306493863b45f Gerrit-Change-Number: 12660 Gerrit-PatchSet: 4 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 07 Mar 2019 23:13:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7917 (Part 3): Decouple Sentry from Impala
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/12684 )
Change subject: IMPALA-7917 (Part 3): Decouple Sentry from Impala
..
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12684/5/fe/src/main/java/org/apache/impala/authorization/AuthorizationFactory.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizationFactory.java:
http://gerrit.cloudera.org:8080/#/c/12684/5/fe/src/main/java/org/apache/impala/authorization/AuthorizationFactory.java@62
PS5, Line 62:*/
: AuthorizationManager newAuthorizationManager(CatalogOpExecutor
catalogOpExecutor);
:
: /**
:* Creates a new instance of {@link AuthorizationManager}.
:*/
: AuthorizationManager newAuthorizationManager(Frontend frontend);
I have a feeling that this AuthzMgr should be tied to a Catalog and not whether
it is frontend or CatalogServer.
newAuthzMgr(Catalog catalog) {}
You can then have children like SentryCatalogdAuthzMgr() and
SentryImpaladAuthzMgr().
For ranger I don't think you need a Catalogd side authz manager at all.
Essentially you could be delegating the impl stuff to the authz engine.
The issue as I see with the current model is that you need to know what methods
you can call, depending on the context you are in.
For ex: if I'm on a coordinator, I should know that I can only call getRoles,
showRoles() etc and not addRole / createRole().
What do you think?
--
To view, visit http://gerrit.cloudera.org:8080/12684
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1a5d3e0a3e86ac2b0329b56247357fca93229dd0
Gerrit-Change-Number: 12684
Gerrit-PatchSet: 5
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Comment-Date: Thu, 07 Mar 2019 23:12:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12696 ) Change subject: [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/253/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12696 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadf183a1579a3a2480d6ee546714251b05620246 Gerrit-Change-Number: 12696 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Mar 2019 23:10:28 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12696 ) Change subject: [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12696 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadf183a1579a3a2480d6ee546714251b05620246 Gerrit-Change-Number: 12696 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Mar 2019 23:10:28 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12696 to look at the new patch set (#2). Change subject: [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP .. [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP Change-Id: Iadf183a1579a3a2480d6ee546714251b05620246 --- M docs/shared/impala_common.xml M docs/topics/impala_literals.xml 2 files changed, 14 insertions(+), 16 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/12696/2 -- To view, visit http://gerrit.cloudera.org:8080/12696 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iadf183a1579a3a2480d6ee546714251b05620246 Gerrit-Change-Number: 12696 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8250: Clean up JNI warnings.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12660 ) Change subject: IMPALA-8250: Clean up JNI warnings. .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3891/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd1709f749a764c1d947704bc64306493863b45f Gerrit-Change-Number: 12660 Gerrit-PatchSet: 4 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 07 Mar 2019 23:09:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8250: Clean up JNI warnings.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/12660 ) Change subject: IMPALA-8250: Clean up JNI warnings. .. Patch Set 3: > "I've kicked off a suite of our tests with code coverage on. I'm > also going to kick off a set of tests with the JNI checking and > 'exhaustive' tests. Maybe there's coverage, but we're not running > the queries against HBase." > > Did anything new come up in this? > Nothing new. I'm still chasing a sequence of these calls which lead to CallObjectMethod() not having its exception checked... checked_jni_CallVoidMethodV checked_jni_ExceptionOccurred checked_jni_ExceptionClear checked_jni_GetObjectClass checked_jni_FindClass checked_jni_GetMethodID checked_jni_CallObjectMethod I can trigger this with set mem_limit=50m;use functional;insert into table alltypesinsert partition (year, month) /* +noclustered */ select at1.id, at1.bool_col, at1.tinyint_col, at1.smallint_col, at1.int_col, at1.bigint_col, at1.float_col, at1.double_col, at1.date_string_col, at1.string_col, at1.timestamp_col, at1.year, at2.id as month from functional.alltypes at1, functional.alltypes at2; when the sink gets closed after a failure. I don't know where this stack of calls is at this point. I'll keep looking, but I don't want to block on it. -- To view, visit http://gerrit.cloudera.org:8080/12660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd1709f749a764c1d947704bc64306493863b45f Gerrit-Change-Number: 12660 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 07 Mar 2019 23:08:14 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12696 Change subject: [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP .. [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP Change-Id: Iadf183a1579a3a2480d6ee546714251b05620246 --- M docs/shared/impala_common.xml 1 file changed, 9 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/12696/1 -- To view, visit http://gerrit.cloudera.org:8080/12696 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iadf183a1579a3a2480d6ee546714251b05620246 Gerrit-Change-Number: 12696 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12696 ) Change subject: [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12696 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadf183a1579a3a2480d6ee546714251b05620246 Gerrit-Change-Number: 12696 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Mar 2019 23:06:21 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12696 ) Change subject: [DOCS] Replaced the spaces and hyphens to non-breaking kind in TIMESTAMP .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/252/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12696 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iadf183a1579a3a2480d6ee546714251b05620246 Gerrit-Change-Number: 12696 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Mar 2019 23:05:57 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Updated the version number for the next release
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12695 ) Change subject: [DOCS] Updated the version number for the next release .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/251/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12695 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7c7a0d8564cc5700eb6c2cfb845b66d84461e4d Gerrit-Change-Number: 12695 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Mar 2019 22:37:57 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Updated the version number for the next release
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12695 ) Change subject: [DOCS] Updated the version number for the next release .. [DOCS] Updated the version number for the next release Change-Id: Ib7c7a0d8564cc5700eb6c2cfb845b66d84461e4d Reviewed-on: http://gerrit.cloudera.org:8080/12695 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins --- M docs/shared/ImpalaVariables.xml 1 file changed, 8 insertions(+), 5 deletions(-) Approvals: Alex Rodoni: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12695 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib7c7a0d8564cc5700eb6c2cfb845b66d84461e4d Gerrit-Change-Number: 12695 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [DOCS] Updated the version number for the next release
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12695 Change subject: [DOCS] Updated the version number for the next release .. [DOCS] Updated the version number for the next release Change-Id: Ib7c7a0d8564cc5700eb6c2cfb845b66d84461e4d --- M docs/shared/ImpalaVariables.xml 1 file changed, 8 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/12695/1 -- To view, visit http://gerrit.cloudera.org:8080/12695 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib7c7a0d8564cc5700eb6c2cfb845b66d84461e4d Gerrit-Change-Number: 12695 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] [DOCS] Updated the version number for the next release
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12695 ) Change subject: [DOCS] Updated the version number for the next release .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/251/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12695 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7c7a0d8564cc5700eb6c2cfb845b66d84461e4d Gerrit-Change-Number: 12695 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Mar 2019 22:17:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7972 Detect self-events to avoid unnecessary invalidates
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12591 ) Change subject: IMPALA-7972 Detect self-events to avoid unnecessary invalidates .. Patch Set 12: (9 comments) I think the new refactor makes sense. Just a few more nits and I think we are good to go. http://gerrit.cloudera.org:8080/#/c/12591/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java: http://gerrit.cloudera.org:8080/#/c/12591/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@743 PS12, Line 743: return result; throw DbNotFound...? http://gerrit.cloudera.org:8080/#/c/12591/12/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@750 PS12, Line 750: result = tbl.getVersionsForInflightEvents(); : return result; merge? http://gerrit.cloudera.org:8080/#/c/12591/12/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/12591/12/fe/src/main/java/org/apache/impala/catalog/Table.java@655 PS12, Line 655: No-op if event processing is disabled doesn't look like it? http://gerrit.cloudera.org:8080/#/c/12591/12/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java: http://gerrit.cloudera.org:8080/#/c/12591/12/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@613 PS12, Line 613: pendingVersionNumbersFromCatalog_ = catalog_ One pending question. Since we are storing these version numbers in the table state, I think we are inherently relying on in-place table modifications (which is kinda true for most cases). However if someone issues an Invalidate, I think we replace the whole table object which essentially wipes of this state. I think the worst that could happen is probably some extra (and unnecessary) invalidates. Is my understanding right? Following the sequence of actions that * i think * could create this. t1. alter table. generates event e t2. invalidate table t3. load table. t4. event 'e' from HMS After t4, pendingVersionNumbersFromCatalog_ is an EMPTY_LIST and hence it kicks off another round of invalidation. Please correct me if I'm wrong. http://gerrit.cloudera.org:8080/#/c/12591/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/12591/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@521 PS12, Line 521: catalog_.getLock().writeLock().unlock(); this looks buggy, what is the corresponding lock for this? http://gerrit.cloudera.org:8080/#/c/12591/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@527 PS12, Line 527:catalog_.getLock().writeLock().unlock() Move this too? http://gerrit.cloudera.org:8080/#/c/12591/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@829 PS12, Line 829: catalog_.isExternalEventProcessingEnabled() Move this inside the addCatalogServiceidentifies()...? http://gerrit.cloudera.org:8080/#/c/12591/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3253 PS12, Line 3253: setLastDdlTime update http://gerrit.cloudera.org:8080/#/c/12591/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3259 PS12, Line 3259: updateLastDdlTimeLocally nit: overrideDdlTime or something like that? -- To view, visit http://gerrit.cloudera.org:8080/12591 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6db0d7f7fe465158fc8cb9d6b6b57a321827b353 Gerrit-Change-Number: 12591 Gerrit-PatchSet: 12 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 07 Mar 2019 21:58:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8284. KuduTableSink spends too much CPU in KuduSchema::Column()
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/12692 ) Change subject: IMPALA-8284. KuduTableSink spends too much CPU in KuduSchema::Column() .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12692 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b4d14d20252bdb190f50ebaaf6179a46eafb932 Gerrit-Change-Number: 12692 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 07 Mar 2019 21:51:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12694 ) Change subject: IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2390/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12694 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0499cee7c532db19cddac3906198d965b27ea604 Gerrit-Change-Number: 12694 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 07 Mar 2019 21:51:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8255: [DOCS] Document the DEFAULT FILE FORMAT query option
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12670 ) Change subject: IMPALA-8255: [DOCS] Document the DEFAULT_FILE_FORMAT query option .. IMPALA-8255: [DOCS] Document the DEFAULT_FILE_FORMAT query option Change-Id: I5dc0e5f66ed59ff516ace2e67b74ade29492186e Reviewed-on: http://gerrit.cloudera.org:8080/12670 Tested-by: Impala Public Jenkins Reviewed-by: Fredy Wijaya --- M docs/impala.ditamap A docs/topics/impala_default_file_format.xml 2 files changed, 98 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Verified Fredy Wijaya: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/12670 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5dc0e5f66ed59ff516ace2e67b74ade29492186e Gerrit-Change-Number: 12670 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored
Hello Thomas Marshall, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/12694 to review the following change. Change subject: IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored .. IMPALA-8283. Order of Kudu PRIMARY KEYs can be silently ignored When creating a Kudu table, we would use the 'PRIMARY KEY (...)' clause to determine which columns made up the primary key, but the order of those columns would be ignored. Thus a statement like: CREATE TABLE (x int, y int, PRIMARY KEY (y, x)) STORED AS KUDU; would silently create a table with an (x,y) primary key instead of a (y,x) key. This can have substantial performance implications. This fixes the frontend to correctly throw an error in this case. This might be incompatible if someone was previously relying on the bug, but I think it's worth fixing because it was clearly doing the wrong thing. Change-Id: I0499cee7c532db19cddac3906198d965b27ea604 --- M fe/src/main/java/org/apache/impala/common/PrintUtils.java M fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java M fe/src/test/java/org/apache/impala/util/PrintUtilsTest.java M testdata/workloads/functional-query/queries/QueryTest/kudu_create.test 4 files changed, 47 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/12694/1 -- To view, visit http://gerrit.cloudera.org:8080/12694 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0499cee7c532db19cddac3906198d965b27ea604 Gerrit-Change-Number: 12694 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-8284. KuduTableSink spends too much CPU in KuduSchema::Column()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12692 ) Change subject: IMPALA-8284. KuduTableSink spends too much CPU in KuduSchema::Column() .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2389/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12692 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b4d14d20252bdb190f50ebaaf6179a46eafb932 Gerrit-Change-Number: 12692 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 07 Mar 2019 20:15:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8279: Revert IMPALA-6658 to avoid ETL performance regression.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12680 ) Change subject: IMPALA-8279: Revert IMPALA-6658 to avoid ETL performance regression. .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2387/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12680 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If6bcbaf564fbbe6dc83ba3afc100b4e5ccc7af40 Gerrit-Change-Number: 12680 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Thu, 07 Mar 2019 20:07:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8138: Reintroduce rpc debugging options
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12297 ) Change subject: IMPALA-8138: Reintroduce rpc debugging options .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/2388/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/12297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2046cb9dadf846ea90c04e95781b2bbde3325941 Gerrit-Change-Number: 12297 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 07 Mar 2019 20:11:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6503: Support reading complex types from ORC
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12168 ) Change subject: IMPALA-6503: Support reading complex types from ORC .. Patch Set 23: Code-Review+2 I'll kick off a dry run to make sure rebased commit is tested. Hopefully Quanlong can hit submit when he wakes up. -- To view, visit http://gerrit.cloudera.org:8080/12168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I244dc9d2b3e425393f90e45632cb8cdbea6cf790 Gerrit-Change-Number: 12168 Gerrit-PatchSet: 23 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 07 Mar 2019 19:57:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 ) Change subject: IMPALA-8100: Add initial support for Ranger .. Patch Set 9: Code-Review+1 Really like how this turned out. Very clean. Would like Bharath to look at the details of semantics. -- To view, visit http://gerrit.cloudera.org:8080/12632 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276 Gerrit-Change-Number: 12632 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Thu, 07 Mar 2019 19:57:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8147: part 2: Remove make impala.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12556 ) Change subject: IMPALA-8147: part 2: Remove make_impala.sh .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3890/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12556 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8688ef9fe651475907cb319eb0b336fd6bef8dfe Gerrit-Change-Number: 12556 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Mar 2019 19:57:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8255: [DOCS] Document the DEFAULT FILE FORMAT query option
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12670 ) Change subject: IMPALA-8255: [DOCS] Document the DEFAULT_FILE_FORMAT query option .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12670 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5dc0e5f66ed59ff516ace2e67b74ade29492186e Gerrit-Change-Number: 12670 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Mar 2019 19:58:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8147: part 2: Remove make impala.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12556 ) Change subject: IMPALA-8147: part 2: Remove make_impala.sh .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12556 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8688ef9fe651475907cb319eb0b336fd6bef8dfe Gerrit-Change-Number: 12556 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Mar 2019 19:57:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6503: Support reading complex types from ORC
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12168 ) Change subject: IMPALA-6503: Support reading complex types from ORC .. Patch Set 23: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3889/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/12168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I244dc9d2b3e425393f90e45632cb8cdbea6cf790 Gerrit-Change-Number: 12168 Gerrit-PatchSet: 23 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 07 Mar 2019 19:57:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7917 (Part 3): Decouple Sentry from Impala
Paul Rogers has posted comments on this change. ( http://gerrit.cloudera.org:8080/12684 ) Change subject: IMPALA-7917 (Part 3): Decouple Sentry from Impala .. Patch Set 5: Code-Review+1 Code looks good. Would like Bharath to double-check the logic flow. -- To view, visit http://gerrit.cloudera.org:8080/12684 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1a5d3e0a3e86ac2b0329b56247357fca93229dd0 Gerrit-Change-Number: 12684 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Thu, 07 Mar 2019 19:52:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8284. KuduTableSink spends too much CPU in KuduSchema::Column()
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/12692 ) Change subject: IMPALA-8284. KuduTableSink spends too much CPU in KuduSchema::Column() .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/12692 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b4d14d20252bdb190f50ebaaf6179a46eafb932 Gerrit-Change-Number: 12692 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Will Berkeley Gerrit-Comment-Date: Thu, 07 Mar 2019 19:51:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8255: [DOCS] Document the DEFAULT FILE FORMAT query option
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12670 ) Change subject: IMPALA-8255: [DOCS] Document the DEFAULT_FILE_FORMAT query option .. Patch Set 3: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/250/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12670 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5dc0e5f66ed59ff516ace2e67b74ade29492186e Gerrit-Change-Number: 12670 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Mar 2019 19:47:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8138: Reintroduce rpc debugging options
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/12297 ) Change subject: IMPALA-8138: Reintroduce rpc debugging options .. Patch Set 2: (3 comments) This is not really ready for full review until https://gerrit.cloudera.org/#/c/12672/ is submitted and it is rebased on top, I just updated it so that reviewers of that patch can see what I've done for the async case. http://gerrit.cloudera.org:8080/#/c/12297/1/be/src/runtime/krpc-backend-client.h File be/src/runtime/krpc-backend-client.h: http://gerrit.cloudera.org:8080/#/c/12297/1/be/src/runtime/krpc-backend-client.h@27 PS1, Line 27: > Please add a class level comment. Done http://gerrit.cloudera.org:8080/#/c/12297/1/be/src/runtime/krpc-backend-client.h@28 PS1, Line 28: > Should this class show that it is a decorated form of > ControlServiceProxy, perhaps by having ControlServiceProxy as part > of it's name? ControlServiceProxyInjected? ControlServiceProxyDecorated? Good idea. "Injected" and "Decorated" seem awkward to me, is it clear enough if I go with ImpalaControlServiceProxy? > > I think this change introduces a convention, that we expect > ControlServiceProxy methods to be overridden in KrpcBackendClient? > If that's so then a comment somewhere should say that. Is it in > fact now a mistake to use ControlServiceProxy? Addressed in the class comment. > > It seems like KrpcBackendClient is mostly boilerplate. Did you > consider extending protoc-gen-krpc.cc to either generate the > decorated methods in ControlServiceProxy, or to generate > KrpcBackendClient? As discussed, while this is an interesting idea, the Kudu people don't have much interest in directly adding such a feature to krpc, and for now at least the benefit of automating this boiler-plate is probably not worth the work. http://gerrit.cloudera.org:8080/#/c/12297/1/be/src/runtime/query-state.h File be/src/runtime/query-state.h: http://gerrit.cloudera.org:8080/#/c/12297/1/be/src/runtime/query-state.h@314 PS1, Line 314: eProxy > Will client_ be a more appropriate name ? Same for other places. I assume that if I change the class name back to something ending in "Proxy" that you're fine leaving this as is? -- To view, visit http://gerrit.cloudera.org:8080/12297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2046cb9dadf846ea90c04e95781b2bbde3325941 Gerrit-Change-Number: 12297 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 07 Mar 2019 19:33:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8284. KuduTableSink spends too much CPU in KuduSchema::Column()
Hello Thomas Marshall, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/12692 to review the following change. Change subject: IMPALA-8284. KuduTableSink spends too much CPU in KuduSchema::Column() .. IMPALA-8284. KuduTableSink spends too much CPU in KuduSchema::Column() The KuduSchema::Column() accessor actually returns a copy of the KuduColumnSchema object, which is not lightweight. We were inadvertently calling this function once for every null cell seen during an insertion. This caused a performance bottleneck for datasets with large numbers of NULL cells. This improves the situation by caching the nullability of the Kudu columns in our own vector. The vector lookups should be inlined and much faster than copying a KuduColumnSchema. No new tests included as this is a perf fix. Change-Id: I1b4d14d20252bdb190f50ebaaf6179a46eafb932 --- M be/src/exec/kudu-table-sink.cc M be/src/exec/kudu-table-sink.h 2 files changed, 16 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/12692/1 -- To view, visit http://gerrit.cloudera.org:8080/12692 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1b4d14d20252bdb190f50ebaaf6179a46eafb932 Gerrit-Change-Number: 12692 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-8138: Reintroduce rpc debugging options
Hello Michael Ho, Andrew Sherman, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12297 to look at the new patch set (#2). Change subject: IMPALA-8138: Reintroduce rpc debugging options .. IMPALA-8138: Reintroduce rpc debugging options In the past, we had fault injection options for backend rpcs implemented in ImpalaBackendClient. With the move the krpc, we lost some of those options. This patch reintroduces equivalent options, except that they now utilize the existing DebugAction framework, which is more flexible and powerful than the FAULT_INJECTION framework used previously. Change-Id: I2046cb9dadf846ea90c04e95781b2bbde3325941 --- A be/src/rpc/impala-control-service-proxy.h A be/src/rpc/impala-data-stream-service-proxy.h M be/src/rpc/rpc-mgr-test.cc M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/service/client-request-state.cc M be/src/service/control-service.cc M be/src/service/control-service.h M be/src/service/data-stream-service.cc M be/src/service/data-stream-service.h M be/src/util/CMakeLists.txt A be/src/util/krpc-debug-util.cc A be/src/util/krpc-debug-util.h M tests/custom_cluster/test_restart_services.py M tests/query_test/test_cancellation.py 19 files changed, 424 insertions(+), 108 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/12297/2 -- To view, visit http://gerrit.cloudera.org:8080/12297 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2046cb9dadf846ea90c04e95781b2bbde3325941 Gerrit-Change-Number: 12297 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-8271: Refactor the use of Thrift enums for query options
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12682 ) Change subject: IMPALA-8271: Refactor the use of Thrift enums for query options .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2386/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12682 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d747aae2c689765be72e117ce030ce4e3ce4641 Gerrit-Change-Number: 12682 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Mar 2019 19:27:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8279: Revert IMPALA-6658 to avoid ETL performance regression.
Andrew Sherman has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12680 ) Change subject: IMPALA-8279: Revert IMPALA-6658 to avoid ETL performance regression. .. IMPALA-8279: Revert IMPALA-6658 to avoid ETL performance regression. IMPALA-6658 changed RleEncoder to have the ability to use run lengths other than 8. It seemed that a slightly more complex RleEncoder could save a small amount of disk space by using the longer run lengths, in particular for bit width of 1. We now see a performance regression on a simple ETL query. Overall it seems that the costs of IMPALA-6658 exceed the benefits. This change removes IMPALA-6658. The strategy for this was that the change to rle-encoding.h, which contains the code, was undone using 'git revert'. I removed the test changes in rle-test.cc that rely on different encoding lengths. This allows us to keep some useful new tests that were written as part of IMPALA-6658 TESTING: Ran all end-to-end tests. Change-Id: If6bcbaf564fbbe6dc83ba3afc100b4e5ccc7af40 --- M be/src/exec/parquet/parquet-bool-decoder-test.cc M be/src/util/rle-encoding.h M be/src/util/rle-test.cc 3 files changed, 141 insertions(+), 383 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/12680/3 -- To view, visit http://gerrit.cloudera.org:8080/12680 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If6bcbaf564fbbe6dc83ba3afc100b4e5ccc7af40 Gerrit-Change-Number: 12680 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-8255: [DOCS] Document the DEFAULT FILE FORMAT query option
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12670 ) Change subject: IMPALA-8255: [DOCS] Document the DEFAULT_FILE_FORMAT query option .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/12670/2/docs/topics/impala_default_file_format.xml File docs/topics/impala_default_file_format.xml: http://gerrit.cloudera.org:8080/#/c/12670/2/docs/topics/impala_default_file_format.xml@46 PS2, Line 46: The SET DEFAULT_FILE_FORMAT statement will not return an : error when the option was set to an unsupported value. The error will : return at the next CREATE TABLE statement. > Yeah that's correct, the validation happens server-side and impala-shell do Please review the reworded paragraph. -- To view, visit http://gerrit.cloudera.org:8080/12670 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5dc0e5f66ed59ff516ace2e67b74ade29492186e Gerrit-Change-Number: 12670 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Mar 2019 19:26:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8255: [DOCS] Document the DEFAULT FILE FORMAT query option
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12670 ) Change subject: IMPALA-8255: [DOCS] Document the DEFAULT_FILE_FORMAT query option .. Patch Set 3: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/250/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12670 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5dc0e5f66ed59ff516ace2e67b74ade29492186e Gerrit-Change-Number: 12670 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Mar 2019 19:25:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8255: [DOCS] Document the DEFAULT FILE FORMAT query option
Hello Bharath Vissapragada, Paul Rogers, Fredy Wijaya, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12670 to look at the new patch set (#3). Change subject: IMPALA-8255: [DOCS] Document the DEFAULT_FILE_FORMAT query option .. IMPALA-8255: [DOCS] Document the DEFAULT_FILE_FORMAT query option Change-Id: I5dc0e5f66ed59ff516ace2e67b74ade29492186e --- M docs/impala.ditamap A docs/topics/impala_default_file_format.xml 2 files changed, 98 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/12670/3 -- To view, visit http://gerrit.cloudera.org:8080/12670 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5dc0e5f66ed59ff516ace2e67b74ade29492186e Gerrit-Change-Number: 12670 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7917 (Part 3): Decouple Sentry from Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12684 ) Change subject: IMPALA-7917 (Part 3): Decouple Sentry from Impala .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2385/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12684 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1a5d3e0a3e86ac2b0329b56247357fca93229dd0 Gerrit-Change-Number: 12684 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Thu, 07 Mar 2019 19:16:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8271: Refactor the use of Thrift enums for query options
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12682 ) Change subject: IMPALA-8271: Refactor the use of Thrift enums for query options .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/12682/2/be/src/service/query-options.cc File be/src/service/query-options.cc: http://gerrit.cloudera.org:8080/#/c/12682/2/be/src/service/query-options.cc@341 PS2, Line 341: ImpalaQueryOptions::REPLICA_PREFERENCE: > optional: if you are already cleaning things up, these could be replaced wi Done http://gerrit.cloudera.org:8080/#/c/12682/2/be/src/service/query-options.cc@343 PS2, Line 343: uery_options->__set_replica_preference(T > Was this query option left out intentionally? Yeah this is an odd one because we don't remove the unused options, .i.e. CACHE_RACK(1) and DISK_RACK(3) but using those options will throw an error. To keep the existing behavior, I leave this option out. -- To view, visit http://gerrit.cloudera.org:8080/12682 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d747aae2c689765be72e117ce030ce4e3ce4641 Gerrit-Change-Number: 12682 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Mar 2019 18:48:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8271: Refactor the use of Thrift enums for query options
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/12682 ) Change subject: IMPALA-8271: Refactor the use of Thrift enums for query options .. IMPALA-8271: Refactor the use of Thrift enums for query options This patch refactors the use of Thrift enums with GetThriftEnum helper function that can automatically validate and convert the query option value into the corresponding Thrift enum value. The validation error message has also been improved to list all possible valid query option values. Testing: - Added missing test cases in both BE and E2E - Ran query-options-test.cc - Ran metadata/test_set.py - Ran query_test/test_nested_types.py - Ran query_test/test_scanners.py Change-Id: I6d747aae2c689765be72e117ce030ce4e3ce4641 --- M be/src/service/query-options-test.cc M be/src/service/query-options.cc M testdata/workloads/functional-query/queries/QueryTest/parquet-ambiguous-list-modern.test M testdata/workloads/functional-query/queries/QueryTest/parquet-resolution-by-name.test M testdata/workloads/functional-query/queries/QueryTest/set.test 5 files changed, 110 insertions(+), 147 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/82/12682/3 -- To view, visit http://gerrit.cloudera.org:8080/12682 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6d747aae2c689765be72e117ce030ce4e3ce4641 Gerrit-Change-Number: 12682 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8250: Clean up JNI warnings.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/12660 ) Change subject: IMPALA-8250: Clean up JNI warnings. .. Patch Set 3: Code-Review+1 "I've kicked off a suite of our tests with code coverage on. I'm also going to kick off a set of tests with the JNI checking and 'exhaustive' tests. Maybe there's coverage, but we're not running the queries against HBase." Did anything new come up in this? Patch LGTM. -- To view, visit http://gerrit.cloudera.org:8080/12660 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd1709f749a764c1d947704bc64306493863b45f Gerrit-Change-Number: 12660 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 07 Mar 2019 18:47:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7917 (Part 3): Decouple Sentry from Impala
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12684 ) Change subject: IMPALA-7917 (Part 3): Decouple Sentry from Impala .. Patch Set 5: Rebased to fix conflicts. -- To view, visit http://gerrit.cloudera.org:8080/12684 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1a5d3e0a3e86ac2b0329b56247357fca93229dd0 Gerrit-Change-Number: 12684 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Comment-Date: Thu, 07 Mar 2019 18:33:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7917 (Part 3): Decouple Sentry from Impala
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/12684 ) Change subject: IMPALA-7917 (Part 3): Decouple Sentry from Impala .. IMPALA-7917 (Part 3): Decouple Sentry from Impala The third part of this patch introduces an interface called AuthorizationManager to perform authorization management related functions, such as granting/revoking a privilege, etc. Some existing authorization management methods have been refactored to reduce the need for if/else conditions to perform certain actions. This patch moves code related to Sentry authorization management code to SentryAuthorizationManager. This patch does not implement AuthorizationManager for Ranger. This patch has no functionality change. Testing: - Ran all FE tests - Ran all E2E authorization tests Change-Id: I1a5d3e0a3e86ac2b0329b56247357fca93229dd0 --- M fe/src/main/java/org/apache/impala/authorization/AuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/AuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationManager.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryProxy.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/testutil/PlannerTestCaseLoader.java 13 files changed, 864 insertions(+), 394 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/84/12684/5 -- To view, visit http://gerrit.cloudera.org:8080/12684 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1a5d3e0a3e86ac2b0329b56247357fca93229dd0 Gerrit-Change-Number: 12684 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12481 ) Change subject: IMPALA-7368: Add initial support for DATE type .. Patch Set 9: I'll leave that up to you. I think it depends whether we're trying to get this into the upstream release or not - that's probably something to figure out between you and Gabor. -- To view, visit http://gerrit.cloudera.org:8080/12481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f Gerrit-Change-Number: 12481 Gerrit-PatchSet: 9 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Mar 2019 18:06:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8271: Refactor the use of Thrift enums for query options
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/12682 ) Change subject: IMPALA-8271: Refactor the use of Thrift enums for query options .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/12682/2/be/src/service/query-options.cc File be/src/service/query-options.cc: http://gerrit.cloudera.org:8080/#/c/12682/2/be/src/service/query-options.cc@341 PS2, Line 341: iequals(value, "true") || iequals(value, "1") optional: if you are already cleaning things up, these could be replaced with a IsTrue(value) after creating an IsTrue() function. http://gerrit.cloudera.org:8080/#/c/12682/2/be/src/service/query-options.cc@343 PS2, Line 343: TImpalaQueryOptions::REPLICA_PREFERENCE: Was this query option left out intentionally? -- To view, visit http://gerrit.cloudera.org:8080/12682 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d747aae2c689765be72e117ce030ce4e3ce4641 Gerrit-Change-Number: 12682 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Mar 2019 17:38:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/12481 ) Change subject: IMPALA-7368: Add initial support for DATE type .. Patch Set 9: > (1 comment) > > Also, is the HBase data representation just a string? Is it > compatible with Hive's representation. I've done some simple ad-hoc testing with HBASE DATE tables, seems to be working fine. The HBASE date representation is string-based and compatible with Hive. On second thought (since we will have an upstream release soon), maybe it makes sense to spend couple more days on adding HBASE tests to this change and then commit everything in one patch. Splitting it into two parts would further complicate things. -- To view, visit http://gerrit.cloudera.org:8080/12481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f Gerrit-Change-Number: 12481 Gerrit-PatchSet: 9 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Mar 2019 17:29:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 ) Change subject: IMPALA-8100: Add initial support for Ranger .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2384/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12632 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276 Gerrit-Change-Number: 12632 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Thu, 07 Mar 2019 17:08:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12481 ) Change subject: IMPALA-7368: Add initial support for DATE type .. Patch Set 9: (1 comment) Also, is the HBase data representation just a string? Is it compatible with Hive's representation. http://gerrit.cloudera.org:8080/#/c/12481/9//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12481/9//COMMIT_MSG@74 PS9, Line 74: - Add HBase and Kudu support. Need to update this. -- To view, visit http://gerrit.cloudera.org:8080/12481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f Gerrit-Change-Number: 12481 Gerrit-PatchSet: 9 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Mar 2019 16:44:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12481 ) Change subject: IMPALA-7368: Add initial support for DATE type .. Patch Set 9: I'm ok with deferring that testing so long as you do a couple of sanity checks to make sure it's not broken. We should file a blocker JIRA to add the testing before the release that this will land in to make sure that it gets done. Otherwise we would need to revert this. In principle we shouldn't add functionality without comprehensive tests but it's worth making an exception here because of the heroic nature of this code change. -- To view, visit http://gerrit.cloudera.org:8080/12481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f Gerrit-Change-Number: 12481 Gerrit-PatchSet: 9 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Mar 2019 16:43:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6503: Support reading complex types from ORC
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12168 ) Change subject: IMPALA-6503: Support reading complex types from ORC .. Patch Set 22: Agreed, thanks a lot for the contribution! -- To view, visit http://gerrit.cloudera.org:8080/12168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I244dc9d2b3e425393f90e45632cb8cdbea6cf790 Gerrit-Change-Number: 12168 Gerrit-PatchSet: 22 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 07 Mar 2019 16:41:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12632 ) Change subject: IMPALA-8100: Add initial support for Ranger .. Patch Set 9: Rebased. Paul/Bharath, this is ready for review. Can you take a look at it? -- To view, visit http://gerrit.cloudera.org:8080/12632 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276 Gerrit-Change-Number: 12632 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Thu, 07 Mar 2019 16:25:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8100: Add initial support for Ranger
Fredy Wijaya has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/12632 ) Change subject: IMPALA-8100: Add initial support for Ranger .. IMPALA-8100: Add initial support for Ranger This patch adds an initial support for Ranger that can be enabled via the following flags in both impalad and catalogd to do enforcement. - ranger_service_type=impala - ranger_app_id=some_app_id - authorization_factory_class=\ org.apache.impala.authorization.ranger.RangerAuthorizationFactory There's a change in DESCRIBE privilege requirement to use ANY privilege instead of VIEW_METADATA privilege as the first-level check to play nicely with Ranger. This is not a security risk since the column-level filtering logic after the first-level check will use VIEW_METADATA privilege to filter out unauthorized column access. In other words, DESCRIBE may return an empty result instead of an authorization error as long as there exists any privilege in the given table. This patch updates AuthorizationStmtTest with a parameterized test that runs the tests against Sentry and Ranger. Testing: - Updated AuthorizationStmtTest with Ranger - Ran all FE tests - Ran all E2E authorization tests Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276 --- M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M bin/rat_exclude_files.txt M common/thrift/BackendGflags.thrift M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java M fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java M fe/src/main/java/org/apache/impala/authorization/Privilege.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationFactory.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaPlugin.java A fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpalaResourceBuilder.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M fe/src/test/java/org/apache/impala/authorization/sentry/SentryProxyTest.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java A fe/src/test/resources/ranger-impala-audit.xml A fe/src/test/resources/ranger-impala-security.xml M impala-parent/pom.xml M testdata/bin/create-load-data.sh M testdata/cluster/.gitignore A testdata/cluster/ranger/setup/impala_policy1.json A testdata/cluster/ranger/setup/impala_policy2.json A testdata/cluster/ranger/setup/impala_policy3.json A testdata/cluster/ranger/setup/impala_service.json A testdata/cluster/ranger/setup/impala_servicedef.json A testdata/cluster/ranger/setup/impala_user.json.template 38 files changed, 1,378 insertions(+), 223 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/12632/9 -- To view, visit http://gerrit.cloudera.org:8080/12632 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8cad9e609d20aae1ff645c84fd58a02afee70276 Gerrit-Change-Number: 12632 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7368: Add initial support for DATE type
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/12481 ) Change subject: IMPALA-7368: Add initial support for DATE type .. Patch Set 6: > (2 comments) > > Do you mean that, with these code changes, you can scan HBase date > tables? Or that we should have negative tests for HBase? > > I think we should have test coverage but I'm open to splitting that > out into a separate patch if that helps this get in faster. Yes, with these code changes Impala can scan/write HBase DATE tables. At the minimum, DATE-specific tests should be added to hbase-inserts.test, hbase-scan-node.test, hbase-compute-stats-incremental.test and hbase-compute-stats.test. This would also involve adding extra HBASE tables to functional_hbase. I'd rather do this in a separate change. -- To view, visit http://gerrit.cloudera.org:8080/12481 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iea8155ef09557e0afa2f8b2d0b2dc9d0896dc30f Gerrit-Change-Number: 12481 Gerrit-PatchSet: 6 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Mar 2019 16:01:56 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-6883: [DOCS] Refactor impala authorization doc
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/12688 ) Change subject: IMPALA-6883: [DOCS] Refactor impala_authorization doc .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12688 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I3df72adb25dcdcbc286934b048645f47d876b33d Gerrit-Change-Number: 12688 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 07 Mar 2019 15:44:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6503: Support reading complex types from ORC
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/12168 ) Change subject: IMPALA-6503: Support reading complex types from ORC .. Patch Set 22: Code-Review+2 Thank you for your great work and patience! It's very cool to have this feature! -- To view, visit http://gerrit.cloudera.org:8080/12168 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I244dc9d2b3e425393f90e45632cb8cdbea6cf790 Gerrit-Change-Number: 12168 Gerrit-PatchSet: 22 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 07 Mar 2019 15:24:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7917 (Part 2): Decouple Sentry from Impala
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12542 ) Change subject: IMPALA-7917 (Part 2): Decouple Sentry from Impala .. IMPALA-7917 (Part 2): Decouple Sentry from Impala This patch adds a new flag --authorization_factory_class and creates some factories to make it easier to switch to a different authorization provider. For backward compatability the default authorization provider is still set to Sentry. The --authorization_factory_class is used to specify which authorization provider implementation to use. This patch has no functional change. Testing: - Ran all FE tests - Ran all authorization E2E tests Change-Id: I051d06451fb14ba2ffb3f7e0d4aae37dee55ab86 Reviewed-on: http://gerrit.cloudera.org:8080/12542 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterDbSetOwnerStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewSetOwnerStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/AuthorizationStmt.java M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java M fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateFunctionStmtBase.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/analysis/DropFunctionStmt.java M fe/src/main/java/org/apache/impala/analysis/HdfsUri.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java A fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java A fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java A fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java A fe/src/main/java/org/apache/impala/authorization/AuthorizableFn.java A fe/src/main/java/org/apache/impala/authorization/AuthorizableServer.java A fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java A fe/src/main/java/org/apache/impala/authorization/AuthorizableUri.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java A fe/src/main/java/org/apache/impala/authorization/AuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java A fe/src/main/java/org/apache/impala/authorization/DefaultAuthorizableFactory.java A fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/PrivilegeRequestBuilder.java A fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizableFactory.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java A fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationFactory.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java M fe/src/test/java/org/apache/impala/analysis/AuditingTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/common/FrontendFixture.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/common/QueryFixture.java M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java M fe/src/test/java/org/apache/impala/testutil/PlannerTestCaseLoader.java 52 files changed, 1,113 insertions(+), 298 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12542 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I051d06451fb14ba2ffb3f7e0d4aae37dee55ab86 Gerrit-Change-Number: 12542 Gerrit-PatchSet: 14
[Impala-ASF-CR] IMPALA-7917 (Part 2): Decouple Sentry from Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12542 ) Change subject: IMPALA-7917 (Part 2): Decouple Sentry from Impala .. Patch Set 13: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12542 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I051d06451fb14ba2ffb3f7e0d4aae37dee55ab86 Gerrit-Change-Number: 12542 Gerrit-PatchSet: 13 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Thu, 07 Mar 2019 10:46:09 + Gerrit-HasComments: No
