[Impala-ASF-CR] IMPALA-6835: Improve Kudu scanner error messages to include the table name and the plan node id
Pooja Nilangekar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10671 Change subject: IMPALA-6835: Improve Kudu scanner error messages to include the table name and the plan node id .. IMPALA-6835: Improve Kudu scanner error messages to include the table name and the plan node id Previously, the error messages in KuduScanner only contained the reason for failure. They did not contain the KuduTable name or the TPlanNode id which made it inconveient to debug. This change add the TPlanNode id to all error messages and the KuduTable name whenever applicable. This change was manually tested by explicitly returning failure while scanning kudu tables. Change-Id: I0377fc8591738dc45092d228fcf292ddbb367825 --- M be/src/exec/kudu-scanner.cc 1 file changed, 24 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/10671/1 -- To view, visit http://gerrit.cloudera.org:8080/10671 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0377fc8591738dc45092d228fcf292ddbb367825 Gerrit-Change-Number: 10671 Gerrit-PatchSet: 1 Gerrit-Owner: Pooja Nilangekar
[Impala-ASF-CR] IMPALA-7158: Fix HdfsScanNodeBase::progress 's init
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/10672 ) Change subject: IMPALA-7158: Fix HdfsScanNodeBase::progress_'s init .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/10672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id2a738edea80ff3fb13ff368b4093c8b4ef34df7 Gerrit-Change-Number: 10672 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 09 Jun 2018 00:53:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7158: Fix HdfsScanNodeBase::progress 's init
Bharath Vissapragada has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10672 Change subject: IMPALA-7158: Fix HdfsScanNodeBase::progress_'s init .. IMPALA-7158: Fix HdfsScanNodeBase::progress_'s init Change-Id: Id2a738edea80ff3fb13ff368b4093c8b4ef34df7 --- M be/src/exec/hdfs-scan-node-base.cc 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/10672/1 -- To view, visit http://gerrit.cloudera.org:8080/10672 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id2a738edea80ff3fb13ff368b4093c8b4ef34df7 Gerrit-Change-Number: 10672 Gerrit-PatchSet: 1 Gerrit-Owner: Bharath Vissapragada
[Impala-ASF-CR](2.x) IMPALA-6394: Disable dfs.namenode.replication.considerLoad
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10664 ) Change subject: IMPALA-6394: Disable dfs.namenode.replication.considerLoad .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2629/ -- To view, visit http://gerrit.cloudera.org:8080/10664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I00010107cbd61bffa38ed8af4395b3a0e3d13c13 Gerrit-Change-Number: 10664 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Sat, 09 Jun 2018 00:15:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7119: Restart whole minicluster when HDFS replication stalls
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10665 to look at the new patch set (#2). Change subject: IMPALA-7119: Restart whole minicluster when HDFS replication stalls .. IMPALA-7119: Restart whole minicluster when HDFS replication stalls After loading data, we wait for HDFS to replicate all of the blocks appropriately. If this takes too long, we restart HDFS. However, HBase can fail if HDFS is restarted and HBase is unable to write its logs. In general, there is no real reason to keep HBase and the other minicluster components running while restarting HDFS. This changes the HDFS health check to restart the whole minicluster and Impala rather than just HDFS. Testing: - Tested with a modified version that always does the restart in the HDFS health check and verified that the tests pass Change-Id: I58ffe301708c78c26ee61aa754a06f46c224c6e2 --- M testdata/bin/create-load-data.sh 1 file changed, 29 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/10665/2 -- To view, visit http://gerrit.cloudera.org:8080/10665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I58ffe301708c78c26ee61aa754a06f46c224c6e2 Gerrit-Change-Number: 10665 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7157: Avoid unnecessarily pretty printing profiles per fragment instance
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/10669 ) Change subject: IMPALA-7157: Avoid unnecessarily pretty printing profiles per fragment instance .. Patch Set 1: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/10669/1/be/src/runtime/fragment-instance-state.cc File be/src/runtime/fragment-instance-state.cc: http://gerrit.cloudera.org:8080/#/c/10669/1/be/src/runtime/fragment-instance-state.cc@a371 PS1, Line 371: : : An alternative we discussed is to bump this log statement to VLOG_ROW() or something higher. That said, it doesn't seem to provide a lot of value to print the profile of a fragment instance in an executor's log esp. in a large cluster. Someone with more historical context of why this was done can feel free to chime in. -- To view, visit http://gerrit.cloudera.org:8080/10669 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0445950385fa6160764feaed9a993fa0e59b242 Gerrit-Change-Number: 10669 Gerrit-PatchSet: 1 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Fri, 08 Jun 2018 23:59:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6929: Support multi-column range partitions for Kudu
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10441 ) Change subject: IMPALA-6929: Support multi-column range partitions for Kudu .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10441 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0141dd3344a4f22b186f513b7406f286668ef1e7 Gerrit-Change-Number: 10441 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 23:45:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7157: Avoid unnecessarily pretty printing profiles per fragment instance
Sailesh Mukil has posted comments on this change. ( http://gerrit.cloudera.org:8080/10669 ) Change subject: IMPALA-7157: Avoid unnecessarily pretty printing profiles per fragment instance .. Patch Set 1: > Uploaded patch set 1. I found this while working on another patch. If anyone feels that printing the profiles for every fragment instance periodically is useful, or has found it useful in the past, please speak up. -- To view, visit http://gerrit.cloudera.org:8080/10669 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0445950385fa6160764feaed9a993fa0e59b242 Gerrit-Change-Number: 10669 Gerrit-PatchSet: 1 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Fri, 08 Jun 2018 23:43:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7157: Avoid unnecessarily pretty printing profiles per fragment instance
Sailesh Mukil has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10669 Change subject: IMPALA-7157: Avoid unnecessarily pretty printing profiles per fragment instance .. IMPALA-7157: Avoid unnecessarily pretty printing profiles per fragment instance In SendReport(), if VLOG_FILE_IS_ON is 'true' (which is not the most verbose logging level, but is higher than default), we pretty print the profile for every fragment instance, which is a very expensive operation, as serializing the profile is non-trivial (look at RuntimeProfile::PrettyPrint()), and printing large amounts of information to the logs isn't cheap as well. Lastly, it is very noisy. This seems unnecessary since this will not benefit us, as all the profiles are merged at the coordinator side. We could argue that this might be necessary when an executor fails to send the profile to the coordinator, but that signifies a network issue which will not be reflected in the profile of any fragment instance. This will help reduce noise in the logs when the log level is bumped up to find other real issues that VLOG_FILE can help with. Change-Id: Ic0445950385fa6160764feaed9a993fa0e59b242 --- M be/src/runtime/fragment-instance-state.cc 1 file changed, 0 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/10669/1 -- To view, visit http://gerrit.cloudera.org:8080/10669 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic0445950385fa6160764feaed9a993fa0e59b242 Gerrit-Change-Number: 10669 Gerrit-PatchSet: 1 Gerrit-Owner: Sailesh Mukil
[native-toolchain-CR] WIP ONLY: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/6521 ) Change subject: WIP ONLY: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. Patch Set 4: (1 comment) > Patch Set 4: > > (1 comment) http://gerrit.cloudera.org:8080/#/c/6521/4/functions.sh File functions.sh: http://gerrit.cloudera.org:8080/#/c/6521/4/functions.sh@502 PS4, Line 502: symlink_lib "$src_lib" "$file" > Yeah I remember looking at some of those binary patching tools but decided Sure. I'll do the review. -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 23:16:37 + Gerrit-HasComments: Yes
[native-toolchain-CR] WIP ONLY: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/6521 ) Change subject: WIP ONLY: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/6521/4/functions.sh File functions.sh: http://gerrit.cloudera.org:8080/#/c/6521/4/functions.sh@502 PS4, Line 502: symlink_lib "$src_lib" "$file" > I don't think symlinking shared objects is a hack in the linux world. That Yeah I remember looking at some of those binary patching tools but decided not to go down that path. I can't remember exactly why. I'm open to trying to move this change forward if you're willing to review all this bash. -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 23:00:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6929: Support multi-column range partitions for Kudu
Thomas Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/10441 )
Change subject: IMPALA-6929: Support multi-column range partitions for Kudu
..
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10441/1/fe/src/main/java/org/apache/impala/analysis/RangePartition.java
File fe/src/main/java/org/apache/impala/analysis/RangePartition.java:
http://gerrit.cloudera.org:8080/#/c/10441/1/fe/src/main/java/org/apache/impala/analysis/RangePartition.java@46
PS1, Line 46: PARTITION (l_val1, ..., l_valn) <[=] VALUES
: * - Single value (no range):
> needs to updated?
Done
http://gerrit.cloudera.org:8080/#/c/10441/1/fe/src/main/java/org/apache/impala/analysis/RangePartition.java@81
PS1, Line 81:
> nit: and ALTER
Done
http://gerrit.cloudera.org:8080/#/c/10441/1/testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
File testdata/workloads/functional-planner/queries/PlannerTest/kudu.test:
http://gerrit.cloudera.org:8080/#/c/10441/1/testdata/workloads/functional-planner/queries/PlannerTest/kudu.test@528
PS1, Line 528: '1'
> why was this change needed?
The point of this test case is that all of the exprs that target partition
columns are constants. Previously, the only partition column in the table was
'test_id', so only the first expr in this select list needed to be constant,
but my change to functional_schema_template.sql makes it so that 'test_name' is
also a partition column, so the expr that is targeting 'test_name' needs to be
a constant now too.
http://gerrit.cloudera.org:8080/#/c/10441/1/testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test
File testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test:
http://gerrit.cloudera.org:8080/#/c/10441/1/testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test@575
PS1, Line 575: ('a', 0) <= values < ('b', 1))
> just for my own info: what happens if the specified ranges overlap? what wi
Overlapping ranges aren't allowed. Detecting this is a bit tricky, so we don't
try and just pass the values off to Kudu, which returns an error.
There was already a test case here for that in the single-range-column case,
and I added one below for the multi-range-column case.
--
To view, visit http://gerrit.cloudera.org:8080/10441
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0141dd3344a4f22b186f513b7406f286668ef1e7
Gerrit-Change-Number: 10441
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Marshall
Gerrit-Reviewer: Thomas Marshall
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Fri, 08 Jun 2018 22:58:25 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6929: Support multi-column range partitions for Kudu
Hello Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10441 to look at the new patch set (#2). Change subject: IMPALA-6929: Support multi-column range partitions for Kudu .. IMPALA-6929: Support multi-column range partitions for Kudu Kudu allows specifying range partitions over multiple columns. Impala already has support for doing this when the partitions are specified with '=', but if the partitions are specified with '<' or '<=', the parser would return an error. This patch modifies the parser to allow for creating Kudu tables like: create table kudu_test (a int, b int, primary key(a, b)) partition by range(a, b) (partition (0, 0) <= values < (1, 1)); and similary to alter partitions like: alter table kudu_test add range partition (1, 1) <= values < (2, 2); Testing: - Modified functional_kudu.jointbl's schema so that we have a table in functional with a multi-column range partition to test things against. - Added FE and E2E tests for CREATE and ALTER. Change-Id: I0141dd3344a4f22b186f513b7406f286668ef1e7 --- M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/RangePartition.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M testdata/datasets/functional/functional_schema_template.sql M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test M testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test 6 files changed, 115 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/10441/2 -- To view, visit http://gerrit.cloudera.org:8080/10441 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0141dd3344a4f22b186f513b7406f286668ef1e7 Gerrit-Change-Number: 10441 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Vuk Ercegovac
[native-toolchain-CR] WIP ONLY: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/6521 ) Change subject: WIP ONLY: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/6521/4/functions.sh File functions.sh: http://gerrit.cloudera.org:8080/#/c/6521/4/functions.sh@502 PS4, Line 502: symlink_lib "$src_lib" "$file" I don't think symlinking shared objects is a hack in the linux world. That said it's not far from patching RPATHs into $origin/../../gcc-4.9.2/lib64 given you already have those matches and functions for generating "../". -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 22:53:37 + Gerrit-HasComments: Yes
[native-toolchain-CR] WIP ONLY: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so
Hello Philip Zeyliger, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/6521 to look at the new patch set (#4). Change subject: WIP ONLY: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so .. WIP ONLY: IMPALA-3926: fix RPATH for libstdc++.so and libgcc.so C++ packages depend on these shared objects. Our toolchain packages should always be run using the toolchain version of the shared objects that it ships. Previously the toolchain artifacts were often (but not always) linked against system version of libgcc.so and libstdc++.so, which can cause compatibility problems. This is a major problem on recent Linux distros like Ubuntu 16.04 that default to the new C++11 ABI because the system libstdc++.so is not compatible with gcc-4.9.2-generated binaries. It also means behaviour of toolchain artifacts may be less consistent across systems. This patch does two things to ensure that toolchain artifacts link to the correct libraries: 1. fixes the RPATHs added to executables and shared objects so that they always point to the ../lib/ and ../lib64/ directories. 2. adds symlinks from the lib/ directory to libstd++.so and libgcc.so where required by executables and shared objects. Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 --- M functions.sh M init-compiler.sh M source/kudu/build.sh 3 files changed, 112 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/native-toolchain refs/changes/21/6521/4 -- To view, visit http://gerrit.cloudera.org:8080/6521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3f8481a8dfe35273a763586e9d2da0d4008ac67 Gerrit-Change-Number: 6521 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6812: Fix flaky Kudu scan tests
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/10503 ) Change subject: IMPALA-6812: Fix flaky Kudu scan tests .. Patch Set 1: So I started playing around with READ_YOUR_WRITES and it doesn't seem to work as I would expect, though perhaps I'm doing something wrong: Suppose I create a new table and scan it without inserting any rows. I get the error message 'Snapshot timestamp is earlier than the ancient history mark' with the 'Snapshot timestamp' listed in the error message as 0. In this case, Impala hasn't called SetLatestObservedTimestamp() since we don't already have a timestamp for the session. Additionally, if I insert rows into the new table, Impala will call SetLatestObservedTimestamp() and I can scan it, but if I leave the session idle for awhile (>15min with default settings) and then try to scan the table, it fails with the 'Snapshot timestamp is earlier than the ancient history mark' error. Note that we never call SetSnapshotMicros() -- To view, visit http://gerrit.cloudera.org:8080/10503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70df84f2cbc663107f2ad029565d3c15bdfbd47c Gerrit-Change-Number: 10503 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 08 Jun 2018 22:18:36 + Gerrit-HasComments: No
[Impala-ASF-CR] THING
Tim Armstrong has abandoned this change. ( http://gerrit.cloudera.org:8080/10641 ) Change subject: THING .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/10641 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I685b3d474c2adb2c6f353722005f8da89d7557d8 Gerrit-Change-Number: 10641 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] THING
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10641 Change subject: THING .. THING Change-Id: I685b3d474c2adb2c6f353722005f8da89d7557d8 --- A test 1 file changed, 0 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/10641/12 -- To view, visit http://gerrit.cloudera.org:8080/10641 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I685b3d474c2adb2c6f353722005f8da89d7557d8 Gerrit-Change-Number: 10641 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6802 (part 4): Clean up authorization tests
Vuk Ercegovac has posted comments on this change. (
http://gerrit.cloudera.org:8080/10442 )
Change subject: IMPALA-6802 (part 4): Clean up authorization tests
..
Patch Set 3:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/10442/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java:
http://gerrit.cloudera.org:8080/#/c/10442/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java@918
PS3, Line 918: @Test
perhaps add a comment explaining what's being tested... from a first glance, it
looks like certain privs can see certain fields, and others cannot? if there
are docs or jiras that explain this more, pls reference them here.
http://gerrit.cloudera.org:8080/#/c/10442/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java@922
PS3, Line 922:
consistent spacing (see comment below)
http://gerrit.cloudera.org:8080/#/c/10442/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java@938
PS3, Line 938:
and several more places below
http://gerrit.cloudera.org:8080/#/c/10442/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java@946
PS3, Line 946: allExcept(
: TPrivilegeLevel.ALL, TPrivilegeLevel.SELECT)
factor out (repeated 3 times in this block, so it adds too much noise).
http://gerrit.cloudera.org:8080/#/c/10442/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java@952
PS3, Line 952: ng[]{"id"},
explain what is being tested here (missing why "id" is special)
http://gerrit.cloudera.org:8080/#/c/10442/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java@1181
PS3, Line 1181:
use consistent spacing for this... from a brief look, seems like no space is
the preference.
--
To view, visit http://gerrit.cloudera.org:8080/10442
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic4cf3a4751b41908ef81ec35d89a2713d9fa0dc4
Gerrit-Change-Number: 10442
Gerrit-PatchSet: 3
Gerrit-Owner: Adam Holley
Gerrit-Reviewer: Adam Holley
Gerrit-Reviewer: Alex Behm
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Fri, 08 Jun 2018 22:02:14 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6812: Fix flaky Kudu scan tests
David Ribeiro Alves has posted comments on this change. ( http://gerrit.cloudera.org:8080/10503 ) Change subject: IMPALA-6812: Fix flaky Kudu scan tests .. Patch Set 1: yeah, switching to RYW makes sense then. would pay close attention to stress tests when testing it out. -- To view, visit http://gerrit.cloudera.org:8080/10503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70df84f2cbc663107f2ad029565d3c15bdfbd47c Gerrit-Change-Number: 10503 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 08 Jun 2018 21:30:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5931: Generates scan ranges in planner for s3/adls
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8523 ) Change subject: IMPALA-5931: Generates scan ranges in planner for s3/adls .. IMPALA-5931: Generates scan ranges in planner for s3/adls Currently, for filesystems that do not include physical block information (e.g., block replica locations, caching), synthetic blocks are generated and stored in the catalog when metadata is loaded. Example file systems for which this is done includes S3, ADLS, and local fs. This change avoids generating these blocks when metadata is loaded. Instead, scan ranges are directly generated from such files by the backend coordinator. Previously, all scan ranges were produced by the planner in HDFSScanNode in the frontend. Now, those files without block information are sent to the coordinator represented by a split specification that determines how the coordinator will create scan ranges to send to executors. This change reduces the space needed in the catalog and reduces the scan range data structures that are passed from the frontend to the backend when planning and coordinating a query. In addition a bug is avoided where non-splittable files were being split anyways to support the query parameter that places a limit on scan ranges. Testing: - added backend scheduler tests - mixed-filesystems test covers tables/queries with multiple fs's. - local fs tests cover the code paths in this change - all core tests pass when configured with s3 - manually tried larger local filesystem tables (tpch) with multiple partitions and observed the same scan ranges. - TODO: adls testing Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Reviewed-on: http://gerrit.cloudera.org:8080/8523 Reviewed-by: Vuk Ercegovac Tested-by: Impala Public Jenkins --- M CMakeLists.txt M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler-test-util.h M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/util/CMakeLists.txt A be/src/util/flat_buffer.cc A be/src/util/flat_buffer.h M common/thrift/Frontend.thrift M common/thrift/PlanNodes.thrift M common/thrift/Planner.thrift M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/HBaseScanNode.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java 22 files changed, 677 insertions(+), 247 deletions(-) Approvals: Vuk Ercegovac: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/8523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Gerrit-Change-Number: 8523 Gerrit-PatchSet: 21 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-5931: Generates scan ranges in planner for s3/adls
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8523 ) Change subject: IMPALA-5931: Generates scan ranges in planner for s3/adls .. Patch Set 20: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/8523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Gerrit-Change-Number: 8523 Gerrit-PatchSet: 20 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 20:28:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7109: fix test multiple partitions same location paths
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10575 ) Change subject: IMPALA-7109: fix test_multiple_partitions_same_location paths .. IMPALA-7109: fix test_multiple_partitions_same_location paths The paths passed to filesystem_util methods is bogus in this test, but it's benign since we don't need to be doing these filesystem operations manually for this managed table. Just delete these lines (fixing them would create the same problem as IMPALA-7099). Change-Id: I7fc0b2aa2e82bfd8224c52546683f23de20cb640 Reviewed-on: http://gerrit.cloudera.org:8080/10575 Reviewed-by: Dan Hecht Tested-by: Impala Public Jenkins --- M tests/metadata/test_partition_metadata.py 1 file changed, 1 insertion(+), 7 deletions(-) Approvals: Dan Hecht: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10575 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7fc0b2aa2e82bfd8224c52546683f23de20cb640 Gerrit-Change-Number: 10575 Gerrit-PatchSet: 4 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7109: fix test multiple partitions same location paths
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10575 ) Change subject: IMPALA-7109: fix test_multiple_partitions_same_location paths .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10575 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fc0b2aa2e82bfd8224c52546683f23de20cb640 Gerrit-Change-Number: 10575 Gerrit-PatchSet: 3 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 20:24:47 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-6394: Disable dfs.namenode.replication.considerLoad
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/10664 ) Change subject: IMPALA-6394: Disable dfs.namenode.replication.considerLoad .. Patch Set 1: Code-Review+2 Thanks for backporting this. -- To view, visit http://gerrit.cloudera.org:8080/10664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I00010107cbd61bffa38ed8af4395b3a0e3d13c13 Gerrit-Change-Number: 10664 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 08 Jun 2018 20:20:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7108: IllegalStateException hit during CardinalityCheckNode.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10605 ) Change subject: IMPALA-7108: IllegalStateException hit during CardinalityCheckNode. .. IMPALA-7108: IllegalStateException hit during CardinalityCheckNode. Since IMPALA-6314 on runtime scalar subqueries we set LIMIT 2 in StmtRewriter.mergeExpr(). We do that because later we add a CardinalityCheckNode on top of such subqueries and with LIMIT 2 we can still check if they return more than one row. In the constructor of CardinalityCheckNode there is a precondition that checks if the child node has LIMIT 2 to be certain that we've set the limit for all the necessary cases. However, some subqueries will get a LIMIT 1 later breaking the precondition in CardinalityCheckNode. An example to these subqueries is a select stmt that selects from an inline view that returns a single row: select * from functional.alltypes where int_col = (select f.id from ( select * from functional.alltypes limit 1) f); Note that we shouldn't add a CardinalityCheckNode to the plan of this query in the first place. To generate a proper plan I updated SelectStmt.returnsSingleRow() because this method didn't handle this case well. I also changed the precondition from Preconditions.checkState(child.getLimit() == 2); to Preconditions.checkState(child.getLimit() <= 2); in order to be more permissive. I added tests for the aforementioned query. Change-Id: I82a7a3fe26db3e12131c030c4ad055a9c4955407 Reviewed-on: http://gerrit.cloudera.org:8080/10605 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test M testdata/workloads/functional-query/queries/QueryTest/subquery.test 4 files changed, 67 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10605 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I82a7a3fe26db3e12131c030c4ad055a9c4955407 Gerrit-Change-Number: 10605 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vincent Tran Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7108: IllegalStateException hit during CardinalityCheckNode.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10605 ) Change subject: IMPALA-7108: IllegalStateException hit during CardinalityCheckNode. .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10605 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82a7a3fe26db3e12131c030c4ad055a9c4955407 Gerrit-Change-Number: 10605 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vincent Tran Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 08 Jun 2018 20:15:49 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-6394: Disable dfs.namenode.replication.considerLoad
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10664 Change subject: IMPALA-6394: Disable dfs.namenode.replication.considerLoad .. IMPALA-6394: Disable dfs.namenode.replication.considerLoad By default namenode excludes heavily-loaded datanodes from being considered for block placement. Impala tests rely on every block to be fully-replicated so this option should be set to false. This is originally from IMPALA-7004 which was for 3.x branch only. The option has been renamed to "dfs.namenode.redundancy.considerLoad" in Hadooop 3.0. Change-Id: I00010107cbd61bffa38ed8af4395b3a0e3d13c13 --- M testdata/cluster/node_templates/common/etc/hadoop/conf/hdfs-site.xml.tmpl 1 file changed, 9 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/10664/1 -- To view, visit http://gerrit.cloudera.org:8080/10664 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: I00010107cbd61bffa38ed8af4395b3a0e3d13c13 Gerrit-Change-Number: 10664 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] [DOCS] Fixed a typo for missing 'not'
Alex Rodoni has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/10657 ) Change subject: [DOCS] Fixed a typo for missing 'not' .. [DOCS] Fixed a typo for missing 'not' Change-Id: I8342bcb47d4a9aa422e234e488dd1dfbdc1694d4 --- M docs/topics/impala_perf_cookbook.xml 1 file changed, 2 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/10657/2 -- To view, visit http://gerrit.cloudera.org:8080/10657 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8342bcb47d4a9aa422e234e488dd1dfbdc1694d4 Gerrit-Change-Number: 10657 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] [DOCS]Fixed a typo for missing 'not'
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10657 Change subject: [DOCS]Fixed a typo for missing 'not' .. [DOCS]Fixed a typo for missing 'not' Change-Id: I8342bcb47d4a9aa422e234e488dd1dfbdc1694d4 --- M docs/topics/impala_perf_cookbook.xml 1 file changed, 2 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/10657/1 -- To view, visit http://gerrit.cloudera.org:8080/10657 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8342bcb47d4a9aa422e234e488dd1dfbdc1694d4 Gerrit-Change-Number: 10657 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-6987: [DOCS] Refactor the INVALIDATE METADATA and REFRESH docs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10339 ) Change subject: IMPALA-6987: [DOCS] Refactor the INVALIDATE METADATA and REFRESH docs .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2124e14900d0f82569c061cc46006447bb054b36 Gerrit-Change-Number: 10339 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 18:18:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6987: [DOCS] Refactor the INVALIDATE METADATA and REFRESH docs
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10339 ) Change subject: IMPALA-6987: [DOCS] Refactor the INVALIDATE METADATA and REFRESH docs .. IMPALA-6987: [DOCS] Refactor the INVALIDATE METADATA and REFRESH docs Change-Id: I2124e14900d0f82569c061cc46006447bb054b36 Reviewed-on: http://gerrit.cloudera.org:8080/10339 Reviewed-by: Vuk Ercegovac Tested-by: Impala Public Jenkins --- M docs/impala.ditamap M docs/shared/impala_common.xml M docs/topics/impala_invalidate_metadata.xml M docs/topics/impala_refresh.xml A docs/topics/impala_refresh_functions.xml 5 files changed, 343 insertions(+), 418 deletions(-) Approvals: Vuk Ercegovac: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I2124e14900d0f82569c061cc46006447bb054b36 Gerrit-Change-Number: 10339 Gerrit-PatchSet: 6 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-6987: [DOCS] Refactor the INVALIDATE METADATA and REFRESH docs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10339 ) Change subject: IMPALA-6987: [DOCS] Refactor the INVALIDATE METADATA and REFRESH docs .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/313/ -- To view, visit http://gerrit.cloudera.org:8080/10339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2124e14900d0f82569c061cc46006447bb054b36 Gerrit-Change-Number: 10339 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 18:15:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7128 (part 2): add an interface for data sources
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10626 ) Change subject: IMPALA-7128 (part 2): add an interface for data sources .. Patch Set 2: Code-Review+2 (3 comments) http://gerrit.cloudera.org:8080/#/c/10626/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10626/2//COMMIT_MSG@12 PS2, Line 12: IMPALA-7131 pls mark these with todo's that reference this jira. http://gerrit.cloudera.org:8080/#/c/10626/2/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java File fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java: http://gerrit.cloudera.org:8080/#/c/10626/2/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java@42 PS2, Line 42: Represents a table backed by an external data source. pull this part up into the interface. http://gerrit.cloudera.org:8080/#/c/10626/2/fe/src/main/java/org/apache/impala/catalog/FeDataSource.java File fe/src/main/java/org/apache/impala/catalog/FeDataSource.java: http://gerrit.cloudera.org:8080/#/c/10626/2/fe/src/main/java/org/apache/impala/catalog/FeDataSource.java@21 PS2, Line 21: * Interface for interacting with data sources from the frontend. nit: use consistent phrasing with the other interfaces "Frontend interface for interacting with data sources." -- To view, visit http://gerrit.cloudera.org:8080/10626 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibe704197dc2ad7c09b8340865f17567096aa630e Gerrit-Change-Number: 10626 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 18:01:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6987: [DOCS] Refactor the INVALIDATE METADATA and REFRESH docs
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10339 ) Change subject: IMPALA-6987: [DOCS] Refactor the INVALIDATE METADATA and REFRESH docs .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2124e14900d0f82569c061cc46006447bb054b36 Gerrit-Change-Number: 10339 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 17:45:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6929: Support multi-column range partitions for Kudu
Vuk Ercegovac has posted comments on this change. (
http://gerrit.cloudera.org:8080/10441 )
Change subject: IMPALA-6929: Support multi-column range partitions for Kudu
..
Patch Set 1:
(4 comments)
looks good, mainly nits and clarifying questions.
http://gerrit.cloudera.org:8080/#/c/10441/1/fe/src/main/java/org/apache/impala/analysis/RangePartition.java
File fe/src/main/java/org/apache/impala/analysis/RangePartition.java:
http://gerrit.cloudera.org:8080/#/c/10441/1/fe/src/main/java/org/apache/impala/analysis/RangePartition.java@46
PS1, Line 46: Multi-value:
: * PARTITION VALUE = (val1, val2, ..., valn)
needs to updated?
http://gerrit.cloudera.org:8080/#/c/10441/1/fe/src/main/java/org/apache/impala/analysis/RangePartition.java@81
PS1, Line 81: CREATE
nit: and ALTER
http://gerrit.cloudera.org:8080/#/c/10441/1/testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
File testdata/workloads/functional-planner/queries/PlannerTest/kudu.test:
http://gerrit.cloudera.org:8080/#/c/10441/1/testdata/workloads/functional-planner/queries/PlannerTest/kudu.test@528
PS1, Line 528: '1'
why was this change needed?
http://gerrit.cloudera.org:8080/#/c/10441/1/testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test
File testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test:
http://gerrit.cloudera.org:8080/#/c/10441/1/testdata/workloads/functional-query/queries/QueryTest/kudu_alter.test@575
PS1, Line 575: ('a', 0) <= values < ('b', 1))
just for my own info: what happens if the specified ranges overlap? what will
happen to tuples that are inserted into the overlap? just wondering if analysis
should attempt to ban such cases.
--
To view, visit http://gerrit.cloudera.org:8080/10441
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I0141dd3344a4f22b186f513b7406f286668ef1e7
Gerrit-Change-Number: 10441
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Fri, 08 Jun 2018 17:35:57 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5931: Generates scan ranges in planner for s3/adls
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8523 ) Change subject: IMPALA-5931: Generates scan ranges in planner for s3/adls .. Patch Set 20: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstrong/34/ -- To view, visit http://gerrit.cloudera.org:8080/8523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Gerrit-Change-Number: 8523 Gerrit-PatchSet: 20 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 16:56:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5931: Generates scan ranges in planner for s3/adls
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8523 ) Change subject: IMPALA-5931: Generates scan ranges in planner for s3/adls .. Patch Set 20: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstrong/34/ -- To view, visit http://gerrit.cloudera.org:8080/8523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Gerrit-Change-Number: 8523 Gerrit-PatchSet: 20 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 16:56:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7109: fix test multiple partitions same location paths
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10575 ) Change subject: IMPALA-7109: fix test_multiple_partitions_same_location paths .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2627/ -- To view, visit http://gerrit.cloudera.org:8080/10575 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fc0b2aa2e82bfd8224c52546683f23de20cb640 Gerrit-Change-Number: 10575 Gerrit-PatchSet: 3 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 16:54:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7109: fix test multiple partitions same location paths
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/10575 ) Change subject: IMPALA-7109: fix test_multiple_partitions_same_location paths .. Patch Set 3: Code-Review+2 Carry -- To view, visit http://gerrit.cloudera.org:8080/10575 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7fc0b2aa2e82bfd8224c52546683f23de20cb640 Gerrit-Change-Number: 10575 Gerrit-PatchSet: 3 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 16:54:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6802 (part 4): Clean up authorization tests
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10442 ) Change subject: IMPALA-6802 (part 4): Clean up authorization tests .. Patch Set 3: Code-Review+1 Vuk, can you take a look at it? -- To view, visit http://gerrit.cloudera.org:8080/10442 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic4cf3a4751b41908ef81ec35d89a2713d9fa0dc4 Gerrit-Change-Number: 10442 Gerrit-PatchSet: 3 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 16:51:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7108: IllegalStateException hit during CardinalityCheckNode.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10605 ) Change subject: IMPALA-7108: IllegalStateException hit during CardinalityCheckNode. .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstrong/33/ -- To view, visit http://gerrit.cloudera.org:8080/10605 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82a7a3fe26db3e12131c030c4ad055a9c4955407 Gerrit-Change-Number: 10605 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vincent Tran Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 08 Jun 2018 16:51:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7108: IllegalStateException hit during CardinalityCheckNode.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10605 ) Change subject: IMPALA-7108: IllegalStateException hit during CardinalityCheckNode. .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstrong/33/ -- To view, visit http://gerrit.cloudera.org:8080/10605 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82a7a3fe26db3e12131c030c4ad055a9c4955407 Gerrit-Change-Number: 10605 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vincent Tran Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 08 Jun 2018 16:51:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7108: IllegalStateException hit during CardinalityCheckNode.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10605 ) Change subject: IMPALA-7108: IllegalStateException hit during CardinalityCheckNode. .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10605 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82a7a3fe26db3e12131c030c4ad055a9c4955407 Gerrit-Change-Number: 10605 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vincent Tran Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 08 Jun 2018 16:51:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6802 (part 4): Clean up authorization tests
Adam Holley has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/10442 ) Change subject: IMPALA-6802 (part 4): Clean up authorization tests .. IMPALA-6802 (part 4): Clean up authorization tests The fourth part of this patch is to rewrite the following authorization tests: - describe Testing: - Added new authorization tests - Ran all front-end tests Cherry-picks: not for 2.x Change-Id: Ic4cf3a4751b41908ef81ec35d89a2713d9fa0dc4 --- M fe/src/test/java/org/apache/impala/analysis/AuthorizationTestV2.java 1 file changed, 251 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/10442/3 -- To view, visit http://gerrit.cloudera.org:8080/10442 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic4cf3a4751b41908ef81ec35d89a2713d9fa0dc4 Gerrit-Change-Number: 10442 Gerrit-PatchSet: 3 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-5931: Generates scan ranges in planner for s3/adls
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8523 ) Change subject: IMPALA-5931: Generates scan ranges in planner for s3/adls .. Patch Set 20: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstrong/32/ -- To view, visit http://gerrit.cloudera.org:8080/8523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Gerrit-Change-Number: 8523 Gerrit-PatchSet: 20 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 16:35:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3307: Add support for IANA time-zone db
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/9986 )
Change subject: IMPALA-3307: Add support for IANA time-zone db
..
Patch Set 14: Code-Review+2
(5 comments)
Thanks for thinking about Zip-Slip!
I have left a few optional comments about the usability of the interfaces.
http://gerrit.cloudera.org:8080/#/c/9986/14//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/9986/14//COMMIT_MSG@34
PS14, Line 34: - Introduces a new startup flag (--hdfs_zone_info_zip) to
impalad to
The Zip slip safe zip-util could be also mentioned in the commit message.
http://gerrit.cloudera.org:8080/#/c/9986/14/be/src/util/filesystem-util.h
File be/src/util/filesystem-util.h:
http://gerrit.cloudera.org:8080/#/c/9986/14/be/src/util/filesystem-util.h@92
PS14, Line 92: Directory(const string& path, bool skip_hidden_entries =
true);
I thought a bit about usability and I vote for removing this parameter and skip
only "." and ".." - I can't imagine any use case when I would be interested in
those.
http://gerrit.cloudera.org:8080/#/c/9986/14/be/src/util/filesystem-util.h@109
PS14, Line 109: static Status GetEntryNames(const string& path,
I would prefer max_result_size to be the last parameter, and give it a default
value of 0.
http://gerrit.cloudera.org:8080/#/c/9986/14/be/src/util/zip-util-test.cc
File be/src/util/zip-util-test.cc:
http://gerrit.cloudera.org:8080/#/c/9986/14/be/src/util/zip-util-test.cc@69
PS14, Line 69: EXPECT_FALSE(filesystem::exists(dest_dir3));
I guess that this is only true if zip decoding failed at the start, and some
files may be already decompressed before reaching an error in the zip. I am not
sure what to do with this, probably nothing. It would be possible add some kind
of cleanup logic to the java util, but I am not sure if this worth the effort.
http://gerrit.cloudera.org:8080/#/c/9986/14/fe/src/main/java/org/apache/impala/util/ZipUtil.java
File fe/src/main/java/org/apache/impala/util/ZipUtil.java:
http://gerrit.cloudera.org:8080/#/c/9986/14/fe/src/main/java/org/apache/impala/util/ZipUtil.java@45
PS14, Line 45: try (ZipFile zip = new ZipFile(params.archive_file)) {
I would move this block to a similar function with (String archiveFile, String
destDir) parameters to make this util usable from Java too. This would be
minimal extra effort and I think that it can be handy to have an easily usable
Zip-Slip safe extract function.
--
To view, visit http://gerrit.cloudera.org:8080/9986
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93c1fbffe81f067919706e30db0a34d0e58e7e77
Gerrit-Change-Number: 9986
Gerrit-PatchSet: 14
Gerrit-Owner: Attila Jeges
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Dan Hecht
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Fri, 08 Jun 2018 16:15:15 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6812: Fix flaky Kudu scan tests
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10503 ) Change subject: IMPALA-6812: Fix flaky Kudu scan tests .. Patch Set 1: So it sounds like we could switch to RYW for testing, see how that works out, and then make a final decision before the next release? -- To view, visit http://gerrit.cloudera.org:8080/10503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70df84f2cbc663107f2ad029565d3c15bdfbd47c Gerrit-Change-Number: 10503 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 08 Jun 2018 16:06:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7144: Re-enable TestDescribeTableResults
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10643 ) Change subject: IMPALA-7144: Re-enable TestDescribeTableResults .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10643 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f Gerrit-Change-Number: 10643 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 16:06:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7144: Re-enable TestDescribeTableResults
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10643 ) Change subject: IMPALA-7144: Re-enable TestDescribeTableResults .. Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/10643/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java: http://gerrit.cloudera.org:8080/#/c/10643/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1979 PS5, Line 1979: rowIdx += 2; // Skips over the first empty line. : Preconditions.checkElementIndex(rowIdx, rows.size()); > simplify: Done http://gerrit.cloudera.org:8080/#/c/10643/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1985 PS5, Line 1985: rowIdx++; > simplify: Done -- To view, visit http://gerrit.cloudera.org:8080/10643 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f Gerrit-Change-Number: 10643 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 16:04:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7144: Re-enable TestDescribeTableResults
Fredy Wijaya has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/10643 ) Change subject: IMPALA-7144: Re-enable TestDescribeTableResults .. IMPALA-7144: Re-enable TestDescribeTableResults This patch makes the TestDescribeTableResults more robust by only comparing the information that the authorization cares about instead of comparing all output in DESCRIBE. This change will avoid any unnecessary changes to AuthorizationTest if HMS updates the DESCRIBE output. The test is also updated to support standalone execution without relying on other tests be executed first since it can cause the test to be flaky especially if the tests in AuthorizationTest are executed in parallel. Testing: - Ran all FE tests Cherry-picks: not for 2.x Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f --- M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java 1 file changed, 174 insertions(+), 186 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/10643/6 -- To view, visit http://gerrit.cloudera.org:8080/10643 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f Gerrit-Change-Number: 10643 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] revert "IMPALA-7149: Skip q7 in test mem usage scaling in erasure coding build"
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10655 ) Change subject: revert "IMPALA-7149: Skip q7 in test_mem_usage_scaling in erasure coding build" .. Patch Set 2: This was required to unblock the 2.x cherry-pick job test failure. I could have added an exclusion but I think this was the fastest way to solve the problem. -- To view, visit http://gerrit.cloudera.org:8080/10655 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib442b61367a236d15d21ee9da2f278907b21d31b Gerrit-Change-Number: 10655 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 16:00:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7144: Re-enable TestDescribeTableResults
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10643 ) Change subject: IMPALA-7144: Re-enable TestDescribeTableResults .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/10643/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java: http://gerrit.cloudera.org:8080/#/c/10643/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1979 PS5, Line 1979: rowIdx++; // Ignore the first empty line. : Preconditions.checkElementIndex(rowIdx + 1, rows.size()); simplify: rowIdx += 2; // Skips over the first empty line Preconditions.checkElementIndex(rowIdx, rows.size()); cols = rows.get(rowIdx).getColVals(); http://gerrit.cloudera.org:8080/#/c/10643/5/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1985 PS5, Line 1985: Preconditions.checkElementIndex(rowIdx + 1, rows.size()); simplify: ++rowIdx; Preconditions.checkElementIndex(rowIdx, rows.size()); cols = rows.get(rowIdx).getColVals(); -- To view, visit http://gerrit.cloudera.org:8080/10643 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f Gerrit-Change-Number: 10643 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 15:59:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7149: Skip q7 in test mem usage scaling in erasure coding build
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10647 ) Change subject: IMPALA-7149: Skip q7 in test_mem_usage_scaling in erasure coding build .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/10647/2/tests/query_test/test_mem_usage_scaling.py File tests/query_test/test_mem_usage_scaling.py: http://gerrit.cloudera.org:8080/#/c/10647/2/tests/query_test/test_mem_usage_scaling.py@35 PS2, Line 35: @SkipIfNotHdfsMinicluster.tuned_for_minicluster What is the purpose of running this test for EC? What test coverage is this providing? Do we expect the memory consumption to be exactly the same for EC as for HDFS, given that it's still fairly timing-dependent. The background is that this test was tuned for the 3-node minicluster and tested for a while in that configuration to make sure it isn't flaky. I've tagged tests like that with the above decorator so that we don't run them in alternative configurations, and I'm not sure that carrying these tests over to EC is the right thing. -- To view, visit http://gerrit.cloudera.org:8080/10647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic9a34a91eef40e1da9c7134cfb7054006d9115de Gerrit-Change-Number: 10647 Gerrit-PatchSet: 2 Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 15:58:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] revert "IMPALA-7149: Skip q7 in test mem usage scaling in erasure coding build"
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10655 ) Change subject: revert "IMPALA-7149: Skip q7 in test_mem_usage_scaling in erasure coding build" .. Patch Set 1: Verified+1 Code-Review+2 Clean revert. -- To view, visit http://gerrit.cloudera.org:8080/10655 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib442b61367a236d15d21ee9da2f278907b21d31b Gerrit-Change-Number: 10655 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 15:53:28 + Gerrit-HasComments: No
[Impala-ASF-CR] revert "IMPALA-7149: Skip q7 in test mem usage scaling in erasure coding build"
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10655 Change subject: revert "IMPALA-7149: Skip q7 in test_mem_usage_scaling in erasure coding build" .. revert "IMPALA-7149: Skip q7 in test_mem_usage_scaling in erasure coding build" This reverts commit 2fee52149e913a12338885cc49c5a86f7d788376. Change-Id: Ib442b61367a236d15d21ee9da2f278907b21d31b --- M tests/query_test/test_mem_usage_scaling.py 1 file changed, 1 insertion(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/10655/1 -- To view, visit http://gerrit.cloudera.org:8080/10655 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib442b61367a236d15d21ee9da2f278907b21d31b Gerrit-Change-Number: 10655 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] revert "IMPALA-7149: Skip q7 in test mem usage scaling in erasure coding build"
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10655 ) Change subject: revert "IMPALA-7149: Skip q7 in test_mem_usage_scaling in erasure coding build" .. revert "IMPALA-7149: Skip q7 in test_mem_usage_scaling in erasure coding build" This reverts commit 2fee52149e913a12338885cc49c5a86f7d788376. Change-Id: Ib442b61367a236d15d21ee9da2f278907b21d31b Reviewed-on: http://gerrit.cloudera.org:8080/10655 Reviewed-by: Tim Armstrong Tested-by: Tim Armstrong --- M tests/query_test/test_mem_usage_scaling.py 1 file changed, 1 insertion(+), 2 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10655 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib442b61367a236d15d21ee9da2f278907b21d31b Gerrit-Change-Number: 10655 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7149: Skip q7 in test mem usage scaling in erasure coding build
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10647 ) Change subject: IMPALA-7149: Skip q7 in test_mem_usage_scaling in erasure coding build .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/10647/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10647/2//COMMIT_MSG@9 PS2, Line 9: The test is flaky in the erasure coding build. Let's disable it for now. This broke the 2.x cherry-picking. 12:19:08 E ImportError: cannot import name SkipIfEC -- To view, visit http://gerrit.cloudera.org:8080/10647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic9a34a91eef40e1da9c7134cfb7054006d9115de Gerrit-Change-Number: 10647 Gerrit-PatchSet: 2 Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 15:53:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-2195: Improper handling of comments in queries
Michael Brown has posted comments on this change. ( http://gerrit.cloudera.org:8080/9933 ) Change subject: IMPALA-2195: Improper handling of comments in queries .. Patch Set 14: +1 since we are trying to ease changes in gently still. -- To view, visit http://gerrit.cloudera.org:8080/9933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ac7cb5a30e6dda73ebe761d9f0eb9ba038e14a7 Gerrit-Change-Number: 9933 Gerrit-PatchSet: 14 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Comment-Date: Fri, 08 Jun 2018 14:57:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2195: Improper handling of comments in queries
Michael Brown has posted comments on this change. ( http://gerrit.cloudera.org:8080/9933 ) Change subject: IMPALA-2195: Improper handling of comments in queries .. Patch Set 14: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/9933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ac7cb5a30e6dda73ebe761d9f0eb9ba038e14a7 Gerrit-Change-Number: 9933 Gerrit-PatchSet: 14 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Comment-Date: Fri, 08 Jun 2018 14:55:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7144: Re-enable TestDescribeTableResults
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/10643 ) Change subject: IMPALA-7144: Re-enable TestDescribeTableResults .. IMPALA-7144: Re-enable TestDescribeTableResults This patch makes the TestDescribeTableResults more robust by only comparing the information that the authorization cares about instead of comparing all output in DESCRIBE. This change will avoid any unnecessary changes to AuthorizationTest if HMS updates the DESCRIBE output. The test is also updated to support standalone execution without relying on other tests be executed first since it can cause the test to be flaky especially if the tests in AuthorizationTest are executed in parallel. Testing: - Ran all FE tests Cherry-picks: not for 2.x Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f --- M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java 1 file changed, 173 insertions(+), 186 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/10643/5 -- To view, visit http://gerrit.cloudera.org:8080/10643 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f Gerrit-Change-Number: 10643 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7144: Re-enable TestDescribeTableResults
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10643 ) Change subject: IMPALA-7144: Re-enable TestDescribeTableResults .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/10643/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java: http://gerrit.cloudera.org:8080/#/c/10643/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1974 PS4, Line 1974: Precondi > does it matter if location is set multiple times for a given set of rows? There can only be one location. I'll add a precondition. http://gerrit.cloudera.org:8080/#/c/10643/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1979 PS4, Line 1979: pty li > that's not the index that is accessed (++rowIdx). Ah yeah. Done. -- To view, visit http://gerrit.cloudera.org:8080/10643 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f Gerrit-Change-Number: 10643 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 13:50:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3307: Add support for IANA time-zone db
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/9986 ) Change subject: IMPALA-3307: Add support for IANA time-zone db .. Patch Set 14: > Uploaded patch set 14. Added one more BE test for extracting files from a zip archive to a non-writable destination directory. Fixed zip-slip vulnerability in ZipUtil.java. -- To view, visit http://gerrit.cloudera.org:8080/9986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93c1fbffe81f067919706e30db0a34d0e58e7e77 Gerrit-Change-Number: 9986 Gerrit-PatchSet: 14 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 08 Jun 2018 13:43:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3307: Add support for IANA time-zone db
Attila Jeges has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/9986 ) Change subject: IMPALA-3307: Add support for IANA time-zone db .. IMPALA-3307: Add support for IANA time-zone db Impala currently uses two different libraries for timestamp manipulations: boost and glibc. Issues with boost: - Time-zone database is currently hard coded in timezone_db.cc. Impala admins cannot update it without upgrading Impala. - Time-zone database is flat, therefore can’t track year-to-year changes. - Time-zone database is not updated on a regular basis. Issues with glibc: - Uses /usr/share/zoneinfo/ database which could be out of sync on some of the nodes in the Impala cluster. - Uses the host system’s local time-zone. Different nodes in the Impala cluster might use a different local time-zone. - Conversion functions take a global lock, which causes severe performance degradation. In addition to the issues above, the fact that /usr/share/zoneinfo/ and the hard-coded boost time-zone database are both in use is a source of inconsistency in itself. This patch makes the following changes: - Instead of boost and glibc, impalad uses Google's CCTZ to implement time-zone conversions. - Introduces a new startup flag (--hdfs_zone_info_zip) to impalad to specify an HDFS/S3/ADLS path to a zip archive that contains the shared compiled IANA time-zone database. If the startup flag is set, impalad will use the specified time-zone database. Otherwise, impalad will use the default /usr/share/zoneinfo time-zone database. - impalad reads the entire time-zone database into an in-memory map on startup for fast lookups. - The name of the coordinator node’s local time-zone is saved to the query context when preparing query execution. This time-zone is used whenever the current time-zone is referred afterwards in an execution node. - Introduces a new startup flag (--hdfs_zone_abbrev_conf) to impalad to specify an HDFS/S3/ADLS path to a shared config file that contains definitions for non-standard time-zone abbreviations. Cherry-picks: not for 2.x. Change-Id: I93c1fbffe81f067919706e30db0a34d0e58e7e77 --- M CMakeLists.txt M be/CMakeLists.txt M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/benchmarks/CMakeLists.txt A be/src/benchmarks/convert-timestamp-benchmark.cc M be/src/common/global-types.h M be/src/common/init.cc M be/src/exec/data-source-scan-node.cc M be/src/exec/data-source-scan-node.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/parquet-column-readers.cc M be/src/exprs/CMakeLists.txt M be/src/exprs/aggregate-functions-ir.cc M be/src/exprs/cast-functions-ir.cc M be/src/exprs/decimal-operators-ir.cc M be/src/exprs/decimal-operators.h M be/src/exprs/expr-test.cc M be/src/exprs/literal.cc M be/src/exprs/timestamp-functions-ir.cc M be/src/exprs/timestamp-functions.cc A be/src/exprs/timezone_db-test.cc M be/src/exprs/timezone_db.cc M be/src/exprs/timezone_db.h M be/src/runtime/raw-value-test.cc M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/runtime/timestamp-test.cc M be/src/runtime/timestamp-value.cc M be/src/runtime/timestamp-value.h M be/src/runtime/timestamp-value.inline.h M be/src/service/frontend.cc M be/src/service/impala-server.cc M be/src/service/impalad-main.cc M be/src/util/CMakeLists.txt M be/src/util/filesystem-util-test.cc M be/src/util/filesystem-util.cc M be/src/util/filesystem-util.h M be/src/util/hdfs-util-test.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M be/src/util/time-test.cc M be/src/util/time.cc M be/src/util/time.h A be/src/util/zip-util-test.cc A be/src/util/zip-util.cc A be/src/util/zip-util.h M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/rat_exclude_files.txt A cmake_modules/FindCctz.cmake M common/thrift/CMakeLists.txt M common/thrift/ImpalaInternalService.thrift A common/thrift/Zip.thrift M common/thrift/metrics.json A fe/src/main/java/org/apache/impala/util/ZipUtil.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M testdata/bin/create-load-data.sh M testdata/data/timezoneverification.csv A testdata/tzdb/2017c-corrupt.zip A testdata/tzdb/2017c.zip A testdata/tzdb/abbrev.conf A testdata/tzdb_tiny/America/New_York A testdata/tzdb_tiny/Etc/GMT+4 A testdata/tzdb_tiny/US/Eastern A testdata/tzdb_tiny/UTC A testdata/tzdb_tiny/Zulu A testdata/tzdb_tiny/posix/UTC A testdata/tzdb_tiny/posixrules M testdata/workloads/functional-query/queries/QueryTest/exprs.test M tests/custom_cluster/test_hive_parquet_timestamp_conversion.py A tests/custom_cluster/test_shared_tzdb.py D tests/query_test/test_timezones.py 72 files changed, 3,117 insertions(+), 1,167 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/9986/14 -- To view, visit http://gerrit.cloudera.org:8080/9986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master
[Impala-ASF-CR] IMPALA-7121: Clean up partitionIds from HdfsTable
Gabor Kaszab has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10654 Change subject: IMPALA-7121: Clean up partitionIds_ from HdfsTable .. IMPALA-7121: Clean up partitionIds_ from HdfsTable The purpose of introducing partitionIds_ member to HdfsTable was to be able to return the IDs of all the current partitions in constant time. Apparently, partitionMap_ also contains these IDs as the key of the map and this is accessible via keySet() also in constant time. It seems reasonable then to remove partitionIds_ and use partitionMap_.keySet() in getPartitionIds() to save some memory. One thing needs extra attention here is that modifying the result of keySet() would also modify partitionMap_ and we should avoid doing this. On every callsites of getPartitionIds() the first step the caller does is to copy the received items to a separate set. So as a solution getPartitionIds() internally creates a copy of the keySet(), removes the default partition and returns this copy to be sure that partitionMap_ can't be altered. The caller sites are also changed not to copy the items but to simpy use the set they received. This will guarantee that we don't regress the computing complexity of getting the partition IDs. Change-Id: I8b5a480e570aeae565fafd4f3e2b279e7a98c7da --- M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java 2 files changed, 16 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/10654/1 -- To view, visit http://gerrit.cloudera.org:8080/10654 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8b5a480e570aeae565fafd4f3e2b279e7a98c7da Gerrit-Change-Number: 10654 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab
[Impala-ASF-CR] IMPALA-6119: Fix issue with multiple partitions sharing same location
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/10543 ) Change subject: IMPALA-6119: Fix issue with multiple partitions sharing same location .. Patch Set 10: (1 comment) http://gerrit.cloudera.org:8080/#/c/10543/9/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/10543/9/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1440 PS9, Line 1440: dropPartitions(partitionsToRemove); This wouldn't work out unfortunately. When you modify the location of a partition than internally the modified partition is set dirty and then it is added to partitionsToRemove at L1439. Here we say that we remove all the other partitions on the same location, but as a result if we originally intended to set the location of a partition to an existing partition's location we end up dropping the other partition as well. Ouch :) In my opinion here we shouldn't call getPartitionsWithSameLocation. The reason we had to do this is that in L1399 Hive returns the available partitions incorrectly (In case we drop a partition with Hive the ones on the same location still appear as valid even though the directory beneath is dropped). So I'd go for not introducing a workaround for this in Impala, I rather opened a Jira for Hive to fix the root cause. Until it is taken care I'd disable the new test_drop_partition_from_hive test and enable it once Hive returns the correct list of partitions. What do you think? -- To view, visit http://gerrit.cloudera.org:8080/10543 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a54bc8224bcefe65b83de2df58bb84629f2aa4a Gerrit-Change-Number: 10543 Gerrit-PatchSet: 10 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 08 Jun 2018 13:36:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6119: Fix issue with multiple partitions sharing same location
Hello Bharath Vissapragada, Zoltan Borok-Nagy, Sailesh Mukil, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10543 to look at the new patch set (#10). Change subject: IMPALA-6119: Fix issue with multiple partitions sharing same location .. IMPALA-6119: Fix issue with multiple partitions sharing same location When multiple partitions point to the same location and a new data file is added to any of them then the expected behaviour is that this new file is added to the other partitions pointing to the same location as well. Apparently, this is not the case and right after the insertion the new file is only visible in the partition where it was inserted to and an invalidate metadata is needed to resolve this inconsistency. This fix addresses this issue with keeping track of a mapping between locations and the HdfsPartitions pointing to it. When new files are inserted into a partition then all the other partition's metadata are reloaded that point to the same location as the one where the files are inserted. The same issue is present when a partition is dropped and there are one or more partitions that share the location of this dropped partition. In this case the actual directory of the partition is erased however, Catalog didn't remove the other partitions on this location and showed them as existing ones. Again, an invalidate metadata helped to make them disappear. This issue is also fixed. Testing: There was an existing test that covered partitions pointing to the same location. However, after each insert it executed a refresh to reload the metadata for the entire table. This reload was removed to cover the changes of this fix. Another test is introduced to cover the case when the location of a partition is altered or a partition is removed. One more test is created to cover when Impala reloads some of it's partitions after Hive had dropped a partition that shares it's location with other partitions. Change-Id: I2a54bc8224bcefe65b83de2df58bb84629f2aa4a --- M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M tests/metadata/test_partition_metadata.py 4 files changed, 177 insertions(+), 13 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/10543/10 -- To view, visit http://gerrit.cloudera.org:8080/10543 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2a54bc8224bcefe65b83de2df58bb84629f2aa4a Gerrit-Change-Number: 10543 Gerrit-PatchSet: 10 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7108: IllegalStateException hit during CardinalityCheckNode.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/10605 ) Change subject: IMPALA-7108: IllegalStateException hit during CardinalityCheckNode. .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/10605/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java: http://gerrit.cloudera.org:8080/#/c/10605/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@1082 PS3, Line 1082: returns at most one r > I think this bit is still inaccurate. Changed to 'returns at most one row'. The name of the method is still inaccurate, but maybe it's okay with the rephrased comments. http://gerrit.cloudera.org:8080/#/c/10605/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@1100 PS3, Line 1100: returns at most one > Does this one need to be changed to be inline with 4. above as well? Right, thanks! -- To view, visit http://gerrit.cloudera.org:8080/10605 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82a7a3fe26db3e12131c030c4ad055a9c4955407 Gerrit-Change-Number: 10605 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vincent Tran Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 08 Jun 2018 09:55:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7108: IllegalStateException hit during CardinalityCheckNode.
Hello Vincent Tran, Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10605 to look at the new patch set (#4). Change subject: IMPALA-7108: IllegalStateException hit during CardinalityCheckNode. .. IMPALA-7108: IllegalStateException hit during CardinalityCheckNode. Since IMPALA-6314 on runtime scalar subqueries we set LIMIT 2 in StmtRewriter.mergeExpr(). We do that because later we add a CardinalityCheckNode on top of such subqueries and with LIMIT 2 we can still check if they return more than one row. In the constructor of CardinalityCheckNode there is a precondition that checks if the child node has LIMIT 2 to be certain that we've set the limit for all the necessary cases. However, some subqueries will get a LIMIT 1 later breaking the precondition in CardinalityCheckNode. An example to these subqueries is a select stmt that selects from an inline view that returns a single row: select * from functional.alltypes where int_col = (select f.id from ( select * from functional.alltypes limit 1) f); Note that we shouldn't add a CardinalityCheckNode to the plan of this query in the first place. To generate a proper plan I updated SelectStmt.returnsSingleRow() because this method didn't handle this case well. I also changed the precondition from Preconditions.checkState(child.getLimit() == 2); to Preconditions.checkState(child.getLimit() <= 2); in order to be more permissive. I added tests for the aforementioned query. Change-Id: I82a7a3fe26db3e12131c030c4ad055a9c4955407 --- M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/planner/CardinalityCheckNode.java M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test M testdata/workloads/functional-query/queries/QueryTest/subquery.test 4 files changed, 67 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/10605/4 -- To view, visit http://gerrit.cloudera.org:8080/10605 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I82a7a3fe26db3e12131c030c4ad055a9c4955407 Gerrit-Change-Number: 10605 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vincent Tran Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7144: Re-enable TestDescribeTableResults
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10643 ) Change subject: IMPALA-7144: Re-enable TestDescribeTableResults .. Patch Set 4: (2 comments) http://gerrit.cloudera.org:8080/#/c/10643/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java: http://gerrit.cloudera.org:8080/#/c/10643/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1974 PS4, Line 1974: location does it matter if location is set multiple times for a given set of rows? http://gerrit.cloudera.org:8080/#/c/10643/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1979 PS4, Line 1979: rowIdx that's not the index that is accessed (++rowIdx). -- To view, visit http://gerrit.cloudera.org:8080/10643 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f Gerrit-Change-Number: 10643 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 06:54:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5931: Generates scan ranges in planner for s3/adls
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/8523 ) Change subject: IMPALA-5931: Generates scan ranges in planner for s3/adls .. Patch Set 19: rebased (clean). re-ran s3 tests (pass). -- To view, visit http://gerrit.cloudera.org:8080/8523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Gerrit-Change-Number: 8523 Gerrit-PatchSet: 19 Gerrit-Owner: Vuk Ercegovac Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Dimitris Tsirogiannis Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 06:43:31 + Gerrit-HasComments: No
