[Impala-ASF-CR] IMPALA-7078: Part 2: reduce queue size based on num scanner threads
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10573 ) Change subject: IMPALA-7078: Part 2: reduce queue size based on num_scanner_threads .. IMPALA-7078: Part 2: reduce queue size based on num_scanner_threads This gives a tighter bound on memory consumption when running with a lower num_scanner_threads value. With IMPALA-7096 we'll revisit the approach to reliably avoid OOM. Cap the maximum row batch queue size at 5 * max_num_scanner_threads_ so that we guarantee lower amounts of memory in the row batch queue when num_scanner_threads is set, rather than just achieving it statistically because of the producer running slower relative to consumer. It does not reduce the default significantly on typical server configurations that would have 24+ cores except under high concurrency or low memory environments where the number of scanner threads is limited. We should evaluate reducing the default further or otherwise better controlling memory consumption in a follow-up, based on experiments. Testing: Tested along with Part 1. Change-Id: I8c0bd14e6bcd9fc1655e344a5307ea0eb4600e8b Reviewed-on: http://gerrit.cloudera.org:8080/10573 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M be/src/exec/hdfs-scan-node.cc 1 file changed, 19 insertions(+), 6 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10573 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8c0bd14e6bcd9fc1655e344a5307ea0eb4600e8b Gerrit-Change-Number: 10573 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7078: Part 1: improve memory consumption of wide Avro scans
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10550 ) Change subject: IMPALA-7078: Part 1: improve memory consumption of wide Avro scans .. IMPALA-7078: Part 1: improve memory consumption of wide Avro scans Revert to the pre-IMPALA-3905 algorithm for deciding when to return a batch from an Avro scan. The post-IMPALA-3905 algorithm is bad for wide tables where there are only a small number of rows per Avro block. Optimise memory transfer for selective scans - don't attach unused decompression buffers to the output batch. Combined with the previous change, this dramatically reduces the amount of memory transferred out of scanner threads for selective scans of wide tables. Includes some observability improvements including additional counters that will help diagnose issues like this more easily: * Add counters to give some insight into row batch queue. Here's an excerpt: - RowBatchBytesEnqueued: 20.89 MB (21903380) - RowBatchQueueCapacity: 5 (5) - RowBatchQueueGetWaitTime: 59.187ms - RowBatchQueuePeakMemoryUsage: 8.85 MB (9279347) - RowBatchQueuePutWaitTime: 0.000ns - RowBatchesEnqueued: 6 (6) * Don't create AverageScannerThreadConcurrency for MT scan node where it's not actually used. * Track the row batch queue memory consumption against a sub-tracker HDFS_SCAN_NODE (id=2): Reservation=48.00 MB OtherMemory=588.00 KB Total=48.57 MB Peak=48.62 MB Queued Batches: Total=588.00 KB Peak=637.00 KB Ran the repro in the JIRA. Memory consumption was reduced from ~500MB to ~220MB on my system. Testing: * Ran stress test for an hour on uncompressed and 3 hours on snappy-compressed avro. * Debug exhaustive tests passed. * ASAN core tests passed. Perf: - Parquet TPC-H scale factor 60 on one impalad showed no change in perf - Avro/Snappy scale factor 20 showed a small improvement: +--+-+-++++ | Workload | File Format | Avg (s) | Delta(Avg) | GeoMean(s) | Delta(GeoMean) | +--+-+-++++ | TPCH(20) | avro / snap / block | 9.86| -2.23% | 7.83 | -2.37% | +--+-+-++++ +--+--+--++-++---++-+---+ | Workload | Query| File Format | Avg(s) | Base Avg(s) | Delta(Avg) | StdDev(%) | Base StdDev(%) | Num Clients | Iters | +--+--+--++-++---++-+---+ | TPCH(20) | TPCH-Q6 | avro / block / block | 5.59 | 5.17| +8.10% | 0.75% | 0.71%| 1 | 30| | TPCH(20) | TPCH-Q14 | avro / block / block | 6.31 | 5.89| +7.21% | 0.68% | 0.74%| 1 | 30| | TPCH(20) | TPCH-Q15 | avro / block / block | 11.32 | 10.64 | +6.41% | 0.37% | 0.46%| 1 | 30| | TPCH(20) | TPCH-Q12 | avro / block / block | 8.57 | 8.14| +5.23% | 0.67% | 0.81%| 1 | 30| | TPCH(20) | TPCH-Q13 | avro / block / block | 6.72 | 6.54| +2.72% | 0.77% | 0.73%| 1 | 30| | TPCH(20) | TPCH-Q4 | avro / block / block | 11.76 | 11.61 | +1.32% | 0.60% | 0.61%| 1 | 30| | TPCH(20) | TPCH-Q7 | avro / block / block | 14.43 | 14.26 | +1.21% | 1.14% | 0.35%| 1 | 30| | TPCH(20) | TPCH-Q21 | avro / block / block | 34.12 | 34.25 | -0.36% | 0.27% | 0.24%| 1 | 30| | TPCH(20) | TPCH-Q20 | avro / block / block | 8.49 | 8.52| -0.38% | 0.45% | 0.54%| 1 | 30| | TPCH(20) | TPCH-Q1 | avro / block / block | 6.99 | 7.02| -0.38% | 0.96% | 0.65%| 1 | 30| | TPCH(20) | TPCH-Q22 | avro / block / block | 2.44 | 2.47| -1.09% | 1.81% | 1.47%| 1 | 30| | TPCH(20) | TPCH-Q11 | avro / block / block | 1.99 | 2.02| -1.57% | 1.95% | 1.90%| 1 | 30| | TPCH(20) | TPCH-Q17 | avro / block / block | 13.57 | 13.79 | -1.63% | 1.53% | 1.31%| 1 | 30| | TPCH(20) | TPCH-Q18 | avro / block / block | 21.93 | 22.31 | -1.72% | 0.31% | 0.34%| 1 | 30| | TPCH(20) | TPCH-Q8 | avro / block / block | 9.05 | 9.31| -2.81% | 0.85% | 0.72%| 1 | 30| | TPCH(20) | TPCH-Q19 | avro / block / block | 7.20 | 7.41| -2.91% | 0.72% | 0.52%| 1 | 30| | TPCH(20) | TPCH-Q9 | avro / block /
[Impala-ASF-CR] IMPALA-7078: Part 1: improve memory consumption of wide Avro scans
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10550 ) Change subject: IMPALA-7078: Part 1: improve memory consumption of wide Avro scans .. Patch Set 12: Verified+1 Tested by https://jenkins.impala.io/job/gerrit-verify-dryrun/2612/ -- To view, visit http://gerrit.cloudera.org:8080/10550 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iebd2600b4784fd19696c9b92eefb7d7e9db0c80b Gerrit-Change-Number: 10550 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vincent Tran Gerrit-Comment-Date: Thu, 07 Jun 2018 04:40:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5552: Add support for authorized proxy groups
Philip Zeyliger has posted comments on this change. (
http://gerrit.cloudera.org:8080/10510 )
Change subject: IMPALA-5552: Add support for authorized proxy groups
..
Patch Set 14:
(8 comments)
Thanks for all the changes. I've made another pass and have some more questions.
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/frontend.cc
File be/src/service/frontend.cc:
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/frontend.cc@61
PS14, Line 61: "Specifies the set of authorized proxy groups (users who can
delegate to other "
: "groups during authorization) and whom they are allowed to
delegate. "
For other components that have done this, how have they documented this?
I don't think it's quite right to say "delegate to other groups", because the
delegation actually happens to a user. This is saying that a given user can
delegate to any user in the named groups, which is slightly different than
delegating to the group.
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/frontend.cc@66
PS14, Line 66: "changed via --authorized_proxy_group_config_delimiter), or
'*' to indicate all "
What does "*" mean here? If you wanted to be able to delegate to anyone, you'd
use the user mechanism. I'm not sure "*" makes sense in context of groups.
Does it?
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/impala-server.h
File be/src/service/impala-server.h:
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/impala-server.h@1031
PS14, Line 1031: /// Map of short usernames of authorized proxy users to the
set of group(s) they are
nit: "set of groups" and ("set of users" on line 1027). There's no reading here
where you want "set of group".
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/impala-server.cc@285
PS14, Line 285: Status status =
exec_env_->frontend()->ValidateSettings(request);
I believe the way we typically do this is TBackendGflags. I think you can
update backend-gflag-util.cc to populate that. Then you don't need this.
See JniFrontend.java's constructor for how it's initialized.
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/impala-server.cc@417
PS14, Line 417: boost::trim(proxy_user);
The old impl didn't trim. I think that's fine, but wanted to check that's a
conscious choice we're making; I don't think it was mentioned in the commit
message.
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/impala-server.cc@1382
PS14, Line 1382: boost::unordered_set users = proxy_user->second;
I think we're slowly migrating from boost::unordered_set to the c++11
equivalent:
$git grep unordered_set | grep '#include' | awk -F: '{ print $2 }' | sort |
uniq -c
19 #include
22 #include
You don't have to, but if you wanted to change this away from boost, I think it
would be welcome and trivial.
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/impala-server.cc@1393
PS14, Line 1393: boost::unordered_set groups = proxy_group->second;
You're referring to proxy_group here without checking if it's groups.end()
(which you do on line 1394). So, something's wrong here, or I'm mis-reading
this.
If this is wrong, please add a test that would have caught it?
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/impala-server.cc@1402
PS14, Line 1402: VLOG_QUERY << "Getting Hadoop groups took " <<
If we're going to log it, let's include the short_user here as well.
--
To view, visit http://gerrit.cloudera.org:8080/10510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb
Gerrit-Change-Number: 10510
Gerrit-PatchSet: 14
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Greg Rahn
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Sailesh Mukil
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Thu, 07 Jun 2018 04:09:18 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7137. Support configuring Frontend to use LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10629 ) Change subject: IMPALA-7137. Support configuring Frontend to use LocalCatalog .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2614/ -- To view, visit http://gerrit.cloudera.org:8080/10629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c9665bd031d23608740b23eef301970af9aa764 Gerrit-Change-Number: 10629 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 07 Jun 2018 03:57:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2751: Matching quotes are not required in comments
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10541 ) Change subject: IMPALA-2751: Matching quotes are not required in comments .. Patch Set 4: Un-revert 84b55c6148dd4e756359717cca30a8eea71cb62a. I tested it on both Python 2.6 and 2.7. Please re-review it again. -- To view, visit http://gerrit.cloudera.org:8080/10541 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2feae34026a7e63f3d31489f757f093a73ca5d2c Gerrit-Change-Number: 10541 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Thu, 07 Jun 2018 03:49:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2751: Matching quotes are not required in comments
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/10541 ) Change subject: IMPALA-2751: Matching quotes are not required in comments .. IMPALA-2751: Matching quotes are not required in comments This patch fixes the issue where non-matching quotes inside comments will cause the shell to not terminate. The fix is to strip any SQL comments before sending to shlex since shlex does not understand SQL comments and will raise an exception when it sees unmatched quotes regardless whether the quotes are in the comments or not. Testing: - Added new shell tests - Ran all end-to-end shell tests on Python 2.6 and Python 2.7 Change-Id: I2feae34026a7e63f3d31489f757f093a73ca5d2c --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 26 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/10541/4 -- To view, visit http://gerrit.cloudera.org:8080/10541 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2feae34026a7e63f3d31489f757f093a73ca5d2c Gerrit-Change-Number: 10541 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-2751: Matching quotes are not required in comments
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/10541 ) Change subject: IMPALA-2751: Matching quotes are not required in comments .. IMPALA-2751: Matching quotes are not required in comments This patch fixes the issue where non-matching quotes inside comments will cause the shell to not terminate. The fix is to strip any SQL comments before sending to shlex since shlex does not understand SQL comments and will raise an exception when it sees unmatched quotes regardless whether the quotes are in the comments or not. Testing: - Added new shell tests - Ran all end-to-end shell tests on Python 2.6 and Python 2.7 Change-Id: I2feae34026a7e63f3d31489f757f093a73ca5d2c --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 26 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/10541/3 -- To view, visit http://gerrit.cloudera.org:8080/10541 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2feae34026a7e63f3d31489f757f093a73ca5d2c Gerrit-Change-Number: 10541 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-5552: Add support for authorized proxy groups
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/10510 ) Change subject: IMPALA-5552: Add support for authorized proxy groups .. Patch Set 14: Code-Review+1 Phil, do you want to +2? -- To view, visit http://gerrit.cloudera.org:8080/10510 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb Gerrit-Change-Number: 10510 Gerrit-PatchSet: 14 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 07 Jun 2018 03:12:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7141. Extract interfaces for partition pruning without loading
Hello Vuk Ercegovac, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10631 to review the following change. Change subject: IMPALA-7141. Extract interfaces for partition pruning without loading .. IMPALA-7141. Extract interfaces for partition pruning without loading This adds a new PrunablePartition interface which HdfsPartition implements. The interface includes only the partition ID and the partition key/values. For the case of the current catalog implementation, this provides no benefit. However, for LocalCatalog, we want to defer loading partition information until after pruning. With this interface, we can construct PrunablePartition objects using just the partition names, and then load the partitions once pruning is complete. Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/FeHdfsTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java A fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java M fe/src/main/java/org/apache/impala/planner/HdfsPartitionFilter.java M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/testutil/BlockIdGenerator.java 18 files changed, 181 insertions(+), 46 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/10631/1 -- To view, visit http://gerrit.cloudera.org:8080/10631 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa Gerrit-Change-Number: 10631 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7082: Show human readable size in query backend page
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10523 ) Change subject: IMPALA-7082: Show human readable size in query backend page .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2615/ -- To view, visit http://gerrit.cloudera.org:8080/10523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I04afb4091bb8b6bc9dedfeceaf9284a8c65b16a1 Gerrit-Change-Number: 10523 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 02:49:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7082: Show human readable size in query backend page
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10523 ) Change subject: IMPALA-7082: Show human readable size in query backend page .. Patch Set 5: Code-Review+2 GVO failed with "java.lang.IllegalStateException: Failed to assign regions to servers after 6 millis.". I'm rebasing and shoving this through GVO again. -- To view, visit http://gerrit.cloudera.org:8080/10523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I04afb4091bb8b6bc9dedfeceaf9284a8c65b16a1 Gerrit-Change-Number: 10523 Gerrit-PatchSet: 5 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 02:49:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7082: Show human readable size in query backend page
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10523 ) Change subject: IMPALA-7082: Show human readable size in query backend page .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2613/ -- To view, visit http://gerrit.cloudera.org:8080/10523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I04afb4091bb8b6bc9dedfeceaf9284a8c65b16a1 Gerrit-Change-Number: 10523 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 02:23:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7140 (part 1). Support fetching schema info in LocalCatalog
Hello Vuk Ercegovac, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10630 to review the following change. Change subject: IMPALA-7140 (part 1). Support fetching schema info in LocalCatalog .. IMPALA-7140 (part 1). Support fetching schema info in LocalCatalog This adds support for loading the Table object from HMS and parsing out the Column and partitioning information from it. With this change, I'm able to connect to an impalad running in "local catalog" mode and run DESCRIBE, DESCRIBE EXTENDED, and SHOW CREATE TABLE commands. Other commands like SHOW PARTITIONS don't work properly yet, and type-specific table functionality (eg views, HBase tables, etc) are not yet supported. Again a simple unit test is included to check that column information is loaded. More thorough testing is deferred until we've reached enough coverage that we can start running e2e tests against a cluster running in "local" mode. Change-Id: I640f27e36198955e057da62a3ce25a858406e496 --- A fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalogException.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 8 files changed, 255 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/10630/1 -- To view, visit http://gerrit.cloudera.org:8080/10630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I640f27e36198955e057da62a3ce25a858406e496 Gerrit-Change-Number: 10630 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-5552: Add support for authorized proxy groups
Fredy Wijaya has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/10510 ) Change subject: IMPALA-5552: Add support for authorized proxy groups .. IMPALA-5552: Add support for authorized proxy groups The patch adds support for mapping of users to a list of proxy groups. The following flags are added in impalad: - authorized_proxy_group_config - authorized_proxy_group_config_delimiter Example: --authorized_proxy_group_config=hue=group1,group2;user1=* This feature is not supported on Shell-based Hadoop groups mapping providers. Testing: - Added FE unit test to check for groups mapping provider - Added BE unit test for the parsing logic - Added a new test in test_authorization.py - Ran all end-to-end test_authorization.py Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb --- M be/src/service/CMakeLists.txt M be/src/service/frontend.cc M be/src/service/frontend.h A be/src/service/impala-server-test.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/backend-gflag-util.cc M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/test/java/org/apache/impala/service/JniFrontendTest.java M tests/authorization/test_authorization.py 11 files changed, 380 insertions(+), 39 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/10510/14 -- To view, visit http://gerrit.cloudera.org:8080/10510 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb Gerrit-Change-Number: 10510 Gerrit-PatchSet: 14 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-5552: Add support for authorized proxy groups
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/10510 )
Change subject: IMPALA-5552: Add support for authorized proxy groups
..
Patch Set 14:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/10510/13/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/10510/13/be/src/service/impala-server.cc@1394
PS13, Line 1394: if (groups.find("*") != groups.end()) return Status::OK();
:
: i
> nit: single line, no braces (same below)
Done
http://gerrit.cloudera.org:8080/#/c/10510/13/be/src/service/impala-server.cc@1406
PS13, Line 1406: );
> nit: PrettyPrinter::Print(MonotinicMillis() - start, TUnit::TIME_MS)
Done
http://gerrit.cloudera.org:8080/#/c/10510/11/tests/authorization/test_authorization.py
File tests/authorization/test_authorization.py:
http://gerrit.cloudera.org:8080/#/c/10510/11/tests/authorization/test_authorization.py@224
PS11, Line 224: ent_resp = self.hs2
> I see. That coverage could be achieved by modifying the args for the test t
That works. Done.
--
To view, visit http://gerrit.cloudera.org:8080/10510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb
Gerrit-Change-Number: 10510
Gerrit-PatchSet: 14
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Greg Rahn
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Sailesh Mukil
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Thu, 07 Jun 2018 01:16:24 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7078: Part 2: reduce queue size based on num scanner threads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10573 ) Change subject: IMPALA-7078: Part 2: reduce queue size based on num_scanner_threads .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10573 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c0bd14e6bcd9fc1655e344a5307ea0eb4600e8b Gerrit-Change-Number: 10573 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 00:45:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7137. Support configuring Frontend to use LocalCatalog
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10629 ) Change subject: IMPALA-7137. Support configuring Frontend to use LocalCatalog .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2614/ -- To view, visit http://gerrit.cloudera.org:8080/10629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c9665bd031d23608740b23eef301970af9aa764 Gerrit-Change-Number: 10629 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 07 Jun 2018 00:36:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7132: Filter out useless output from run clang tidy.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10615 ) Change subject: IMPALA-7132: Filter out useless output from run_clang_tidy.sh .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10615 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I63c46a7d57295eba38fac8ab49c7a15d2802df1d Gerrit-Change-Number: 10615 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 07 Jun 2018 00:36:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7110. Fix errors from error-prone
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10583 ) Change subject: IMPALA-7110. Fix errors from error-prone .. Patch Set 3: This change did not cherrypick successfully into branch 2.x. To resolve this, please do the cherry-pick manually and submit it to Gerrit at refs/for/2.x or add an exception to the branch 2.x copy of bin/ignored_commits.json. Thanks, your friendly bot at https://jenkins.impala.io/job/cherrypick-2.x-and-test/590/ . -- To view, visit http://gerrit.cloudera.org:8080/10583 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9bfadecb5b92bba4fc7921a6f87f249ed90e771 Gerrit-Change-Number: 10583 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Thu, 07 Jun 2018 00:25:07 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Fix Indents from IMPALA-4970
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10622 ) Change subject: Fix Indents from IMPALA-4970 .. Fix Indents from IMPALA-4970 fixing the mistake in indentation made previously Change-Id: I4ea53e0780cca9f791808c2a73c7f72ba27c2159 Reviewed-on: http://gerrit.cloudera.org:8080/10560 Reviewed-by: Sailesh Mukil Tested-by: Tim Armstrong Reviewed-on: http://gerrit.cloudera.org:8080/10622 Reviewed-by: Tim Armstrong --- M be/src/runtime/coordinator.cc 1 file changed, 4 insertions(+), 4 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10622 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: merged Gerrit-Change-Id: I4ea53e0780cca9f791808c2a73c7f72ba27c2159 Gerrit-Change-Number: 10622 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Rahul Shivu Mahadev Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR](2.x) IMPALA-6956: deflake and logging for query expiration test
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10621 ) Change subject: IMPALA-6956: deflake and logging for query_expiration test .. IMPALA-6956: deflake and logging for query_expiration test There was a recent flake where the number of in-flight queries that were executing differed from the expected number. The reason for the false negative is that one of the queries expired before the check for in-flight queries: it took too long to issue the queries. This change modifies the timeout for the expired query to not expire so quickly. Additional logging is added to the check for in-flight queries so that we can distinguish the case of too few queries vs. queries that have the wrong state. Change-Id: I01a8762d28ad920b9ec8a0b1b82469618c66768f Reviewed-on: http://gerrit.cloudera.org:8080/10602 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins Reviewed-on: http://gerrit.cloudera.org:8080/10621 Tested-by: Tim Armstrong --- M tests/custom_cluster/test_query_expiration.py 1 file changed, 9 insertions(+), 6 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10621 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: merged Gerrit-Change-Id: I01a8762d28ad920b9ec8a0b1b82469618c66768f Gerrit-Change-Number: 10621 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR](2.x) Guard S3 access check with variable
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10616 ) Change subject: Guard S3 access check with variable .. Patch Set 1: Verified+1 Verified with https://gerrit.cloudera.org/#/c/10623/ -- To view, visit http://gerrit.cloudera.org:8080/10616 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I1df16b2d037afafb7748f52b64732f680883f16c Gerrit-Change-Number: 10616 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 00:23:56 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7101: Fix race between Fetch and Close RPCs that can lead to hang
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10619 ) Change subject: IMPALA-7101: Fix race between Fetch and Close RPCs that can lead to hang .. IMPALA-7101: Fix race between Fetch and Close RPCs that can lead to hang If we hit EOS, we'll wait for all the backends to report status (to try to get a complete profile). But if the query is closed after this point, then we can get stuck waiting since once the query is closed, ImpalaServer won't know about this coordinator and so it will stop forwarding on the ReportStatus RPCs. The real fix for this is IMPALA-6984, but in the mean time, add another special case for this JIRA (see the other TODO IMPALA-6984 in coordinator.cc). The cancellation test only finds this race once in a while (several hours) indirectly in a COMPUTE STATS query because the ChildQueryExecutor will do a CloseOperation() while the execution thread is inside Fetch(). To make this more reproducible, modify the cancellation test to allow the close and fetch rpcs to execute concurrently (don't join the test's fetch thread until after close). This makes the race reproducible in a few iterations and a few minutes. Testing: - Loop test_cancellation.py Change-Id: I7c147550f86d81b818ecbdd34cf2919ced7ff8c5 Reviewed-on: http://gerrit.cloudera.org:8080/10601 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins Reviewed-on: http://gerrit.cloudera.org:8080/10619 Tested-by: Tim Armstrong --- M be/src/runtime/coordinator.cc M tests/query_test/test_cancellation.py 2 files changed, 47 insertions(+), 9 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10619 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: merged Gerrit-Change-Id: I7c147550f86d81b818ecbdd34cf2919ced7ff8c5 Gerrit-Change-Number: 10619 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR](2.x) IMPALA-7105. Ensure fe tests pass when running standalone
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10620 ) Change subject: IMPALA-7105. Ensure fe tests pass when running standalone .. IMPALA-7105. Ensure fe tests pass when running standalone CatalogTest and HdfsStorageDescriptorTest would fail when run individually because they relied on various initialization that only occurs when FeSupport is loaded. These tests weren't explicitly loading FeSupport, but instead relying on earlier tests in 'mvn test' to do so. Tested that, with this change, I'm able to run these tests in Eclipse and they pass. I also tested 'mvn test -DreuseForks=false' and verified that all tests now pass in this mode. Change-Id: I85286ba44f03526b487212a8ddadf71a9f427555 Reviewed-on: http://gerrit.cloudera.org:8080/10570 Reviewed-by: Vuk Ercegovac Tested-by: Impala Public Jenkins Reviewed-on: http://gerrit.cloudera.org:8080/10620 Reviewed-by: Tim Armstrong Tested-by: Tim Armstrong --- M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/catalog/HdfsStorageDescriptorTest.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java 4 files changed, 14 insertions(+), 3 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10620 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: merged Gerrit-Change-Id: I85286ba44f03526b487212a8ddadf71a9f427555 Gerrit-Change-Number: 10620 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR](2.x) IMPALA-7101 addendum: loosen exception check on 2.x
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10623 ) Change subject: IMPALA-7101 addendum: loosen exception check on 2.x .. IMPALA-7101 addendum: loosen exception check on 2.x This works around IMPALA-7133 by accepting the bogus message in the test. We can revert this change once IMPALA-7133 is fixed. Change-Id: Id94c658223a4e5f6b12aa62120f20136b1065081 Reviewed-on: http://gerrit.cloudera.org:8080/10623 Reviewed-by: Dan Hecht Tested-by: Impala Public Jenkins --- M tests/query_test/test_cancellation.py 1 file changed, 2 insertions(+), 1 deletion(-) Approvals: Dan Hecht: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: merged Gerrit-Change-Id: Id94c658223a4e5f6b12aa62120f20136b1065081 Gerrit-Change-Number: 10623 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR](2.x) Guard S3 access check with variable
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10616 ) Change subject: Guard S3 access check with variable .. Guard S3 access check with variable We've recently seen automated build failures where multiple inclusions of impala-config.sh led to the S3 access check being performed multiple times. This would trigger rate limiting on S3's side, which then caused the jobs to fail. This change adds a guard to only perform the check once, even when including impala-config.sh multiple times. To test this I included impala-config.sh multiple times on my local shell and observed that the check was only performed once. Change-Id: I1df16b2d037afafb7748f52b64732f680883f16c Reviewed-on: http://gerrit.cloudera.org:8080/10607 Reviewed-by: Lars Volker Tested-by: Impala Public Jenkins Reviewed-on: http://gerrit.cloudera.org:8080/10616 Tested-by: Tim Armstrong --- M bin/impala-config.sh 1 file changed, 11 insertions(+), 4 deletions(-) Approvals: Lars Volker: Looks good to me, approved Tim Armstrong: Verified -- To view, visit http://gerrit.cloudera.org:8080/10616 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: merged Gerrit-Change-Id: I1df16b2d037afafb7748f52b64732f680883f16c Gerrit-Change-Number: 10616 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR](2.x) IMPALA-7012: Fix NPE when parsing unexpected tokens
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10617 ) Change subject: IMPALA-7012: Fix NPE when parsing unexpected tokens .. IMPALA-7012: Fix NPE when parsing unexpected tokens Currently some token are not added to tokenIdMap in the sql scanner and an NPE will be thrown if such are parsed as unexpected tokens. This patch fixes this bug and adds a case to ParserTest. Change-Id: I9c846fdfb22ba37bfc3b1985b9a044014ab58968 Reviewed-on: http://gerrit.cloudera.org:8080/10512 Reviewed-by: Tianyi Wang Tested-by: Impala Public Jenkins Reviewed-on: http://gerrit.cloudera.org:8080/10617 Reviewed-by: Tim Armstrong Tested-by: Tim Armstrong --- M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java 3 files changed, 36 insertions(+), 21 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: merged Gerrit-Change-Id: I9c846fdfb22ba37bfc3b1985b9a044014ab58968 Gerrit-Change-Number: 10617 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR](2.x) Add missing namespace qualifiers
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10618 ) Change subject: Add missing namespace qualifiers .. Add missing namespace qualifiers As part of an upcoming rebase of the KRPC code, this change adds missing namespace qualifiers in some places. Change-Id: I3ef8f5c8f05100418dec90c2cd2e2b3dcd827c27 Reviewed-on: http://gerrit.cloudera.org:8080/10608 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins Reviewed-on: http://gerrit.cloudera.org:8080/10618 Reviewed-by: Lars Volker Tested-by: Tim Armstrong --- M be/src/exec/kudu-util.h M be/src/runtime/descriptors.h 2 files changed, 2 insertions(+), 2 deletions(-) Approvals: Lars Volker: Looks good to me, approved Tim Armstrong: Verified -- To view, visit http://gerrit.cloudera.org:8080/10618 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: merged Gerrit-Change-Id: I3ef8f5c8f05100418dec90c2cd2e2b3dcd827c27 Gerrit-Change-Number: 10618 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR](2.x) Add missing namespace qualifiers
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10618 ) Change subject: Add missing namespace qualifiers .. Patch Set 1: Verified+1 Verified with https://gerrit.cloudera.org/#/c/10623/ -- To view, visit http://gerrit.cloudera.org:8080/10618 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I3ef8f5c8f05100418dec90c2cd2e2b3dcd827c27 Gerrit-Change-Number: 10618 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 00:23:32 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7105. Ensure fe tests pass when running standalone
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10620 ) Change subject: IMPALA-7105. Ensure fe tests pass when running standalone .. Patch Set 1: Verified+1 Verified with https://gerrit.cloudera.org/#/c/10623/ -- To view, visit http://gerrit.cloudera.org:8080/10620 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I85286ba44f03526b487212a8ddadf71a9f427555 Gerrit-Change-Number: 10620 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 07 Jun 2018 00:23:23 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Fix Indents from IMPALA-4970
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10622 ) Change subject: Fix Indents from IMPALA-4970 .. Patch Set 1: Verified+1 Verified with https://gerrit.cloudera.org/#/c/10623/ -- To view, visit http://gerrit.cloudera.org:8080/10622 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I4ea53e0780cca9f791808c2a73c7f72ba27c2159 Gerrit-Change-Number: 10622 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Rahul Shivu Mahadev Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 00:23:14 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7101: Fix race between Fetch and Close RPCs that can lead to hang
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10619 ) Change subject: IMPALA-7101: Fix race between Fetch and Close RPCs that can lead to hang .. Patch Set 1: Verified+1 Verified with https://gerrit.cloudera.org/#/c/10623/ -- To view, visit http://gerrit.cloudera.org:8080/10619 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I7c147550f86d81b818ecbdd34cf2919ced7ff8c5 Gerrit-Change-Number: 10619 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 00:23:27 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-6956: deflake and logging for query expiration test
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10621 ) Change subject: IMPALA-6956: deflake and logging for query_expiration test .. Patch Set 1: Verified+1 Verified with https://gerrit.cloudera.org/#/c/10623/ -- To view, visit http://gerrit.cloudera.org:8080/10621 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I01a8762d28ad920b9ec8a0b1b82469618c66768f Gerrit-Change-Number: 10621 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 07 Jun 2018 00:23:18 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7012: Fix NPE when parsing unexpected tokens
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10617 ) Change subject: IMPALA-7012: Fix NPE when parsing unexpected tokens .. Patch Set 1: Verified+1 Verified with https://gerrit.cloudera.org/#/c/10623/ -- To view, visit http://gerrit.cloudera.org:8080/10617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I9c846fdfb22ba37bfc3b1985b9a044014ab58968 Gerrit-Change-Number: 10617 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 00:23:37 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7101 addendum: loosen exception check on 2.x
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10623 ) Change subject: IMPALA-7101 addendum: loosen exception check on 2.x .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: Id94c658223a4e5f6b12aa62120f20136b1065081 Gerrit-Change-Number: 10623 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 07 Jun 2018 00:19:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7132: Filter out useless output from run clang tidy.sh
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/10615 ) Change subject: IMPALA-7132: Filter out useless output from run_clang_tidy.sh .. Patch Set 1: Just realized that I had patched run-clang-tidy.py in my env to further reduce its output, so there is still some noise that would be nice to remove. This is in the right direction, but I'm going to see if I can filter out that other output. -- To view, visit http://gerrit.cloudera.org:8080/10615 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I63c46a7d57295eba38fac8ab49c7a15d2802df1d Gerrit-Change-Number: 10615 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 07 Jun 2018 00:18:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5552: Add support for authorized proxy groups
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/10510 )
Change subject: IMPALA-5552: Add support for authorized proxy groups
..
Patch Set 13:
(3 comments)
LGTM once my comments are addressed.
http://gerrit.cloudera.org:8080/#/c/10510/13/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/10510/13/be/src/service/impala-server.cc@1394
PS13, Line 1394: if (groups.find("*") != groups.end()) {
: return Status::OK();
: }
nit: single line, no braces (same below)
http://gerrit.cloudera.org:8080/#/c/10510/13/be/src/service/impala-server.cc@1406
PS13, Line 1406: end
nit: PrettyPrinter::Print(MonotinicMillis() - start, TUnit::TIME_MS)
remove L1404
http://gerrit.cloudera.org:8080/#/c/10510/11/tests/authorization/test_authorization.py
File tests/authorization/test_authorization.py:
http://gerrit.cloudera.org:8080/#/c/10510/11/tests/authorization/test_authorization.py@224
PS11, Line 224: er is 'hue'
> Yes, for this particular test, the code path to getHadoopGroups will not be
I see. That coverage could be achieved by modifying the args for the test
test_group_impersonation (above) by adding
--authorized_proxy_user_config=hue=something_that_is_never_used
Can we do that instead?
--
To view, visit http://gerrit.cloudera.org:8080/10510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb
Gerrit-Change-Number: 10510
Gerrit-PatchSet: 13
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Greg Rahn
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Sailesh Mukil
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Thu, 07 Jun 2018 00:05:57 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7132: Filter out useless output from run clang tidy.sh
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/10615 )
Change subject: IMPALA-7132: Filter out useless output from run_clang_tidy.sh
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10615/1/bin/run_clang_tidy.sh
File bin/run_clang_tidy.sh:
http://gerrit.cloudera.org:8080/#/c/10615/1/bin/run_clang_tidy.sh@57
PS1, Line 57: run-clang-tidy.py -header-filter "${PIPE_DIRS%?}" -j"${CORES}"
${DIRS} 2> /dev/null
> Would -quiet work?
Unfortunately, -quiet only handles a few types of output. Most of the noise is
from output like this:
"""
Fix conflicts with existing fix! The new insertion has the same insert location
as an existing replacement.
New replacement: /home/joe/view1/Impala/be/src/util/string-parser.h:
11416:+0:"break; "
Existing replacement: /home/joe/view1/Impala/be/src/util/string-parser.h:
11416:+0:"FALLTHROUGH_INTENDED; "
"""
Apparently, -quiet doesn't handle that.
--
To view, visit http://gerrit.cloudera.org:8080/10615
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63c46a7d57295eba38fac8ab49c7a15d2802df1d
Gerrit-Change-Number: 10615
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple
Gerrit-Reviewer: Joe McDonnell
Gerrit-Comment-Date: Thu, 07 Jun 2018 00:05:12 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7132: Filter out useless output from run clang tidy.sh
Jim Apple has posted comments on this change. (
http://gerrit.cloudera.org:8080/10615 )
Change subject: IMPALA-7132: Filter out useless output from run_clang_tidy.sh
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10615/1/bin/run_clang_tidy.sh
File bin/run_clang_tidy.sh:
http://gerrit.cloudera.org:8080/#/c/10615/1/bin/run_clang_tidy.sh@57
PS1, Line 57: run-clang-tidy.py -header-filter "${PIPE_DIRS%?}" -j"${CORES}"
${DIRS} 2> /dev/null
Would -quiet work?
--
To view, visit http://gerrit.cloudera.org:8080/10615
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I63c46a7d57295eba38fac8ab49c7a15d2802df1d
Gerrit-Change-Number: 10615
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple
Gerrit-Comment-Date: Wed, 06 Jun 2018 23:43:25 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7137. Support configuring Frontend to use LocalCatalog
Hello Vuk Ercegovac, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10629 to review the following change. Change subject: IMPALA-7137. Support configuring Frontend to use LocalCatalog .. IMPALA-7137. Support configuring Frontend to use LocalCatalog This adds a new flag -use_local_catalog which is passed through to the frontend and causes it to use LocalCatalog instead of ImpaladCatalog. Additionally, when this flag is configured, the impalad does not subscribe to catalog topic updates from the statestore. The patch is slightly more complex than simply picking which class to instantiate, because the lifecycle is designed a bit differently between the two implementations: - LocalCatalog is instantiated once per query/request. - ImpaladCatalog is instantiated once and stateful across queries, except when a full catalog update is received. In order to abstract this difference in lifecycle from the frontend, I introduced a new FeCatalogManager class with different implementations for the two lifecycles. I also had to add a simple test implementation since some tests rely on directly injecting a Catalog implementation into the Frontend. This patch also includes a few small changes to the local catalog implementation objects to enable the impalad to start and accept connections. With this patch, I was able to manually test as follows: I started just the statestore and the impalad in the new mode: - ./bin/start-statestored.sh - ./bin/start-impalad.sh --use_local_catalog I connected with impala-shell as usual and was able to run the most simple queries: - SHOW DATABASES; - USE functional; - SHOW TABLES; All other functionality results in error messages due to the various TODOs in the current skeleton implementation. Change-Id: I8c9665bd031d23608740b23eef301970af9aa764 --- M be/src/runtime/exec-env.cc M be/src/service/impala-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java A fe/src/main/java/org/apache/impala/service/FeCatalogManager.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java 11 files changed, 232 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/10629/1 -- To view, visit http://gerrit.cloudera.org:8080/10629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8c9665bd031d23608740b23eef301970af9aa764 Gerrit-Change-Number: 10629 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-3040: Force a rescan of HDFS caching stats
Tianyi Wang has abandoned this change. ( http://gerrit.cloudera.org:8080/10625 ) Change subject: IMPALA-3040: Force a rescan of HDFS caching stats .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/10625 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I2b4f3241a784d6c85306411aa0c8e7527feaef86 Gerrit-Change-Number: 10625 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang
[Impala-ASF-CR] IMPALA-7115: set a default THREAD RESERVATION LIMIT value
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10628 Change subject: IMPALA-7115: set a default THREAD_RESERVATION_LIMIT value .. IMPALA-7115: set a default THREAD_RESERVATION_LIMIT value The value is chosen to allow only queries that have a reasonable chance of succeeding, albeit with poor performance because of the high number of threads. Testing: Added a test to make sure that the default value rejects a large query. Change-Id: I31d3fa3f6305c360922649dba53a9026c9563384 --- M common/thrift/ImpalaInternalService.thrift M testdata/workloads/functional-query/queries/QueryTest/thread-limits.test 2 files changed, 3,010 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/10628/1 -- To view, visit http://gerrit.cloudera.org:8080/10628 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I31d3fa3f6305c360922649dba53a9026c9563384 Gerrit-Change-Number: 10628 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-5552: Add support for authorized proxy groups
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/10510 )
Change subject: IMPALA-5552: Add support for authorized proxy groups
..
Patch Set 13:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/10510/11/be/src/service/frontend.h
File be/src/service/frontend.h:
http://gerrit.cloudera.org:8080/#/c/10510/11/be/src/service/frontend.h@156
PS11, Line 156: /// Returns (in the output parameter) the list of groups for
the given user.
> Comment
Done
http://gerrit.cloudera.org:8080/#/c/10510/8/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/10510/8/be/src/service/impala-server.cc@401
PS8, Line 401: const string& authorized_proxy_config_delimiter,
: AuthorizedProxyMap* authorized_proxy_config_map
> I'm not the expert here but I don't think we worry about the nullability as
Done
http://gerrit.cloudera.org:8080/#/c/10510/8/be/src/service/impala-server.cc@1389
PS8, Line 1389: () > 0) {
> Yep, let's attach a timer to this and log it.
Done
http://gerrit.cloudera.org:8080/#/c/10510/11/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/10510/11/be/src/service/impala-server.cc@1405
PS11, Line 1405: OG_QUERY << "Getting Hadoop groups took " <<
> Move out of the loop?
Done
http://gerrit.cloudera.org:8080/#/c/10510/11/be/src/service/impala-server.cc@1406
PS11, Line 1406: rettyPrinter::Print(end - start,
> Can we move it before L1394? If this passes, we don't need to do a group lo
Yeah good idea. Done.
http://gerrit.cloudera.org:8080/#/c/10510/11/common/thrift/Frontend.thrift
File common/thrift/Frontend.thrift:
http://gerrit.cloudera.org:8080/#/c/10510/11/common/thrift/Frontend.thrift@834
PS11, Line 834: A flag to indicate wh
> update
Done
http://gerrit.cloudera.org:8080/#/c/10510/11/fe/src/main/java/org/apache/impala/service/JniFrontend.java
File fe/src/main/java/org/apache/impala/service/JniFrontend.java:
http://gerrit.cloudera.org:8080/#/c/10510/11/fe/src/main/java/org/apache/impala/service/JniFrontend.java@627
PS11, Line 627: TGetHadoopGroupsRequest request =
> Do we need to do this everytime?
Oops. That was unintentional. Done.
http://gerrit.cloudera.org:8080/#/c/10510/11/fe/src/main/java/org/apache/impala/service/JniFrontend.java@653
PS11, Line 653:* provider implementation.
> Returns.
Done
http://gerrit.cloudera.org:8080/#/c/10510/11/tests/authorization/test_authorization.py
File tests/authorization/test_authorization.py:
http://gerrit.cloudera.org:8080/#/c/10510/11/tests/authorization/test_authorization.py@200
PS11, Line 200: @pytest.mark.execute_serially
: @CustomClusterTestSuite.with_args("--server_name=server1\
: --authorization_policy_file=%s\
: --authorized_proxy_user_config=hue=%s\
: --authorized_proxy_group_config=hue=%s\
: --abort_on_failed_audit_event=false\
: --audit_event_log_dir=%s" % (AUTH_POLICY_FILE,
: getuser(),
: ','.join(get_groups()),
: AUDIT_LOG_DIR))
: def test_user_and_group_impersonation(self):
: """End-to-end user and grou
> I mean "custom delimiters are being parsed correctly."
Will remove. Done.
http://gerrit.cloudera.org:8080/#/c/10510/11/tests/authorization/test_authorization.py@224
PS11, Line 224: er is 'hue'
> We don't seem to be delegating it to a group user in __test_impersonation()
Yes, for this particular test, the code path to getHadoopGroups will not be
called. The goal is to make sure having the two authorized user and groups
configs still work. I can remove this if you think this test doesn't make sense.
--
To view, visit http://gerrit.cloudera.org:8080/10510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb
Gerrit-Change-Number: 10510
Gerrit-PatchSet: 13
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Greg Rahn
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Sailesh Mukil
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Wed, 06 Jun 2018 22:56:56 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7082: Show human readable size in query backend page
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10523 ) Change subject: IMPALA-7082: Show human readable size in query backend page .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I04afb4091bb8b6bc9dedfeceaf9284a8c65b16a1 Gerrit-Change-Number: 10523 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 06 Jun 2018 22:57:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7082: Show human readable size in query backend page
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10523 ) Change subject: IMPALA-7082: Show human readable size in query backend page .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2613/ -- To view, visit http://gerrit.cloudera.org:8080/10523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I04afb4091bb8b6bc9dedfeceaf9284a8c65b16a1 Gerrit-Change-Number: 10523 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 06 Jun 2018 22:57:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5552: Add support for authorized proxy groups
Fredy Wijaya has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/10510 ) Change subject: IMPALA-5552: Add support for authorized proxy groups .. IMPALA-5552: Add support for authorized proxy groups The patch adds support for mapping of users to a list of proxy groups. The following flags are added in impalad: - authorized_proxy_group_config - authorized_proxy_group_config_delimiter Example: --authorized_proxy_group_config=hue=group1,group2;user1=* This feature is not supported on Shell-based Hadoop groups mapping providers. Testing: - Added FE unit test to check for groups mapping provider - Added BE unit test for the parsing logic - Added a new test in test_authorization.py - Ran all end-to-end test_authorization.py Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb --- M be/src/service/CMakeLists.txt M be/src/service/frontend.cc M be/src/service/frontend.h A be/src/service/impala-server-test.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/backend-gflag-util.cc M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/test/java/org/apache/impala/service/JniFrontendTest.java M tests/authorization/test_authorization.py 11 files changed, 396 insertions(+), 39 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/10510/13 -- To view, visit http://gerrit.cloudera.org:8080/10510 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb Gerrit-Change-Number: 10510 Gerrit-PatchSet: 13 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-2195: Improper handling of comments in queries
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/9933 ) Change subject: IMPALA-2195: Improper handling of comments in queries .. Patch Set 13: Code-Review+1 Carry + 1, rebased and also tested on both Python 2.6 and 2.7. Anyone willing to review this? -- To view, visit http://gerrit.cloudera.org:8080/9933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ac7cb5a30e6dda73ebe761d9f0eb9ba038e14a7 Gerrit-Change-Number: 9933 Gerrit-PatchSet: 13 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Taras Bobrovytsky Gerrit-Comment-Date: Wed, 06 Jun 2018 22:30:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2195: Improper handling of comments in queries
Fredy Wijaya has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/9933 ) Change subject: IMPALA-2195: Improper handling of comments in queries .. IMPALA-2195: Improper handling of comments in queries This patch fixes an issue where parseline is unable to deduce the correct command when a statement has a leading comment. Before: > -- comment > insert into table t values(100); Fetched 1 row(s) in 0.01s After: > -- comment > insert into table t values(100); Modified 1 row(s) in 0.01s Before (FE syntax error): > /*comment*/ help; After (show help correctly): > /*comment*/ help; Testing: - Added shell tests - Ran end-to-end shell tests on Python 2.6 and Python 2.7 Change-Id: I7ac7cb5a30e6dda73ebe761d9f0eb9ba038e14a7 --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 157 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/9933/13 -- To view, visit http://gerrit.cloudera.org:8080/9933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7ac7cb5a30e6dda73ebe761d9f0eb9ba038e14a7 Gerrit-Change-Number: 9933 Gerrit-PatchSet: 13 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Taras Bobrovytsky
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Hello Vuk Ercegovac, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10611 to look at the new patch set (#2). Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition This refactors out interfaces in the frontend for the interaction between the analysis/planning code and the classes that implement these catalog objects. This takes care of the most commonly used objects but defers some others (e.g. functions, cache pools, data sources, etc) to follow-on patches. There are a few spots remaining in the frontend that still downcast to implementation classes, particularly where the frontend actually makes modifications to catalog objects in-place. I left TODOs in those spots and will come back later as necessary. Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 --- M fe/src/main/java/org/apache/impala/analysis/AlterTableAddDropRangePartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAddPartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAddReplaceColsStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAlterColStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableDropColStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableDropPartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewRenameStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetFileFormatStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetRowFormatStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSortByStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateFunctionStmtBase.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/DropDbStmt.java M fe/src/main/java/org/apache/impala/analysis/DropFunctionStmt.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/PartitionDef.java M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java M fe/src/main/java/org/apache/impala/analysis/PartitionSpecBase.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowCreateFunctionStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/main/java/org/apache/impala/analysis/WithClause.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java A fe/src/main/java/org/apache/impala/catalog/FeCatalog.java A fe/src/main/java/org/apache/impala/catalog/FeDb.java A
[Impala-ASF-CR] IMPALA-7135. Skeleton implementation of LocalCatalog
Hello Vuk Ercegovac, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10627 to review the following change. Change subject: IMPALA-7135. Skeleton implementation of LocalCatalog .. IMPALA-7135. Skeleton implementation of LocalCatalog This adds some of the high level classes for implementing the local catalog: - LocalCatalog is the top level implementation. The plan is to instantiate this once per query, so that no thread safety is required. - It loads metadata from a MetaProvider interface. The current implementation fetches directly from HMS and provides no caching. A future subtask will add a CachingMetaProvider implementation. Separating out caching will make it easier to experiment with different policies or storage mechanisms. - It instantiates LocalDb and LocalTable objects to implement FeDb and FeTable. These are mostly stubbed out except for the most basic functionality. Functionality will be filled in incrementally in further patches. Since it's not yet possible to hook this up to most of the existing tests, a very simple new unit test is included to cover the bits of functionality that are not stubbed out. I didn't concentrate on too much test coverage here, since once we've implemented more functionality we can switch over all of the existing tests to get coverage of the new implementation. Change-Id: Iab653371188b21c72f50ee1ec4e94950aa6fb9ee --- M fe/src/main/java/org/apache/impala/catalog/Catalog.java A fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java A fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java A fe/src/main/java/org/apache/impala/catalog/local/LocalCatalogException.java A fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java A fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java A fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java A fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 8 files changed, 765 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/10627/1 -- To view, visit http://gerrit.cloudera.org:8080/10627 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iab653371188b21c72f50ee1ec4e94950aa6fb9ee Gerrit-Change-Number: 10627 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7128 (part 2): add an interface for data sources
Hello Vuk Ercegovac, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10626 to review the following change. Change subject: IMPALA-7128 (part 2): add an interface for data sources .. IMPALA-7128 (part 2): add an interface for data sources This changes most of the usage of DataSource and DataSourceTable to use interfaces instead of implementation classes. There are still various usages of the implementation for functionality like creating and dropping data sources. We'll address those as part of IMPALA-7131 at a later date. Change-Id: Ibe704197dc2ad7c09b8340865f17567096aa630e --- M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/catalog/DataSource.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java A fe/src/main/java/org/apache/impala/catalog/FeDataSource.java A fe/src/main/java/org/apache/impala/catalog/FeDataSourceTable.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java 13 files changed, 99 insertions(+), 26 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/10626/1 -- To view, visit http://gerrit.cloudera.org:8080/10626 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ibe704197dc2ad7c09b8340865f17567096aa630e Gerrit-Change-Number: 10626 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR](2.x) IMPALA-7012: Fix NPE when parsing unexpected tokens
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10617 ) Change subject: IMPALA-7012: Fix NPE when parsing unexpected tokens .. Patch Set 1: I'm running the precommit job on the head of all these commits. When that passes, I'll submit all of them together: https://gerrit.cloudera.org/#/c/10623/ -- To view, visit http://gerrit.cloudera.org:8080/10617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I9c846fdfb22ba37bfc3b1985b9a044014ab58968 Gerrit-Change-Number: 10617 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 06 Jun 2018 22:27:45 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7012: Fix NPE when parsing unexpected tokens
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10617 ) Change subject: IMPALA-7012: Fix NPE when parsing unexpected tokens .. Patch Set 1: Sorry I got it wrong. We should be able to merge it if the parser test passed in the build you mentioned. -- To view, visit http://gerrit.cloudera.org:8080/10617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I9c846fdfb22ba37bfc3b1985b9a044014ab58968 Gerrit-Change-Number: 10617 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 06 Jun 2018 22:27:17 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7012: Fix NPE when parsing unexpected tokens
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10617 ) Change subject: IMPALA-7012: Fix NPE when parsing unexpected tokens .. Patch Set 1: So we need to manually verify it, right? I will build it and test the parser part. -- To view, visit http://gerrit.cloudera.org:8080/10617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I9c846fdfb22ba37bfc3b1985b9a044014ab58968 Gerrit-Change-Number: 10617 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 06 Jun 2018 22:22:57 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7012: Fix NPE when parsing unexpected tokens
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10617 ) Change subject: IMPALA-7012: Fix NPE when parsing unexpected tokens .. Patch Set 1: Aren't clean picks automatic? -- To view, visit http://gerrit.cloudera.org:8080/10617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I9c846fdfb22ba37bfc3b1985b9a044014ab58968 Gerrit-Change-Number: 10617 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 06 Jun 2018 22:01:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5380: [DOCS] Added additional filesystems supported in URI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10551 ) Change subject: IMPALA-5380: [DOCS] Added additional filesystems supported in URI .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10551 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d57efd564fb5779539a88d7b209cee93d313c6 Gerrit-Change-Number: 10551 Gerrit-PatchSet: 6 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Wed, 06 Jun 2018 21:30:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6714: [DOCS] ORC file format support
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10525 ) Change subject: IMPALA-6714: [DOCS] ORC file format support .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10525 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib1ee23ed844653c274babdce5a332dbe5c79b630 Gerrit-Change-Number: 10525 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 06 Jun 2018 21:22:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7108: IllegalStateException hit during CardinalityCheckNode.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10605 ) Change subject: IMPALA-7108: IllegalStateException hit during CardinalityCheckNode. .. Patch Set 2: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/10605/2/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java: http://gerrit.cloudera.org:8080/#/c/10605/2/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@1082 PS2, Line 1082:* Check if the stmt returns a single row. This can happen Should this be "returns at most one row"? -- To view, visit http://gerrit.cloudera.org:8080/10605 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82a7a3fe26db3e12131c030c4ad055a9c4955407 Gerrit-Change-Number: 10605 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vincent Tran Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 06 Jun 2018 21:18:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR](2.x) IMPALA-6956: deflake and logging for query expiration test
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10621 ) Change subject: IMPALA-6956: deflake and logging for query_expiration test .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10621 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I01a8762d28ad920b9ec8a0b1b82469618c66768f Gerrit-Change-Number: 10621 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 06 Jun 2018 21:15:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5380: [DOCS] Added additional filesystems supported in URI
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10551 ) Change subject: IMPALA-5380: [DOCS] Added additional filesystems supported in URI .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/310/ -- To view, visit http://gerrit.cloudera.org:8080/10551 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d57efd564fb5779539a88d7b209cee93d313c6 Gerrit-Change-Number: 10551 Gerrit-PatchSet: 6 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Wed, 06 Jun 2018 21:06:46 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-6956: deflake and logging for query expiration test
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10621 ) Change subject: IMPALA-6956: deflake and logging for query_expiration test .. Patch Set 1: Clean pick -- To view, visit http://gerrit.cloudera.org:8080/10621 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I01a8762d28ad920b9ec8a0b1b82469618c66768f Gerrit-Change-Number: 10621 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 06 Jun 2018 21:05:56 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Fix Indents from IMPALA-4970
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10622 ) Change subject: Fix Indents from IMPALA-4970 .. Patch Set 1: Code-Review+2 Clean pick -- To view, visit http://gerrit.cloudera.org:8080/10622 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I4ea53e0780cca9f791808c2a73c7f72ba27c2159 Gerrit-Change-Number: 10622 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Rahul Shivu Mahadev Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 06 Jun 2018 21:06:01 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7012: Fix NPE when parsing unexpected tokens
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10617 ) Change subject: IMPALA-7012: Fix NPE when parsing unexpected tokens .. Patch Set 1: Code-Review+2 Clean pick -- To view, visit http://gerrit.cloudera.org:8080/10617 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I9c846fdfb22ba37bfc3b1985b9a044014ab58968 Gerrit-Change-Number: 10617 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 06 Jun 2018 21:05:36 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7101: Fix race between Fetch and Close RPCs that can lead to hang
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10619 ) Change subject: IMPALA-7101: Fix race between Fetch and Close RPCs that can lead to hang .. Patch Set 1: Code-Review+2 Clean pick -- To view, visit http://gerrit.cloudera.org:8080/10619 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I7c147550f86d81b818ecbdd34cf2919ced7ff8c5 Gerrit-Change-Number: 10619 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 06 Jun 2018 21:05:43 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7105. Ensure fe tests pass when running standalone
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10620 ) Change subject: IMPALA-7105. Ensure fe tests pass when running standalone .. Patch Set 1: Code-Review+2 Clean pick -- To view, visit http://gerrit.cloudera.org:8080/10620 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I85286ba44f03526b487212a8ddadf71a9f427555 Gerrit-Change-Number: 10620 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 06 Jun 2018 21:05:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7078: Part 2: reduce queue size based on num scanner threads
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10573 ) Change subject: IMPALA-7078: Part 2: reduce queue size based on num_scanner_threads .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2612/ -- To view, visit http://gerrit.cloudera.org:8080/10573 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c0bd14e6bcd9fc1655e344a5307ea0eb4600e8b Gerrit-Change-Number: 10573 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 06 Jun 2018 21:01:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7132: Filter out useless output from run clang tidy.sh
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10615 ) Change subject: IMPALA-7132: Filter out useless output from run_clang_tidy.sh .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2611/ -- To view, visit http://gerrit.cloudera.org:8080/10615 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I63c46a7d57295eba38fac8ab49c7a15d2802df1d Gerrit-Change-Number: 10615 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 06 Jun 2018 20:56:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7078: Part 1: improve memory consumption of wide Avro scans
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10550 ) Change subject: IMPALA-7078: Part 1: improve memory consumption of wide Avro scans .. Patch Set 11: (3 comments) http://gerrit.cloudera.org:8080/#/c/10550/11/be/src/exec/hdfs-scan-node.cc File be/src/exec/hdfs-scan-node.cc: http://gerrit.cloudera.org:8080/#/c/10550/11/be/src/exec/hdfs-scan-node.cc@186 PS11, Line 186: AVERAGE_SCANNER_THREAD_CONCURRENCY, _scanner_thread_counter_); > would it be better to move this above line 183 so that other code doesn't h It's so the average time doesn't include the time when the scan node is unopened. E.g. if the scan is opened late in the query lifecycle (maybe it's on the right side of a join or something), we don't want to include all the time beforehand when no scanner threads are expected to be running. http://gerrit.cloudera.org:8080/#/c/10550/11/be/src/exec/hdfs-scan-node.cc@192 PS11, Line 192: > nit: Intentional newline? Not really but I move the code anyway in part 2 of the patch, so I won't bother changing it. http://gerrit.cloudera.org:8080/#/c/10550/11/be/src/exec/scan-node.h File be/src/exec/scan-node.h: http://gerrit.cloudera.org:8080/#/c/10550/11/be/src/exec/scan-node.h@191 PS11, Line 191: /// Cumulative number of scanner threads created during the scan. Some may be created : /// and then destroyed, so this can exceed the peak number of threads. > This is a great observability addition to this counter. But to someone who Honestly, I think this counter isn't very useful, but I didn't necessarily want to remove it as part of this change. I figured we should at least document it. -- To view, visit http://gerrit.cloudera.org:8080/10550 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iebd2600b4784fd19696c9b92eefb7d7e9db0c80b Gerrit-Change-Number: 10550 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vincent Tran Gerrit-Comment-Date: Wed, 06 Jun 2018 20:55:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR](2.x) Add missing namespace qualifiers
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/10618 ) Change subject: Add missing namespace qualifiers .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10618 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I3ef8f5c8f05100418dec90c2cd2e2b3dcd827c27 Gerrit-Change-Number: 10618 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Wed, 06 Jun 2018 20:52:54 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Guard S3 access check with variable
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/10616 ) Change subject: Guard S3 access check with variable .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10616 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I1df16b2d037afafb7748f52b64732f680883f16c Gerrit-Change-Number: 10616 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Wed, 06 Jun 2018 20:51:07 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-6956: deflake and logging for query expiration test
Hello Impala Public Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10621 to review the following change. Change subject: IMPALA-6956: deflake and logging for query_expiration test .. IMPALA-6956: deflake and logging for query_expiration test There was a recent flake where the number of in-flight queries that were executing differed from the expected number. The reason for the false negative is that one of the queries expired before the check for in-flight queries: it took too long to issue the queries. This change modifies the timeout for the expired query to not expire so quickly. Additional logging is added to the check for in-flight queries so that we can distinguish the case of too few queries vs. queries that have the wrong state. Change-Id: I01a8762d28ad920b9ec8a0b1b82469618c66768f Reviewed-on: http://gerrit.cloudera.org:8080/10602 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M tests/custom_cluster/test_query_expiration.py 1 file changed, 9 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/10621/1 -- To view, visit http://gerrit.cloudera.org:8080/10621 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: I01a8762d28ad920b9ec8a0b1b82469618c66768f Gerrit-Change-Number: 10621 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR](2.x) IMPALA-7101: Fix race between Fetch and Close RPCs that can lead to hang
Hello Impala Public Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10619 to review the following change. Change subject: IMPALA-7101: Fix race between Fetch and Close RPCs that can lead to hang .. IMPALA-7101: Fix race between Fetch and Close RPCs that can lead to hang If we hit EOS, we'll wait for all the backends to report status (to try to get a complete profile). But if the query is closed after this point, then we can get stuck waiting since once the query is closed, ImpalaServer won't know about this coordinator and so it will stop forwarding on the ReportStatus RPCs. The real fix for this is IMPALA-6984, but in the mean time, add another special case for this JIRA (see the other TODO IMPALA-6984 in coordinator.cc). The cancellation test only finds this race once in a while (several hours) indirectly in a COMPUTE STATS query because the ChildQueryExecutor will do a CloseOperation() while the execution thread is inside Fetch(). To make this more reproducible, modify the cancellation test to allow the close and fetch rpcs to execute concurrently (don't join the test's fetch thread until after close). This makes the race reproducible in a few iterations and a few minutes. Testing: - Loop test_cancellation.py Change-Id: I7c147550f86d81b818ecbdd34cf2919ced7ff8c5 Reviewed-on: http://gerrit.cloudera.org:8080/10601 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M be/src/runtime/coordinator.cc M tests/query_test/test_cancellation.py 2 files changed, 47 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/10619/1 -- To view, visit http://gerrit.cloudera.org:8080/10619 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: I7c147550f86d81b818ecbdd34cf2919ced7ff8c5 Gerrit-Change-Number: 10619 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR](2.x) IMPALA-7101 addendum: loosen exception check on 2.x
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10623 ) Change subject: IMPALA-7101 addendum: loosen exception check on 2.x .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2610/ -- To view, visit http://gerrit.cloudera.org:8080/10623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: Id94c658223a4e5f6b12aa62120f20136b1065081 Gerrit-Change-Number: 10623 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 06 Jun 2018 20:50:30 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Fix Indents from IMPALA-4970
Hello Sailesh Mukil, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10622 to review the following change. Change subject: Fix Indents from IMPALA-4970 .. Fix Indents from IMPALA-4970 fixing the mistake in indentation made previously Change-Id: I4ea53e0780cca9f791808c2a73c7f72ba27c2159 Reviewed-on: http://gerrit.cloudera.org:8080/10560 Reviewed-by: Sailesh Mukil Tested-by: Tim Armstrong --- M be/src/runtime/coordinator.cc 1 file changed, 4 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/10622/1 -- To view, visit http://gerrit.cloudera.org:8080/10622 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: I4ea53e0780cca9f791808c2a73c7f72ba27c2159 Gerrit-Change-Number: 10622 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Rahul Shivu Mahadev Gerrit-Reviewer: Sailesh Mukil
[Impala-ASF-CR](2.x) Add missing namespace qualifiers
Hello Impala Public Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10618 to review the following change. Change subject: Add missing namespace qualifiers .. Add missing namespace qualifiers As part of an upcoming rebase of the KRPC code, this change adds missing namespace qualifiers in some places. Change-Id: I3ef8f5c8f05100418dec90c2cd2e2b3dcd827c27 Reviewed-on: http://gerrit.cloudera.org:8080/10608 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M be/src/exec/kudu-util.h M be/src/runtime/descriptors.h 2 files changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/10618/1 -- To view, visit http://gerrit.cloudera.org:8080/10618 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: I3ef8f5c8f05100418dec90c2cd2e2b3dcd827c27 Gerrit-Change-Number: 10618 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR](2.x) IMPALA-7101 addendum: loosen exception check on 2.x
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10623 Change subject: IMPALA-7101 addendum: loosen exception check on 2.x .. IMPALA-7101 addendum: loosen exception check on 2.x This works around IMPALA-7133 by accepting the bogus message in the test. We can revert this change once IMPALA-7133 is fixed. Change-Id: Id94c658223a4e5f6b12aa62120f20136b1065081 --- M tests/query_test/test_cancellation.py 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/10623/1 -- To view, visit http://gerrit.cloudera.org:8080/10623 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: Id94c658223a4e5f6b12aa62120f20136b1065081 Gerrit-Change-Number: 10623 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR](2.x) Guard S3 access check with variable
Hello Lars Volker, Impala Public Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10616 to review the following change. Change subject: Guard S3 access check with variable .. Guard S3 access check with variable We've recently seen automated build failures where multiple inclusions of impala-config.sh led to the S3 access check being performed multiple times. This would trigger rate limiting on S3's side, which then caused the jobs to fail. This change adds a guard to only perform the check once, even when including impala-config.sh multiple times. To test this I included impala-config.sh multiple times on my local shell and observed that the check was only performed once. Change-Id: I1df16b2d037afafb7748f52b64732f680883f16c Reviewed-on: http://gerrit.cloudera.org:8080/10607 Reviewed-by: Lars Volker Tested-by: Impala Public Jenkins --- M bin/impala-config.sh 1 file changed, 11 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/10616/1 -- To view, visit http://gerrit.cloudera.org:8080/10616 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: I1df16b2d037afafb7748f52b64732f680883f16c Gerrit-Change-Number: 10616 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR](2.x) IMPALA-7105. Ensure fe tests pass when running standalone
Hello Vuk Ercegovac, Impala Public Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/10620 to review the following change. Change subject: IMPALA-7105. Ensure fe tests pass when running standalone .. IMPALA-7105. Ensure fe tests pass when running standalone CatalogTest and HdfsStorageDescriptorTest would fail when run individually because they relied on various initialization that only occurs when FeSupport is loaded. These tests weren't explicitly loading FeSupport, but instead relying on earlier tests in 'mvn test' to do so. Tested that, with this change, I'm able to run these tests in Eclipse and they pass. I also tested 'mvn test -DreuseForks=false' and verified that all tests now pass in this mode. Change-Id: I85286ba44f03526b487212a8ddadf71a9f427555 Reviewed-on: http://gerrit.cloudera.org:8080/10570 Reviewed-by: Vuk Ercegovac Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/service/FeSupport.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/catalog/HdfsStorageDescriptorTest.java M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java 4 files changed, 14 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/10620/1 -- To view, visit http://gerrit.cloudera.org:8080/10620 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: I85286ba44f03526b487212a8ddadf71a9f427555 Gerrit-Change-Number: 10620 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7078: Part 1: improve memory consumption of wide Avro scans
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10550 ) Change subject: IMPALA-7078: Part 1: improve memory consumption of wide Avro scans .. Patch Set 12: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10550 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iebd2600b4784fd19696c9b92eefb7d7e9db0c80b Gerrit-Change-Number: 10550 Gerrit-PatchSet: 12 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vincent Tran Gerrit-Comment-Date: Wed, 06 Jun 2018 20:40:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7078: Part 2: reduce queue size based on num scanner threads
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10573 ) Change subject: IMPALA-7078: Part 2: reduce queue size based on num_scanner_threads .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10573 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c0bd14e6bcd9fc1655e344a5307ea0eb4600e8b Gerrit-Change-Number: 10573 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 06 Jun 2018 20:40:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10611 ) Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 Gerrit-Change-Number: 10611 Gerrit-PatchSet: 1 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Wed, 06 Jun 2018 20:33:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7132: Filter out useless output from run clang tidy.sh
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10615 Change subject: IMPALA-7132: Filter out useless output from run_clang_tidy.sh .. IMPALA-7132: Filter out useless output from run_clang_tidy.sh Clang's run-clang-tidy.py script produces a large amount of output to stderr even when there are no warnings or errors. None of this output is useful. This ignores the stderr output from run-clang-tidy.py. Testing with a known clang tidy issue shows that warnings and errors are still in the output. Change-Id: I63c46a7d57295eba38fac8ab49c7a15d2802df1d --- M bin/run_clang_tidy.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/10615/1 -- To view, visit http://gerrit.cloudera.org:8080/10615 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I63c46a7d57295eba38fac8ab49c7a15d2802df1d Gerrit-Change-Number: 10615 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-6910/IMPALA-7070: Increase log level for HDFS S3 code
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10596 ) Change subject: IMPALA-6910/IMPALA-7070: Increase log level for HDFS S3 code .. IMPALA-6910/IMPALA-7070: Increase log level for HDFS S3 code This change increases the log level for the HDFS S3 code to DEBUG to get more information on both Jiras. Change-Id: If5f9650a322f9c987f5f4e322681ab290cf487ee Reviewed-on: http://gerrit.cloudera.org:8080/10596 Reviewed-by: Lars Volker Reviewed-by: Sailesh Mukil Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M fe/src/test/resources/log4j.properties.template 1 file changed, 3 insertions(+), 0 deletions(-) Approvals: Lars Volker: Looks good to me, but someone else must approve Sailesh Mukil: Looks good to me, but someone else must approve Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10596 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If5f9650a322f9c987f5f4e322681ab290cf487ee Gerrit-Change-Number: 10596 Gerrit-PatchSet: 3 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6910/IMPALA-7070: Increase log level for HDFS S3 code
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10596 ) Change subject: IMPALA-6910/IMPALA-7070: Increase log level for HDFS S3 code .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10596 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If5f9650a322f9c987f5f4e322681ab290cf487ee Gerrit-Change-Number: 10596 Gerrit-PatchSet: 2 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 06 Jun 2018 20:21:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5380: [DOCS] Added additional filesystems supported in URI
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/10551 ) Change subject: IMPALA-5380: [DOCS] Added additional filesystems supported in URI .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10551 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d57efd564fb5779539a88d7b209cee93d313c6 Gerrit-Change-Number: 10551 Gerrit-PatchSet: 6 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Wed, 06 Jun 2018 19:22:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7016: Implement ALTER DATABASE SET OWNER
Fredy Wijaya has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/10471 ) Change subject: IMPALA-7016: Implement ALTER DATABASE SET OWNER .. IMPALA-7016: Implement ALTER DATABASE SET OWNER Alter the database owner to either user or role. On database creation, the database owner will be set to the current user, which can be viewed via DESCRIBE DATABASE db command. Having an owner information allows implementing a feature where an owner can be given certain privileges automatically upon a database creation. The ALTER DATABASE SET OWNER will be a useful command for transferring ownership (a set of owner privileges) from the current owner to another owner. Syntax: ALTER DATABASE db SET OWNER USER user ALTER DATABASE db SET OWNER ROLE role Testing: - Added new front-end tests - Added new end-to-end tests Change-Id: Ie3b923021ebce5192d2d64784e7ddb952ba82bc3 --- M common/thrift/CatalogService.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup A fe/src/main/java/org/apache/impala/analysis/AlterDbSetOwnerStmt.java A fe/src/main/java/org/apache/impala/analysis/AlterDbStmt.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/Owner.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M tests/metadata/test_ddl.py M tests/metadata/test_ddl_base.py 15 files changed, 361 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/10471/8 -- To view, visit http://gerrit.cloudera.org:8080/10471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3b923021ebce5192d2d64784e7ddb952ba82bc3 Gerrit-Change-Number: 10471 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7016: Implement ALTER DATABASE SET OWNER
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/10471 )
Change subject: IMPALA-7016: Implement ALTER DATABASE SET OWNER
..
Patch Set 7:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/10471/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10471/5//COMMIT_MSG@9
PS5, Line 9: wner
> Is there some info that explains how the owner attribute will be used in Im
Done
http://gerrit.cloudera.org:8080/#/c/10471/3/common/thrift/JniCatalog.thrift
File common/thrift/JniCatalog.thrift:
http://gerrit.cloudera.org:8080/#/c/10471/3/common/thrift/JniCatalog.thrift@66
PS3, Line 66: SET_OWNER
> is there a plan (or jira) that points to more options here?
I remember seeing this: https://issues.apache.org/jira/browse/IMPALA-462
It will also be nice in the future if Impala can support ALTER DATABASE RENAME
in the future.
http://gerrit.cloudera.org:8080/#/c/10471/3/fe/src/main/cup/sql-parser.cup
File fe/src/main/cup/sql-parser.cup:
http://gerrit.cloudera.org:8080/#/c/10471/3/fe/src/main/cup/sql-parser.cup@268
PS3, Line 268: , KW_DRO
> that's not reserved nor on the list of planned keywords. will need to see w
I think you were looking at the wrong patch set. I have removed KW_OWNER from
the scanner.
http://gerrit.cloudera.org:8080/#/c/10471/5/fe/src/main/cup/sql-parser.cup
File fe/src/main/cup/sql-parser.cup:
http://gerrit.cloudera.org:8080/#/c/10471/5/fe/src/main/cup/sql-parser.cup@1049
PS5, Line 1049: parser.checkIdentKeyword("OWNER", owner_id);
: RESULT = new AlterDbSetOwnerStmt(db, new
Owner(TOwnerType.ROLE, role));
: :}
> repeated comment... I'd prefer if the version on L1039 is used once and pla
Done
http://gerrit.cloudera.org:8080/#/c/10471/5/fe/src/main/java/org/apache/impala/analysis/AlterDbSetOwnerStmt.java
File fe/src/main/java/org/apache/impala/analysis/AlterDbSetOwnerStmt.java:
http://gerrit.cloudera.org:8080/#/c/10471/5/fe/src/main/java/org/apache/impala/analysis/AlterDbSetOwnerStmt.java@34
PS5, Line 34: super(
> noticed that there is no analysis method for this stmt. do we require that
In Hive, the user/role can be arbitrary. I'll add a check for the length though.
http://gerrit.cloudera.org:8080/#/c/10471/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/10471/5/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3535
PS5, Line 3535: Preconditions.
> move these two methods to L3516
Done
http://gerrit.cloudera.org:8080/#/c/10471/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:
http://gerrit.cloudera.org:8080/#/c/10471/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@3917
PS5, Line 3917:
> is there a length limit on this string?
Done
http://gerrit.cloudera.org:8080/#/c/10471/5/fe/src/test/java/org/apache/impala/analysis/ParserTest.java
File fe/src/test/java/org/apache/impala/analysis/ParserTest.java:
http://gerrit.cloudera.org:8080/#/c/10471/5/fe/src/test/java/org/apache/impala/analysis/ParserTest.java@3751
PS5, Line 3751: er"
> add the following variations for foo:
Done
--
To view, visit http://gerrit.cloudera.org:8080/10471
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie3b923021ebce5192d2d64784e7ddb952ba82bc3
Gerrit-Change-Number: 10471
Gerrit-PatchSet: 7
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Adam Holley
Gerrit-Reviewer: Alex Behm
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Wed, 06 Jun 2018 19:19:57 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7016: Implement ALTER DATABASE SET OWNER
Fredy Wijaya has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/10471 ) Change subject: IMPALA-7016: Implement ALTER DATABASE SET OWNER .. IMPALA-7016: Implement ALTER DATABASE SET OWNER Alter the database owner to either user or role. On database creation, the database owner will be set to the current user, which can be viewed via DESCRIBE DATABASE db command. Having an owner information allows implementing a feature where an owner can be given certain privileges automatically upon a database creation. The ALTER DATABASE SET OWNER will be a useful command for transferring ownership (a set of owner privileges) from the current owner to another owner. Syntax: ALTER DATABASE db SET OWNER USER user ALTER DATABASE db SET OWNER ROLE role Testing: - Added new front-end tests - Added new end-to-end tests Change-Id: Ie3b923021ebce5192d2d64784e7ddb952ba82bc3 --- M common/thrift/CatalogService.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup A fe/src/main/java/org/apache/impala/analysis/AlterDbSetOwnerStmt.java A fe/src/main/java/org/apache/impala/analysis/AlterDbStmt.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/Owner.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M tests/metadata/test_ddl.py M tests/metadata/test_ddl_base.py 15 files changed, 361 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/10471/7 -- To view, visit http://gerrit.cloudera.org:8080/10471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3b923021ebce5192d2d64784e7ddb952ba82bc3 Gerrit-Change-Number: 10471 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-5380: [DOCS] Added additional filesystems supported in URI
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10551 ) Change subject: IMPALA-5380: [DOCS] Added additional filesystems supported in URI .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/10551/5/docs/topics/impala_authorization.xml File docs/topics/impala_authorization.xml: http://gerrit.cloudera.org:8080/#/c/10551/5/docs/topics/impala_authorization.xml@924 PS5, Line 924: will be automatically appended > nit: It seems to be prepended, no? You could also rephrase as "If..., it wi Done http://gerrit.cloudera.org:8080/#/c/10551/5/docs/topics/impala_authorization.xml@924 PS5, Line 924: will be automatically appended > You could even include the example from Fredy's comment. Done -- To view, visit http://gerrit.cloudera.org:8080/10551 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d57efd564fb5779539a88d7b209cee93d313c6 Gerrit-Change-Number: 10551 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Wed, 06 Jun 2018 19:16:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5380: [DOCS] Added additional filesystems supported in URI
Hello Lars Volker, Fredy Wijaya, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10551 to look at the new patch set (#6). Change subject: IMPALA-5380: [DOCS] Added additional filesystems supported in URI .. IMPALA-5380: [DOCS] Added additional filesystems supported in URI Change-Id: I11d57efd564fb5779539a88d7b209cee93d313c6 --- M docs/topics/impala_authorization.xml 1 file changed, 33 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/10551/6 -- To view, visit http://gerrit.cloudera.org:8080/10551 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I11d57efd564fb5779539a88d7b209cee93d313c6 Gerrit-Change-Number: 10551 Gerrit-PatchSet: 6 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-6812: Fix flaky Kudu scan tests
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/10503 ) Change subject: IMPALA-6812: Fix flaky Kudu scan tests .. Patch Set 1: As Tim said, Impala keeps track of KuduClient::GetLatestObservedTimestamp() on a per-session basis and propagates it around to all impalads. Having RYW consistency at the session level would be sufficient for our purposes here. Impala also uses just a single KuduClient per impalad, though of course a scan may be performed by multiple impalads in parallel. So it sounds like going to READ_YOUR_WRITES is the right way forward, at least for tests. There's still a question of if we want to set that as the default, given things like: is there a perf cost for the extra consistency that the typical user might not want to pay? Would changing this default count as a breaking change that we might not want to do just in a random release? -- To view, visit http://gerrit.cloudera.org:8080/10503 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I70df84f2cbc663107f2ad029565d3c15bdfbd47c Gerrit-Change-Number: 10503 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: David Ribeiro Alves Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 06 Jun 2018 18:59:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5380: [DOCS] Added additional filesystems supported in URI
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/10551 ) Change subject: IMPALA-5380: [DOCS] Added additional filesystems supported in URI .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/10551/5/docs/topics/impala_authorization.xml File docs/topics/impala_authorization.xml: http://gerrit.cloudera.org:8080/#/c/10551/5/docs/topics/impala_authorization.xml@924 PS5, Line 924: will be automatically appended > nit: It seems to be prepended, no? You could also rephrase as "If..., it wi You could even include the example from Fredy's comment. -- To view, visit http://gerrit.cloudera.org:8080/10551 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d57efd564fb5779539a88d7b209cee93d313c6 Gerrit-Change-Number: 10551 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Wed, 06 Jun 2018 18:57:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5380: [DOCS] Added additional filesystems supported in URI
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/10551 ) Change subject: IMPALA-5380: [DOCS] Added additional filesystems supported in URI .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/10551/5/docs/topics/impala_authorization.xml File docs/topics/impala_authorization.xml: http://gerrit.cloudera.org:8080/#/c/10551/5/docs/topics/impala_authorization.xml@924 PS5, Line 924: will be automatically appended nit: It seems to be prepended, no? You could also rephrase as "If..., it will be appended to the default filesystem prefix." -- To view, visit http://gerrit.cloudera.org:8080/10551 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d57efd564fb5779539a88d7b209cee93d313c6 Gerrit-Change-Number: 10551 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Wed, 06 Jun 2018 18:56:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5380: [DOCS] Added additional filesystems supported in URI
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10551 ) Change subject: IMPALA-5380: [DOCS] Added additional filesystems supported in URI .. Patch Set 5: Lars, Could you +2 this? I try not to +2 my own commits. Thanks! -- To view, visit http://gerrit.cloudera.org:8080/10551 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d57efd564fb5779539a88d7b209cee93d313c6 Gerrit-Change-Number: 10551 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Wed, 06 Jun 2018 18:49:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5380: [DOCS] Added additional filesystems supported in URI
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10551 ) Change subject: IMPALA-5380: [DOCS] Added additional filesystems supported in URI .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/10551/3/docs/topics/impala_authorization.xml File docs/topics/impala_authorization.xml: http://gerrit.cloudera.org:8080/#/c/10551/3/docs/topics/impala_authorization.xml@921 PS3, Line 921: If a URI starts with anything else, it will : cause an exception > I don't think this is correct. If a URI starts with an absolute path, it wi The sentence was updated. -- To view, visit http://gerrit.cloudera.org:8080/10551 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d57efd564fb5779539a88d7b209cee93d313c6 Gerrit-Change-Number: 10551 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Wed, 06 Jun 2018 18:46:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5380: [DOCS] Added additional filesystems supported in URI
Hello Lars Volker, Fredy Wijaya, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10551 to look at the new patch set (#5). Change subject: IMPALA-5380: [DOCS] Added additional filesystems supported in URI .. IMPALA-5380: [DOCS] Added additional filesystems supported in URI Change-Id: I11d57efd564fb5779539a88d7b209cee93d313c6 --- M docs/topics/impala_authorization.xml 1 file changed, 24 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/10551/5 -- To view, visit http://gerrit.cloudera.org:8080/10551 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I11d57efd564fb5779539a88d7b209cee93d313c6 Gerrit-Change-Number: 10551 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-3307: Add support for IANA time-zone db
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/9986 ) Change subject: IMPALA-3307: Add support for IANA time-zone db .. Patch Set 11: (2 comments) Thanks for adding zip support! We should add some tests for zip_util, especially for error handling, which is an untested path at the moment if didn't miss something. I am ok with moving this (and dealing with my other comments) to a later commit. http://gerrit.cloudera.org:8080/#/c/9986/11/be/src/exprs/timezone_db.cc File be/src/exprs/timezone_db.cc: http://gerrit.cloudera.org:8080/#/c/9986/11/be/src/exprs/timezone_db.cc@198 PS11, Line 198: GetNextDirectoryEntry This is subjective, but I do not like this interface too much. I would prefer to wrap dir_stream to a class/struct, or create a function like this: static STATUS ListDirEntries(string path, vector& result, int max_result_num = 0). Both could be moved to util/filesystem_util. http://gerrit.cloudera.org:8080/#/c/9986/11/be/src/exprs/timezone_db.cc@213 PS11, Line 213: readdir_r There was a discussion about readdir_r() vs readdir() in https://gerrit.cloudera.org/#/c/8546/8 , and readdir() was preferred in the end. -- To view, visit http://gerrit.cloudera.org:8080/9986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93c1fbffe81f067919706e30db0a34d0e58e7e77 Gerrit-Change-Number: 9986 Gerrit-PatchSet: 11 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 06 Jun 2018 18:35:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7108: IllegalStateException hit during CardinalityCheckNode.
Vincent Tran has posted comments on this change. ( http://gerrit.cloudera.org:8080/10605 ) Change subject: IMPALA-7108: IllegalStateException hit during CardinalityCheckNode. .. Patch Set 2: Code-Review+1 LGTM Tested against a few other variations of subqueries that return a single scalar values. -- To view, visit http://gerrit.cloudera.org:8080/10605 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82a7a3fe26db3e12131c030c4ad055a9c4955407 Gerrit-Change-Number: 10605 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vincent Tran Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 06 Jun 2018 18:25:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5380: [DOCS] Added additional filesystems supported in URI
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10551 ) Change subject: IMPALA-5380: [DOCS] Added additional filesystems supported in URI .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/10551/3/docs/topics/impala_authorization.xml File docs/topics/impala_authorization.xml: http://gerrit.cloudera.org:8080/#/c/10551/3/docs/topics/impala_authorization.xml@921 PS3, Line 921: If a URI starts with anything else, it will : cause an exception I don't think this is correct. If a URI starts with an absolute path, it will automatically append the default prefix filesystem. For example: > grant all on uri '/tmp' to role test_role; > show grant role test_role; Query: show grant role test_role +---+--+---++--+---+--+---+ | scope | database | table | column | uri | privilege | grant_option | create_time | +---+--+---++--+---+--+---+ | uri | | || hdfs://localhost:20500/tmp | all | false| Wed, Jun 06 2018 11:22:26.283 | -- To view, visit http://gerrit.cloudera.org:8080/10551 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I11d57efd564fb5779539a88d7b209cee93d313c6 Gerrit-Change-Number: 10551 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Wed, 06 Jun 2018 18:24:39 + Gerrit-HasComments: Yes
