date:20180529

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10523 )

Change subject: IMPALA-7082: Show human readable size in query backend page
..


Patch Set 4: Code-Review+1

This looks fine to me. If we end up doing a lot of this, we'll want to have a 
javascript unit test framework, but I don't think we're there yet :)

I'd give a +2, but I think we're currently trying to not add more things until 
the builds stabilize a bit more. (See e-mails from Thomas on the dev list.)


--
To view, visit http://gerrit.cloudera.org:8080/10523
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04afb4091bb8b6bc9dedfeceaf9284a8c65b16a1
Gerrit-Change-Number: 10523
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Comment-Date: Wed, 30 May 2018 03:48:55 +
Gerrit-HasComments: No

[Impala-ASF-CR] Revert IMPALA-2751: Matching quotes are not requirerd in comments

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10537 )

Change subject: Revert IMPALA-2751: Matching quotes are not requirerd in 
comments
..


Patch Set 1: Code-Review+2

If this is a clean revert, let's do that first. There's no shame in reverts!

I think we've got a pause on new commits for build stabilization, so the 
unrevert is probably post-pause.


--
To view, visit http://gerrit.cloudera.org:8080/10537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id9995a91408d86a5ae1ecd70d07b02622ae26b43
Gerrit-Change-Number: 10537
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Wed, 30 May 2018 03:44:14 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6338: Disable more flaky bloom filter tests

2018-05-29 Thread Bikramjeet Vig (Code Review)

Bikramjeet Vig has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10530 )

Change subject: IMPALA-6338: Disable more flaky bloom filter tests
..


Patch Set 1: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/10530
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc9da892efa3b27d63056ad8e3befac82808ffdb
Gerrit-Change-Number: 10530
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Comment-Date: Wed, 30 May 2018 03:32:45 +
Gerrit-HasComments: No

[Impala-ASF-CR] Revert IMPALA-2751: Matching quotes are not requirerd in comments

2018-05-29 Thread Fredy Wijaya (Code Review)

Fredy Wijaya has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10537 )

Change subject: Revert IMPALA-2751: Matching quotes are not requirerd in 
comments
..


Patch Set 1:

I'm making a fix for this right now. If the fix works, this should fix 
IMPALA-7089.


--
To view, visit http://gerrit.cloudera.org:8080/10537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id9995a91408d86a5ae1ecd70d07b02622ae26b43
Gerrit-Change-Number: 10537
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Comment-Date: Wed, 30 May 2018 03:30:33 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7082: Show human readable size in query backend page

Quanlong Huang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10523 )

Change subject: IMPALA-7082: Show human readable size in query backend page
..


Patch Set 4:

> This file (and a .gitignore) slipped into your git commit somehow.

Yeah, just deleted this change. It's due to I removed the generated-sources dir 
to rebuild the project (for thrift update to 0.9.3-p4). I used to dev in 2.x 
branch which's still using thrift-0.9.0...


--
To view, visit http://gerrit.cloudera.org:8080/10523
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04afb4091bb8b6bc9dedfeceaf9284a8c65b16a1
Gerrit-Change-Number: 10523
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Comment-Date: Wed, 30 May 2018 03:21:57 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7082: Show human readable size in query backend page

Hello Philip Zeyliger, Sailesh Mukil,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/10523

to look at the new patch set (#4).

Change subject: IMPALA-7082: Show human readable size in query backend page
..

IMPALA-7082: Show human readable size in query backend page

This patch reuse the JS function introduced in IMPALA-6966,
which can convert Long size value into human readable size,
to render the "Peak mem. consumption" column in the Query
details's Backends page.

Change-Id: I04afb4091bb8b6bc9dedfeceaf9284a8c65b16a1
---
M www/catalog.tmpl
M www/common-header.tmpl
M www/query_backends.tmpl
A www/scripts/util.js
4 files changed, 39 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/10523/4
--
To view, visit http://gerrit.cloudera.org:8080/10523
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I04afb4091bb8b6bc9dedfeceaf9284a8c65b16a1
Gerrit-Change-Number: 10523
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Sailesh Mukil

[Impala-ASF-CR] IMPALA-7082: Show human readable size in query backend page

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10523 )

Change subject: IMPALA-7082: Show human readable size in query backend page
..


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/10523/3/be/generated-sources/gen-cpp/CMakeLists.txt
File be/generated-sources/gen-cpp/CMakeLists.txt:

http://gerrit.cloudera.org:8080/#/c/10523/3/be/generated-sources/gen-cpp/CMakeLists.txt@a1
PS3, Line 1:
This file (and a .gitignore) slipped into your git commit somehow.



--
To view, visit http://gerrit.cloudera.org:8080/10523
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04afb4091bb8b6bc9dedfeceaf9284a8c65b16a1
Gerrit-Change-Number: 10523
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Comment-Date: Wed, 30 May 2018 03:11:16 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7091: Address NullPointerException in HBaseTable.getRegionSize().

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10531 )

Change subject: IMPALA-7091: Address NullPointerException in 
HBaseTable.getRegionSize().
..


Patch Set 1:

https://jenkins.impala.io/job/test-with-docker-parameterized/16/ ran just fine 
with this change.


--
To view, visit http://gerrit.cloudera.org:8080/10531
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02f06daf69e7f7e97c9ecc13997147530c2f9d3f
Gerrit-Change-Number: 10531
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Wed, 30 May 2018 03:09:41 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7091: Address NullPointerException in HBaseTable.getRegionSize().

Philip Zeyliger has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/10531


Change subject: IMPALA-7091: Address NullPointerException in 
HBaseTable.getRegionSize().
..

IMPALA-7091: Address NullPointerException in HBaseTable.getRegionSize().

It's possible for "serverLoad.getRegionsLoad().get(info.getRegionName())"
to be null, which causes a NullPointerException in the planner, and
visible to the user. The code around it already says that it handles
errors by returning 0 for the size, and I've extended that to one more
case.

In practice, I have seen this come up in failures of the following test:

  failure.test_failpoints.TestFailpoints.test_failpoints[table_format: 
hbase/none | exec_option: {'batch_size': 0, 'num_nodes': 0, 
'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 
'abort_on_error': 1, 'debug_action': None, 'exec_single_node_rows_threshold': 
0} | mt_dop: 4 | location: OPEN | action: MEM_LIMIT_EXCEEDED | query: select * 
from alltypessmall union all select * from alltypessmall]

I saw this failure only happen in some test-with-docker runs,
inconsistently.  The error is a little bit hard to spot, but by
correlating the timestamp of the failing test (which just complains
about NullPointerException), you can find a Java stack trace complaining
of a NPE in "regionLoad.getStorefileSizeMB()". I think the likely cause
is regionLoad being null.

Change-Id: I02f06daf69e7f7e97c9ecc13997147530c2f9d3f
---
M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java
1 file changed, 6 insertions(+), 2 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/10531/1
--
To view, visit http://gerrit.cloudera.org:8080/10531
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I02f06daf69e7f7e97c9ecc13997147530c2f9d3f
Gerrit-Change-Number: 10531
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Philip Zeyliger

[Impala-ASF-CR] IMPALA-7090: Limit the size of expr created by EqualityDisjunctsToInRule

2018-05-29 Thread Tianyi Wang (Code Review)

Tianyi Wang has uploaded a new patch set (#3). ( 
http://gerrit.cloudera.org:8080/10528 )

Change subject: IMPALA-7090: Limit the size of expr created by 
EqualityDisjunctsToInRule
..

IMPALA-7090: Limit the size of expr created by EqualityDisjunctsToInRule

Currently EqualityDisjunctsToInRule introduced in IMPALA-5280 might
create an expr with unlimited number of children and fails a query,
which should be avoided. The easy solution is to not apply the
rewrite when the number of children is large.

Change-Id: Ie40c3210271a9e3c7f1b2b869a8c2ec8bacaa72a
---
M fe/src/main/java/org/apache/impala/rewrite/EqualityDisjunctsToInRule.java
M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java
2 files changed, 75 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/10528/3
--
To view, visit http://gerrit.cloudera.org:8080/10528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40c3210271a9e3c7f1b2b869a8c2ec8bacaa72a
Gerrit-Change-Number: 10528
Gerrit-PatchSet: 3
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Reviewer: Vuk Ercegovac

[Impala-ASF-CR] IMPALA-5522: Add support for authorized proxy groups

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10510 )

Change subject: IMPALA-5522: Add support for authorized proxy groups
..


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/10510/2/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/10510/2/be/src/service/impala-server.cc@319
PS2, Line 319: [](const string& config) {
 : return Substitute("Invalid proxy user configuration. 
No mapping value "
 : "specified for the proxy user. For more 
information review usage of the "
 : "--authorized_proxy_user_config flag: $0", 
config);
 : });
> I don't think we can call CLEAN_EXIT_WITH_ERROR since it needs the config o
The error can be more generic saying that there's something invalid about the 
configuration. This isn't a huge deal, but I've not seen use use lambdas for 
error handling.


http://gerrit.cloudera.org:8080/#/c/10510/2/fe/src/main/java/org/apache/impala/service/JniFrontend.java
File fe/src/main/java/org/apache/impala/service/JniFrontend.java:

http://gerrit.cloudera.org:8080/#/c/10510/2/fe/src/main/java/org/apache/impala/service/JniFrontend.java@623
PS2, Line 623:   result.setGroups(GROUPS.getGroups(request.getUser()));
> It depends on the Hadoop mapping provider implementation and yes it can inv
For common Impala deployments, do you know what's in use?

For the deployment that I checked, on a nightly cluster at my employer, I found 
that Cloudera Manager seems to default to 
org.apache.hadoop.security.ShellBasedUnixGroupsMapping. If that's the case, 
we'll be forking per-query, and my sense of the comments at 
https://issues.apache.org/jira/browse/IMPALA-5624 is that we've been trying 
pretty hard to avoid it.

I'd go so far as saying that we should check to see what class the user group 
mapping provider is, and, if it's in a list of well-known forkers, we should 
fail.



--
To view, visit http://gerrit.cloudera.org:8080/10510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb
Gerrit-Change-Number: 10510
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Wed, 30 May 2018 03:00:26 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5522: Add support for authorized proxy groups

2018-05-29 Thread Fredy Wijaya (Code Review)

Fredy Wijaya has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10510 )

Change subject: IMPALA-5522: Add support for authorized proxy groups
..


Patch Set 3:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/10510/2/be/src/service/frontend.cc
File be/src/service/frontend.cc:

http://gerrit.cloudera.org:8080/#/c/10510/2/be/src/service/frontend.cc@69
PS2, Line 69: "Specifies the delimiter used in 
authorized_proxy_group_config. ");
> I see that we're copying an older pattern, but this seems unfortunate. How
We can specify a custom delimiter to something other than comma. It just 
defaults it to comma when the authorized_proxy_user/group_config_delimiter is 
not specified.

I'll add some tests for custom delimiters.


http://gerrit.cloudera.org:8080/#/c/10510/2/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/10510/2/be/src/service/impala-server.cc@319
PS2, Line 319: [](const string& config) {
 : return Substitute("Invalid proxy user configuration. 
No mapping value "
 : "specified for the proxy user. For more 
information review usage of the "
 : "--authorized_proxy_user_config flag: $0", 
config);
 : });
> Others can comment, but I don't think this pattern (passing a lambda to err
I don't think we can call CLEAN_EXIT_WITH_ERROR since it needs the config 
object for the string substitution, which comes from a loop.


http://gerrit.cloudera.org:8080/#/c/10510/2/be/src/service/impala-server.cc@398
PS2, Line 398: void ImpalaServer::AddAuthorizedProxyConfig(
> You're doing parsing here. I think it'd be lovely to see a C++ unit test th
Done.


http://gerrit.cloudera.org:8080/#/c/10510/2/fe/src/main/java/org/apache/impala/service/JniFrontend.java
File fe/src/main/java/org/apache/impala/service/JniFrontend.java:

http://gerrit.cloudera.org:8080/#/c/10510/2/fe/src/main/java/org/apache/impala/service/JniFrontend.java@623
PS2, Line 623:   result.setGroups(GROUPS.getGroups(request.getUser()));
It depends on the Hadoop mapping provider implementation and yes it can invoke 
a subprocess as a fallback. Let me know if there's a better way to get Hadoop 
groups without using Hadoop API.

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/GroupsMapping.html

> The default implementation. It will determine if the Java Native Interface 
> (JNI) is available. If JNI is available, the implementation will use the API 
> within hadoop to resolve a list of groups for a user. If JNI is not available 
> then the shell-based implementation, ShellBasedUnixGroupsMapping, is used.


http://gerrit.cloudera.org:8080/#/c/10510/2/tests/authorization/test_authorization.py
File tests/authorization/test_authorization.py:

http://gerrit.cloudera.org:8080/#/c/10510/2/tests/authorization/test_authorization.py@193
PS2, Line 193: --audit_event_log_dir=%s" % (AUTH_POLICY_FILE,
> Please explain what this test is doing.
Yeah that's true. It only needs to be in one group for it to work. I wanted to 
test to make sure the code works with multiple groups, too.

Good point on having authorized user and group settings at the same time. I'll 
add another test for that.



--
To view, visit http://gerrit.cloudera.org:8080/10510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb
Gerrit-Change-Number: 10510
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Wed, 30 May 2018 01:53:33 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5522: Add support for authorized proxy groups

2018-05-29 Thread Fredy Wijaya (Code Review)

Fredy Wijaya has uploaded a new patch set (#3). ( 
http://gerrit.cloudera.org:8080/10510 )

Change subject: IMPALA-5522: Add support for authorized proxy groups
..

IMPALA-5522: Add support for authorized proxy groups

The patch adds support for mapping of users to a list of proxy groups.

The following flags are added in impalad:
- authorized_proxy_group_config
- authorized_proxy_group_config_delimiter

Example:
--authorized_proxy_group_config=hue=group1,group2;user1=*

Testing:
- Added BE unit test for the parsing logic
- Added a new test in test_authorization.py
- Ran all end-to-end test_authorization.py

Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb
---
M be/src/service/CMakeLists.txt
M be/src/service/frontend.cc
M be/src/service/frontend.h
A be/src/service/impala-server-test.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/util/backend-gflag-util.cc
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M tests/authorization/test_authorization.py
10 files changed, 274 insertions(+), 31 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/10510/3
--
To view, visit http://gerrit.cloudera.org:8080/10510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb
Gerrit-Change-Number: 10510
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Vuk Ercegovac

[Impala-ASF-CR] IMPALA-7082: Show human readable size in query backend page

Quanlong Huang has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10523 )

Change subject: IMPALA-7082: Show human readable size in query backend page
..


Patch Set 3:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/10523/2/www/common-header.tmpl
File www/common-header.tmpl:

http://gerrit.cloudera.org:8080/#/c/10523/2/www/common-header.tmpl@25
PS2, Line 25: 
> Sailesh is right, of course. We could include a new script ~here.
Done


http://gerrit.cloudera.org:8080/#/c/10523/2/www/common-header.tmpl@54
PS2, Line 54: {{/error}}
> Is this is the right file to put these utility functions in?
Sure! Done.



--
To view, visit http://gerrit.cloudera.org:8080/10523
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04afb4091bb8b6bc9dedfeceaf9284a8c65b16a1
Gerrit-Change-Number: 10523
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Comment-Date: Wed, 30 May 2018 01:37:40 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7082: Show human readable size in query backend page

Hello Philip Zeyliger, Sailesh Mukil,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/10523

to look at the new patch set (#3).

Change subject: IMPALA-7082: Show human readable size in query backend page
..

IMPALA-7082: Show human readable size in query backend page

This patch reuse the JS function introduced in IMPALA-6966,
which can convert Long size value into human readable size,
to render the "Peak mem. consumption" column in the Query
details's Backends page.

Change-Id: I04afb4091bb8b6bc9dedfeceaf9284a8c65b16a1
---
D be/generated-sources/gen-cpp/CMakeLists.txt
D be/generated-sources/impala-ir/.gitignore
M www/catalog.tmpl
M www/common-header.tmpl
M www/query_backends.tmpl
A www/scripts/util.js
6 files changed, 39 insertions(+), 122 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/10523/3
--
To view, visit http://gerrit.cloudera.org:8080/10523
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I04afb4091bb8b6bc9dedfeceaf9284a8c65b16a1
Gerrit-Change-Number: 10523
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Sailesh Mukil

[Impala-ASF-CR](2.x) Ignore IMPALA-6953 which has already been picked

Thomas Marshall has abandoned this change. ( 
http://gerrit.cloudera.org:8080/10539 )

Change subject: Ignore IMPALA-6953 which has already been picked
..


Abandoned
--
To view, visit http://gerrit.cloudera.org:8080/10539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: abandon
Gerrit-Change-Id: Ifc8aff499c00f78e8d3c82756bb47ed0e943ea75
Gerrit-Change-Number: 10539
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR](2.x) Ignore IMPALA-6953 which has already been picked

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10539 )

Change subject: Ignore IMPALA-6953 which has already been picked
..


Patch Set 1:

I think I accidentally fixed this by submitting my version of the patch, which 
has the original Change-Id but no content.


--
To view, visit http://gerrit.cloudera.org:8080/10539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc8aff499c00f78e8d3c82756bb47ed0e943ea75
Gerrit-Change-Number: 10539
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 30 May 2018 01:16:08 +
Gerrit-HasComments: No

[Impala-ASF-CR](2.x) Ignore IMPALA-6953 which has already been picked

Thomas Marshall has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/10539


Change subject: Ignore IMPALA-6953 which has already been picked
..

Ignore IMPALA-6953 which has already been picked

IMPALA-6953 was already cherry-picked to 2.x, but
with a different Change-Id than in master.

Change-Id: Ifc8aff499c00f78e8d3c82756bb47ed0e943ea75
---
M bin/ignored_commits.json
1 file changed, 3 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/10539/1
--
To view, visit http://gerrit.cloudera.org:8080/10539
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifc8aff499c00f78e8d3c82756bb47ed0e943ea75
Gerrit-Change-Number: 10539
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall

[Impala-ASF-CR](2.x) IMPALA-6953: part 1: clean up DiskIoMgr

Hello Impala Public Jenkins,

I'd like you to do a code review. Please visit

http://gerrit.cloudera.org:8080/10532

to review the following change.


Change subject: IMPALA-6953: part 1: clean up DiskIoMgr
..

IMPALA-6953: part 1: clean up DiskIoMgr

There should be no behavioural changes as a result of
this refactoring.

Make DiskQueue an encapsulated class.

Remove friend classes where possible, either by using public
methods or moving code between classes.

Move method into protected in some cases.

Split GetNextRequestRange() into two methods that
operate on DiskQueue and RequestContext state. The methods
belong to the respective classes.

Reduce transitive #include dependencies to hopefully help
with build time.

Testing:
Ran core tests.

Change-Id: I5a6e393f8c01d10143cbac91108af37f6498c1b1
Reviewed-on: http://gerrit.cloudera.org:8080/10245
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins 
---
M be/src/exec/blocking-join-node.cc
M be/src/exec/hdfs-orc-scanner.h
M be/src/exec/hdfs-parquet-table-writer.cc
M be/src/exec/hdfs-scan-node-mt.cc
M be/src/exec/hdfs-scan-node-mt.h
M be/src/exec/hdfs-scan-node.h
M be/src/exec/hdfs-scanner.h
M be/src/exec/kudu-scan-node.cc
M be/src/exec/kudu-table-sink.cc
M be/src/exec/parquet-column-readers.cc
M be/src/runtime/io/disk-io-mgr-internal.h
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/runtime/io/error-converter.cc
M be/src/runtime/io/local-file-system.cc
M be/src/runtime/io/request-context.cc
M be/src/runtime/io/request-context.h
M be/src/runtime/io/request-ranges.h
M be/src/runtime/io/scan-range.cc
M be/src/runtime/row-batch.h
20 files changed, 590 insertions(+), 533 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/10532/1
--
To view, visit http://gerrit.cloudera.org:8080/10532
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5a6e393f8c01d10143cbac91108af37f6498c1b1
Gerrit-Change-Number: 10532
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Marshall

[Impala-ASF-CR](2.x) IMPALA-6953: part 1: clean up DiskIoMgr

Tim Armstrong has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/10532 )

Change subject: IMPALA-6953: part 1: clean up DiskIoMgr
..

IMPALA-6953: part 1: clean up DiskIoMgr

There should be no behavioural changes as a result of
this refactoring.

Make DiskQueue an encapsulated class.

Remove friend classes where possible, either by using public
methods or moving code between classes.

Move method into protected in some cases.

Split GetNextRequestRange() into two methods that
operate on DiskQueue and RequestContext state. The methods
belong to the respective classes.

Reduce transitive #include dependencies to hopefully help
with build time.

Testing:
Ran core tests.

Change-Id: I5a6e393f8c01d10143cbac91108af37f6498c1b1
Reviewed-on: http://gerrit.cloudera.org:8080/10245
Reviewed-by: Tim Armstrong 
Tested-by: Impala Public Jenkins 
Reviewed-on: http://gerrit.cloudera.org:8080/10532
Reviewed-by: Thomas Marshall 
Tested-by: Thomas Marshall 
---
0 files changed, 0 insertions(+), 0 deletions(-)

Approvals:
  Thomas Marshall: Looks good to me, approved; Verified
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/10532
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: merged
Gerrit-Change-Id: I5a6e393f8c01d10143cbac91108af37f6498c1b1
Gerrit-Change-Number: 10532
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR](2.x) IMPALA-6953: part 1: clean up DiskIoMgr

Thomas Marshall has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10538 )

Change subject: IMPALA-6953: part 1: clean up DiskIoMgr
..


Patch Set 1: Verified+1 Code-Review+2

Cherry-pick, no interesting conflicts, passed gvo: 
https://jenkins.impala.io/job/gerrit-verify-dryrun/2564/


--
To view, visit http://gerrit.cloudera.org:8080/10538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I50b448834b832a0ee0dc5d85541691cd2f308e12
Gerrit-Change-Number: 10538
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Wed, 30 May 2018 01:03:36 +
Gerrit-HasComments: No

[Impala-ASF-CR](2.x) IMPALA-6953: part 1: clean up DiskIoMgr

Thomas Marshall has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/10538 )

Change subject: IMPALA-6953: part 1: clean up DiskIoMgr
..

IMPALA-6953: part 1: clean up DiskIoMgr

There should be no behavioural changes as a result of
this refactoring.

Make DiskQueue an encapsulated class.

Remove friend classes where possible, either by using public
methods or moving code between classes.

Move method into protected in some cases.

Split GetNextRequestRange() into two methods that
operate on DiskQueue and RequestContext state. The methods
belong to the respective classes.

Reduce transitive #include dependencies to hopefully help
with build time.

Testing:
Ran core tests.

Change-Id: I50b448834b832a0ee0dc5d85541691cd2f308e12
Reviewed-on: http://gerrit.cloudera.org:8080/10538
Reviewed-by: Thomas Marshall 
Tested-by: Thomas Marshall 
---
M be/src/exec/blocking-join-node.cc
M be/src/exec/hdfs-orc-scanner.h
M be/src/exec/hdfs-parquet-table-writer.cc
M be/src/exec/hdfs-scan-node-mt.cc
M be/src/exec/hdfs-scan-node-mt.h
M be/src/exec/hdfs-scan-node.h
M be/src/exec/hdfs-scanner.h
M be/src/exec/kudu-scan-node.cc
M be/src/exec/kudu-table-sink.cc
M be/src/exec/parquet-column-readers.cc
M be/src/runtime/io/disk-io-mgr-internal.h
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/runtime/io/error-converter.cc
M be/src/runtime/io/local-file-system.cc
M be/src/runtime/io/request-context.cc
M be/src/runtime/io/request-context.h
M be/src/runtime/io/request-ranges.h
M be/src/runtime/io/scan-range.cc
M be/src/runtime/row-batch.h
20 files changed, 590 insertions(+), 533 deletions(-)

Approvals:
  Thomas Marshall: Looks good to me, approved; Verified

--
To view, visit http://gerrit.cloudera.org:8080/10538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: merged
Gerrit-Change-Id: I50b448834b832a0ee0dc5d85541691cd2f308e12
Gerrit-Change-Number: 10538
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Marshall 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-7090: Limit the size of expr created by EqualityDisjunctsToInRule

2018-05-29 Thread Vuk Ercegovac (Code Review)

Vuk Ercegovac has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10528 )

Change subject: IMPALA-7090: Limit the size of expr created by 
EqualityDisjunctsToInRule
..


Patch Set 2: Code-Review+2

(2 comments)

http://gerrit.cloudera.org:8080/#/c/10528/1/fe/src/main/java/org/apache/impala/rewrite/EqualityDisjunctsToInRule.java
File fe/src/main/java/org/apache/impala/rewrite/EqualityDisjunctsToInRule.java:

http://gerrit.cloudera.org:8080/#/c/10528/1/fe/src/main/java/org/apache/impala/rewrite/EqualityDisjunctsToInRule.java@82
PS1, Line 82: else if (otherPred instanceof InPredicate && !((InPredicate) 
otherPred).isNotIn()
> While more fault tolerance is often good, I don't think we want an unoptimi
As we add more rules and independently change analysis for expressions, I 
expect unforeseen cases like this to be more common. Currently, if such a case 
comes up (as without this change), the end user needs to turn off rewrites 
while waiting for a bug fix. Do we have other cases where optimizations are not 
applied, but the query is run (with a warning)?


http://gerrit.cloudera.org:8080/#/c/10528/2/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java
File fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java:

http://gerrit.cloudera.org:8080/#/c/10528/2/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java@172
PS2, Line 172: /** co
nit: reformat this comment:
/**
 * Construct an in-list: string_col in (offset ... offset + length)
 */

... to be consistent with other single line method comments.



--
To view, visit http://gerrit.cloudera.org:8080/10528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40c3210271a9e3c7f1b2b869a8c2ec8bacaa72a
Gerrit-Change-Number: 10528
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Wed, 30 May 2018 01:02:59 +
Gerrit-HasComments: Yes

[Impala-ASF-CR](2.x) IMPALA-6953: part 1: clean up DiskIoMgr

Thomas Marshall has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/10538


Change subject: IMPALA-6953: part 1: clean up DiskIoMgr
..

IMPALA-6953: part 1: clean up DiskIoMgr

There should be no behavioural changes as a result of
this refactoring.

Make DiskQueue an encapsulated class.

Remove friend classes where possible, either by using public
methods or moving code between classes.

Move method into protected in some cases.

Split GetNextRequestRange() into two methods that
operate on DiskQueue and RequestContext state. The methods
belong to the respective classes.

Reduce transitive #include dependencies to hopefully help
with build time.

Testing:
Ran core tests.

Change-Id: I50b448834b832a0ee0dc5d85541691cd2f308e12
---
M be/src/exec/blocking-join-node.cc
M be/src/exec/hdfs-orc-scanner.h
M be/src/exec/hdfs-parquet-table-writer.cc
M be/src/exec/hdfs-scan-node-mt.cc
M be/src/exec/hdfs-scan-node-mt.h
M be/src/exec/hdfs-scan-node.h
M be/src/exec/hdfs-scanner.h
M be/src/exec/kudu-scan-node.cc
M be/src/exec/kudu-table-sink.cc
M be/src/exec/parquet-column-readers.cc
M be/src/runtime/io/disk-io-mgr-internal.h
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/disk-io-mgr.h
M be/src/runtime/io/error-converter.cc
M be/src/runtime/io/local-file-system.cc
M be/src/runtime/io/request-context.cc
M be/src/runtime/io/request-context.h
M be/src/runtime/io/request-ranges.h
M be/src/runtime/io/scan-range.cc
M be/src/runtime/row-batch.h
20 files changed, 590 insertions(+), 533 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/10538/1
--
To view, visit http://gerrit.cloudera.org:8080/10538
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: newchange
Gerrit-Change-Id: I50b448834b832a0ee0dc5d85541691cd2f308e12
Gerrit-Change-Number: 10538
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-7073: skip TestScannerReservation on non-miniclusters

2018-05-29 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/10533 )

Change subject: IMPALA-7073: skip TestScannerReservation on non-miniclusters
..

IMPALA-7073: skip TestScannerReservation on non-miniclusters

The test is (sort of) tuned for miniclusters and is very targeted
to testing a specific code path, rather than general functional
correctness, so we don't really need coverage on all filesystems.

Change-Id: I7952f780cff80c08a6cbef898bf7b95c9bba5f6a
Reviewed-on: http://gerrit.cloudera.org:8080/10533
Reviewed-by: Thomas Marshall 
Tested-by: Impala Public Jenkins 
---
M tests/query_test/test_scanners.py
1 file changed, 3 insertions(+), 1 deletion(-)

Approvals:
  Thomas Marshall: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/10533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I7952f780cff80c08a6cbef898bf7b95c9bba5f6a
Gerrit-Change-Number: 10533
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Marshall

[Impala-ASF-CR] IMPALA-7073: skip TestScannerReservation on non-miniclusters

2018-05-29 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10533 )

Change subject: IMPALA-7073: skip TestScannerReservation on non-miniclusters
..


Patch Set 1: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/10533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7952f780cff80c08a6cbef898bf7b95c9bba5f6a
Gerrit-Change-Number: 10533
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Wed, 30 May 2018 00:38:20 +
Gerrit-HasComments: No

[Impala-ASF-CR] Revert IMPALA-2751: Matching quotes are not requirerd in comments

Thomas Marshall has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/10537


Change subject: Revert IMPALA-2751: Matching quotes are not requirerd in 
comments
..

Revert IMPALA-2751: Matching quotes are not requirerd in comments

This patch is causing a large number of builds to fail, see
IMPALA-7089.

Change-Id: Id9995a91408d86a5ae1ecd70d07b02622ae26b43
---
M shell/impala_shell.py
M tests/shell/test_shell_interactive.py
2 files changed, 5 insertions(+), 25 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/10537/1
--
To view, visit http://gerrit.cloudera.org:8080/10537
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id9995a91408d86a5ae1ecd70d07b02622ae26b43
Gerrit-Change-Number: 10537
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall

[Impala-ASF-CR] IMPALA-7090: Limit the size of expr created by EqualityDisjunctsToInRule

2018-05-29 Thread Tianyi Wang (Code Review)

Tianyi Wang has uploaded a new patch set (#2). ( 
http://gerrit.cloudera.org:8080/10528 )

Change subject: IMPALA-7090: Limit the size of expr created by 
EqualityDisjunctsToInRule
..

IMPALA-7090: Limit the size of expr created by EqualityDisjunctsToInRule

Currently EqualityDisjunctsToInRule introduced in IMPALA-5280 might
create an expr with unlimited number of children and fails a query,
which should be avoided. The easy solution is to not apply the
rewrite when the number of children is large.

Change-Id: Ie40c3210271a9e3c7f1b2b869a8c2ec8bacaa72a
---
M fe/src/main/java/org/apache/impala/rewrite/EqualityDisjunctsToInRule.java
M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java
2 files changed, 73 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/10528/2
--
To view, visit http://gerrit.cloudera.org:8080/10528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie40c3210271a9e3c7f1b2b869a8c2ec8bacaa72a
Gerrit-Change-Number: 10528
Gerrit-PatchSet: 2
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Reviewer: Vuk Ercegovac

[Impala-ASF-CR] IMPALA-6990: TestClientSsl.test tls v12 failing due to Python SSL error

Sailesh Mukil has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10529 )

Change subject: IMPALA-6990: TestClientSsl.test_tls_v12 failing due to Python 
SSL error
..


Patch Set 2:

(2 comments)

> (2 comments)
 >
 > I wrote a comment on the JIRA upstream.
 >
 > I think the easier thing to do is to check Python version and skip
 > if you're using a too-old Python, with an explanation. Possibly add
 > a second copy of the test with the flag changes you suggest.
 >
 > But, regardless, I still don't quite understand whether the test
 > was just failing silently before or something fundamentally broke.

I just realized that the test wasn't running before at all since all non-RHEL7 
systems that we use for testing don't have OpenSSL 1.0.1+, which is the minimum 
required version for TLSv1.2.

I updated the patch to skip this specific test if we don't have an up-to-date 
Python that supports the client using TLSv1.2.

http://gerrit.cloudera.org:8080/#/c/10529/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/10529/2//COMMIT_MSG@42
PS2, Line 42: which does not force the use of specific ciphers, so we get away 
without
> What doesn't force the user of ciphers?
Actually, I've found a flaw in my reasoning, I didn't realize but this test was 
being skipped on older-than-RHEL7-systems due to them having older versions of 
OpenSSL, that don't support TLSv1.2.

So, the ciphers had nothing to do with it and I've removed this para.


http://gerrit.cloudera.org:8080/#/c/10529/2//COMMIT_MSG@49
PS2, Line 49: least unblock our builds while we can upgrade the AMIs for RHEL7. 
This
> RHEL7 just doesn't ship a newer Python based on my quick check with Docker.
Makes sense. I've skipped the test unless a TLSv1.2 compatible version of 
Python is detected.



--
To view, visit http://gerrit.cloudera.org:8080/10529
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I92c66ecaeb94b0c83ee6f1396c082709c21b3187
Gerrit-Change-Number: 10529
Gerrit-PatchSet: 2
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Comment-Date: Tue, 29 May 2018 22:55:20 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6990: TestClientSsl.test tls v12 failing due to Python SSL error

Hello Philip Zeyliger,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/10529

to look at the new patch set (#3).

Change subject: IMPALA-6990: TestClientSsl.test_tls_v12 failing due to Python 
SSL error
..

IMPALA-6990: TestClientSsl.test_tls_v12 failing due to Python SSL error

When we upgraded to thrift-0.9.3, the TSSLSocket.py logic changed quite
a bit. Our RHEL7 machines come equipped with Python 2.7.5. Looking at
these comments, that means that we'll be unable to create a 'SSLContext'
but be able to explicitly specify ciphers:
https://github.com/apache/thrift/blob/master/lib/py/src/transport/TSSLSocket.py#L37-L41

# SSLContext is not available for Python < 2.7.9
_has_ssl_context = sys.hexversion >= 0x020709F0

# ciphers argument is not available for Python < 2.7.0
_has_ciphers = sys.hexversion >= 0x020700F0

If we cannot create a 'SSLContext', then we cannot use TLSv1.2 and have
to use TLSv1:
https://github.com/apache/thrift/blob/master/lib/py/src/transport/TSSLSocket.py#L48-L49

# For python >= 2.7.9, use latest TLS that both client and server
# supports.
# SSL 2.0 and 3.0 are disabled via ssl.OP_NO_SSLv2 and ssl.OP_NO_SSLv3.
# For python < 2.7.9, use TLS 1.0 since TLSv1_X nor OP_NO_SSLvX is
# unavailable.
_default_protocol = ssl.PROTOCOL_SSLv23 if _has_ssl_context else \
ssl.PROTOCOL_TLSv1

Our custom cluster test forces the server to use TLSv1.2 and also forces
a specific cipher:
https://github.com/apache/impala/blob/master/tests/custom_cluster/test_client_ssl.py#L118-L119

So this combination of configuration values causes a failure in RHEL7
because we only allow a specific cipher which works with TLSv1.2, but
the client cannot use TLSv1.2 due to the Python version as mentioned above.

We've not noticed these failures on older-than-RHEL7-systems since the
OpenSSL versions on those systems don't support TLSv1.2. (< OpenSSL 1.0.1)

To fix this, we need to change the Python version on RHEL 7 to be
>= Python 2.7.9. This patch skips the test if an older version of
Python than 2.7.9 is detected.

Change-Id: I92c66ecaeb94b0c83ee6f1396c082709c21b3187
---
M tests/custom_cluster/test_client_ssl.py
1 file changed, 6 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/10529/3
--
To view, visit http://gerrit.cloudera.org:8080/10529
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I92c66ecaeb94b0c83ee6f1396c082709c21b3187
Gerrit-Change-Number: 10529
Gerrit-PatchSet: 3
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Philip Zeyliger

[native-toolchain-CR] Revert "Bump libunwind version to 1.3-rc1"

Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10534 )

Change subject: Revert "Bump libunwind version to 1.3-rc1"
..


Patch Set 1: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/10534
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5662fba751067f6fed87d6180962fb1d31d872d0
Gerrit-Change-Number: 10534
Gerrit-PatchSet: 1
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Tue, 29 May 2018 22:25:28 +
Gerrit-HasComments: No

[native-toolchain-CR] Revert "Bump libunwind version to 1.3-rc1"

Lars Volker has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/10534 )

Change subject: Revert "Bump libunwind version to 1.3-rc1"
..

Revert "Bump libunwind version to 1.3-rc1"

This reverts commit c33b6fa03f7f29152b475caeb820b0df952d9538.

Change-Id: I5662fba751067f6fed87d6180962fb1d31d872d0
Reviewed-on: http://gerrit.cloudera.org:8080/10534
Reviewed-by: Tim Armstrong 
Tested-by: Lars Volker 
---
M buildall.sh
M source/libunwind/build.sh
A 
source/libunwind/libunwind-1.1-patches/0001-Build-libunwind-1.1-on-ppc64le.patch
D 
source/libunwind/libunwind-1.3-rc1-patches/0001-libunwind-Use-syscall-directly-in-write_validate-to-avoid-ASAN.patch
D 
source/libunwind/libunwind-1.3-rc1-patches/0002-libunwind-trace-cache-destructor.patch
5 files changed, 11 insertions(+), 78 deletions(-)

Approvals:
  Tim Armstrong: Looks good to me, approved
  Lars Volker: Verified

--
To view, visit http://gerrit.cloudera.org:8080/10534
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I5662fba751067f6fed87d6180962fb1d31d872d0
Gerrit-Change-Number: 10534
Gerrit-PatchSet: 2
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Tim Armstrong

[native-toolchain-CR] Revert "Bump libunwind version to 1.3-rc1"

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10534 )

Change subject: Revert "Bump libunwind version to 1.3-rc1"
..


Patch Set 1: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/10534
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5662fba751067f6fed87d6180962fb1d31d872d0
Gerrit-Change-Number: 10534
Gerrit-PatchSet: 1
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Tue, 29 May 2018 22:15:53 +
Gerrit-HasComments: No

[native-toolchain-CR] Revert "Bump libunwind version to 1.3-rc1"

Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10534 )

Change subject: Revert "Bump libunwind version to 1.3-rc1"
..


Patch Set 1:

Previous change broke building the toolchain on CentOS 5.


--
To view, visit http://gerrit.cloudera.org:8080/10534
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5662fba751067f6fed87d6180962fb1d31d872d0
Gerrit-Change-Number: 10534
Gerrit-PatchSet: 1
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Tue, 29 May 2018 22:11:54 +
Gerrit-HasComments: No

[native-toolchain-CR] Revert "Bump libunwind version to 1.3-rc1"

Lars Volker has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/10534


Change subject: Revert "Bump libunwind version to 1.3-rc1"
..

Revert "Bump libunwind version to 1.3-rc1"

This reverts commit c33b6fa03f7f29152b475caeb820b0df952d9538.

Change-Id: I5662fba751067f6fed87d6180962fb1d31d872d0
---
M buildall.sh
M source/libunwind/build.sh
A 
source/libunwind/libunwind-1.1-patches/0001-Build-libunwind-1.1-on-ppc64le.patch
D 
source/libunwind/libunwind-1.3-rc1-patches/0001-libunwind-Use-syscall-directly-in-write_validate-to-avoid-ASAN.patch
D 
source/libunwind/libunwind-1.3-rc1-patches/0002-libunwind-trace-cache-destructor.patch
5 files changed, 11 insertions(+), 78 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/native-toolchain 
refs/changes/34/10534/1
--
To view, visit http://gerrit.cloudera.org:8080/10534
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5662fba751067f6fed87d6180962fb1d31d872d0
Gerrit-Change-Number: 10534
Gerrit-PatchSet: 1
Gerrit-Owner: Lars Volker

[Impala-ASF-CR] IMPALA-7089: xfail test kudu dml reporting

2018-05-29 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/10527 )

Change subject: IMPALA-7089: xfail test_kudu_dml_reporting
..

IMPALA-7089: xfail test_kudu_dml_reporting

test_kudu_dml_reporting has been causing a large number of build
failures. Temporarily disable it while we figure out what's going on.

Also improve output of test_kudu_dml_reporting on failure.

Change-Id: I222e4c86a50f2450201fbad8b937e8fcf4fac31d
Reviewed-on: http://gerrit.cloudera.org:8080/10527
Reviewed-by: Joe McDonnell 
Tested-by: Impala Public Jenkins 
---
M tests/shell/test_shell_commandline.py
1 file changed, 2 insertions(+), 1 deletion(-)

Approvals:
  Joe McDonnell: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/10527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I222e4c86a50f2450201fbad8b937e8fcf4fac31d
Gerrit-Change-Number: 10527
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Marshall 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell

[Impala-ASF-CR] IMPALA-7089: xfail test kudu dml reporting

2018-05-29 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10527 )

Change subject: IMPALA-7089: xfail test_kudu_dml_reporting
..


Patch Set 1: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/10527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I222e4c86a50f2450201fbad8b937e8fcf4fac31d
Gerrit-Change-Number: 10527
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Comment-Date: Tue, 29 May 2018 21:49:34 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7088: Fix uninitialized variable in cluster dataload

2018-05-29 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10526 )

Change subject: IMPALA-7088: Fix uninitialized variable in cluster dataload
..


Patch Set 2: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/10526
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5581a45460dc341842d77eaa09647e50f35be6c7
Gerrit-Change-Number: 10526
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Tue, 29 May 2018 21:34:01 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7088: Fix uninitialized variable in cluster dataload

2018-05-29 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/10526 )

Change subject: IMPALA-7088: Fix uninitialized variable in cluster dataload
..

IMPALA-7088: Fix uninitialized variable in cluster dataload

bin/load-data.py uses a unique directory for local Hive
execution to avoid a race condition when executing multiple
Hive commands at once. This unique directory is not needed
when loading on a real cluster. However, the code to remove
the unique directory at the end does not handle this
correctly.

This skips the code to remove the unique directory when
it is uninitialized.

Change-Id: I5581a45460dc341842d77eaa09647e50f35be6c7
Reviewed-on: http://gerrit.cloudera.org:8080/10526
Reviewed-by: Joe McDonnell 
Tested-by: Impala Public Jenkins 
---
M bin/load-data.py
1 file changed, 2 insertions(+), 1 deletion(-)

Approvals:
  Joe McDonnell: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/10526
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I5581a45460dc341842d77eaa09647e50f35be6c7
Gerrit-Change-Number: 10526
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger

[Impala-ASF-CR] IMPALA-7073: skip TestScannerReservation on non-miniclusters

2018-05-29 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10533 )

Change subject: IMPALA-7073: skip TestScannerReservation on non-miniclusters
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2565/


--
To view, visit http://gerrit.cloudera.org:8080/10533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7952f780cff80c08a6cbef898bf7b95c9bba5f6a
Gerrit-Change-Number: 10533
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Tue, 29 May 2018 21:31:11 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7073: skip TestScannerReservation on non-miniclusters

Thomas Marshall has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10533 )

Change subject: IMPALA-7073: skip TestScannerReservation on non-miniclusters
..


Patch Set 1: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/10533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7952f780cff80c08a6cbef898bf7b95c9bba5f6a
Gerrit-Change-Number: 10533
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Thomas Marshall 
Gerrit-Comment-Date: Tue, 29 May 2018 21:29:41 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7073: skip TestScannerReservation on non-miniclusters

Tim Armstrong has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/10533


Change subject: IMPALA-7073: skip TestScannerReservation on non-miniclusters
..

IMPALA-7073: skip TestScannerReservation on non-miniclusters

The test is (sort of) tuned for miniclusters and is very targeted
to testing a specific code path, rather than general functional
correctness, so we don't really need coverage on all filesystems.

Change-Id: I7952f780cff80c08a6cbef898bf7b95c9bba5f6a
---
M tests/query_test/test_scanners.py
1 file changed, 3 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/10533/1
--
To view, visit http://gerrit.cloudera.org:8080/10533
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I7952f780cff80c08a6cbef898bf7b95c9bba5f6a
Gerrit-Change-Number: 10533
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong

[Impala-ASF-CR] IMPALA-7090: Limit the size of expr created by EqualityDisjunctsToInRule

2018-05-29 Thread Vuk Ercegovac (Code Review)

Vuk Ercegovac has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10528 )

Change subject: IMPALA-7090: Limit the size of expr created by 
EqualityDisjunctsToInRule
..


Patch Set 1:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/10528/1/fe/src/main/java/org/apache/impala/rewrite/EqualityDisjunctsToInRule.java
File fe/src/main/java/org/apache/impala/rewrite/EqualityDisjunctsToInRule.java:

http://gerrit.cloudera.org:8080/#/c/10528/1/fe/src/main/java/org/apache/impala/rewrite/EqualityDisjunctsToInRule.java@67
PS1, Line 67:
: el
while here, pls fix this formatting.


http://gerrit.cloudera.org:8080/#/c/10528/1/fe/src/main/java/org/apache/impala/rewrite/EqualityDisjunctsToInRule.java@79
PS1, Line 79: newInList.add(otherPred.getChild(1));
if newInList.size() == Expr.EXPR_CHILDREN_LIMIT, this operation will exceed it, 
and we'll have the same error, right?

pls add a test for this edge case.


http://gerrit.cloudera.org:8080/#/c/10528/1/fe/src/main/java/org/apache/impala/rewrite/EqualityDisjunctsToInRule.java@82
PS1, Line 82: if (newInList.size() + otherPred.getChildren().size() > 
Expr.EXPR_CHILDREN_LIMIT)
pls add a jira, if we don't already have one, to back out of rewrites that 
produce an expression that fails analysis. ideally, we should be able to remove 
this specific guard (which is tested in analysis) with that support. it would 
be good to avoid duplicating checks that happen in analysis.


http://gerrit.cloudera.org:8080/#/c/10528/1/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java
File fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java:

http://gerrit.cloudera.org:8080/#/c/10528/1/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java@174
PS1, Line 174: a
nit: an


http://gerrit.cloudera.org:8080/#/c/10528/1/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java@178
PS1, Line 178: disjuct
nit: disjunct


http://gerrit.cloudera.org:8080/#/c/10528/1/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java@194
PS1, Line 194:   }
add a test to check a partial rewrite. for example, a where clause with an 
in-list of size EXPR_CHILDREN_LIMIT - 2 or'd with 10 eq predicates. if I 
understand it correctly, 2 of the or'd predicates will be merged into the 
in-list and 8 will be merged into another in-list, leaving a disjunction of two 
in-lists.



--
To view, visit http://gerrit.cloudera.org:8080/10528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie40c3210271a9e3c7f1b2b869a8c2ec8bacaa72a
Gerrit-Change-Number: 10528
Gerrit-PatchSet: 1
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Tue, 29 May 2018 21:18:56 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5522: Add support for authorized proxy groups

Philip Zeyliger has posted comments on this change. (
http://gerrit.cloudera.org:8080/10510 )

Change subject: IMPALA-5522: Add support for authorized proxy groups
..

Patch Set 2:

(5 comments)

My main concern is whether this forks when evaluating the groups.

http://gerrit.cloudera.org:8080/#/c/10510/2/be/src/service/frontend.cc
File be/src/service/frontend.cc:

http://gerrit.cloudera.org:8080/#/c/10510/2/be/src/service/frontend.cc@69
PS2, Line 69: "Specifies the delimiter used in
authorized_proxy_group_config. ");
I see that we're copying an older pattern, but this seems unfortunate. How do
other systems deal with this? Do they just disallow groups with commas in them?

Note that I didn't see a test for custom delimeters.

http://gerrit.cloudera.org:8080/#/c/10510/2/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/10510/2/be/src/service/impala-server.cc@319
PS2, Line 319: [](const string& config) {
: return Substitute("Invalid proxy user configuration.
No mapping value "
: "specified for the proxy user. For more
information review usage of the "
: "--authorized_proxy_user_config flag: $0",
config);
: });
Others can comment, but I don't think this pattern (passing a lambda to error
out with) adds anything as opposed to:

if (!AddAuthorizedProxyConfig(...).ok()) {
CLEAN_EXIT_WITH_ERROR("")
}

seems clearer and is consistent with other stuff nearby.

http://gerrit.cloudera.org:8080/#/c/10510/2/be/src/service/impala-server.cc@398
PS2, Line 398: void ImpalaServer::AddAuthorizedProxyConfig(
You're doing parsing here. I think it'd be lovely to see a C++ unit test that
checks the parsing explicitly.

http://gerrit.cloudera.org:8080/#/c/10510/2/fe/src/main/java/org/apache/impala/service/JniFrontend.java
File fe/src/main/java/org/apache/impala/service/JniFrontend.java:

http://gerrit.cloudera.org:8080/#/c/10510/2/fe/src/main/java/org/apache/impala/service/JniFrontend.java@623
PS2, Line 623: result.setGroups(GROUPS.getGroups(request.getUser()));
Does this cause a fork()/clone() or equivalent? We've seen problems with Impala
calling out to kinit, causing a fork(), causing failures because individual
impalad's can be quite big and run the system out of memory. Historically,
implementations of Hadoop's user-group-mapping stuff called out to "groups".

http://gerrit.cloudera.org:8080/#/c/10510/2/tests/authorization/test_authorization.py
File tests/authorization/test_authorization.py:

http://gerrit.cloudera.org:8080/#/c/10510/2/tests/authorization/test_authorization.py@193
PS2, Line 193: ','.join(get_groups()),
Please explain what this test is doing.

I think this test requires that a user be in at least one group. For line 193,
I don't think you need all the groups; you just need to be in one group.

Do we need a test that lets you in via user but not via groups, or vice-versa?
i.e., do the two settings work together reasonably?

--
To view, visit http://gerrit.cloudera.org:8080/10510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb
Gerrit-Change-Number: 10510
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Sailesh Mukil
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Tue, 29 May 2018 21:17:45 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6947: Kudu tests flaky due to rpc timeout

Michael Brown has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10466 )

Change subject: IMPALA-6947: Kudu tests flaky due to rpc timeout
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/10466/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/10466/1//COMMIT_MSG@18
PS1, Line 18: Testing:
: - Passed a full run of core tests.
Also, since this affects ASAN, I suggest trying this against an ASAN build.



--
To view, visit http://gerrit.cloudera.org:8080/10466
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c4a2d87934cd4eb98509512f7060a596894ed53
Gerrit-Change-Number: 10466
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall 
Gerrit-Reviewer: Michael Brown 
Gerrit-Comment-Date: Tue, 29 May 2018 20:33:58 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6947: Kudu tests flaky due to rpc timeout

Michael Brown has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10466 )

Change subject: IMPALA-6947: Kudu tests flaky due to rpc timeout
..


Patch Set 1:

(2 comments)

Please add another appropriate reviewer for the changes to the .cc files. It 
looks fine to me but someone more familiar should look as well.

http://gerrit.cloudera.org:8080/#/c/10466/1/be/src/common/global-flags.cc
File be/src/common/global-flags.cc:

http://gerrit.cloudera.org:8080/#/c/10466/1/be/src/common/global-flags.cc@162
PS1, Line 162: DEFINE_int32(kudu_rpc_timeout_ms, 0, "Timeout (milliseconds) set 
for Kudu rpcs. This "
 : "must be a positive value or it will be ignored and Kudu's 
default of 10s will be "
 : "used. There is no way to disable timeouts.");
 :
Is this an option that ought to be hidden away from users/customers? If yes, 
please make sure that in defining this it's properly hidden. Also, it's not 
clear to me on how this relates to kudu_operation_timeout_ms. Can you clean up 
the help text?


http://gerrit.cloudera.org:8080/#/c/10466/1/bin/run-all-tests.sh
File bin/run-all-tests.sh:

http://gerrit.cloudera.org:8080/#/c/10466/1/bin/run-all-tests.sh@84
PS1, Line 84: TEST_START_CLUSTER_ARGS="${TEST_START_CLUSTER_ARGS} 
--impalad_args=--kudu_rpc_timeout_ms=6"
Since this only happens with Kudu on ASAN, I think we only want to enable this 
if we are under those conditions.  A better place to do this is in 
start_impalad_instances() in bin/start-impala-cluster.py. You can use 
tests.common.environ specific_build_type_timeout() to tell you whether you're 
in ASAN or not and set a timeout appropriately. Do a git grep for 
specific_build_type_timeout and you'll see plenty of examples.



--
To view, visit http://gerrit.cloudera.org:8080/10466
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c4a2d87934cd4eb98509512f7060a596894ed53
Gerrit-Change-Number: 10466
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall 
Gerrit-Reviewer: Michael Brown 
Gerrit-Comment-Date: Tue, 29 May 2018 20:18:03 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6933: Avoids db name collisions for Kudu tests

2018-05-29 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10513 )

Change subject: IMPALA-6933: Avoids db name collisions for Kudu tests
..


Patch Set 2: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/10513
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7c2f8a35fec90ae0dabe80237d83954668b47f6e
Gerrit-Change-Number: 10513
Gerrit-PatchSet: 2
Gerrit-Owner: Vuk Ercegovac 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Tue, 29 May 2018 20:03:53 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6933: Avoids db name collisions for Kudu tests

2018-05-29 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/10513 )

Change subject: IMPALA-6933: Avoids db name collisions for Kudu tests
..

IMPALA-6933: Avoids db name collisions for Kudu tests

Kudu tests generate temporary db names in a way so that its
unlikely, yet possible to collide. A recent test failure
indicates such a collision came up. The fix changes the
way that the name is generated so that it includes the
classes name for which the db name is generated. This db name
will make it easier to see which test created it and the name
will not collide with other names generated by other tests.

Testing:
- ran the updated test locally

Change-Id: I7c2f8a35fec90ae0dabe80237d83954668b47f6e
Reviewed-on: http://gerrit.cloudera.org:8080/10513
Reviewed-by: Michael Brown 
Tested-by: Impala Public Jenkins 
---
M tests/common/kudu_test_suite.py
M tests/conftest.py
2 files changed, 29 insertions(+), 5 deletions(-)

Approvals:
  Michael Brown: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/10513
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I7c2f8a35fec90ae0dabe80237d83954668b47f6e
Gerrit-Change-Number: 10513
Gerrit-PatchSet: 3
Gerrit-Owner: Vuk Ercegovac 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Vuk Ercegovac

[Impala-ASF-CR] IMPALA-6338: Disable more flaky bloom filter tests

Thomas Marshall has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/10530


Change subject: IMPALA-6338: Disable more flaky bloom filter tests
..

IMPALA-6338: Disable more flaky bloom filter tests

Until IMPALA-6338 is fixed, temporarily disable tests that are
affected by it - any test that has a 'limit' and relies on the
contents of the runtime profile. This patch disables the runtime
profile check for all such tests in bloom_filter.test

Change-Id: Ifc9da892efa3b27d63056ad8e3befac82808ffdb
---
M testdata/workloads/functional-query/queries/QueryTest/bloom_filters.test
1 file changed, 12 insertions(+), 8 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/10530/1
--
To view, visit http://gerrit.cloudera.org:8080/10530
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifc9da892efa3b27d63056ad8e3befac82808ffdb
Gerrit-Change-Number: 10530
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall

[Impala-ASF-CR] IMPALA-7082: Show human readable size in query backend page

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10523 )

Change subject: IMPALA-7082: Show human readable size in query backend page
..


Patch Set 2: -Code-Review

(1 comment)

http://gerrit.cloudera.org:8080/#/c/10523/2/www/common-header.tmpl
File www/common-header.tmpl:

http://gerrit.cloudera.org:8080/#/c/10523/2/www/common-header.tmpl@25
PS2, Line 25: 
Sailesh is right, of course. We could include a new script ~here.



--
To view, visit http://gerrit.cloudera.org:8080/10523
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04afb4091bb8b6bc9dedfeceaf9284a8c65b16a1
Gerrit-Change-Number: 10523
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Comment-Date: Tue, 29 May 2018 19:52:12 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6990: TestClientSsl.test tls v12 failing due to Python SSL error

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10529 )

Change subject: IMPALA-6990: TestClientSsl.test_tls_v12 failing due to Python 
SSL error
..


Patch Set 2:

(2 comments)

I wrote a comment on the JIRA upstream.

I think the easier thing to do is to check Python version and skip if you're 
using a too-old Python, with an explanation. Possibly add a second copy of the 
test with the flag changes you suggest.

But, regardless, I still don't quite understand whether the test was just 
failing silently before or something fundamentally broke.

http://gerrit.cloudera.org:8080/#/c/10529/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/10529/2//COMMIT_MSG@42
PS2, Line 42: which does not force the use of specific ciphers, so we get away 
without
What doesn't force the user of ciphers?


http://gerrit.cloudera.org:8080/#/c/10529/2//COMMIT_MSG@49
PS2, Line 49: least unblock our builds while we can upgrade the AMIs for RHEL7. 
This
RHEL7 just doesn't ship a newer Python based on my quick check with Docker. I 
don't think this will get resolved this way.



--
To view, visit http://gerrit.cloudera.org:8080/10529
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I92c66ecaeb94b0c83ee6f1396c082709c21b3187
Gerrit-Change-Number: 10529
Gerrit-PatchSet: 2
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Tue, 29 May 2018 19:50:02 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6990: TestClientSsl.test tls v12 failing due to Python SSL error

Sailesh Mukil has uploaded a new patch set (#2). ( 
http://gerrit.cloudera.org:8080/10529 )

Change subject: IMPALA-6990: TestClientSsl.test_tls_v12 failing due to Python 
SSL error
..

IMPALA-6990: TestClientSsl.test_tls_v12 failing due to Python SSL error

When we upgraded to thrift-0.9.3, the TSSLSocket.py logic changed quite
a bit. Our RHEL7 machines come equipped with Python 2.7.5. Looking at
these comments, that means that we'll be unable to create a 'SSLContext'
but be able to explicitly specify ciphers:
https://github.com/apache/thrift/blob/master/lib/py/src/transport/TSSLSocket.py#L37-L41

# SSLContext is not available for Python < 2.7.9
_has_ssl_context = sys.hexversion >= 0x020709F0

# ciphers argument is not available for Python < 2.7.0
_has_ciphers = sys.hexversion >= 0x020700F0

If we cannot create a 'SSLContext', then we cannot use TLSv1.2 and have
to use TLSv1:
https://github.com/apache/thrift/blob/master/lib/py/src/transport/TSSLSocket.py#L48-L49

# For python >= 2.7.9, use latest TLS that both client and server
# supports.
# SSL 2.0 and 3.0 are disabled via ssl.OP_NO_SSLv2 and ssl.OP_NO_SSLv3.
# For python < 2.7.9, use TLS 1.0 since TLSv1_X nor OP_NO_SSLvX is
# unavailable.
_default_protocol = ssl.PROTOCOL_SSLv23 if _has_ssl_context else \
ssl.PROTOCOL_TLSv1

Our custom cluster test forces the server to use TLSv1.2 and also forces
a specific cipher:
https://github.com/apache/impala/blob/master/tests/custom_cluster/test_client_ssl.py#L118-L119

So this combination of configuration values causes a failure in RHEL7
because we only allow a specific cipher which works with TLSv1.2, but
the client cannot use TLSv1.2 due to the Python version as mentioned above.

On systems lower than RHEL7, the machines come equipped with Python 2.6.6,
which does not force the use of specific ciphers, so we get away without
a failure.

To fix this, we either need to change the Python version on RHEL 7 to
be >= Python 2.7.9, or reduce the 'test_client_ssl' limitation to run TLSv1.

The second option is the quickest, although not ideal, but it should at
least unblock our builds while we can upgrade the AMIs for RHEL7. This
patch does just that.

Change-Id: I92c66ecaeb94b0c83ee6f1396c082709c21b3187
---
M tests/custom_cluster/test_client_ssl.py
1 file changed, 1 insertion(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/10529/2
--
To view, visit http://gerrit.cloudera.org:8080/10529
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I92c66ecaeb94b0c83ee6f1396c082709c21b3187
Gerrit-Change-Number: 10529
Gerrit-PatchSet: 2
Gerrit-Owner: Sailesh Mukil

[Impala-ASF-CR] IMPALA-6990: TestClientSsl.test tls v12 failing due to Python SSL error

Sailesh Mukil has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/10529


Change subject: IMPALA-6990: TestClientSsl.test_tls_v12 failing due to Python 
SSL error
..

IMPALA-6990: TestClientSsl.test_tls_v12 failing due to Python SSL error

Spent some more time looking at this and found that 'requests' wasn't the 
culprit.

When we upgraded to thrift-0.9.3, the TSSLSocket.py logic changed quite a bit.
Our RHEL7 machines come equipped with Python 2.7.5. Looking at these comments,
that means that we'll be unable to create a 'SSLContext' but able to explicitly
specify ciphers:
https://github.com/apache/thrift/blob/master/lib/py/src/transport/TSSLSocket.py#L37-L41

# SSLContext is not available for Python < 2.7.9
_has_ssl_context = sys.hexversion >= 0x020709F0

# ciphers argument is not available for Python < 2.7.0
_has_ciphers = sys.hexversion >= 0x020700F0

If we cannot create a 'SSLContext', then we cannot use TLSv1.2 and have to use 
TLSv1:
https://github.com/apache/thrift/blob/master/lib/py/src/transport/TSSLSocket.py#L48-L49

# For python >= 2.7.9, use latest TLS that both client and server
# supports.
# SSL 2.0 and 3.0 are disabled via ssl.OP_NO_SSLv2 and ssl.OP_NO_SSLv3.
# For python < 2.7.9, use TLS 1.0 since TLSv1_X nor OP_NO_SSLvX is
# unavailable.
_default_protocol = ssl.PROTOCOL_SSLv23 if _has_ssl_context else \
ssl.PROTOCOL_TLSv1
Our custom cluster test forces the server to use TLSv1.2 and also forces a 
specific cipher:
https://github.com/apache/impala/blob/master/tests/custom_cluster/test_client_ssl.py#L118-L119

So this combination of configuration values causes a failure in RHEL7
because we only allow a specific cipher which works with TLSv1.2, but
the client cannot use TLSv1.2 due to the Python version as mentioned above.

On systems lower than RHEL7, the machines come equipped with Python 2.6.6,
which does not force the use of specific ciphers, so we get away without
a failure.

To fix this, we either need to change the Python version on RHEL 7 to
be >= Python 2.7.9, or reduce the 'test_client_ssl' limitation to run TLSv1.

The second option is the quickest, although not ideal, but it should at
least unblock our builds while we can upgrade the AMIs for RHEL7. This
patch does just that.

Change-Id: I92c66ecaeb94b0c83ee6f1396c082709c21b3187
---
M tests/custom_cluster/test_client_ssl.py
1 file changed, 1 insertion(+), 2 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/10529/1
--
To view, visit http://gerrit.cloudera.org:8080/10529
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I92c66ecaeb94b0c83ee6f1396c082709c21b3187
Gerrit-Change-Number: 10529
Gerrit-PatchSet: 1
Gerrit-Owner: Sailesh Mukil

[Impala-ASF-CR] IMPALA-7082: Show human readable size in query backend page

Sailesh Mukil has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10523 )

Change subject: IMPALA-7082: Show human readable size in query backend page
..


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/10523/2/www/common-header.tmpl
File www/common-header.tmpl:

http://gerrit.cloudera.org:8080/#/c/10523/2/www/common-header.tmpl@54
PS2, Line 54:

[Impala-ASF-CR] IMPALA-7090: Limit the size of expr created by EqualityDisjunctsToInRule

2018-05-29 Thread Tianyi Wang (Code Review)

Tianyi Wang has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/10528


Change subject: IMPALA-7090: Limit the size of expr created by 
EqualityDisjunctsToInRule
..

IMPALA-7090: Limit the size of expr created by EqualityDisjunctsToInRule

Currently EqualityDisjunctsToInRule introduced in IMPALA-5280 might
create an expr with unlimited number of children and fails a query,
which should be avoided. The easy solution is to not apply the
rewrite when the number of children is large.

Change-Id: Ie40c3210271a9e3c7f1b2b869a8c2ec8bacaa72a
---
M fe/src/main/java/org/apache/impala/rewrite/EqualityDisjunctsToInRule.java
M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java
2 files changed, 34 insertions(+), 7 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/28/10528/1
--
To view, visit http://gerrit.cloudera.org:8080/10528
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie40c3210271a9e3c7f1b2b869a8c2ec8bacaa72a
Gerrit-Change-Number: 10528
Gerrit-PatchSet: 1
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Vuk Ercegovac

[Impala-ASF-CR] IMPALA-7082: Show human readable size in query backend page

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10523 )

Change subject: IMPALA-7082: Show human readable size in query backend page
..


Patch Set 2: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/10523
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I04afb4091bb8b6bc9dedfeceaf9284a8c65b16a1
Gerrit-Change-Number: 10523
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Tue, 29 May 2018 19:08:17 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7089: xfail test kudu dml reporting

2018-05-29 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10527 )

Change subject: IMPALA-7089: xfail test_kudu_dml_reporting
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2563/


--
To view, visit http://gerrit.cloudera.org:8080/10527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I222e4c86a50f2450201fbad8b937e8fcf4fac31d
Gerrit-Change-Number: 10527
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Comment-Date: Tue, 29 May 2018 18:42:13 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7089: xfail test kudu dml reporting

Joe McDonnell has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10527 )

Change subject: IMPALA-7089: xfail test_kudu_dml_reporting
..


Patch Set 1: Code-Review+2

Makes sense to XFAIL for now since this is impacting so many builds and 
preventing other builds from running.


--
To view, visit http://gerrit.cloudera.org:8080/10527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I222e4c86a50f2450201fbad8b937e8fcf4fac31d
Gerrit-Change-Number: 10527
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Comment-Date: Tue, 29 May 2018 18:39:41 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6034: Add Cpu and scanned bytes limits per query

2018-05-29 Thread Bikramjeet Vig (Code Review)

Bikramjeet Vig has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10415 )

Change subject: IMPALA-6034: Add Cpu and scanned bytes limits per query
..


Patch Set 3:

(21 comments)

http://gerrit.cloudera.org:8080/#/c/10415/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/10415/3//COMMIT_MSG@10
PS3, Line 10: apply
nit: applied


http://gerrit.cloudera.org:8080/#/c/10415/3//COMMIT_MSG@14
PS3, Line 14: upated
updated


http://gerrit.cloudera.org:8080/#/c/10415/3//COMMIT_MSG@18
PS3, Line 18:
nit: extra whitespace


http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.h
File be/src/runtime/coordinator-backend-state.h:

http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.h@113
PS3, Line 113: fragments
nit: fragment


http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.h@200
PS3, Line 200: /// used by Coordinator::BackendState::AggregateBackendStats
maybe mention units. (same for cpu_sys_)


http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.h@201
PS3, Line 201: cpu_user_
nit: how about cpu_user_time_

(same for cpu_sys_)


http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.h@207
PS3, Line 207: BYTES_READ_COUNTERs in profile_
nit: Collection of BYTES_READ_COUNTERs of all the scan nodes in this fragment 
instance.


http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.cc
File be/src/runtime/coordinator-backend-state.cc:

http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.cc@218
PS3, Line 218:   resource_utilization->backend_scanned_bytes = 
resource_utilization_.backend_scanned_bytes;
long line


http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.cc@341
PS3, Line 341: AggregateBackendStats
do we need this call here? are any of the instance stats used in 
AggregateBackendStats updated in this method?


http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.cc@527
PS3, Line 527:   RuntimeProfile::Counter* profile_user_time_counter = 
profile_->GetCounter("TotalThreadsUserTime");
nit: long line


http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.cc@533
PS3, Line 533:   RuntimeProfile::Counter* profile_system_time_counter = 
profile_->GetCounter("TotalThreadsSysTime");
nit: long line


http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.cc@638
PS3, Line 638:   "peak_mem_consumption", 
resource_utilization_.peak_consumption, document->GetAllocator());
nit:long line


http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator.h
File be/src/runtime/coordinator.h:

http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator.h@173
PS3, Line 173: peak_consumption
nit: how about peak_mem_consumption


http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator.h@176
PS3, Line 176: backend_
nit: we can probably get rid of this prefix now that all these counters are 
under a struct named "BackendResourceUtilization"


http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator.h@179
PS3, Line 179: cpu_user
nit: cpu_user_time_
(same below)


http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator.h@185
PS3, Line 185: Aggregate CPU and bytes read metrics
update comment: Aggregate CPU, scanned bytes and peak memory consumption 
metrics across all backends


http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:

http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator.cc@725
PS3, Line 725: info
unused variable


http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator.cc@732
PS3, Line 732: peak_memory = backend_resource_utilization.peak_consumption;
 : backend_user_cpu = 
backend_resource_utilization.backend_cpu_user;
 : backend_sys_cpu = 
backend_resource_utilization.backend_cpu_sys;
 : backend_scanned_bytes = 
backend_resource_utilization.backend_scanned_bytes;
nit: can we directly use the values in the struct?
If the objective of doing this was to make the names of the variables shorter 
for readability, perhaps we can shorten "backend_resource_utilization" to  just 
"resource_usage"


http://gerrit.cloudera.org:8080/#/c/10415/1/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/10415/1/be/src/service/impala-server.cc@1015
PS1, Line 1015: _QUERY << "Qu
> maybe have a named constant for this or a default constructor specifically
can you address this too?


http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

[Impala-ASF-CR] IMPALA-7089: xfail test kudu dml reporting

Thomas Marshall has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/10527


Change subject: IMPALA-7089: xfail test_kudu_dml_reporting
..

IMPALA-7089: xfail test_kudu_dml_reporting

test_kudu_dml_reporting has been causing a large number of build
failures. Temporarily disable it while we figure out what's going on.

Also improve output of test_kudu_dml_reporting on failure.

Change-Id: I222e4c86a50f2450201fbad8b937e8fcf4fac31d
---
M tests/shell/test_shell_commandline.py
1 file changed, 2 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/10527/1
--
To view, visit http://gerrit.cloudera.org:8080/10527
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I222e4c86a50f2450201fbad8b937e8fcf4fac31d
Gerrit-Change-Number: 10527
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Marshall

[Impala-ASF-CR] IMPALA-7088: Fix uninitialized variable in cluster dataload

2018-05-29 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10526 )

Change subject: IMPALA-7088: Fix uninitialized variable in cluster dataload
..


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2562/


--
To view, visit http://gerrit.cloudera.org:8080/10526
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5581a45460dc341842d77eaa09647e50f35be6c7
Gerrit-Change-Number: 10526
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Tue, 29 May 2018 18:09:01 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7088: Fix uninitialized variable in cluster dataload

Joe McDonnell has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10526 )

Change subject: IMPALA-7088: Fix uninitialized variable in cluster dataload
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/10526/1/bin/load-data.py
File bin/load-data.py:

http://gerrit.cloudera.org:8080/#/c/10526/1/bin/load-data.py@170
PS1, Line 170:   if (unique_dir):
> python nit: you don't need the parens here.
Good point.



--
To view, visit http://gerrit.cloudera.org:8080/10526
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5581a45460dc341842d77eaa09647e50f35be6c7
Gerrit-Change-Number: 10526
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Tue, 29 May 2018 17:54:11 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7088: Fix uninitialized variable in cluster dataload

Hello Philip Zeyliger,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/10526

to look at the new patch set (#2).

Change subject: IMPALA-7088: Fix uninitialized variable in cluster dataload
..

IMPALA-7088: Fix uninitialized variable in cluster dataload

bin/load-data.py uses a unique directory for local Hive
execution to avoid a race condition when executing multiple
Hive commands at once. This unique directory is not needed
when loading on a real cluster. However, the code to remove
the unique directory at the end does not handle this
correctly.

This skips the code to remove the unique directory when
it is uninitialized.

Change-Id: I5581a45460dc341842d77eaa09647e50f35be6c7
---
M bin/load-data.py
1 file changed, 2 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/10526/2
--
To view, visit http://gerrit.cloudera.org:8080/10526
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5581a45460dc341842d77eaa09647e50f35be6c7
Gerrit-Change-Number: 10526
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger

[Impala-ASF-CR] IMPALA-7088: Fix uninitialized variable in cluster dataload

Joe McDonnell has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10526 )

Change subject: IMPALA-7088: Fix uninitialized variable in cluster dataload
..


Patch Set 2: Code-Review+2

Carry +2


--
To view, visit http://gerrit.cloudera.org:8080/10526
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5581a45460dc341842d77eaa09647e50f35be6c7
Gerrit-Change-Number: 10526
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Tue, 29 May 2018 17:54:28 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7088: Fix uninitialized variable in cluster dataload

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10526 )

Change subject: IMPALA-7088: Fix uninitialized variable in cluster dataload
..


Patch Set 1: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/10526/1/bin/load-data.py
File bin/load-data.py:

http://gerrit.cloudera.org:8080/#/c/10526/1/bin/load-data.py@170
PS1, Line 170:   if (unique_dir):
python nit: you don't need the parens here.



--
To view, visit http://gerrit.cloudera.org:8080/10526
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5581a45460dc341842d77eaa09647e50f35be6c7
Gerrit-Change-Number: 10526
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Tue, 29 May 2018 17:44:41 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7088: Fix uninitialized variable in cluster dataload

Joe McDonnell has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/10526


Change subject: IMPALA-7088: Fix uninitialized variable in cluster dataload
..

IMPALA-7088: Fix uninitialized variable in cluster dataload

bin/load-data.py uses a unique directory for local Hive
execution to avoid a race condition when executing multiple
Hive commands at once. This unique directory is not needed
when loading on a real cluster. However, the code to remove
the unique directory at the end does not handle this
correctly.

This skips the code to remove the unique directory when
it is uninitialized.

Change-Id: I5581a45460dc341842d77eaa09647e50f35be6c7
---
M bin/load-data.py
1 file changed, 2 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/10526/1
--
To view, visit http://gerrit.cloudera.org:8080/10526
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5581a45460dc341842d77eaa09647e50f35be6c7
Gerrit-Change-Number: 10526
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell

[Impala-ASF-CR] IMPALA-6714: [DOCS] ORC file format support

Michael Brown has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10525 )

Change subject: IMPALA-6714: [DOCS] ORC file format support
..


Patch Set 1:

Alex Rodoni is currently most familiar with docs so making them a reviewer. 
Thanks.


--
To view, visit http://gerrit.cloudera.org:8080/10525
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib1ee23ed844653c274babdce5a332dbe5c79b630
Gerrit-Change-Number: 10525
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Balazs Jeszenszky 
Gerrit-Reviewer: Michael Brown 
Gerrit-Comment-Date: Tue, 29 May 2018 17:05:28 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5642: [DOCS] An additional restriction for Hive/Java UDFs

2018-05-29 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/10520 )

Change subject: IMPALA-5642: [DOCS] An additional restriction for Hive/Java UDFs
..

IMPALA-5642: [DOCS] An additional restriction for Hive/Java UDFs

Change-Id: I79f5fcbb570fda48f9ac03f6c3760366aa1859d2
Reviewed-on: http://gerrit.cloudera.org:8080/10520
Reviewed-by: Bharath Vissapragada 
Tested-by: Impala Public Jenkins 
---
M docs/topics/impala_udf.xml
1 file changed, 5 insertions(+), 0 deletions(-)

Approvals:
  Bharath Vissapragada: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/10520
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I79f5fcbb570fda48f9ac03f6c3760366aa1859d2
Gerrit-Change-Number: 10520
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-5642: [DOCS] An additional restriction for Hive/Java UDFs

2018-05-29 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10520 )

Change subject: IMPALA-5642: [DOCS] An additional restriction for Hive/Java UDFs
..


Patch Set 1: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/10520
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79f5fcbb570fda48f9ac03f6c3760366aa1859d2
Gerrit-Change-Number: 10520
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Tue, 29 May 2018 16:53:39 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5642: [DOCS] An additional restriction for Hive/Java UDFs

2018-05-29 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10520 )

Change subject: IMPALA-5642: [DOCS] An additional restriction for Hive/Java UDFs
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-docs-submit/299/


--
To view, visit http://gerrit.cloudera.org:8080/10520
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79f5fcbb570fda48f9ac03f6c3760366aa1859d2
Gerrit-Change-Number: 10520
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Tue, 29 May 2018 16:37:24 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5642: [DOCS] An additional restriction for Hive/Java UDFs

2018-05-29 Thread Alex Rodoni (Code Review)

Alex Rodoni has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10520 )

Change subject: IMPALA-5642: [DOCS] An additional restriction for Hive/Java UDFs
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/10520/1/docs/topics/impala_udf.xml
File docs/topics/impala_udf.xml:

http://gerrit.cloudera.org:8080/#/c/10520/1/docs/topics/impala_udf.xml@188
PS1, Line 188: Hive/Java UDFs
> I guess this part is kinda obvious from the section heading (that we are re
To avoid having to do a new +2, I will keep this as it is for now. Thank you!



--
To view, visit http://gerrit.cloudera.org:8080/10520
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I79f5fcbb570fda48f9ac03f6c3760366aa1859d2
Gerrit-Change-Number: 10520
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Tue, 29 May 2018 16:36:30 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6933: Avoids db name collisions for Kudu tests

2018-05-29 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/10513 )

Change subject: IMPALA-6933: Avoids db name collisions for Kudu tests
..


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2561/


--
To view, visit http://gerrit.cloudera.org:8080/10513
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7c2f8a35fec90ae0dabe80237d83954668b47f6e
Gerrit-Change-Number: 10513
Gerrit-PatchSet: 2
Gerrit-Owner: Vuk Ercegovac 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Tue, 29 May 2018 16:34:03 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6933: Avoids db name collisions for Kudu tests