[Impala-ASF-CR] IMPALA-5427: Fix race between CRS::UpdateQueryStatus() and beeswax RPCs

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged.

Change subject: IMPALA-5427: Fix race between CRS::UpdateQueryStatus() and 
beeswax RPCs
..


IMPALA-5427: Fix race between CRS::UpdateQueryStatus() and beeswax RPCs

There was a race between ClientRequestState::UpdateQueryStatus() and the
beeswax get_state()/get_log() RPCs leading to the rare situation that a
query would abort with an error, but the error message would be empty.

The fix is to take the ClientRequestState lock in the beeswax RPCs
before obtaining the status.

To test this I ran test_corrupt_files in a loop for a day. Without this
fix, it would usually fail within a few hours.

I changed the test to allow running it in parallel like so:

@pytest.mark.parametrize('multiplier', xrange(32))
def test_corrupt_files(self, vector, multiplier):

Then I ran it in a loop like so:

i=0; while [ $? -eq 0 ]; do ((++i)); echo "Run: $i"; impala-py.test \
tests/query_test/test_scanners.py::TestParquet::test_corrupt_files \
--exploration_strategy=exhaustive -n8; done

Change-Id: Ib4494fe3f933cc23841db0e7da407eec5650f2b5
Reviewed-on: http://gerrit.cloudera.org:8080/7155
Reviewed-by: Lars Volker 
Tested-by: Impala Public Jenkins
---
M be/src/service/impala-beeswax-server.cc
1 file changed, 28 insertions(+), 19 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Lars Volker: Looks good to me, approved



-- 
To view, visit http://gerrit.cloudera.org:8080/7155
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib4494fe3f933cc23841db0e7da407eec5650f2b5
Gerrit-PatchSet: 15
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Sailesh Mukil 

[Impala-ASF-CR] IMPALA-5427: Fix race between CRS::UpdateQueryStatus() and beeswax RPCs

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5427: Fix race between CRS::UpdateQueryStatus() and 
beeswax RPCs
..


Patch Set 14: Verified+1

-- 
To view, visit http://gerrit.cloudera.org:8080/7155
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4494fe3f933cc23841db0e7da407eec5650f2b5
Gerrit-PatchSet: 14
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5513: Fix display message exception when using invalid KEYVAL

[Donghui Xu (Code Review)]

Donghui Xu has posted comments on this change.

Change subject: IMPALA-5513: Fix display message exception when using invalid 
KEYVAL
..


Patch Set 4:

(2 comments)

I have modified the code according to your opinion.
I think your advice is fine, but how do i know who is familiar with that file?

http://gerrit.cloudera.org:8080/#/c/7187/2//COMMIT_MSG
Commit Message:

PS2, Line 9: 
> Please do not end lines in spaces.
Done


PS2, Line 13:  print_to_stder
> Functions are usually written "print_to_stderr()" - preserving capitalizati
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/7187
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ib883499a88f39d91b69bea4291f1ce5dd264ccf6
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Donghui Xu 
Gerrit-Reviewer: Donghui Xu 
Gerrit-Reviewer: Jim Apple 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5513: Fix display message exception when using invalid KEYVAL

[Donghui Xu (Code Review)]

Donghui Xu has uploaded a new patch set (#4).

Change subject: IMPALA-5513: Fix display message exception when using invalid 
KEYVAL
..

IMPALA-5513: Fix display message exception when using invalid KEYVAL

Function Print_to_stderr has a syntax error when an error message is
 displayed.

I solved this problem by exchanging the position of variable and the
 subsequent strings in function print_to_stderr().

Change-Id: Ib883499a88f39d91b69bea4291f1ce5dd264ccf6
---
M shell/impala_shell.py
1 file changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/7187/4
-- 
To view, visit http://gerrit.cloudera.org:8080/7187
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib883499a88f39d91b69bea4291f1ce5dd264ccf6
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Donghui Xu 
Gerrit-Reviewer: Donghui Xu 
Gerrit-Reviewer: Jim Apple 

[Impala-ASF-CR] IMPALA-5513: Fix display message exception when using invalid KEYVAL

[Donghui Xu (Code Review)]

Donghui Xu has uploaded a new patch set (#3).

Change subject: IMPALA-5513: Fix display message exception when using invalid 
KEYVAL
..

IMPALA-5513: Fix display message exception when using invalid KEYVAL

Function Print_to_stderr has a syntax error when an error message is
 displayed.

I solved this problem by exchanging the position of variable and the 
subsequent strings in function print_to_stderr().

Change-Id: Ib883499a88f39d91b69bea4291f1ce5dd264ccf6
---
M shell/impala_shell.py
1 file changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/7187/3
-- 
To view, visit http://gerrit.cloudera.org:8080/7187
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib883499a88f39d91b69bea4291f1ce5dd264ccf6
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Donghui Xu 
Gerrit-Reviewer: Donghui Xu 
Gerrit-Reviewer: Jim Apple 

[Impala-ASF-CR] IMPALA-5507: Add clear description to help information of KEYVAL option

[Donghui Xu (Code Review)]

Donghui Xu has posted comments on this change.

Change subject: IMPALA-5507: Add clear description to help information of 
KEYVAL option
..


Patch Set 4:

(3 comments)

I have modified the code according to your opinion, please check again.
Thank you.

http://gerrit.cloudera.org:8080/#/c/7179/3/shell/option_parser.py
File shell/option_parser.py:

PS3, Line 88: rea
> "will"
Done


PS3, Line 162: follow th
> What exactly do you mean by character - do you mean a-z and A-Z?
Done


PS3, Line 163: KEY has 
> What contains characters and/or numbers - KEY? VALUE?
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/7179
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I68cfc16838c6c0e7813f03dd4296f9eb54ec4c63
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Donghui Xu 
Gerrit-Reviewer: Donghui Xu 
Gerrit-Reviewer: Jim Apple 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5507: Add clear description to help information of KEYVAL option

[Donghui Xu (Code Review)]

Donghui Xu has uploaded a new patch set (#4).

Change subject: IMPALA-5507: Add clear description to help information of 
KEYVAL option
..

IMPALA-5507: Add clear description to help information of KEYVAL option

Help information of KEYVAL option in impala-shell is not clear enough.

I fix this issue by adding clear description to help information of
KEYVAL option.

Change-Id: I68cfc16838c6c0e7813f03dd4296f9eb54ec4c63
---
M shell/option_parser.py
1 file changed, 7 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/7179/4
-- 
To view, visit http://gerrit.cloudera.org:8080/7179
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I68cfc16838c6c0e7813f03dd4296f9eb54ec4c63
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Donghui Xu 
Gerrit-Reviewer: Donghui Xu 
Gerrit-Reviewer: Jim Apple 

[Impala-ASF-CR] IMPALA-2636: HS2 GetTables() returns TABLE TYPE as TABLE for VIEW

[sandeep akinapelli (Code Review)]

sandeep akinapelli has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/7353

Change subject: IMPALA-2636: HS2 GetTables() returns TABLE_TYPE as TABLE for 
VIEW
..

IMPALA-2636: HS2 GetTables() returns TABLE_TYPE as TABLE for VIEW

Added code to read the table type from metastore table but defaults to
"TABLE" if the metastore table is not loaded.
After the change, GetTabletypes also returns "VIEW" apart from "TABLE"
Changed unit and jdbc testcases for GetTableTypes.
Added new Frontend test for reading views.

Change-Id: I90616388e6181cf342b3de389af940214ed46428
---
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/service/FrontendTest.java
M fe/src/test/java/org/apache/impala/service/JdbcTest.java
4 files changed, 74 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/7353/1
-- 
To view, visit http://gerrit.cloudera.org:8080/7353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I90616388e6181cf342b3de389af940214ed46428
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: sandeep akinapelli 

[Impala-ASF-CR] IMPALA-5427: Fix race between CRS::UpdateQueryStatus() and beeswax RPCs

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5427: Fix race between CRS::UpdateQueryStatus() and 
beeswax RPCs
..


Patch Set 14:

Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/821/

-- 
To view, visit http://gerrit.cloudera.org:8080/7155
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4494fe3f933cc23841db0e7da407eec5650f2b5
Gerrit-PatchSet: 14
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5427: Fix race between CRS::UpdateQueryStatus() and beeswax RPCs

[Lars Volker (Code Review)]

Lars Volker has posted comments on this change.

Change subject: IMPALA-5427: Fix race between CRS::UpdateQueryStatus() and 
beeswax RPCs
..


Patch Set 14: Code-Review+2

PS14 removes a newline in case of empty coordinator error log. Carrying Dan's 
+2 again.

-- 
To view, visit http://gerrit.cloudera.org:8080/7155
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4494fe3f933cc23841db0e7da407eec5650f2b5
Gerrit-PatchSet: 14
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5427: Fix race between CRS::UpdateQueryStatus() and beeswax RPCs

[Lars Volker (Code Review)]

Hello Impala Public Jenkins, Dan Hecht,

I'd like you to reexamine a change.  Please visit

http://gerrit.cloudera.org:8080/7155

to look at the new patch set (#14).

Change subject: IMPALA-5427: Fix race between CRS::UpdateQueryStatus() and 
beeswax RPCs
..

IMPALA-5427: Fix race between CRS::UpdateQueryStatus() and beeswax RPCs

There was a race between ClientRequestState::UpdateQueryStatus() and the
beeswax get_state()/get_log() RPCs leading to the rare situation that a
query would abort with an error, but the error message would be empty.

The fix is to take the ClientRequestState lock in the beeswax RPCs
before obtaining the status.

To test this I ran test_corrupt_files in a loop for a day. Without this
fix, it would usually fail within a few hours.

I changed the test to allow running it in parallel like so:

@pytest.mark.parametrize('multiplier', xrange(32))
def test_corrupt_files(self, vector, multiplier):

Then I ran it in a loop like so:

i=0; while [ $? -eq 0 ]; do ((++i)); echo "Run: $i"; impala-py.test \
tests/query_test/test_scanners.py::TestParquet::test_corrupt_files \
--exploration_strategy=exhaustive -n8; done

Change-Id: Ib4494fe3f933cc23841db0e7da407eec5650f2b5
---
M be/src/service/impala-beeswax-server.cc
1 file changed, 28 insertions(+), 19 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/7155/14
-- 
To view, visit http://gerrit.cloudera.org:8080/7155
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4494fe3f933cc23841db0e7da407eec5650f2b5
Gerrit-PatchSet: 14
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Sailesh Mukil 

[Impala-ASF-CR] IMPALA-5513: Fix display message exception when using invalid KEYVAL

[Jim Apple (Code Review)]

Jim Apple has posted comments on this change.

Change subject: IMPALA-5513: Fix display message exception when using invalid 
KEYVAL
..


Patch Set 2:

(2 comments)

Thank you for your patch.

You might get faster reviews in the future if you pick a person who has worked 
on the file in question and add them as a reviewer here in Gerrit.

http://gerrit.cloudera.org:8080/#/c/7187/2//COMMIT_MSG
Commit Message:

PS2, Line 9:  
Please do not end lines in spaces.


PS2, Line 13: Print_to_stderr
Functions are usually written "print_to_stderr()" - preserving capitalization 
and adding parens.


-- 
To view, visit http://gerrit.cloudera.org:8080/7187
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ib883499a88f39d91b69bea4291f1ce5dd264ccf6
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Donghui Xu 
Gerrit-Reviewer: Donghui Xu 
Gerrit-Reviewer: Jim Apple 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5507: Add clear description to help information of KEYVAL option

[Jim Apple (Code Review)]

Jim Apple has posted comments on this change.

Change subject: IMPALA-5507: Add clear description to help information of 
KEYVAL option
..


Patch Set 3:

(4 comments)

Thank you for sending this patch. A few comments:

http://gerrit.cloudera.org:8080/#/c/7179/3/shell/option_parser.py
File shell/option_parser.py:

PS3, Line 88: may
"will"


PS3, Line 88: stdin
Add back comment about ctrl+d?


PS3, Line 162: character
What exactly do you mean by character - do you mean a-z and A-Z?


PS3, Line 163: contains
What contains characters and/or numbers - KEY? VALUE?


-- 
To view, visit http://gerrit.cloudera.org:8080/7179
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I68cfc16838c6c0e7813f03dd4296f9eb54ec4c63
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Donghui Xu 
Gerrit-Reviewer: Donghui Xu 
Gerrit-Reviewer: Jim Apple 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5036: Parquet count star optimization

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-5036: Parquet count star optimization
..


Patch Set 9: Verified-1

Build failed: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/820/

-- 
To view, visit http://gerrit.cloudera.org:8080/6812
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I536b85c014821296aed68a0c68faadae96005e62
Gerrit-PatchSet: 9
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Marcel Kornacker 
Gerrit-Reviewer: Mostafa Mokhtar 
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-Reviewer: Zach Amsden 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5514: Add the joint judgment of ldap password cmd and ldap

[Donghui Xu (Code Review)]

Donghui Xu has posted comments on this change.

Change subject: IMPALA-5514: Add the joint judgment of ldap_password_cmd and 
ldap
..


Patch Set 2:

Can anyone review this issue please? Thanks.

-- 
To view, visit http://gerrit.cloudera.org:8080/7188
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I3711d8a0eca2fa8612e2943fa9121945db6b012e
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Donghui Xu 
Gerrit-Reviewer: Donghui Xu 
Gerrit-HasComments: No