[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[Jim Apple (Code Review)]

Jim Apple has submitted this change and it was merged.

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..


IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

The current version of pytest in the Impala python environment is
quite old (2.7.2) and there have been bug fixes in later versions
that we could benefit from.

Also, since the passing of params to pytest.main() as a string will
be deprecated in upcoming versions of pytest, edit run-tests.py to
instead pass params as a list. (This also means we don't need to
worry about esoteric bash limitations re: single quotes in strings.)

While working on this file, the filtering of commandline args when
running the verfier tests was made a little more robust.

Tested by doing a standard (non-exhaustive) test run on centos 6.4
and ubuntu 14.04, plus an exhaustive test run on RHEL7.

Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Reviewed-on: http://gerrit.cloudera.org:8080/5640
Tested-by: Impala Public Jenkins
Reviewed-by: Jim Apple 
---
M infra/python/deps/requirements.txt
M tests/run-tests.py
2 files changed, 82 insertions(+), 67 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Jim Apple: Looks good to me, approved



-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 10
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[Jim Apple (Code Review)]

Jim Apple has posted comments on this change.

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..


Patch Set 9: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 9
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..


Patch Set 9: Verified+1

-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 9
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change.

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..


Patch Set 9:

Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/228/

-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 9
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[Jim Apple (Code Review)]

Jim Apple has posted comments on this change.

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..


Patch Set 8:

David, do you want me to rebase and commit this?

-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[Jim Apple (Code Review)]

Jim Apple has posted comments on this change.

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..


Patch Set 8: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[Michael Brown (Code Review)]

Michael Brown has posted comments on this change.

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..


Patch Set 8: Code-Review+1

-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[David Knupp (Code Review)]

David Knupp has posted comments on this change.

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..


Patch Set 7:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/5640/7/tests/run-tests.py
File tests/run-tests.py:

PS7, Line 144: other_tests
> Maybe use the name explicit_tests ?
Done. (And, with hindsight, changed 'params' to 'config_options'.)


-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[David Knupp (Code Review)]

Hello Lars Volker,

I'd like you to reexamine a change.  Please visit

http://gerrit.cloudera.org:8080/5640

to look at the new patch set (#8).

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..

IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

The current version of pytest in the Impala python environment is
quite old (2.7.2) and there have been bug fixes in later versions
that we could benefit from.

Also, since the passing of params to pytest.main() as a string will
be deprecated in upcoming versions of pytest, edit run-tests.py to
instead pass params as a list. (This also means we don't need to
worry about esoteric bash limitations re: single quotes in strings.)

While working on this file, the filtering of commandline args when
running the verfier tests was made a little more robust.

Tested by doing a standard (non-exhaustive) test run on centos 6.4
and ubuntu 14.04, plus an exhaustive test run on RHEL7.

Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
---
M infra/python/deps/requirements.txt
M tests/run-tests.py
2 files changed, 82 insertions(+), 67 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/5640/8
-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[Michael Brown (Code Review)]

Michael Brown has posted comments on this change.

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..


Patch Set 7:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/5640/7/tests/run-tests.py
File tests/run-tests.py:

PS7, Line 144: other_tests
Maybe use the name explicit_tests ?


-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[David Knupp (Code Review)]

David Knupp has posted comments on this change.

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..


Patch Set 6:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/5640/6/tests/run-tests.py
File tests/run-tests.py:

PS6, Line 151: 
> Why did you get rid of this string replacement?
Because the primary goal of this change is to pass the args as a list, not as 
one big monolithic string. We only needed to do this weird string replacement 
because we were building that one long string.


PS6, Line 102: https://issues.cloudera.org/browse/IMPALA-4510
> Just say IMPALA-4510 so that when the Jira site is moved from Cloudera to A
Done


PS6, Line 119: logging_args += (arg, os.path.join(RESULT_DIR, 
log.format(base_name)))
> I think it would be more readable to use logging_args.extend() in this case
That's fair. I think I decided against .extend([...]) simply to reduce the 
visual clutter to deeply nested parens/brackets.


PS6, Line 125: verifcation
> verification
Done


PS6, Line 128: for arg in commandline_args:
 :   try:
 : pytest.config.getvalue(arg.strip('-'))  # Raises 
ValueError if invalid arg
 : kept_args += [arg, str(commandline_args.next())]
> Cheeky. Are there any py.test config options that don't take arguments? If 
Note that I didn't actually change the existing logic here, I just changed the 
name of 'raw_args' to the more descriptive 'commandline_args'.

That said, this seems like a valid point. I poked around in pytest's source 
code for a while, and it appears that any tests modules, classes, and functions 
explicitly called out on the command line can be retrieved by calling 
pytest.config.getoption('file_or_dir'). We can use this fact to know which 
items to remove from from commandline_args.


-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[Michael Brown (Code Review)]

Michael Brown has posted comments on this change.

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..


Patch Set 6:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/5640/6/tests/run-tests.py
File tests/run-tests.py:

PS6, Line 151: 
Why did you get rid of this string replacement?


PS6, Line 102: https://issues.cloudera.org/browse/IMPALA-4510
Just say IMPALA-4510 so that when the Jira site is moved from Cloudera to ASF, 
the comment is still valid.


PS6, Line 119: logging_args += (arg, os.path.join(RESULT_DIR, 
log.format(base_name)))
I think it would be more readable to use logging_args.extend() in this case. I 
spent time in this review trying to see if you unpacked what I assumed were 
tuples, because I thought you were apending these as tuples.


PS6, Line 125: verifcation
verification


PS6, Line 128: for arg in commandline_args:
 :   try:
 : pytest.config.getvalue(arg.strip('-'))  # Raises 
ValueError if invalid arg
 : kept_args += [arg, str(commandline_args.next())]
Cheeky. Are there any py.test config options that don't take arguments? If yes, 
could that result in:

If you have py.test --foo --bar 1,

Then in the first iteration:
arg will be foo, commandline_args.next() will be bar

In the second iteration:
arg will be 1.  You'll do a getvalue() on it, raise ValueError, and ultimately 
1 won't be applied to bar.


-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[David Knupp (Code Review)]

David Knupp has posted comments on this change.

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..


Patch Set 6:

Started http://jenkins.impala.io:8080/view/Utility/job/pre-review-test/10/

-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[David Knupp (Code Review)]

Hello Lars Volker,

I'd like you to reexamine a change.  Please visit

http://gerrit.cloudera.org:8080/5640

to look at the new patch set (#6).

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..

IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

The current version of pytest in the Impala python environment is
quite old (2.7.2) and there have been bug fixes in later versions
that we could benefit from.

Also, since the passing of params to pytest.main() as a string will
be deprecated in upcoming versions of pytest, edit run-tests.py to
instead pass params as a list. (This also means we don't need to
worry about esoteric bash limitations re: single quotes in strings.)

Tested by doing a standard (non-exhaustive) test run on centos 6.4
and ubuntu 14.04, plus an exhaustive test run on RHEL7.

Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
---
M infra/python/deps/requirements.txt
M tests/run-tests.py
2 files changed, 67 insertions(+), 63 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/5640/6
-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[David Knupp (Code Review)]

David Knupp has posted comments on this change.

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/5640/3//COMMIT_MSG
Commit Message:

Line 13: In addition to bumping the version of pytest (and related modules),
> I think that would be good. In my experience; formatting or renaming change
https://gerrit.cloudera.org/#/c/5794/ has been submitted for review.


-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[Jim Apple (Code Review)]

Jim Apple has posted comments on this change.

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..


Patch Set 5:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/5640/3//COMMIT_MSG
Commit Message:

Line 13: In addition to bumping the version of pytest (and related modules),
> I can do that if you feel strongly about it.
I think that would be good. In my experience; formatting or renaming changes 
that touch a large number of files or lines can obscure the tricky bits of more 
substantive changes when they are bundled.

I'd expect the renames could be +2ed easily and so I'd suggest that one first 
before the pytest upgrade.


Line 24: sed -i 's/TestMatrix/ImpalaTestMatrix/g'
> OK, so this seemingly innocuous question opened up (what seems to me) a rea
Filed https://issues.cloudera.org/browse/IMPALA-4812 and 
https://issues.cloudera.org/browse/IMPALA-4818.

Can you comment out that one test and then verify that the "core" exploration 
strategy collects the same tests?

Don't worry about "pairwise"; as far as I can tell that strategy is broken and 
unused.


-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[David Knupp (Code Review)]

David Knupp has posted comments on this change.

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..


Patch Set 5:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/5640/3/tests/common/kudu_test_suite.py
File tests/common/kudu_test_suite.py:

Line 61: cls.ImpalaTestMatrix.add_dimension(
> long line
Done


http://gerrit.cloudera.org:8080/#/c/5640/3/tests/custom_cluster/test_permanent_udfs.py
File tests/custom_cluster/test_permanent_udfs.py:

Line 48: cls.ImpalaTestMatrix.add_dimension(
> long line
Done


http://gerrit.cloudera.org:8080/#/c/5640/3/tests/experiments/test_targeted_perf.py
File tests/experiments/test_targeted_perf.py:

Line 33: cls.TestMatrix.add_constraint(
> long line, here and elsewhere
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[David Knupp (Code Review)]

Hello Lars Volker,

I'd like you to reexamine a change.  Please visit

http://gerrit.cloudera.org:8080/5640

to look at the new patch set (#5).

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..

IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

The current version of pytest in the Impala python environment is
quite old (2.7.2) and there have been bug fixes in later versions
that we could benefit from.

In addition to bumping the version of pytest (and related modules),
this patch also makes some changes to address warning messages that
showed up from pytest re: the imported TestMatrix, TestVector, and
TestDimension classes, which were being misidentified as potential
test classes. The fix was to simply prepend the class names with
Impala-

git grep -l 'TestDimension' | xargs \
sed -i 's/TestDimension/ImpalaTestDimension/g'

git grep -l 'TestMatrix' | xargs \
sed -i 's/TestMatrix/ImpalaTestMatrix/g'

git grep -l 'TestVector' | xargs \
sed -i 's/TestVector/ImpalaTestVector/g'

Finally, the passing of params to pytest.main() as a string will
be deprecated in future versions of pytest, so change args in
test/run-tests.py to a list. (This also means we don't need to
worry about esoteric bash limitations re: single quotes in strings.)

Tested by doing a standard (non-exhaustive) test run on centos 6.4
and ubuntu 14.04, plus an exhaustive test run on RHEL7.

Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
---
M infra/python/deps/requirements.txt
M tests/authorization/test_grant_revoke.py
M tests/catalog_service/test_catalog_service_client.py
M tests/catalog_service/test_hms_failure.py
M tests/catalog_service/test_large_num_partitions.py
M tests/common/base_test_suite.py
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/kudu_test_suite.py
M tests/common/test_dimensions.py
M tests/common/test_vector.py
M tests/conftest.py
M tests/custom_cluster/test_admission_controller.py
M tests/custom_cluster/test_hive_parquet_timestamp_conversion.py
M tests/custom_cluster/test_parquet_max_page_header.py
M tests/custom_cluster/test_permanent_udfs.py
M tests/custom_cluster/test_spilling.py
M tests/data_errors/test_data_errors.py
M tests/experiments/test_targeted_perf.py
M tests/failure/test_failpoints.py
M tests/metadata/test_compute_stats.py
M tests/metadata/test_ddl.py
M tests/metadata/test_ddl_base.py
M tests/metadata/test_explain.py
M tests/metadata/test_hdfs_encryption.py
M tests/metadata/test_hdfs_permissions.py
M tests/metadata/test_hidden_files.py
M tests/metadata/test_hms_integration.py
M tests/metadata/test_last_ddl_time_update.py
M tests/metadata/test_load.py
M tests/metadata/test_metadata_query_statements.py
M tests/metadata/test_partition_metadata.py
M tests/metadata/test_recover_partitions.py
M tests/metadata/test_refresh_partition.py
M tests/metadata/test_set.py
M tests/metadata/test_show_create_table.py
M tests/metadata/test_stale_metadata.py
M tests/metadata/test_views_compatibility.py
M tests/query_test/test_aggregation.py
M tests/query_test/test_analytic_tpcds.py
M tests/query_test/test_avro_schema_resolution.py
M tests/query_test/test_cancellation.py
M tests/query_test/test_chars.py
M tests/query_test/test_compressed_formats.py
M tests/query_test/test_decimal_casting.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_errorlog.py
M tests/query_test/test_exprs.py
M tests/query_test/test_hash_join_timer.py
M tests/query_test/test_hbase_queries.py
M tests/query_test/test_hdfs_caching.py
M tests/query_test/test_hdfs_fd_caching.py
M tests/query_test/test_insert.py
M tests/query_test/test_insert_parquet.py
M tests/query_test/test_insert_permutation.py
M tests/query_test/test_invalid_test_header.py
M tests/query_test/test_join_queries.py
M tests/query_test/test_limit.py
M tests/query_test/test_local_fs.py
M tests/query_test/test_mem_usage_scaling.py
M tests/query_test/test_mt_dop.py
M tests/query_test/test_multiple_filesystems.py
M tests/query_test/test_nested_types.py
M tests/query_test/test_partitioning.py
M tests/query_test/test_queries.py
M tests/query_test/test_query_mem_limit.py
M tests/query_test/test_query_opts.py
M tests/query_test/test_rows_availability.py
M tests/query_test/test_runtime_filters.py
M tests/query_test/test_scanners.py
M tests/query_test/test_scanners_fuzz.py
M tests/query_test/test_scratch_limit.py
M tests/query_test/test_sort.py
M tests/query_test/test_timezones.py
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_nested_queries.py
M tests/query_test/test_tpch_queries.py
M tests/query_test/test_udfs.py
M tests/run-tests.py
M tests/stress/test_ddl_stress.py
M tests/stress/test_mini_stress.py
M tests/unittests/test_result_verifier.py
83 files changed, 522 insertions(+), 453 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF 

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[David Knupp (Code Review)]

Hello Lars Volker,

I'd like you to reexamine a change.  Please visit

http://gerrit.cloudera.org:8080/5640

to look at the new patch set (#4).

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..

IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

The current version of pytest in the Impala python environment is
quite old (2.7.2) and there have been bug fixes in later versions
that we could benefit from.

In addition to bumping the version of pytest (and related modules),
this patch also makes some changes to address warning messages that
showed up from pytest re: the imported TestMatrix and TestDimension
classes, which were being misidentified as potential test classes.
The fix was to simply prepend the class names with Impala-

git grep -l 'TestDimension' | xargs \
sed -i 's/TestDimension/ImpalaTestDimension/g'

git grep -l 'TestMatrix' | xargs \
sed -i 's/TestMatrix/ImpalaTestMatrix/g'

git grep -l 'TestVector' | xargs \
sed -i 's/TestVector/ImpalaTestVector/g'

Finally, the passing of params to pytest.main() as a string will
be deprecated in future versions of pytest, so change args in
test/run-tests.py to a list. (This also means we don't need to
worry about esoteric bash limitations re: single quotes in strings.)

Tested by doing a standard (non-exhaustive) test run on centos 6.4
and ubuntu 14.04, plus an exhaustive test run on RHEL7.

Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
---
M infra/python/deps/requirements.txt
M tests/authorization/test_grant_revoke.py
M tests/catalog_service/test_catalog_service_client.py
M tests/catalog_service/test_hms_failure.py
M tests/catalog_service/test_large_num_partitions.py
M tests/common/base_test_suite.py
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/kudu_test_suite.py
M tests/common/test_dimensions.py
M tests/common/test_vector.py
M tests/conftest.py
M tests/custom_cluster/test_admission_controller.py
M tests/custom_cluster/test_hive_parquet_timestamp_conversion.py
M tests/custom_cluster/test_parquet_max_page_header.py
M tests/custom_cluster/test_permanent_udfs.py
M tests/custom_cluster/test_spilling.py
M tests/data_errors/test_data_errors.py
M tests/experiments/test_targeted_perf.py
M tests/failure/test_failpoints.py
M tests/metadata/test_compute_stats.py
M tests/metadata/test_ddl.py
M tests/metadata/test_ddl_base.py
M tests/metadata/test_explain.py
M tests/metadata/test_hdfs_encryption.py
M tests/metadata/test_hdfs_permissions.py
M tests/metadata/test_hidden_files.py
M tests/metadata/test_hms_integration.py
M tests/metadata/test_last_ddl_time_update.py
M tests/metadata/test_load.py
M tests/metadata/test_metadata_query_statements.py
M tests/metadata/test_partition_metadata.py
M tests/metadata/test_recover_partitions.py
M tests/metadata/test_refresh_partition.py
M tests/metadata/test_set.py
M tests/metadata/test_show_create_table.py
M tests/metadata/test_stale_metadata.py
M tests/metadata/test_views_compatibility.py
M tests/query_test/test_aggregation.py
M tests/query_test/test_analytic_tpcds.py
M tests/query_test/test_avro_schema_resolution.py
M tests/query_test/test_cancellation.py
M tests/query_test/test_chars.py
M tests/query_test/test_compressed_formats.py
M tests/query_test/test_decimal_casting.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_errorlog.py
M tests/query_test/test_exprs.py
M tests/query_test/test_hash_join_timer.py
M tests/query_test/test_hbase_queries.py
M tests/query_test/test_hdfs_caching.py
M tests/query_test/test_hdfs_fd_caching.py
M tests/query_test/test_insert.py
M tests/query_test/test_insert_parquet.py
M tests/query_test/test_insert_permutation.py
M tests/query_test/test_invalid_test_header.py
M tests/query_test/test_join_queries.py
M tests/query_test/test_limit.py
M tests/query_test/test_local_fs.py
M tests/query_test/test_mem_usage_scaling.py
M tests/query_test/test_mt_dop.py
M tests/query_test/test_multiple_filesystems.py
M tests/query_test/test_nested_types.py
M tests/query_test/test_partitioning.py
M tests/query_test/test_queries.py
M tests/query_test/test_query_mem_limit.py
M tests/query_test/test_query_opts.py
M tests/query_test/test_rows_availability.py
M tests/query_test/test_runtime_filters.py
M tests/query_test/test_scanners.py
M tests/query_test/test_scanners_fuzz.py
M tests/query_test/test_scratch_limit.py
M tests/query_test/test_sort.py
M tests/query_test/test_timezones.py
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_nested_queries.py
M tests/query_test/test_tpch_queries.py
M tests/query_test/test_udfs.py
M tests/run-tests.py
M tests/stress/test_ddl_stress.py
M tests/stress/test_mini_stress.py
M tests/unittests/test_result_verifier.py
83 files changed, 522 insertions(+), 453 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF 

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[David Knupp (Code Review)]

David Knupp has posted comments on this change.

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..


Patch Set 3:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/5640/3//COMMIT_MSG
Commit Message:

PS3, Line 7: 2.9.2
> Why not 3.0.5?
I was going to add a comment about this to the JIRA.

Essentially, I couldn't find the workaround/fix to get rid of this 
3.0.5-specific warning, and I thought an incremental upgrade would at least 
solve some of our issues:

   WP1 None Module already imported so can not be re-written: random_plugin
   WP1 None Module already imported so can not be re-written: xdist
   WP1 None Module already imported so can not be re-written: xdist
   WP1 None Module already imported so can not be re-written: xdist


Line 13: In addition to bumping the version of pytest (and related modules),
> It might make more sense to split these changes up.
I can do that if you feel strongly about it.


Line 15: showed up from pytest re: the imported TestMatrix and TestDimension
> Could you put in the commit message the sed script (or whatever) you used t
Done


Line 24: Tested by doing a standard (non-exhaustive) test run on centos 6.4
> Presumably that checks that the tests that run do not fail. How did you ver
OK, so this seemingly innocuous question opened up (what seems to me) a really 
weird can of worms.

First, it is possible to to do a dry-run of sorts if you specify the 
--collect-only option. This will show exactly which tests would have been 
executed, taking into consideration all test vectors.

Since run-tests.py doesn't seem to honor --collect-only, you have to use 
impala-py.test directly, which means you have to explicitly pass in the test 
directories that run-tests.py would have typically white-listed out for you. 
E.g., this is the command for getting an exhaustive dry-run:

  $ impala-py.test --exploration_strategy=exhaustive --collect-only failure/ 
query_test/ stress/ aux_query_tests/ shell/ hs2/ catalog_service/ metadata/ 
data_errors/ statestore/ unittests/


To test, I ran it first from asf-gerrit-master branch (pytest v2.7.2), and then 
my dev branch (pytest v2.9.2), each time blowing away the python virtual 
environment and bootstrapping a new one. If I grep for "collected" in each 
file, the number was always the same -- 20881 items...

  $ grep collected ~/tmp/pytest_logs/pytest*exhaustive*log
  /home/dknupp/tmp/pytest_logs/pytest272_exhaustive_collect2.log:collecting ... 
collected 20881 items
  /home/dknupp/tmp/pytest_logs/pytest272_exhaustive_collect.log:collecting ... 
collected 20881 items
  /home/dknupp/tmp/pytest_logs/pytest292_exhaustive_collect2.log:collecting ... 
collected 20881 items
  /home/dknupp/tmp/pytest_logs/pytest292_exhaustive_collect.log:collecting ... 
collected 20881 items


However, I was surprised to learn that this number tended to vary when 
collecting tests with the core exploration_strategy -- even when comparing 
successive runs on the same branch. I narrowed it down to a single test -- 
query_test::test_cancellation::TestCancellationSerial::test_cancel_insert.

If I log output from the following command 10 times (regardless of version of 
pytest we use) in quick succession:

  $ impala-py.test --exploration_strategy=core --collect-only 
query_test/test_cancellation.py


...and again grep for the collected items, the result varies between 51 and 57 
items:

  $ grep collected ~/tmp/pytest_logs/pytest272_test_cancellation_collect*
  
/home/dknupp/tmp/pytest_logs/pytest272_test_cancellation_collect0.log:collecting
 ... collected 51 items
  
/home/dknupp/tmp/pytest_logs/pytest272_test_cancellation_collect1.log:collecting
 ... collected 52 items
  
/home/dknupp/tmp/pytest_logs/pytest272_test_cancellation_collect2.log:collecting
 ... collected 53 items
  
/home/dknupp/tmp/pytest_logs/pytest272_test_cancellation_collect3.log:collecting
 ... collected 54 items
  
/home/dknupp/tmp/pytest_logs/pytest272_test_cancellation_collect4.log:collecting
 ... collected 53 items
  
/home/dknupp/tmp/pytest_logs/pytest272_test_cancellation_collect5.log:collecting
 ... collected 55 items
  
/home/dknupp/tmp/pytest_logs/pytest272_test_cancellation_collect6.log:collecting
 ... collected 53 items
  
/home/dknupp/tmp/pytest_logs/pytest272_test_cancellation_collect7.log:collecting
 ... collected 57 items
  
/home/dknupp/tmp/pytest_logs/pytest272_test_cancellation_collect8.log:collecting
 ... collected 53 items
  
/home/dknupp/tmp/pytest_logs/pytest272_test_cancellation_collect9.log:collecting
 ... collected 52 items


Compare this to the result if we try another test module. In this case, the 
result is consistent every time:

  $ grep collected ~/tmp/pytest_logs/pytest292_test_queries_collect*
  /home/dknupp/tmp/pytest_logs/pytest292_test_queries_collect0.log:collecting 
... collected 38 items
  /home/dknupp/tmp/pytest_logs/pytest292_test_queries_collect1.log:collecting 

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[David Knupp (Code Review)]

David Knupp has uploaded a new patch set (#2).

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..

IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

The current version of pytest in the Impala python environment is
quite old (2.7.2) and there have been bug fixes in later versions
that we could benefit from.

In addition to bumping the version of pytest (and related modules),
this patch also makes some changes to address warning messages that
showed up from pytest re: the imported TestMatrix and TestDimension
classes, which were being misidentified as potential test classes.
(Prepending the class names with Impala- silenced the warnings.)

Finally, the passing of params to pytest.main() as a string will
be deprecated in future versions of pytest, so change args in
test/run-tests.py to a list. (This also means we don't need to
worry about esoteric bash limitations re: single quotes in strings.)

Tested by doing a standard (non-exhaustive) test run on centos 6.4
and ubuntu 14.04, plus an exhaustive test run on RHEL7.

Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
---
M infra/python/deps/requirements.txt
M tests/authorization/test_grant_revoke.py
M tests/catalog_service/test_catalog_service_client.py
M tests/catalog_service/test_hms_failure.py
M tests/catalog_service/test_large_num_partitions.py
M tests/common/base_test_suite.py
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/kudu_test_suite.py
M tests/common/test_dimensions.py
M tests/common/test_vector.py
M tests/conftest.py
M tests/custom_cluster/test_admission_controller.py
M tests/custom_cluster/test_hive_parquet_timestamp_conversion.py
M tests/custom_cluster/test_parquet_max_page_header.py
M tests/custom_cluster/test_permanent_udfs.py
M tests/custom_cluster/test_spilling.py
M tests/data_errors/test_data_errors.py
M tests/experiments/test_targeted_perf.py
M tests/failure/test_failpoints.py
M tests/metadata/test_compute_stats.py
M tests/metadata/test_ddl.py
M tests/metadata/test_ddl_base.py
M tests/metadata/test_explain.py
M tests/metadata/test_hdfs_encryption.py
M tests/metadata/test_hdfs_permissions.py
M tests/metadata/test_hidden_files.py
M tests/metadata/test_hms_integration.py
M tests/metadata/test_last_ddl_time_update.py
M tests/metadata/test_load.py
M tests/metadata/test_metadata_query_statements.py
M tests/metadata/test_partition_metadata.py
M tests/metadata/test_recover_partitions.py
M tests/metadata/test_refresh_partition.py
M tests/metadata/test_set.py
M tests/metadata/test_show_create_table.py
M tests/metadata/test_stale_metadata.py
M tests/metadata/test_views_compatibility.py
M tests/query_test/test_aggregation.py
M tests/query_test/test_analytic_tpcds.py
M tests/query_test/test_avro_schema_resolution.py
M tests/query_test/test_cancellation.py
M tests/query_test/test_chars.py
M tests/query_test/test_compressed_formats.py
M tests/query_test/test_decimal_casting.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_errorlog.py
M tests/query_test/test_exprs.py
M tests/query_test/test_hash_join_timer.py
M tests/query_test/test_hbase_queries.py
M tests/query_test/test_hdfs_caching.py
M tests/query_test/test_hdfs_fd_caching.py
M tests/query_test/test_insert.py
M tests/query_test/test_insert_parquet.py
M tests/query_test/test_insert_permutation.py
M tests/query_test/test_invalid_test_header.py
M tests/query_test/test_join_queries.py
M tests/query_test/test_limit.py
M tests/query_test/test_local_fs.py
M tests/query_test/test_mem_usage_scaling.py
M tests/query_test/test_mt_dop.py
M tests/query_test/test_multiple_filesystems.py
M tests/query_test/test_nested_types.py
M tests/query_test/test_partitioning.py
M tests/query_test/test_queries.py
M tests/query_test/test_query_mem_limit.py
M tests/query_test/test_query_opts.py
M tests/query_test/test_rows_availability.py
M tests/query_test/test_runtime_filters.py
M tests/query_test/test_scanners.py
M tests/query_test/test_scanners_fuzz.py
M tests/query_test/test_scratch_limit.py
M tests/query_test/test_sort.py
M tests/query_test/test_timezones.py
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_nested_queries.py
M tests/query_test/test_tpch_queries.py
M tests/query_test/test_udfs.py
M tests/run-tests.py
M tests/stress/test_ddl_stress.py
M tests/stress/test_mini_stress.py
M tests/unittests/test_result_verifier.py
83 files changed, 417 insertions(+), 418 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/5640/2
-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[David Knupp (Code Review)]

David Knupp has uploaded a new patch set (#3).

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..

IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

The current version of pytest in the Impala python environment is
quite old (2.7.2) and there have been bug fixes in later versions
that we could benefit from.

In addition to bumping the version of pytest (and related modules),
this patch also makes some changes to address warning messages that
showed up from pytest re: the imported TestMatrix and TestDimension
classes, which were being misidentified as potential test classes.
(Prepending the class names with Impala- silenced the warnings.)

Finally, the passing of params to pytest.main() as a string will
be deprecated in future versions of pytest, so change args in
test/run-tests.py to a list. (This also means we don't need to
worry about esoteric bash limitations re: single quotes in strings.)

Tested by doing a standard (non-exhaustive) test run on centos 6.4
and ubuntu 14.04, plus an exhaustive test run on RHEL7.

Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
---
M infra/python/deps/requirements.txt
M tests/authorization/test_grant_revoke.py
M tests/catalog_service/test_catalog_service_client.py
M tests/catalog_service/test_hms_failure.py
M tests/catalog_service/test_large_num_partitions.py
M tests/common/base_test_suite.py
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/kudu_test_suite.py
M tests/common/test_dimensions.py
M tests/common/test_vector.py
M tests/conftest.py
M tests/custom_cluster/test_admission_controller.py
M tests/custom_cluster/test_hive_parquet_timestamp_conversion.py
M tests/custom_cluster/test_parquet_max_page_header.py
M tests/custom_cluster/test_permanent_udfs.py
M tests/custom_cluster/test_spilling.py
M tests/data_errors/test_data_errors.py
M tests/experiments/test_targeted_perf.py
M tests/failure/test_failpoints.py
M tests/metadata/test_compute_stats.py
M tests/metadata/test_ddl.py
M tests/metadata/test_ddl_base.py
M tests/metadata/test_explain.py
M tests/metadata/test_hdfs_encryption.py
M tests/metadata/test_hdfs_permissions.py
M tests/metadata/test_hidden_files.py
M tests/metadata/test_hms_integration.py
M tests/metadata/test_last_ddl_time_update.py
M tests/metadata/test_load.py
M tests/metadata/test_metadata_query_statements.py
M tests/metadata/test_partition_metadata.py
M tests/metadata/test_recover_partitions.py
M tests/metadata/test_refresh_partition.py
M tests/metadata/test_set.py
M tests/metadata/test_show_create_table.py
M tests/metadata/test_stale_metadata.py
M tests/metadata/test_views_compatibility.py
M tests/query_test/test_aggregation.py
M tests/query_test/test_analytic_tpcds.py
M tests/query_test/test_avro_schema_resolution.py
M tests/query_test/test_cancellation.py
M tests/query_test/test_chars.py
M tests/query_test/test_compressed_formats.py
M tests/query_test/test_decimal_casting.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_errorlog.py
M tests/query_test/test_exprs.py
M tests/query_test/test_hash_join_timer.py
M tests/query_test/test_hbase_queries.py
M tests/query_test/test_hdfs_caching.py
M tests/query_test/test_hdfs_fd_caching.py
M tests/query_test/test_insert.py
M tests/query_test/test_insert_parquet.py
M tests/query_test/test_insert_permutation.py
M tests/query_test/test_invalid_test_header.py
M tests/query_test/test_join_queries.py
M tests/query_test/test_limit.py
M tests/query_test/test_local_fs.py
M tests/query_test/test_mem_usage_scaling.py
M tests/query_test/test_mt_dop.py
M tests/query_test/test_multiple_filesystems.py
M tests/query_test/test_nested_types.py
M tests/query_test/test_partitioning.py
M tests/query_test/test_queries.py
M tests/query_test/test_query_mem_limit.py
M tests/query_test/test_query_opts.py
M tests/query_test/test_rows_availability.py
M tests/query_test/test_runtime_filters.py
M tests/query_test/test_scanners.py
M tests/query_test/test_scanners_fuzz.py
M tests/query_test/test_scratch_limit.py
M tests/query_test/test_sort.py
M tests/query_test/test_timezones.py
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_nested_queries.py
M tests/query_test/test_tpch_queries.py
M tests/query_test/test_udfs.py
M tests/run-tests.py
M tests/stress/test_ddl_stress.py
M tests/stress/test_mini_stress.py
M tests/unittests/test_result_verifier.py
83 files changed, 416 insertions(+), 418 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/5640/3
-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[Jim Apple (Code Review)]

Jim Apple has posted comments on this change.

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..


Patch Set 3:

(10 comments)

Thank you for making this patch

http://gerrit.cloudera.org:8080/#/c/5640/3//COMMIT_MSG
Commit Message:

PS3, Line 7: 2.9.2
Why not 3.0.5?


Line 13: In addition to bumping the version of pytest (and related modules),
It might make more sense to split these changes up.


Line 15: showed up from pytest re: the imported TestMatrix and TestDimension
Could you put in the commit message the sed script (or whatever) you used to do 
the change?


Line 24: Tested by doing a standard (non-exhaustive) test run on centos 6.4
Presumably that checks that the tests that run do not fail. How did you verify 
that this patch does not accidentally disable any tests?

Is there a pytest dry-run mode that just lists the tests? You could compare the 
lists before and after this patch.


http://gerrit.cloudera.org:8080/#/c/5640/3/tests/common/kudu_test_suite.py
File tests/common/kudu_test_suite.py:

Line 61: 
cls.ImpalaTestMatrix.add_dimension(create_uncompressed_text_dimension(cls.get_workload()))
long line


http://gerrit.cloudera.org:8080/#/c/5640/3/tests/custom_cluster/test_permanent_udfs.py
File tests/custom_cluster/test_permanent_udfs.py:

Line 48: 
cls.ImpalaTestMatrix.add_dimension(create_uncompressed_text_dimension(cls.get_workload()))
long line


http://gerrit.cloudera.org:8080/#/c/5640/3/tests/experiments/test_targeted_perf.py
File tests/experiments/test_targeted_perf.py:

Line 33: cls.ImpalaTestMatrix.add_constraint(lambda v: 
v.get_value('exec_option')['batch_size'] == 0)
long line, here and elsewhere


http://gerrit.cloudera.org:8080/#/c/5640/3/tests/run-tests.py
File tests/run-tests.py:

PS3, Line 101: we need to account for
Why? What bad thing could happen otherwise?


PS3, Line 103: random
"arbitrary", not "random"


PS3, Line 143: Because of the way our repo is organized.
This doesn't explain a lot as it stands. I'm also not sure if it is a 
continuation of the above sentence or the beginning of the next one.


-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp 
Gerrit-Reviewer: David Knupp 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Brown 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

[David Knupp (Code Review)]

Hello Lars Volker,

I'd like you to reexamine a change.  Please visit

http://gerrit.cloudera.org:8080/5640

to look at the new patch set (#2).

Change subject: IMPALA-4735: Upgrade pytest in python env to version 2.9.2.
..

IMPALA-4735: Upgrade pytest in python env to version 2.9.2.

The current version of pytest in the Impala python environment is
quite old (2.7.2) and there have been bug fixes in later versions
that we could benefit from.

In addition to bumping the version of pytest (and related modules),
this patch also makes some changes to address warning messages that
showed up from pytest re: the imported TestMatrix and TestDimension
classes, which were being misidentified as potential test classes.
(Prepending the class names with Impala- silenced the warnings.)

Finally, the passing of params to pytest.main() as a string will
be deprecated in future versions of pytest, so change args in
test/run-tests.py to a list. (This also means we don't need to
worry about esoteric bash limitations re: single quotes in strings.)

Tested by doing a standard (non-exhaustive) test run on centos 6.4
and ubuntu 14.04, plus an exhaustive test run on RHEL7.

Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
---
M infra/python/deps/requirements.txt
M tests/authorization/test_grant_revoke.py
M tests/catalog_service/test_catalog_service_client.py
M tests/catalog_service/test_hms_failure.py
M tests/catalog_service/test_large_num_partitions.py
M tests/common/base_test_suite.py
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/common/kudu_test_suite.py
M tests/common/test_dimensions.py
M tests/common/test_vector.py
M tests/conftest.py
M tests/custom_cluster/test_admission_controller.py
M tests/custom_cluster/test_hive_parquet_timestamp_conversion.py
M tests/custom_cluster/test_parquet_max_page_header.py
M tests/custom_cluster/test_permanent_udfs.py
M tests/custom_cluster/test_spilling.py
M tests/data_errors/test_data_errors.py
M tests/experiments/test_targeted_perf.py
M tests/failure/test_failpoints.py
M tests/metadata/test_compute_stats.py
M tests/metadata/test_ddl.py
M tests/metadata/test_ddl_base.py
M tests/metadata/test_explain.py
M tests/metadata/test_hdfs_encryption.py
M tests/metadata/test_hdfs_permissions.py
M tests/metadata/test_hidden_files.py
M tests/metadata/test_hms_integration.py
M tests/metadata/test_last_ddl_time_update.py
M tests/metadata/test_load.py
M tests/metadata/test_metadata_query_statements.py
M tests/metadata/test_partition_metadata.py
M tests/metadata/test_recover_partitions.py
M tests/metadata/test_refresh_partition.py
M tests/metadata/test_set.py
M tests/metadata/test_show_create_table.py
M tests/metadata/test_stale_metadata.py
M tests/metadata/test_views_compatibility.py
M tests/query_test/test_aggregation.py
M tests/query_test/test_analytic_tpcds.py
M tests/query_test/test_avro_schema_resolution.py
M tests/query_test/test_cancellation.py
M tests/query_test/test_chars.py
M tests/query_test/test_compressed_formats.py
M tests/query_test/test_decimal_casting.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_errorlog.py
M tests/query_test/test_exprs.py
M tests/query_test/test_hash_join_timer.py
M tests/query_test/test_hbase_queries.py
M tests/query_test/test_hdfs_caching.py
M tests/query_test/test_hdfs_fd_caching.py
M tests/query_test/test_insert.py
M tests/query_test/test_insert_parquet.py
M tests/query_test/test_insert_permutation.py
M tests/query_test/test_invalid_test_header.py
M tests/query_test/test_join_queries.py
M tests/query_test/test_limit.py
M tests/query_test/test_local_fs.py
M tests/query_test/test_mem_usage_scaling.py
M tests/query_test/test_mt_dop.py
M tests/query_test/test_multiple_filesystems.py
M tests/query_test/test_nested_types.py
M tests/query_test/test_partitioning.py
M tests/query_test/test_queries.py
M tests/query_test/test_query_mem_limit.py
M tests/query_test/test_query_opts.py
M tests/query_test/test_rows_availability.py
M tests/query_test/test_runtime_filters.py
M tests/query_test/test_scanners.py
M tests/query_test/test_scanners_fuzz.py
M tests/query_test/test_scratch_limit.py
M tests/query_test/test_sort.py
M tests/query_test/test_timezones.py
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_nested_queries.py
M tests/query_test/test_tpch_queries.py
M tests/query_test/test_udfs.py
M tests/run-tests.py
M tests/stress/test_ddl_stress.py
M tests/stress/test_mini_stress.py
M tests/unittests/test_result_verifier.py
83 files changed, 417 insertions(+), 418 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/5640/2
-- 
To view, visit http://gerrit.cloudera.org:8080/5640
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I40d129e0e63ca5bee126bac6ac923abb3c7e0a67
Gerrit-PatchSet: 2