[jira] [Commented] (IMPALA-5534) Fix and re-enable run-process-failure-tests.sh
[
https://issues.apache.org/jira/browse/IMPALA-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18051749#comment-18051749
]
ASF subversion and git services commented on IMPALA-5534:
-
Commit 78ec9903cc317dad3879358fb8ae6f2db2f4802e in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=78ec9903c ]
IMPALA-14668: Upgrade to pytest 6.2.5
This patch upgrades pytest version from 2.9.2 to 6.2.5, the highest
pytest version available without the need to upgrade setuptools.
The pytest requirement is moved to py3-requirements.txt. We can not go
back to test with Python2 + pytest-2.9.2 after this patch because our
test script need to be adjusted as well. This is OK since we have
default testing using Python3 since IMPALA-14333 and stop building
impala-shell for Python2 since IMPALA-14606. The adjustment are follows:
- Replace deprecated @pytest.yield_fixture with plain @pytest.fixture.
- Replace --resultlog parameter (removed in version 6.0) with
--report-log from pytest-reportlog plugin.
- Make impala-shell.sh bootstrap Python3
venv (infra/python/env-gcc10.4.0-py3/) by default. Python2
venv (infra/python/env-gcc10.4.0/) is not bootstrapped automatically
anymore.
- Upgrade execnet to version 1.9.0. This is required by
pytest-xdist==2.4.0.
- Remove unused pytest-runner plugin
- Add -Wonce argument in pytest.ini to slightly suppress warnings.
- Add "junit_logging = system-err" option at pytest.ini to continue
logging stderr output to junit xml file.
- Fix rootdir and pytest.ini path programmatically at run-tests.py
because individual pytest-xdist worker often does not pick this up
correctly during parallel run.
With pytest-xdist==2.4.0, parallel EE tests does not show verbose
individual test names. It only shows pytest progress marker like
following line
...ssss.
This is a known limitation in pytest-xdist because execnet, the
underlying library used for communication between master and workers,
does not support transferring stdout/stderr from workers.
https://pytest-xdist.readthedocs.io/en/stable/known-limitations.html
Read following links for more detail about the deprecation notes:
https://docs.pytest.org/en/stable/deprecations.html
Change SKIP_SSL_MSG default to empty string because skipif does not
accept None reason anymore. Removed run-process-failure-tests.sh (unused
after IMPALA-5534) and unused pytest fixtures. Fixed erroneous log
formatting in test_restart_services.py.
Fixed small warnings found by pytest-6.2.5 that help stabilize
exhaustive tests run at:
- test_automatic_invalidation.py
- test_calcite_planner.py
- test_events_custom_configs.py
- test_session_expiration.py
- test_shell_interactive.py
- auto_scaler.py
- concurrent_workload.py
- hdfs_util.py
Most of the remaining warnings are about not closing resources properly
at the end of test. These warnings should be addressed in follow up
patches.
Testing:
- Pass exhaustive tests.
Change-Id: Ic3812fe976ef09ac48753dee30151714f4752c24
Reviewed-on: http://gerrit.cloudera.org:8080/23842
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> Fix and re-enable run-process-failure-tests.sh
> --
>
> Key: IMPALA-5534
> URL: https://issues.apache.org/jira/browse/IMPALA-5534
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.5.0, Impala 2.4.0, Impala 2.6.0, Impala 2.7.0,
> Impala 2.8.0
>Reporter: Alexander Behm
>Assignee: Sahil Takiar
>Priority: Major
> Labels: test
> Fix For: Impala 4.0.0
>
>
> See bin/run-all-tests.sh:
> {code}
> ...
> # Finally, run the process failure tests.
> # Disabled temporarily until we figure out the proper timeouts required to
> make the test
> # succeed.
> # ${IMPALA_HOME}/tests/run-process-failure-tests.sh
> ...
> {code}
> We should fix and re-enable these tests or alternatively re-implement the
> tests in a different way to get the same coverage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (IMPALA-5534) Fix and re-enable run-process-failure-tests.sh
[
https://issues.apache.org/jira/browse/IMPALA-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17157051#comment-17157051
]
ASF subversion and git services commented on IMPALA-5534:
-
Commit fc19e70cbc94735c38e64573c1e015878f218281 in impala's branch
refs/heads/master from Sahil Takiar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=fc19e70 ]
IMPALA-5534: Fix and enable experimental failure tests
Moves the test_catalog_hms_failures.py and test_process_failures.py from
the experimental tests to custom cluster tests.
catalog_service/test_hms_failure.py is combined with
custom_cluster/test_catalog_hms_failure.py as well in order to unify all
tests for HMS failures. Several modifications to the tests were
necessary to get them working again, but for the most part, the logic of
the tests remained the same. A few additional fault tolerance tests
(e.g. TestHiveMetaStoreFailure::test_hms_client_retries) were added as
well. The overall goal is to increase the process failure test coverage
for all components: impalads, statestore, catalogd, HMS, etc.
test_restart_catalogd in test_process_failures.py fails due to
IMPALA-9848, so it is skipped for now.
Testing:
* Ran new tests locally
Change-Id: I9dbb98017fb6c40cea349e7c63a35c325cbbc288
Reviewed-on: http://gerrit.cloudera.org:8080/16157
Reviewed-by: Sahil Takiar
Tested-by: Impala Public Jenkins
> Fix and re-enable run-process-failure-tests.sh
> --
>
> Key: IMPALA-5534
> URL: https://issues.apache.org/jira/browse/IMPALA-5534
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.5.0, Impala 2.4.0, Impala 2.6.0, Impala 2.7.0,
> Impala 2.8.0
>Reporter: Alexander Behm
>Assignee: Sahil Takiar
>Priority: Major
> Labels: test
>
> See bin/run-all-tests.sh:
> {code}
> ...
> # Finally, run the process failure tests.
> # Disabled temporarily until we figure out the proper timeouts required to
> make the test
> # succeed.
> # ${IMPALA_HOME}/tests/run-process-failure-tests.sh
> ...
> {code}
> We should fix and re-enable these tests or alternatively re-implement the
> tests in a different way to get the same coverage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (IMPALA-5534) Fix and re-enable run-process-failure-tests.sh
[
https://issues.apache.org/jira/browse/IMPALA-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107566#comment-17107566
]
Sahil Takiar commented on IMPALA-5534:
--
Thanks for pointing me to this Tim. I was actually taking a look at these tests
recently, and considering what parts of them are useful going forward. Will
consolidate this JIRA with some of my other efforts.
> Fix and re-enable run-process-failure-tests.sh
> --
>
> Key: IMPALA-5534
> URL: https://issues.apache.org/jira/browse/IMPALA-5534
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.5.0, Impala 2.4.0, Impala 2.6.0, Impala 2.7.0,
> Impala 2.8.0
>Reporter: Alexander Behm
>Assignee: Sahil Takiar
>Priority: Major
> Labels: test
>
> See bin/run-all-tests.sh:
> {code}
> ...
> # Finally, run the process failure tests.
> # Disabled temporarily until we figure out the proper timeouts required to
> make the test
> # succeed.
> # ${IMPALA_HOME}/tests/run-process-failure-tests.sh
> ...
> {code}
> We should fix and re-enable these tests or alternatively re-implement the
> tests in a different way to get the same coverage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (IMPALA-5534) Fix and re-enable run-process-failure-tests.sh
[
https://issues.apache.org/jira/browse/IMPALA-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107537#comment-17107537
]
Tim Armstrong commented on IMPALA-5534:
---
[~stakiar] maybe we should just delete these tests, since we're doing different
thigns for fault tolerance testing now.
> Fix and re-enable run-process-failure-tests.sh
> --
>
> Key: IMPALA-5534
> URL: https://issues.apache.org/jira/browse/IMPALA-5534
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 2.5.0, Impala 2.4.0, Impala 2.6.0, Impala 2.7.0,
> Impala 2.8.0
>Reporter: Alexander Behm
>Assignee: Sahil Takiar
>Priority: Major
> Labels: test
>
> See bin/run-all-tests.sh:
> {code}
> ...
> # Finally, run the process failure tests.
> # Disabled temporarily until we figure out the proper timeouts required to
> make the test
> # succeed.
> # ${IMPALA_HOME}/tests/run-process-failure-tests.sh
> ...
> {code}
> We should fix and re-enable these tests or alternatively re-implement the
> tests in a different way to get the same coverage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
