Andrew Kyle Purtell created PHOENIX-7980:
--------------------------------------------
Summary: Improve handling of test timeouts and resource loss
Key: PHOENIX-7980
URL: https://issues.apache.org/jira/browse/PHOENIX-7980
Project: Phoenix
Issue Type: Bug
Components: test
Reporter: Andrew Kyle Purtell
Assignee: Andrew Kyle Purtell
Several Phoenix test classes run with reuseForks=true, sharing a single in-JVM
minicluster across all classes that land on that same fork (via
BaseTest.setUpTestDriver). This is important for reducing overall test runtime
but can be problematic. When the fork's in-JVM minicluster dies this poisons
every subsequent class assigned to that same reused fork and causes all manner
of malfunction that currently are not handled all that well.
We should at least bound the failure blast radius with the following basic test
improvements:
- Add {{<forkedProcessTimeoutInSeconds>1800</forkedProcessTimeoutInSeconds>}}
to surefire/failsafe configuration. This kills any fork that hasn't finished
its class in 30 min.
- Add a global ClassRule {{Timeout(20, MINUTES)}} to {{BaseTest}}. Triggers
from inside the fork and produces a failure report XML for the wedged test
class.
- Detect and fail fast on discovery of a dead in-JVM cluster in a shared fork.
In {{BaseTest.doSetup}} after the utility starts, add a health check that
verifies that the master is up. This alone may reduce failure case wastage of
test runner resources to seconds from hours.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
