Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21485 )
Change subject: IMPALA-12616: Fix test_restart_services.py::TestRestart tests for S3 ...................................................................... IMPALA-12616: Fix test_restart_services.py::TestRestart tests for S3 The test_restart_catalogd_while_handling_rpc_response* tests from custom_cluster/test_restart_services.py have been failing consistently on s3. The alter table statement is expected to succeed, but instead it fails with: "CatalogException: Detected catalog service ID changes" This manifests as a timeout waiting for the statement to reach the finished state. The test relies on specific timing with a sleep injected via a debug action. The failure stems from the catalog being slower on s3. The alter table wakes up before the catalog service ID change has fully completed, and it fails when it sees the catalog service ID change. This increases two sleep times: 1. This increases the sleep time before restarting the catalogd from 0.5 seconds to 5 seconds. This gives the catalogd longer to receive the message about the alter table and respond back to the impalad. 2. This increases the WAIT_BEFORE_PROCESSING_CATALOG_UPDATE sleep from 10 seconds to 30 seconds so the alter table statement doesn't wake up until the catalog service ID change is finalized. The test is verifying that the right messages are in the impalad logs, so we know this is still testing the same condition. This modifies the tests to use wait_for_finished_timeout() rather than wait_for_state(). This bails out immediately if the query fails rather than waiting unnecessarily for the full timeout. This also clears the query options so that later statements don't inherit the debug_action that the alter table statement used. Testing: - Ran the tests 100x in a loop on s3 - Ran the tests 100x in a loop on HDFS Change-Id: Ieb5699b8fb0b2ad8bad4ac30922a7b4d7fa17d29 Reviewed-on: http://gerrit.cloudera.org:8080/21485 Tested-by: Impala Public Jenkins <[email protected]> Reviewed-by: Daniel Becker <[email protected]> --- M tests/custom_cluster/test_restart_services.py 1 file changed, 31 insertions(+), 9 deletions(-) Approvals: Impala Public Jenkins: Verified Daniel Becker: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/21485 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ieb5699b8fb0b2ad8bad4ac30922a7b4d7fa17d29 Gerrit-Change-Number: 21485 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Daniel Becker <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]>
