Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24026 )
Change subject: IMPALA-14776 (part 1): Auto-close manually created impala clients ...................................................................... IMPALA-14776 (part 1): Auto-close manually created impala clients There are several APIs in ImpalaTestSuite that tests can use to create a new client: - create_impala_client() - create_impala_client_from_vector() - create_client_for_nth_impalad() If the tests don't close the client properly, pytest will report a leak of a socket. It seems possible that this type of leak can also result in leaving an unclosed session on the impalad. This can trigger TestValidateMetrics.test_metrics_are_zero(), which has been an issue since switching to pytest 6.2.5. This modifies those APIs to register the clients in a list and close them during test method teardown. That should prevent resource leaks (sockets, sessions) associated with these clients. The clients may have been cleaned up by other means (with-as or try-finally or a manual close), so it is heavily relying on close() being idempotent. The list is maintained at the instance level, and it is set in setup_method() and freed in teardown_method(). This means that test classes need to call ImpalaTestSuite::setup_method() and teardown_method(). This modifies several test classes to call these methods appropriately. This also means that the APIs are no longer class methods, so callers need to call them on their instance rather than on the class itself. This required changing a handful of locations. ImpalaTestSuite's built-in clients use an internal API so that they can be reused across test methods. They instead are scoped at the class level. Testing: - Ran an exhaustive job Change-Id: Ib550527838a81cd2aaf69bb715080f6ac6da3786 Reviewed-on: http://gerrit.cloudera.org:8080/24026 Reviewed-by: Joe McDonnell <[email protected]> Tested-by: Joe McDonnell <[email protected]> Reviewed-by: Csaba Ringhofer <[email protected]> --- M tests/common/custom_cluster_test_suite.py M tests/common/impala_test_suite.py M tests/conftest.py M tests/custom_cluster/test_blacklist.py M tests/custom_cluster/test_breakpad.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/custom_cluster/test_query_retries.py M tests/custom_cluster/test_redaction.py M tests/custom_cluster/test_scratch_disk.py M tests/metadata/test_hdfs_encryption.py M tests/metadata/test_hdfs_permissions.py M tests/metadata/test_load.py M tests/query_test/test_compressed_formats.py M tests/query_test/test_insert_permutation.py M tests/stress/test_acid_stress.py M tests/stress/test_update_stress.py 17 files changed, 162 insertions(+), 55 deletions(-) Approvals: Joe McDonnell: Looks good to me, but someone else must approve; Verified Csaba Ringhofer: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/24026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib550527838a81cd2aaf69bb715080f6ac6da3786 Gerrit-Change-Number: 24026 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Jason Fehr <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]>
