[
https://issues.apache.org/jira/browse/IMPALA-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18062487#comment-18062487
]
ASF subversion and git services commented on IMPALA-14776:
----------------------------------------------------------
Commit 3e495c44112bd0c01acd0bdf744c4cbdcd394e4e in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=3e495c441 ]
IMPALA-14776 (part 1): Auto-close manually created impala clients
There are several APIs in ImpalaTestSuite that tests can use to
create a new client:
- create_impala_client()
- create_impala_client_from_vector()
- create_client_for_nth_impalad()
If the tests don't close the client properly, pytest will report
a leak of a socket. It seems possible that this type of leak can
also result in leaving an unclosed session on the impalad. This can
trigger TestValidateMetrics.test_metrics_are_zero(), which has
been an issue since switching to pytest 6.2.5.
This modifies those APIs to register the clients in a list and
close them during test method teardown. That should prevent
resource leaks (sockets, sessions) associated with these clients.
The clients may have been cleaned up by other means (with-as or
try-finally or a manual close), so it is heavily relying on
close() being idempotent.
The list is maintained at the instance level, and it is set in
setup_method() and freed in teardown_method(). This means that
test classes need to call ImpalaTestSuite::setup_method() and
teardown_method(). This modifies several test classes to call
these methods appropriately.
This also means that the APIs are no longer class methods, so callers
need to call them on their instance rather than on the class
itself. This required changing a handful of locations.
ImpalaTestSuite's built-in clients use an internal API so that
they can be reused across test methods. They instead are scoped
at the class level.
Testing:
- Ran an exhaustive job
Change-Id: Ib550527838a81cd2aaf69bb715080f6ac6da3786
Reviewed-on: http://gerrit.cloudera.org:8080/24026
Reviewed-by: Joe McDonnell <[email protected]>
Tested-by: Joe McDonnell <[email protected]>
Reviewed-by: Csaba Ringhofer <[email protected]>
> Fix pytest ResourceWarnings
> ---------------------------
>
> Key: IMPALA-14776
> URL: https://issues.apache.org/jira/browse/IMPALA-14776
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure, Test
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Critical
>
> After switching to pytest 6.2.5, it now outputs information about leaked
> resources (sockets, open files, thread pools):
> {noformat}
> 18:15:34
> authorization/test_ranger.py::TestRangerIndependent::test_grant_revoke_with_role["table_format:
> text/none" | "protocol: hs2" | "exec_option: {'abort_on_error': 1,
> 'batch_size': 0, 'disable_codegen': False, 'disable_codegen_rows_threshold':
> 0, 'exec_single_node_rows_threshold': 0, 'num_nodes': 0, 'test_replan': 1}"]
> 18:15:34
> /data0/jenkins/workspace/impala-cdw-master-staging-core-asan/repos/Impala/tests/common/impala_test_suite.py:951:
> ResourceWarning: unclosed <socket.socket fd=47,
> family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0,
> laddr=('127.0.0.1', 44380), raddr=('127.0.0.1', 21050)>{noformat}
> We should fix these warnings. In particular, tests that don't clean up
> clients can cause there to be orphaned HS2 sessions that trigger
> TestValidateMetrics.test_metrics_are_zero().
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
