[
https://issues.apache.org/jira/browse/IMPALA-14776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18063333#comment-18063333
]
ASF subversion and git services commented on IMPALA-14776:
----------------------------------------------------------
Commit 24611f7dba77723a805518a09ca6e334c4794796 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=24611f7db ]
IMPALA-14776 (part 3): Fix misc resource leaks
This fixes a variety of misc resource leaks and other warnings.
This is purely about reducing the noise, so these are not
fixing issues that would impact build stability.
There are a few different categories:
1. Leaked open files
- There are a couple locations that open /dev/null manually
to use with subprocess. This switches them to subprocess.DEVNULL
- Otherwise, use a context manager
2. Leaked thread pools - Manually call terminate at the end
3. Leaked processes - Add a wait() call after killing them
4. Warnings about deprecated pyparsing APIs - switched to the
new API in db_connection.py
5. Warnings about invalid escapes - used raw strings for regex
6. Warnings from pytest failing to collect tests from
TestLastDdlTimeUpdate's TestHelper
- Renamed TestHelper to Helper so pytest doesn't try to find
tests in it
7. A few minor socket leaks
Testing:
- Ran a core job
Change-Id: Iee2c7733f477eadac7cde020b9aa47077985eb0d
Reviewed-on: http://gerrit.cloudera.org:8080/24066
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Csaba Ringhofer <[email protected]>
Reviewed-by: Michael Smith <[email protected]>
> Fix pytest ResourceWarnings
> ---------------------------
>
> Key: IMPALA-14776
> URL: https://issues.apache.org/jira/browse/IMPALA-14776
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure, Test
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Critical
>
> After switching to pytest 6.2.5, it now outputs information about leaked
> resources (sockets, open files, thread pools):
> {noformat}
> 18:15:34
> authorization/test_ranger.py::TestRangerIndependent::test_grant_revoke_with_role["table_format:
> text/none" | "protocol: hs2" | "exec_option: {'abort_on_error': 1,
> 'batch_size': 0, 'disable_codegen': False, 'disable_codegen_rows_threshold':
> 0, 'exec_single_node_rows_threshold': 0, 'num_nodes': 0, 'test_replan': 1}"]
> 18:15:34
> /data0/jenkins/workspace/impala-cdw-master-staging-core-asan/repos/Impala/tests/common/impala_test_suite.py:951:
> ResourceWarning: unclosed <socket.socket fd=47,
> family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0,
> laddr=('127.0.0.1', 44380), raddr=('127.0.0.1', 21050)>{noformat}
> We should fix these warnings. In particular, tests that don't clean up
> clients can cause there to be orphaned HS2 sessions that trigger
> TestValidateMetrics.test_metrics_are_zero().
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
