[
https://issues.apache.org/jira/browse/IMPALA-12681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17804530#comment-17804530
]
ASF subversion and git services commented on IMPALA-12681:
----------------------------------------------------------
Commit 56a7514ba6e55f04d883067144b786c9cee7601a in impala's branch
refs/heads/master from Yida Wu
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=56a7514ba ]
IMPALA-12681: Release file descriptors for partially written temporary files
This patch fixes a bug where partially written temporary files are
removed without releasing the file descriptors. This patch fixes
the bug by adding a call to Close() of the local file writer
during the Delete() of the DiskFile class, which could be called
when the local buffer file is being evicted or the query ends,
ensuring proper release of the file handle.
Testing:
Passed core tests.
Additionally, a check has been added in the test
test_scratch_disk.py to verify that there are no deleted
files in the /proc/x/fd/ directory.
Change-Id: I58a2bac419ced806d6f5a32bcdf24d79e078ab14
Reviewed-on: http://gerrit.cloudera.org:8080/20852
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Some local file descriptors not released when using remote spilling
> -------------------------------------------------------------------
>
> Key: IMPALA-12681
> URL: https://issues.apache.org/jira/browse/IMPALA-12681
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.0.0, Impala 4.1.0, Impala 4.2.0
> Reporter: Yida Wu
> Assignee: Yida Wu
> Priority: Major
>
> The bug occurs during remote spilling when writing spilled data to local
> buffers. In this process, if the files are not completely filled, such as
> when no more data is incoming at the end of spilling, there is a possibility
> of partial writes to the files, the files might be physically removed without
> properly releasing the associated file descriptor. This issue can be observed
> in cases like the one described below.
> {code:java}
> find /proc/*/fd -ls | grep '(deleted)'
> 288574785 0 lrwx------ 1 impala impala 64 Jan 3 14:24 /proc/x/fd/xxxx ->
> /opt/impala/scratch/impala-scratch/impala-scratch-xxxxxxxxx-xxxx-xxxx\
> (deleted) {code}
> In such a scenario, the disk space occupied by the file may not be reclaimed
> because the file descriptor still maintains a reference to the file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
