Yida Wu created IMPALA-12681:
--------------------------------
Summary: Some local file descriptors not released when using
remote spilling
Key: IMPALA-12681
URL: https://issues.apache.org/jira/browse/IMPALA-12681
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 4.2.0, Impala 4.1.0, Impala 4.0.0
Reporter: Yida Wu
Assignee: Yida Wu
The bug occurs during remote spilling when writing spilled data to local
buffers. In this process, if the files are not completely filled, such as when
no more data is incoming at the end of spilling, there is a possibility of
partial writes to the files, the files might be physically removed without
properly releasing the associated file descriptor. This issue can be observed
in cases like the one described below.
{code:java}
find /proc/*/fd -ls | grep '(deleted)'
288574785 0 lrwx------ 1 impala impala 64 Jan 3 14:24 /proc/x/fd/xxxx ->
/opt/impala/scratch/impala-scratch/impala-scratch-xxxxxxxxx-xxxx-xxxx\
(deleted) {code}
In such a scenario, the disk space occupied by the file may not be reclaimed
because the file descriptor still maintains a reference to the file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
