[
https://issues.apache.org/jira/browse/DRILL-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15219896#comment-15219896
]
ASF GitHub Bot commented on DRILL-2100:
---------------------------------------
Github user vdiravka commented on a diff in the pull request:
https://github.com/apache/drill/pull/454#discussion_r58057778
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java
---
@@ -110,11 +111,12 @@
private LinkedList<BatchGroup> batchGroups = Lists.newLinkedList();
private LinkedList<BatchGroup> spilledBatchGroups =
Lists.newLinkedList();
private SelectionVector4 sv4;
- private FileSystem fs;
+ private static FileSystem fs;
--- End diff --
It was my first decision to put `closeSpillFileSystem` into close(). But it
in this case we can have failed query when one leaf fragment executes `close()`
and delete spill folder while another leaf fragment still spooling to the disk.
That's why we need to delete the spill folder only when root fragment
(foreman) is closed.
And `fs` is initialized only once that's why every instance of class uses
the same FileSystem.
If I am right one drillbit can perform only one query at the same time. And
another drillbit performing an external sort uses own static FileSystem
instance.
> Drill not deleting spooling files
> ---------------------------------
>
> Key: DRILL-2100
> URL: https://issues.apache.org/jira/browse/DRILL-2100
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
> Affects Versions: 0.8.0
> Reporter: Abhishek Girish
> Assignee: Vitalii Diravka
> Fix For: Future
>
>
> Currently, after forcing queries to use an external sort by switching off
> hash join/agg causes spill-to-disk files accumulating.
> This causes issues with disk space availability when the spill is configured
> to be on the local file system (/tmp/drill). Also not optimal when configured
> to use DFS (custom).
> Drill must clean up all temporary files created after a query completes or
> after a drillbit restart.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
