[ https://issues.apache.org/jira/browse/ASTERIXDB-1796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15882005#comment-15882005 ]
ASF subversion and git services commented on ASTERIXDB-1796: ------------------------------------------------------------ Commit 34f2384ea66a4b62627cb5a3ff3544b6d4558469 in asterixdb's branch refs/heads/master from [~buyingyi] [ https://git-wip-us.apache.org/repos/asf?p=asterixdb.git;h=34f2384 ] ASTERIXDB-1791, ASTERIXDB-1796: fix failure handling in runtime operators. This change includes the following parts: - Fix the implementation of fail() and close() in several runtime operators to avoid file handle leak and job hang; - Add an erase method to RunFileWriter which closes files before deleting them in order release the holding disk space; - Call RunFileWriter.close() and RunFileReader.close() in "finally" blocks. - Fix RunFileReader to not truncate files to be deleted - it is not the root cause of un-released disk space - open deleted files are the root cuase; - Check file handle leaks in LangExecutionUtil.tearDown(). Change-Id: I203168171e6dac16b57d2eda960823e3810e22a3 Reviewed-on: https://asterix-gerrit.ics.uci.edu/1513 Tested-by: Jenkins <jenk...@fulliautomatix.ics.uci.edu> Integration-Tests: Jenkins <jenk...@fulliautomatix.ics.uci.edu> Reviewed-by: Till Westmann <ti...@apache.org> > Job hanging after "no space on device" > -------------------------------------- > > Key: ASTERIXDB-1796 > URL: https://issues.apache.org/jira/browse/ASTERIXDB-1796 > Project: Apache AsterixDB > Issue Type: Bug > Components: Hyracks Core > Reporter: Yingyi Bu > Assignee: Yingyi Bu > > Setting: > 2 NCs > 1GB RAM per NC > one partition per NC > TPC-H data: 2GB > Query: > {noformat} > USE tpch; > SET `compiler.joinmemory` "8MB" > SET `compiler.groupmemory` "8MB" > SET `compiler.sortmemory` "8MB" > WITH tmp AS > ( > SELECT l_orderkey, SUM(l_quantity) t_sum_quantity > FROM LineItem > GROUP BY l_orderkey > ) > SELECT c.c_name, c.c_custkey, o.o_orderkey, o.o_orderdate, o.o_totalprice, > SUM(l.l_quantity) sum_quantity > FROM LineItem l > JOIN tmp t ON t.l_orderkey = l.l_orderkey > JOIN Orders o ON o.o_orderkey = t.l_orderkey > JOIN Customer c ON c.c_custkey = o.o_custkey > WHERE t.t_sum_quantity > 30 > GROUP BY c.c_name, c.c_custkey, o.o_orderkey, o.o_orderdate, o.o_totalprice > ORDER BY o.o_totalprice DESC, o.o_orderdate > LIMIT 100 > ; > {noformat} > {noformat} > org.apache.hyracks.api.exceptions.HyracksDataException: No space left on > device > at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:376) > at org.apache.hyracks.control.nc.Task.run(Task.java:308) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: No space > left on device > at org.apache.hyracks.control.nc.io.IOManager.syncWrite(IOManager.java:143) > at > org.apache.hyracks.dataflow.common.io.RunFileWriter.nextFrame(RunFileWriter.java:60) > at > org.apache.hyracks.dataflow.std.buffermanager.VPartitionTupleBufferManager.flushPartition(VPartitionTupleBufferManager.java:285) > at > org.apache.hyracks.dataflow.std.join.OptimizedHybridHashJoin.probe(OptimizedHybridHashJoin.java:514) > at > org.apache.hyracks.dataflow.std.join.OptimizedHybridHashJoinOperatorDescriptor$ProbeAndJoinActivityNode$1.nextFrame(OptimizedHybridHashJoinOperatorDescriptor.java:405) > at org.apache.hyracks.control.nc.Task.pushFrames(Task.java:360) > ... 4 more > Caused by: java.io.IOException: No space left on device > at sun.nio.ch.FileDispatcherImpl.pwrite0(Native Method) > at sun.nio.ch.FileDispatcherImpl.pwrite(FileDispatcherImpl.java:66) > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89) > at sun.nio.ch.IOUtil.write(IOUtil.java:65) > at sun.nio.ch.FileChannelImpl.writeInternal(FileChannelImpl.java:778) > at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:764) > at org.apache.hyracks.control.nc.io.IOManager.syncWrite(IOManager.java:130) > ... 9 more > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346)