[jira] [Updated] (PIG-2812) Spill InternalCachedBag into only 1 file
[ https://issues.apache.org/jira/browse/PIG-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Le Dem updated PIG-2812: --- Fix Version/s: (was: 0.11) I'm detaching this from pig-0.11 as it is not ready yet Spill InternalCachedBag into only 1 file Key: PIG-2812 URL: https://issues.apache.org/jira/browse/PIG-2812 Project: Pig Issue Type: Bug Components: data Reporter: Haitao Yao Assignee: Haitao Yao Attachments: aa.jpg, spill.patch I encountered a reducer's OOM because of java.io.DeleteOnExitHook. And I found out that the InternalCachedBag creates a seperate tmp file, and the tmp files is deleted on exit. So the file delete hook caused the OOM. Why not just hold the tmp file handle and spill only one tmp file? Too many tmp files may block the tasktracker start process, if the tmp files are not cleaned on time and the tasktracker restarts at this specific time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2812) Spill InternalCachedBag into only 1 file
[ https://issues.apache.org/jira/browse/PIG-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haitao Yao updated PIG-2812: Patch Info: Patch Available Spill InternalCachedBag into only 1 file Key: PIG-2812 URL: https://issues.apache.org/jira/browse/PIG-2812 Project: Pig Issue Type: Bug Components: data Reporter: Haitao Yao Fix For: 0.11 Attachments: aa.jpg, spill.patch I encountered a reducer's OOM because of java.io.DeleteOnExitHook. And I found out that the InternalCachedBag creates a seperate tmp file, and the tmp files is deleted on exit. So the file delete hook caused the OOM. Why not just hold the tmp file handle and spill only one tmp file? Too many tmp files may block the tasktracker start process, if the tmp files are not cleaned on time and the tasktracker restarts at this specific time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2812) Spill InternalCachedBag into only 1 file
[ https://issues.apache.org/jira/browse/PIG-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haitao Yao updated PIG-2812: Attachment: spill.patch patch for spill spill into only 1 directory and use shutdownhook to delete the dir Spill InternalCachedBag into only 1 file Key: PIG-2812 URL: https://issues.apache.org/jira/browse/PIG-2812 Project: Pig Issue Type: Bug Components: data Reporter: Haitao Yao Fix For: 0.11 Attachments: aa.jpg, spill.patch I encountered a reducer's OOM because of java.io.DeleteOnExitHook. And I found out that the InternalCachedBag creates a seperate tmp file, and the tmp files is deleted on exit. So the file delete hook caused the OOM. Why not just hold the tmp file handle and spill only one tmp file? Too many tmp files may block the tasktracker start process, if the tmp files are not cleaned on time and the tasktracker restarts at this specific time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2812) Spill InternalCachedBag into only 1 file
[ https://issues.apache.org/jira/browse/PIG-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-2812: Fix Version/s: 0.11 Spill InternalCachedBag into only 1 file Key: PIG-2812 URL: https://issues.apache.org/jira/browse/PIG-2812 Project: Pig Issue Type: Bug Components: data Reporter: Haitao Yao Fix For: 0.11 Attachments: aa.jpg I encountered a reducer's OOM because of java.io.DeleteOnExitHook. And I found out that the InternalCachedBag creates a seperate tmp file, and the tmp files is deleted on exit. So the file delete hook caused the OOM. Why not just hold the tmp file handle and spill only one tmp file? Too many tmp files may block the tasktracker start process, if the tmp files are not cleaned on time and the tasktracker restarts at this specific time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
