Support for Multiple tmp directories and clean up
-------------------------------------------------
Key: PIG-138
URL: https://issues.apache.org/jira/browse/PIG-138
Project: Pig
Issue Type: Improvement
Environment: Pig Local
Reporter: Amir Youssefi
This is to separate additional improvements and original requirement on PIG-129
issue.
Pasting comments of Pi Song.
"I think the concept of multi-dir temp file creator (LocalDirAllocator in
Hadoop) should be adopted to Pig. What it does is:-
* You can set up a set of tmp file dirs in configuration (They can be on
different physical drives so you can utilize more disk space)
* When a temp file is being created, the system will probe the given temp
dirs in round-robin fashion
* For a selected temp dir, if it exists and you have permission to write,
temp file will be created
* For a selected temp dir, it it doesn't exist or you don't have permission
to write, the temp dir will be kept in the black list, thus not being used
later on.
* For the next temp file, move on to the next temp dir
"
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
