No (easy) way to tell PigInputFormat not to combine splits
----------------------------------------------------------
Key: PIG-2646
URL: https://issues.apache.org/jira/browse/PIG-2646
Project: Pig
Issue Type: Bug
Components: impl
Reporter: Alex Levenson
Priority: Minor
It seems like the only way to turn off split combining is by setting
pig.noSplitCombination in the configuration.
In PigInputFormat line 252:
{code}
boolean combinable = !(loadFunc instanceof MergeJoinIndexer) &&
!(IndexableLoadFunc.class.isAssignableFrom(loadFunc.getClass())) &&
!(CollectableLoadFunc.class.isAssignableFrom(loadFunc.getClass()) &&
OrderedLoadFunc.class.isAssignableFrom(loadFunc.getClass()));
if (combinable)
combinable = !conf.getBoolean("pig.noSplitCombination", false);
{code}
However, if you're writing a LoadFunc whose underlying InputFormat is going to
do the split combining itself, the only way to set this parameter in the
configuration is
to do it in the LoadFunc.setLocation() method (which seems like a bad place to
do that) because it's the only method that gives you acces to the configuration.
It would be nice if LoadFunc could either:
a) Have an init method that gets passed the Configuration
b) Have access to a static method like PigInputFormat.setCombineSplits(false);
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
