[
https://issues.apache.org/jira/browse/PIG-2578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13433772#comment-13433772
]
Rohini Palaniswamy commented on PIG-2578:
-----------------------------------------
Spoke with Daniel. He said it was intentional to make the JobConf read-only so
that each store does not override another. But the problem with that is it does
not allow addition of Credentials and setting of JT specific config like
Distributed cache configuration on the Job. We need a more cleaner solution to
solve it and prevent StoreFunc implementations to not put something in JobConf
that will not work correctly with multiple stores. We got rid of the multiple
stores messing up problem in hcat by putting the properties in UDFContext
instead of Job. But cannot expect all StoreFunc implementations to do that
unless forced to which was the intention of this JIRA.
> Multiple Store-commands mess up mapred.output.dir.
> --------------------------------------------------
>
> Key: PIG-2578
> URL: https://issues.apache.org/jira/browse/PIG-2578
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.1, 0.9.2
> Reporter: Mithun Radhakrishnan
> Assignee: Daniel Dai
> Fix For: 0.10.0, 0.11
>
> Attachments: PIG-2578-1.patch
>
>
> When one runs a pig-script with multiple storers, one sees the following:
> 1. When run as a script, Pig launches a single job.
> 2. PigOutputCommitter::setupJob() calls the
> underlyingOutputCommitter::setupJob(), once for each storer. But the
> mapred.output.dir is the same for both calls, even though the storers write
> to different locations.
> This was originally seen in HCATALOG-276, when HCatalog's end-to-end tests
> are run against Pig.
> (https://issues.apache.org/jira/browse/HCATALOG-276)
> Sample pig-script (near identical to HCatalog's Pig_Checkin_4 test):
> a = load 'keyvals' using org.apache.hcatalog.pig.HCatLoader();
> split a into b if key<200, c if key >=200;
> store b into 'keyvals_lt200' using org.apache.hcatalog.pig.HCatStorer();
> store c into 'keyvals_ge200' using org.apache.hcatalog.pig.HCatStorer();
> I've suggested a workaround in HCat for the time being, but I think this
> might be something that needs fixing in Pig.
> Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
