Hi Dan, Thanks for looking into it. I just commented in the PR again. I think we can simplify the interaction by moving more of the conf and udfcontext interaction to setInput. as setInput is called in all methods that take a location in the Pig Loader API, we don't need to worry about order of calls anymore. That with the use of isFrontEnd() you can do the two cases of initializing from parameters in frontend or reading from conf/context in backend. As you mentioned the Pig API is not very clear here and I think it is worth refining this part of the code. Julien
On Tue, Aug 12, 2014 at 9:06 AM, Daniel Weeks <[email protected]> wrote: > Looks like the job config properties needed to be set in the front end as > well, which is why travis failed. > > At this point, all we're saving by using 'isFrontend()' is resetting the > property on the backend. I don't think there's any functional difference. > > -Dan > > > On Mon, Aug 11, 2014 at 5:10 PM, Daniel Weeks <[email protected]> wrote: > >> I cleaned it up using the isFrontend() method. I'm not sure if it >> matters whether the logic is in setLocation or setInput, but I moved it as >> you suggested. >> >> I spoke with Cheolsoo about loader behavior and it isn't well documented >> as to order of operations and how many times a loader will be instantiated, >> so hopefully this will be enough to get around that behavior. >> >> -Dan >> >> >> On Mon, Aug 11, 2014 at 4:04 PM, Julien Le Dem <[email protected]> >> wrote: >> >>> I just reviewed it. >>> It seems we want tighter logic around what we do in Frontend vs backend? >>> >>> >>> On Mon, Aug 11, 2014 at 2:53 PM, Daniel Weeks <[email protected]> >>> wrote: >>> >>>> +parquet-dev >>>> >>>> >>>> On Mon, Aug 11, 2014 at 2:49 PM, Daniel Weeks <[email protected]> >>>> wrote: >>>> >>>>> Julien, >>>>> >>>>> We ran into a rather critical issue for us with the ParquetLoader and >>>>> how it manages the pig schema. The fix is trivial (need to store schema >>>>> and other loader specific options in udfcontext). >>>>> >>>>> Can we get someone to review this? You might be the most familiar >>>>> with this code. >>>>> >>>>> JIRA: https://issues.apache.org/jira/browse/PARQUET-70 >>>>> Pull Request: https://github.com/apache/incubator-parquet-mr/pull/36 >>>>> >>>>> Thanks, >>>>> Dan >>>>> >>>>> >>>>> >>>> >>> >> >
