Scott,
I made an effort to address the documentation in https://issues.apache.org/jira/browse/PIG-1370
If you have a chance take a look and let me know if it deals with
the issues you have or if more work is needed.
Alan.
On May 24, 2010, at 11:00 AM, Scott Carey wrote:
I have been using these documents for a couple weeks, implementing
various store and load functionality, and they have been very helpful.
However, there is room for improvement. What is most unclear is
when the API methods get called. Each method should clearly state
in these documents (and the javadoc) when it is called -- front-end
only? back-end only? both? Sometimes this is obvious, other times
it is not.
For example, without looking at the source code its not possible to
tell or infer if pushProjection() is called on the front-end or back-
end, or both. It could be implemented by being called on the front-
end, expecting the loader implementation to persist necessary state
to UDFContext for the back-end, or be called only on the back-end,
or both. One has to look at PigStorage source to see that it
persists the pushProjection information into UDFContext, so its
_probably_ only called on the front-end.
There are also a few types that these interfaces return or are
provided that are completely undocumented. I had to look at the
source code to figure out what ResourceStatistics does, and how
ResourceSchema should be used. RequiredField, RequiredFieldList,
and RequiredFieldResponse are all poorly documented aspects of a
public interface.
On May 21, 2010, at 11:42 AM, Pradeep Kamath wrote:
To add to this, there is also a how-to document on how to go about
writing load/store functions from scratch in Pig 0.7 at
http://wiki.apache.org/pig/Pig070LoadStoreHowTo.
Pradeep
-----Original Message-----
From: Alan Gates [mailto:[email protected]]
Sent: Friday, May 21, 2010 11:33 AM
To: [email protected]
Cc: Eli Collins
Subject: Pig loader 0.6 to 0.7 migration guide
At the Bay Area HUG on Wednesday someone (Eli I think, though I might
be remembering incorrectly) asked if there was a migration guide for
moving Pig load and store functions from 0.6 to 0.7. I said there
was
but I couldn't remember if it had been posted yet or not. In fact it
had already been posted to
http://wiki.apache.org/pig/LoadStoreMigrationGuide
. Also, you can find the list of all incompatible changes for 0.7 at
http://wiki.apache.org/pig/Pig070IncompatibleChanges
. Sorry, I should have included those links in my original slides.
Alan.
