You can also look at what Vertica did for their Pig connector: https://github.com/vertica/Vertica-Hadoop-Connector/blob/master/pig-connector/com/vertica/pig/VerticaLoader.java
(it's apache licensed, so if you reuse any code, you have to indicate the Vertica copyright and apache license in credits). D On Fri, Aug 31, 2012 at 5:09 PM, Russell Jurney <[email protected]> wrote: > I've thought about that, but getting stuff into Piggybank is hard - you > have to peg it to a Pig release. My plan is to get MySQL working in github, > then generalize into DbStorage in piggybank for 0.11. > > On Fri, Aug 31, 2012 at 4:50 PM, Ruslan Al-Fakikh < > [email protected]> wrote: > >> Terry, Russell, >> >> Just a proposal: >> maybe it should be added to DBStorage? >> >> http://pig.apache.org/docs/r0.8.1/api/org/apache/pig/piggybank/storage/DBStorage.html >> As far as I know it only stores data for now, but i think it can be >> extended to load and store, like PigStorage. >> >> Ruslan >> >> On Sat, Sep 1, 2012 at 3:03 AM, Russell Jurney <[email protected]> >> wrote: >> > That would be awesome - I will generalize it and blog about what a great >> > person you are :D >> > >> > On Fri, Aug 31, 2012 at 3:12 PM, Terry Siu <[email protected]> >> wrote: >> > >> >> Thanks, Russell, I'll dig in to your recommendations. I'd be happy to >> open >> >> source it, but at the moment, it's not exactly general enough. However, >> I >> >> can certainly put it on github for your perusal. >> >> >> >> -Terry >> >> >> >> -----Original Message----- >> >> From: Russell Jurney [mailto:[email protected]] >> >> Sent: Friday, August 31, 2012 3:03 PM >> >> To: [email protected] >> >> Subject: Re: Custom DB Loader UDF >> >> >> >> I don't have an answer, and I'm only learning these APIs myself, but >> >> you're writing something I'm planning on writing very soon - a >> >> MySQL-specific LoadFunc for Pig. I would greatly appreciate it if you >> would >> >> open source it on github or contribute it to Piggybank :) >> >> >> >> The InputSplits should determine the number of mappers, but to debug you >> >> might try forcing it by setting some properties in your script re: >> >> inputsplits (see >> >> >> >> >> https://groups.google.com/a/cloudera.org/forum/?fromgroups=#!topic/cdh-user/1QtL9bBwL0c >> >> ): >> >> >> >> The input split size is detemined by map.min.split.size, dfs.block.size >> >> and mapred.map.tasks. >> >> >> >> goalSize = totalSize / mapred.map.tasks >> >> minSize = max {mapred.min.split.size, minSplitSize} splitSize= max >> >> (minSize, min(goalSize, dfs.block.size)) >> >> >> >> minSplitSize is determined by each InputFormat such as >> >> SequenceFileInputFormat. >> >> >> >> >> >> I'd play around with those and see if you can get it doing what you >> want. >> >> >> >> On Fri, Aug 31, 2012 at 2:02 PM, Terry Siu <[email protected]> >> >> wrote: >> >> >> >> > Hi all, >> >> > >> >> > I know this question has probably been posed multiple times, but I'm >> >> > having difficulty figuring out a couple of aspects of a custom >> >> > LoaderFunc to read from a DB. And yes, I did try to Google my way to >> an >> >> answer. >> >> > Anyhoo, for what it's worth, I have a MySql table that I wish to load >> >> > via Pig. I have the LoaderFunc working using PigServer in a Java app, >> >> > but I noticed the following when my job gets submitted to my MR >> >> > cluster. I generated 6 InputSplits in my custom InputFormat, where >> >> > each split specifies a non-overlapping range/page of records to read >> >> > from. I thought that each InputSplit would correspond to a map task, >> >> > but what I see in the JobTracker is that the submitted job only has 1 >> >> > map task which executes each split serially. Is my understanding even >> >> > correct that a split can be effectively assigned to a single map task? >> >> > If so, can I coerce the submitted MR job to properly get each of my >> >> > splits to execute in its own map task? >> >> > >> >> > Thanks, >> >> > -Terry >> >> > >> >> >> >> >> >> >> >> -- >> >> Russell Jurney twitter.com/rjurney [email protected] >> >> datasyndrome.com >> >> >> > >> > >> > >> > -- >> > Russell Jurney twitter.com/rjurney [email protected] >> datasyndrome.com >> >> >> >> -- >> Best Regards, >> Ruslan Al-Fakikh >> > > > > -- > Russell Jurney twitter.com/rjurney [email protected] datasyndrome.com
