I've thought about that, but getting stuff into Piggybank is hard - you have to peg it to a Pig release. My plan is to get MySQL working in github, then generalize into DbStorage in piggybank for 0.11.
On Fri, Aug 31, 2012 at 4:50 PM, Ruslan Al-Fakikh < [email protected]> wrote: > Terry, Russell, > > Just a proposal: > maybe it should be added to DBStorage? > > http://pig.apache.org/docs/r0.8.1/api/org/apache/pig/piggybank/storage/DBStorage.html > As far as I know it only stores data for now, but i think it can be > extended to load and store, like PigStorage. > > Ruslan > > On Sat, Sep 1, 2012 at 3:03 AM, Russell Jurney <[email protected]> > wrote: > > That would be awesome - I will generalize it and blog about what a great > > person you are :D > > > > On Fri, Aug 31, 2012 at 3:12 PM, Terry Siu <[email protected]> > wrote: > > > >> Thanks, Russell, I'll dig in to your recommendations. I'd be happy to > open > >> source it, but at the moment, it's not exactly general enough. However, > I > >> can certainly put it on github for your perusal. > >> > >> -Terry > >> > >> -----Original Message----- > >> From: Russell Jurney [mailto:[email protected]] > >> Sent: Friday, August 31, 2012 3:03 PM > >> To: [email protected] > >> Subject: Re: Custom DB Loader UDF > >> > >> I don't have an answer, and I'm only learning these APIs myself, but > >> you're writing something I'm planning on writing very soon - a > >> MySQL-specific LoadFunc for Pig. I would greatly appreciate it if you > would > >> open source it on github or contribute it to Piggybank :) > >> > >> The InputSplits should determine the number of mappers, but to debug you > >> might try forcing it by setting some properties in your script re: > >> inputsplits (see > >> > >> > https://groups.google.com/a/cloudera.org/forum/?fromgroups=#!topic/cdh-user/1QtL9bBwL0c > >> ): > >> > >> The input split size is detemined by map.min.split.size, dfs.block.size > >> and mapred.map.tasks. > >> > >> goalSize = totalSize / mapred.map.tasks > >> minSize = max {mapred.min.split.size, minSplitSize} splitSize= max > >> (minSize, min(goalSize, dfs.block.size)) > >> > >> minSplitSize is determined by each InputFormat such as > >> SequenceFileInputFormat. > >> > >> > >> I'd play around with those and see if you can get it doing what you > want. > >> > >> On Fri, Aug 31, 2012 at 2:02 PM, Terry Siu <[email protected]> > >> wrote: > >> > >> > Hi all, > >> > > >> > I know this question has probably been posed multiple times, but I'm > >> > having difficulty figuring out a couple of aspects of a custom > >> > LoaderFunc to read from a DB. And yes, I did try to Google my way to > an > >> answer. > >> > Anyhoo, for what it's worth, I have a MySql table that I wish to load > >> > via Pig. I have the LoaderFunc working using PigServer in a Java app, > >> > but I noticed the following when my job gets submitted to my MR > >> > cluster. I generated 6 InputSplits in my custom InputFormat, where > >> > each split specifies a non-overlapping range/page of records to read > >> > from. I thought that each InputSplit would correspond to a map task, > >> > but what I see in the JobTracker is that the submitted job only has 1 > >> > map task which executes each split serially. Is my understanding even > >> > correct that a split can be effectively assigned to a single map task? > >> > If so, can I coerce the submitted MR job to properly get each of my > >> > splits to execute in its own map task? > >> > > >> > Thanks, > >> > -Terry > >> > > >> > >> > >> > >> -- > >> Russell Jurney twitter.com/rjurney [email protected] > >> datasyndrome.com > >> > > > > > > > > -- > > Russell Jurney twitter.com/rjurney [email protected] > datasyndrome.com > > > > -- > Best Regards, > Ruslan Al-Fakikh > -- Russell Jurney twitter.com/rjurney [email protected] datasyndrome.com
