I think that you are right. But the most weird thing is that there is no LOG outputed.
I can see that the last method invoked was getLocations. And anything after that was not recorded anywhere. Even in the pseudo-distributed mode of Hadoop, if I use System.out.println, the stdout under userlogs should contain a job's log, right? But that did not happen after getLocations. It drove me crazy. I have no idea of if an important method like PigInputFormat.getRecordReader has ever been invoked. Please enlighten me! Thanks, Richard > Date: Wed, 28 Oct 2009 10:57:54 -0400 > Subject: Re: Custom Loadfunc problem! > From: dvrya...@gmail.com > To: pig-dev@hadoop.apache.org > > Richard, > I would strongly advise you against trying to mess with PigInputFormat > and JobControlCompiler. We know that slicers work (after all, people > manage to write hbase loaders and such). Plus the whole > input/output/slice/slicer thing is being revised right now, there is a > proposal in PIG-966 you can look at if you would like to comment. > > In fact I just took your code and substituted some placeholders for > various methods and classes (like HSSchema, etc), and I can see Slice > methods getting called and everything. > > What do your location strings look like? Pig tries to be helpful by > transforming locations into HDFS paths if the locations are not > qualified by a scheme ( something:/ ). This makes file-based LoadFuncs > easier to write, but it also means that you have to specify a scheme > if you are doing something that's not file-based. Try changing your > code to expect the location string to start with some scheme, such as > abcd:/ (or perhaps something more meaningful) and things should work. > > -D > > 2009/10/28 RichardGUO Fei <gladiato...@hotmail.com>: > > > > It seems that the real problem is JobControlCompiler.getJob, because this > > method removes POLoad and sets PigInputFormat as the input format but the > > path passed to my storage is actually a table name rather than a file path! > > I think that the LoadFunc should be more flexible. I am trying to fix this > > at this moment. > > > > > > > > Thanks, > > > > Richard > > > >> From: zjf...@gmail.com > >> To: pig-dev@hadoop.apache.org > >> Subject: RE: Custom Loadfunc problem! > >> Date: Tue, 27 Oct 2009 23:40:43 -0800 > >> > >> > >> I mean hadoop's local mode not pig's own local mode > >> > >> > >> -----Original Message----- > >> From: Dmitriy Ryaboy [mailto:dvrya...@gmail.com] > >> Sent: 2009年10月26日 6:33 > >> To: pig-dev@hadoop.apache.org; pig-dev@hadoop.apache.org > >> Subject: RE: Custom Loadfunc problem! > >> > >> Jeff, > >> Slicers dont work in local mode, there is an ancient ticket for that on the > >> Jira. > >> > >> Richard -- hard to say whats going on without more code. Think you can come > >> up with a simplified version of your loadfunc that fails in a similar > >> manner, and share it? > >> > >> > >> > >> -----Original Message----- > >> From: "zjffdu" <zjf...@gmail.com> > >> To: pig-dev@hadoop.apache.org > >> Sent: 10/27/2009 1:45 AM > >> Subject: RE: Custom Loadfunc problem! > >> > >> Illustrate will not execute the job, while dump and store will execute it. > >> So I think there must be something wrong with your custom slicer. I suggest > >> you set breakpoint in your slicer and debug it in map reduce mode locally > >> > >> > >> > >> -----Original Message----- > >> From: RichardGUO Fei [mailto:gladiato...@hotmail.com] > >> Sent: 20091026 0:43 > >> To: pig-dev@hadoop.apache.org > >> Subject: RE: Custom Loadfunc problem! > >> > >> > >> Hi, > >> > >> > >> > >> Btw, my program works with ILLUSTRATE but n > >> > >> [truncated by sender] > >> > > > > _________________________________________________________________ > > MSN十周年庆典，查看MSN注册时间，赢取神秘大奖 > > http://10.msn.com.cn _________________________________________________________________ 约会说不清地方？来试试微软地图最新msn互动功能！ http://ditu.live.com/?form=TL&swm=1
