Richard, I would strongly advise you against trying to mess with PigInputFormat and JobControlCompiler. We know that slicers work (after all, people manage to write hbase loaders and such). Plus the whole input/output/slice/slicer thing is being revised right now, there is a proposal in PIG-966 you can look at if you would like to comment.
In fact I just took your code and substituted some placeholders for various methods and classes (like HSSchema, etc), and I can see Slice methods getting called and everything. What do your location strings look like? Pig tries to be helpful by transforming locations into HDFS paths if the locations are not qualified by a scheme ( something:/ ). This makes file-based LoadFuncs easier to write, but it also means that you have to specify a scheme if you are doing something that's not file-based. Try changing your code to expect the location string to start with some scheme, such as abcd:/ (or perhaps something more meaningful) and things should work. -D 2009/10/28 RichardGUO Fei <gladiato...@hotmail.com>: > > It seems that the real problem is JobControlCompiler.getJob, because this > method removes POLoad and sets PigInputFormat as the input format but the > path passed to my storage is actually a table name rather than a file path! I > think that the LoadFunc should be more flexible. I am trying to fix this at > this moment. > > > > Thanks, > > Richard > >> From: zjf...@gmail.com >> To: pig-dev@hadoop.apache.org >> Subject: RE: Custom Loadfunc problem! >> Date: Tue, 27 Oct 2009 23:40:43 -0800 >> >> >> I mean hadoop's local mode not pig's own local mode >> >> >> -----Original Message----- >> From: Dmitriy Ryaboy [mailto:dvrya...@gmail.com] >> Sent: 2009年10月26日 6:33 >> To: pig-dev@hadoop.apache.org; pig-dev@hadoop.apache.org >> Subject: RE: Custom Loadfunc problem! >> >> Jeff, >> Slicers dont work in local mode, there is an ancient ticket for that on the >> Jira. >> >> Richard -- hard to say whats going on without more code. Think you can come >> up with a simplified version of your loadfunc that fails in a similar >> manner, and share it? >> >> >> >> -----Original Message----- >> From: "zjffdu" <zjf...@gmail.com> >> To: pig-dev@hadoop.apache.org >> Sent: 10/27/2009 1:45 AM >> Subject: RE: Custom Loadfunc problem! >> >> Illustrate will not execute the job, while dump and store will execute it. >> So I think there must be something wrong with your custom slicer. I suggest >> you set breakpoint in your slicer and debug it in map reduce mode locally >> >> >> >> -----Original Message----- >> From: RichardGUO Fei [mailto:gladiato...@hotmail.com] >> Sent: 20091026 0:43 >> To: pig-dev@hadoop.apache.org >> Subject: RE: Custom Loadfunc problem! >> >> >> Hi, >> >> >> >> Btw, my program works with ILLUSTRATE but n >> >> [truncated by sender] >> > > _________________________________________________________________ > MSN十周年庆典，查看MSN注册时间，赢取神秘大奖 > http://10.msn.com.cn
