Hey all - I moved this to https://issues.apache.org/jira/browse/HCATALOG-345
--travis On Fri, Mar 23, 2012 at 2:32 PM, Travis Crawford <[email protected]>wrote: > Hey Sushanth - > > I poked around with this a bit more and have more details. Here's the > snippit that's blowing up. Prebioously it was because there was no arg-less > constructor. Now it blows up because LoadFincBasedInptuFormat uses the > "mapreduce" classes, but now a "mapred" InputFormat is needed. Given that > HCatBaseInputFormat is already using the "mapreduce" packages I'm guessing > it could be modified to wrap input formats of either namespace. I don't > have a proposal about actually doing that yet though. > > private static > org.apache.hadoop.mapred.InputFormat<WritableComparable, Writable> > getMapRedInputFormat (JobConf job, Class inputFormatClass) throws > IOException { > return ( > org.apache.hadoop.mapred.InputFormat<WritableComparable, > Writable>) > ReflectionUtils.newInstance(inputFormatClass, job); > } > > And the log: > > 2012-03-23 21:23:31,269 [main] ERROR org.apache.pig.tools.grunt.Grunt - > ERROR 6017: org.apache.pig.backend.executionengine.ExecException: ERROR > 2118: org.apache.hcatalog.pig.drivers.LoadFuncBasedInputFormat cannot be > cast to org.apache.hadoop.mapred.InputFormat > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:282) > <SNIPPED FOR BREVITY> > *Caused by: java.lang.ClassCastException: > org.apache.hcatalog.pig.drivers.LoadFuncBasedInputFormat cannot be cast to > org.apache.hadoop.mapred.InputFormat* > at > org.apache.hcatalog.mapreduce.HCatBaseInputFormat.getMapRedInputFormat(HCatBaseInputFormat.java:102) > at > org.apache.hcatalog.mapreduce.HCatBaseInputFormat.getSplits(HCatBaseInputFormat.java:159) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:270) > ... 20 more > > Any tips on how best to make this work with existing LoadFunc's? We have a > lot floating around and I'm very interested in making that work. > > Thanks! > Travis > > > > > > On Fri, Mar 23, 2012 at 10:18 AM, Sushanth Sowmyan <[email protected]>wrote: > >> There are no new StorageDriver equivalents for them (no >> LoadFuncBasedSerDe/IF) yet, we just made text and json work directly. >> >> For now though, the idea was that irrespective of what your metadata >> says, by way of ISD/OSD info, it should still work from pig if it >> works from hive. From the stacktrace you pasted on the other thread, >> my guess would be that pig is still picking up the old hcatalog jar >> (using the older HCatLoader) >> >> On Fri, Mar 23, 2012 at 10:06 AM, Francis Liu <[email protected]> wrote: >> > I believe those storage drivers weren't migrated to storagehandlers. So >> > there is no support for it in 0.4 yet. >> > >> > -Francis >> > >> > On 3/22/12 4:15 PM, "Travis Crawford" <[email protected]> wrote: >> > >> >>Hey hcatalog gurus - >> >> >> >>Using a trunk build of HCatalog against our current HiveMetaStore fails, >> >>because data was loaded with references to storage drivers. For example, >> >>my >> >>PART_ID table has records like the following: >> >> >> >>hcat.isd org.apache.hcatalog.pig.drivers.LoadFuncBasedInputDriver >> >>hcat.osd org.apache.hcatalog.pig.drivers.StoreFuncBasedOutputDriver >> >> >> >>With a pre-serde version this works fine. Any tips on the migration path >> >>to >> >>the new serde-based system? >> >> >> >>Thanks! >> >>Travis >> > >> > >> > >
