Hey Josh, Great, very useful!
Thanks a lot for sharing this. Cheers, Kaspar > On 04.11.2014, at 23:13, Josh Wills <[email protected]> wrote: > > Hey Kaspar, > > I wrote a PType for supporting arbitrarily serializables as part of some work > I was doing on Oryx: > > https://github.com/cloudera/oryx/blob/master/computation-common/src/main/java/com/cloudera/oryx/computation/common/types/Serializables.java > > I never committed it to Crunch proper b/c it's not the sort of thing we try > to encourage people to do, though like you, I have been in situations where > it was the most expedient solution. ;-) > > J > >> On Tue, Nov 4, 2014 at 10:50 PM, Kaspar Fischer <[email protected]> >> wrote: >> Hi everybody, >> >> I am using Crunch with Avros serialization. Some intermediate objects of my >> pipeline are POJO's that have fields with generics. When running the >> pipeline I get: >> >> Exception in thread "main" org.apache.avro.AvroTypeException: Unknown type: T >> at >> org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:220) >> at >> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:374) >> at >> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:283) >> at >> org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:430) >> at >> org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:72) >> at >> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:354) >> at >> org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:197) >> at >> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:374) >> at >> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:283) >> at >> org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:430) >> at >> org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:72) >> at >> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:354) >> at >> org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:154) >> at org.apache.crunch.types.avro.Avros.reflects(Avros.java:275) >> at org.apache.crunch.types.avro.Avros.containers(Avros.java:264) >> at org.apache.crunch.types.avro.Avros.records(Avros.java:252) >> >> How can I get around this? Is there a way to serialize the object using Java >> serialization (or Kryo)? >> >> Thanks for any pointers, >> Kaspar > > > > -- > Director of Data Science > Cloudera > Twitter: @josh_wills
