Hey Kaspar, I wrote a PType for supporting arbitrarily serializables as part of some work I was doing on Oryx:
https://github.com/cloudera/oryx/blob/master/computation-common/src/main/java/com/cloudera/oryx/computation/common/types/Serializables.java I never committed it to Crunch proper b/c it's not the sort of thing we try to encourage people to do, though like you, I have been in situations where it was the most expedient solution. ;-) J On Tue, Nov 4, 2014 at 10:50 PM, Kaspar Fischer <[email protected]> wrote: > Hi everybody, > > I am using Crunch with Avros serialization. Some intermediate objects of > my pipeline are POJO's that have fields with generics. When running the > pipeline I get: > > Exception in thread "main" org.apache.avro.AvroTypeException: Unknown > type: T > > at > org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:220) > > at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:374) > > at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:283) > > at > org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:430) > > at > org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:72) > > at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:354) > > at > org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:197) > > at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:374) > > at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:283) > > at > org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:430) > > at > org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:72) > > at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:354) > > at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:154) > > at org.apache.crunch.types.avro.Avros.reflects(Avros.java:275) > > at org.apache.crunch.types.avro.Avros.containers(Avros.java:264) > > at org.apache.crunch.types.avro.Avros.records(Avros.java:252) > > How can I get around this? Is there a way to serialize the object using > Java serialization (or Kryo)? > > Thanks for any pointers, > Kaspar > -- Director of Data Science Cloudera <http://www.cloudera.com> Twitter: @josh_wills <http://twitter.com/josh_wills>
