[ https://issues.apache.org/jira/browse/CRUNCH-540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14617568#comment-14617568 ]
Tyler Kohn commented on CRUNCH-540: ----------------------------------- added initialize before calling getDetachedValue: ptype.initialize(new Configuration()); S perfInfo = ptype.getDetachedValue(s); this did work (my tests passed) but is that the suggested implementation? > AvroReflectDeepCopier not serializable (but crunch is trying!) > -------------------------------------------------------------- > > Key: CRUNCH-540 > URL: https://issues.apache.org/jira/browse/CRUNCH-540 > Project: Crunch > Issue Type: Bug > Components: Core > Affects Versions: 0.11.0, 0.12.0 > Reporter: Tyler Kohn > Assignee: Josh Wills > Attachments: CRUNCH-540.patch > > > When I try to use avro objects in a crunch job and it uses reflect to > generate the schema crunch tries to verify the serialization of the entire > ptype object which includes AvroReflectDeepCopier and throws the following > exception: > java.lang.IllegalStateException: named 'null' cannot be serialized > at > org.apache.crunch.impl.mem.collect.MemCollection.verifySerializable(MemCollection.java:104) > at > org.apache.crunch.impl.mem.collect.MemCollection.parallelDo(MemCollection.java:123) > at > org.apache.crunch.impl.mem.collect.MemCollection.parallelDo(MemCollection.java:117) > at > org.apache.crunch.impl.mem.collect.MemCollection.parallelDo(MemCollection.java:112) > ... > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:68) > Caused by: org.apache.commons.lang.SerializationException: > java.io.NotSerializableException: org.apache.avro.io.BinaryDecoder > - field (class > "org.apache.crunch.types.avro.AvroDeepCopier$AvroReflectDeepCopier", name: > "binaryDecoder", type: "class org.apache.avro.io.BinaryDecoder") > - object (class > "org.apache.crunch.types.avro.AvroDeepCopier$AvroReflectDeepCopier", > org.apache.crunch.types.avro.AvroDeepCopier$AvroReflectDeepCopier@1c25b8a7) > - field (class "org.apache.crunch.types.avro.AvroType", name: > "deepCopier", type: "interface org.apache.crunch.types.DeepCopier") > - object (class "org.apache.crunch.types.avro.AvroType", > org.apache.crunch.types.avro.AvroType@6889d368) > - field (class "com.rr.crunch.util.AggregationUtils$5", name: > "val$ptype", type: "interface org.apache.crunch.types.PType") > - root object (class "com.rr.crunch.util.AggregationUtils$5", > com.rr.crunch.util.AggregationUtils$5@fac80) > at > org.apache.commons.lang.SerializationUtils.serialize(SerializationUtils.java:112) > at > org.apache.commons.lang.SerializationUtils.serialize(SerializationUtils.java:134) > at > org.apache.crunch.impl.mem.collect.MemCollection.verifySerializable(MemCollection.java:102) > ... 29 more > I'm assuming that other than the valueClass, all the member variables below > should be transient. > public static class AvroReflectDeepCopier<T> extends AvroDeepCopier<T> { > private DatumReader<T> datumReader; > private DatumWriter<T> datumWriter; > private BinaryEncoder binaryEncoder; > private BinaryDecoder binaryDecoder; > private final Class<T> valueClass; > Help greatly appreciated and let me know if this isn't an actual bug and is > user error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)