[
https://issues.apache.org/jira/browse/PIG-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16010247#comment-16010247
]
Nandor Kollar commented on PIG-5134:
------------------------------------
[~rohini]
Attached PIG-5134_3.patch. Fixed the wildcard imports and removed the test from
the excluded tests. I'm wondering if this we should do the same patch for the
other two Avro wrapper classes (bag and map wrapper), what do you think?
> Fix TestAvroStorage unit test in Spark mode
> -------------------------------------------
>
> Key: PIG-5134
> URL: https://issues.apache.org/jira/browse/PIG-5134
> Project: Pig
> Issue Type: Bug
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: Nandor Kollar
> Fix For: spark-branch
>
> Attachments: PIG-5134_2.patch, PIG-5134_3.patch, PIG-5134.patch
>
>
> It seems that test fails, because Avro GenericData#Record doesn't implement
> Serializable interface:
> {code}
> 2017-02-23 09:14:41,887 ERROR [main] spark.JobGraphBuilder
> (JobGraphBuilder.java:sparkOperToRDD(183)) - throw exception in
> sparkOperToRDD:
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0
> in stage 9.0 (TID 9) had a not serializable result:
> org.apache.avro.generic.GenericData$Record
> Serialization stack:
> - object not serializable (class:
> org.apache.avro.generic.GenericData$Record, value: {"key": "stuff in closet",
> "value1": {"thing": "hat", "count": 7}, "value2": {"thing": "coat", "count":
> 2}})
> - field (class: org.apache.pig.impl.util.avro.AvroTupleWrapper, name:
> avroObject, type: interface org.apache.avro.generic.IndexedRecord)
> - object (class org.apache.pig.impl.util.avro.AvroTupleWrapper,
> org.apache.pig.impl.util.avro.AvroTupleWrapper@3d3a58c1)
> at
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
> at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
> {code}
> The failing tests is a new test introduced with merging trunk to spark
> branch, that's why we didn't see this error before.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
