[
https://issues.apache.org/jira/browse/PIG-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated PIG-4859:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to Spark branch. Thanks, Liyun!
> Need upgrade snappy-java.version to 1.1.1.3
> -------------------------------------------
>
> Key: PIG-4859
> URL: https://issues.apache.org/jira/browse/PIG-4859
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4859.patch
>
>
> run pig on spark on yarn-client env as following:
> export SPARK_MASTER="yarn-client"
> ./pig -x spark xxx.pig
> Throw error like following:
> {code}
> main] 2016-03-30 16:52:26,115 INFO scheduler.DAGScheduler
> (Logging.scala:logInfo(59)) - Job 0 failed: saveAsNewAPIHadoopDataset at
> StoreConverter.java:101, took 73.980147 s
> 19895 [main] 2016-03-30 16:52:26,119 ERROR spark.JobGraphBuilder
> (JobGraphBuilder.java:sparkOperToRDD(166)) - throw exception in
> sparkOperToRDD:
> 19896 org.apache.spark.SparkException: Job aborted due to stage failure: Task
> 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage
> 0.0 (TID 3, zly1.sh.intel.com): java .lang.UnsatisfiedLinkError:
> org.xerial.snappy.SnappyNative.uncompressedLength(Ljava/lang/Object;II)I
> 19897 at org.xerial.snappy.SnappyNative.uncompressedLength(Native
> Method)
> 19898 at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:541)
> 19899 at
> org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:350)
> 19900 at
> org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:158)
> 19901 at
> org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:142)
> 19902 at
> java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2313)
> 19903 at
> java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2326)
> 19904 at
> java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2797)
> 19905 at
> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802)
> 19906 at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
> 19907 at
> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:64)
> 19908 at
> org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:64)
> 19909 at
> org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:103)
> 19910 at
> org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:216)
> 19911 at
> org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:178)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
