[
https://issues.apache.org/jira/browse/TINKERPOP-1426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Xin Yu updated TINKERPOP-1426:
-----------------------------------
Description:
There is description for Serializer in spark:
* Implementations of this trait should implement:
*
* 1. a zero-arg constructor or a constructor that accepts a
[[org.apache.spark.SparkConf]]
* as parameter. If both constructors are defined, the latter takes precedence.
*
* 2. Java serialization interface.
Class GryoSerializer in Tinkerepop extends Serializer, but does not implement
java.io.Serializable.
It works well before Spark 2.0. But with Spark 2.0, it changed by SPARK-13926
for Dependency,scala.
Gyro and all its fields must implement Java serialisation interface, otherwise
hundreds of test cases are failed as:
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
Task not serializable: java.io.NotSerializableException:
org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer
Serialization stack:
- object not serializable (class:
org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer, value:
org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer@1b12ec8e)
- field (class: org.apache.spark.ShuffleDependency, name: serializer,
type: class org.apache.spark.serializer.Serializer)
- object (class org.apache.spark.ShuffleDependency,
org.apache.spark.ShuffleDependency@7a4f876a)
- field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
- object (class scala.Tuple2, (MapPartitionsRDD[1] at mapToPair at
InputFormatRDD.java:46,org.apache.spark.ShuffleDependency@7a4f876a))
was:
In spark description for Serializer, there is:
* Implementations of this trait should implement:
*
* 1. a zero-arg constructor or a constructor that accepts a
[[org.apache.spark.SparkConf]]
* as parameter. If both constructors are defined, the latter takes precedence.
*
* 2. Java serialization interface.
Tinkerepop Class GryoSerializer extends Serializer, but does not implement
serialisation. It works well before Spark 2.0. But with Spark 2.0, with the
change for SPARK-13926 for Dependency,scala. Gyro must implement Java
serialisation interface, otherwise hundreds of test cases are failed as:
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
Task not serializable: java.io.NotSerializableException:
org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer
Serialization stack:
- object not serializable (class:
org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer, value:
org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer@1b12ec8e)
- field (class: org.apache.spark.ShuffleDependency, name: serializer,
type: class org.apache.spark.serializer.Serializer)
- object (class org.apache.spark.ShuffleDependency,
org.apache.spark.ShuffleDependency@7a4f876a)
- field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
- object (class scala.Tuple2, (MapPartitionsRDD[1] at mapToPair at
InputFormatRDD.java:46,org.apache.spark.ShuffleDependency@7a4f876a))
> GryoSerializer should implement Java serialization interface
> ------------------------------------------------------------
>
> Key: TINKERPOP-1426
> URL: https://issues.apache.org/jira/browse/TINKERPOP-1426
> Project: TinkerPop
> Issue Type: Bug
> Components: io
> Affects Versions: 3.2.1
> Reporter: Chen Xin Yu
>
> There is description for Serializer in spark:
> * Implementations of this trait should implement:
> *
> * 1. a zero-arg constructor or a constructor that accepts a
> [[org.apache.spark.SparkConf]]
> * as parameter. If both constructors are defined, the latter takes
> precedence.
> *
> * 2. Java serialization interface.
> Class GryoSerializer in Tinkerepop extends Serializer, but does not implement
> java.io.Serializable.
> It works well before Spark 2.0. But with Spark 2.0, it changed by SPARK-13926
> for Dependency,scala.
> Gyro and all its fields must implement Java serialisation interface,
> otherwise hundreds of test cases are failed as:
> Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
> Task not serializable: java.io.NotSerializableException:
> org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer
> Serialization stack:
> - object not serializable (class:
> org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer, value:
> org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer@1b12ec8e)
> - field (class: org.apache.spark.ShuffleDependency, name: serializer,
> type: class org.apache.spark.serializer.Serializer)
> - object (class org.apache.spark.ShuffleDependency,
> org.apache.spark.ShuffleDependency@7a4f876a)
> - field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
> - object (class scala.Tuple2, (MapPartitionsRDD[1] at mapToPair at
> InputFormatRDD.java:46,org.apache.spark.ShuffleDependency@7a4f876a))
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
