[
https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15430215#comment-15430215
]
DjvuLee edited comment on SPARK-3630 at 8/22/16 7:10 AM:
---------------------------------------------------------
Can I know how much data do you test? We encounter this error in our
production, our data is about several TB. The Spark version is 1.6.1, and the
snappy version is 1.1.2.4。When the data is small, we never encounter this error.
was (Author: djvulee):
How much data do you test? we encounter this error in our production. Our data
is about several TB. The Spark version is 1.6.1, and the snappy version is
1.1.2.4。
> Identify cause of Kryo+Snappy PARSING_ERROR
> -------------------------------------------
>
> Key: SPARK-3630
> URL: https://issues.apache.org/jira/browse/SPARK-3630
> Project: Spark
> Issue Type: Task
> Components: Spark Core
> Affects Versions: 1.1.0, 1.2.0
> Reporter: Andrew Ash
> Assignee: Josh Rosen
>
> A recent GraphX commit caused non-deterministic exceptions in unit tests so
> it was reverted (see SPARK-3400).
> Separately, [~aash] observed the same exception stacktrace in an
> application-specific Kryo registrator:
> {noformat}
> com.esotericsoftware.kryo.KryoException: java.io.IOException: failed to
> uncompress the chunk: PARSING_ERROR(2)
> com.esotericsoftware.kryo.io.Input.fill(Input.java:142)
> com.esotericsoftware.kryo.io.Input.require(Input.java:169)
> com.esotericsoftware.kryo.io.Input.readInt(Input.java:325)
> com.esotericsoftware.kryo.io.Input.readFloat(Input.java:624)
> com.esotericsoftware.kryo.serializers.DefaultSerializers$FloatSerializer.read(DefaultSerializers.java:127)
>
> com.esotericsoftware.kryo.serializers.DefaultSerializers$FloatSerializer.read(DefaultSerializers.java:117)
>
> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
>
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
>
> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
> ...
> {noformat}
> This ticket is to identify the cause of the exception in the GraphX commit so
> the faulty commit can be fixed and merged back into master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
