[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps
[
https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014055#comment-15014055
]
Davies Liu commented on SPARK-11016:
[~sowen] I tried to assigned this to [~drcrallen] (who did the job), but it
always tell me that the user does not exists, does my account not have the
permission to access some profiles?
> Spark fails when running with a task that requires a more recent version of
> RoaringBitmaps
> --
>
> Key: SPARK-11016
> URL: https://issues.apache.org/jira/browse/SPARK-11016
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.0
>Reporter: Charles Allen
> Fix For: 1.6.0
>
>
> The following error appears during Kryo init whenever a more recent version
> (>0.5.0) of Roaring bitmaps is required by a job.
> org/roaringbitmap/RoaringArray$Element was removed in 0.5.0
> {code}
> A needed class was not found. This could be due to an error in your runpath.
> Missing class: org/roaringbitmap/RoaringArray$Element
> java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringArray$Element
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala:338)
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala)
> at
> org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:93)
> at
> org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:237)
> at
> org.apache.spark.serializer.KryoSerializerInstance.(KryoSerializer.scala:222)
> at
> org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:138)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201)
> at
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102)
> at
> org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:85)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
> at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1318)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1006)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1003)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:1003)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:818)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:816)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.textFile(SparkContext.scala:816)
> {code}
> See https://issues.apache.org/jira/browse/SPARK-5949 for related info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps
[
https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014056#comment-15014056
]
Sean Owen commented on SPARK-11016:
---
No it's just that people have to be in the Contributors role to be assigned.
Long story. It's done now.
> Spark fails when running with a task that requires a more recent version of
> RoaringBitmaps
> --
>
> Key: SPARK-11016
> URL: https://issues.apache.org/jira/browse/SPARK-11016
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.0
>Reporter: Charles Allen
>Assignee: Charles Allen
> Fix For: 1.6.0
>
>
> The following error appears during Kryo init whenever a more recent version
> (>0.5.0) of Roaring bitmaps is required by a job.
> org/roaringbitmap/RoaringArray$Element was removed in 0.5.0
> {code}
> A needed class was not found. This could be due to an error in your runpath.
> Missing class: org/roaringbitmap/RoaringArray$Element
> java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringArray$Element
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala:338)
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala)
> at
> org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:93)
> at
> org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:237)
> at
> org.apache.spark.serializer.KryoSerializerInstance.(KryoSerializer.scala:222)
> at
> org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:138)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201)
> at
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102)
> at
> org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:85)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
> at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1318)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1006)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1003)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:1003)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:818)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:816)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.textFile(SparkContext.scala:816)
> {code}
> See https://issues.apache.org/jira/browse/SPARK-5949 for related info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps
[
https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014078#comment-15014078
]
Davies Liu commented on SPARK-11016:
Thanks!
> Spark fails when running with a task that requires a more recent version of
> RoaringBitmaps
> --
>
> Key: SPARK-11016
> URL: https://issues.apache.org/jira/browse/SPARK-11016
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.0
>Reporter: Charles Allen
>Assignee: Charles Allen
> Fix For: 1.6.0
>
>
> The following error appears during Kryo init whenever a more recent version
> (>0.5.0) of Roaring bitmaps is required by a job.
> org/roaringbitmap/RoaringArray$Element was removed in 0.5.0
> {code}
> A needed class was not found. This could be due to an error in your runpath.
> Missing class: org/roaringbitmap/RoaringArray$Element
> java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringArray$Element
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala:338)
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala)
> at
> org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:93)
> at
> org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:237)
> at
> org.apache.spark.serializer.KryoSerializerInstance.(KryoSerializer.scala:222)
> at
> org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:138)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201)
> at
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102)
> at
> org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:85)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
> at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1318)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1006)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1003)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:1003)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:818)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:816)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.textFile(SparkContext.scala:816)
> {code}
> See https://issues.apache.org/jira/browse/SPARK-5949 for related info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps
[
https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007565#comment-15007565
]
Davies Liu commented on SPARK-11016:
[~charles.al...@acxiom.com] Could you send your patch to
github.com/apache/spark ?
> Spark fails when running with a task that requires a more recent version of
> RoaringBitmaps
> --
>
> Key: SPARK-11016
> URL: https://issues.apache.org/jira/browse/SPARK-11016
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.0
>Reporter: Charles Allen
> Fix For: 1.6.0
>
>
> The following error appears during Kryo init whenever a more recent version
> (>0.5.0) of Roaring bitmaps is required by a job.
> org/roaringbitmap/RoaringArray$Element was removed in 0.5.0
> {code}
> A needed class was not found. This could be due to an error in your runpath.
> Missing class: org/roaringbitmap/RoaringArray$Element
> java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringArray$Element
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala:338)
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala)
> at
> org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:93)
> at
> org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:237)
> at
> org.apache.spark.serializer.KryoSerializerInstance.(KryoSerializer.scala:222)
> at
> org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:138)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201)
> at
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102)
> at
> org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:85)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
> at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1318)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1006)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1003)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:1003)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:818)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:816)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.textFile(SparkContext.scala:816)
> {code}
> See https://issues.apache.org/jira/browse/SPARK-5949 for related info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps
[
https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007640#comment-15007640
]
Charles Allen commented on SPARK-11016:
---
[~davies] Was in a meeting, looks like you got it :)
> Spark fails when running with a task that requires a more recent version of
> RoaringBitmaps
> --
>
> Key: SPARK-11016
> URL: https://issues.apache.org/jira/browse/SPARK-11016
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.0
>Reporter: Charles Allen
> Fix For: 1.6.0
>
>
> The following error appears during Kryo init whenever a more recent version
> (>0.5.0) of Roaring bitmaps is required by a job.
> org/roaringbitmap/RoaringArray$Element was removed in 0.5.0
> {code}
> A needed class was not found. This could be due to an error in your runpath.
> Missing class: org/roaringbitmap/RoaringArray$Element
> java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringArray$Element
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala:338)
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala)
> at
> org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:93)
> at
> org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:237)
> at
> org.apache.spark.serializer.KryoSerializerInstance.(KryoSerializer.scala:222)
> at
> org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:138)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201)
> at
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102)
> at
> org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:85)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
> at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1318)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1006)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1003)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:1003)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:818)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:816)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.textFile(SparkContext.scala:816)
> {code}
> See https://issues.apache.org/jira/browse/SPARK-5949 for related info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps
[
https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007620#comment-15007620
]
Apache Spark commented on SPARK-11016:
--
User 'davies' has created a pull request for this issue:
https://github.com/apache/spark/pull/9748
> Spark fails when running with a task that requires a more recent version of
> RoaringBitmaps
> --
>
> Key: SPARK-11016
> URL: https://issues.apache.org/jira/browse/SPARK-11016
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.0
>Reporter: Charles Allen
> Fix For: 1.6.0
>
>
> The following error appears during Kryo init whenever a more recent version
> (>0.5.0) of Roaring bitmaps is required by a job.
> org/roaringbitmap/RoaringArray$Element was removed in 0.5.0
> {code}
> A needed class was not found. This could be due to an error in your runpath.
> Missing class: org/roaringbitmap/RoaringArray$Element
> java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringArray$Element
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala:338)
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala)
> at
> org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:93)
> at
> org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:237)
> at
> org.apache.spark.serializer.KryoSerializerInstance.(KryoSerializer.scala:222)
> at
> org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:138)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201)
> at
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102)
> at
> org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:85)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
> at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1318)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1006)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1003)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:1003)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:818)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:816)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.textFile(SparkContext.scala:816)
> {code}
> See https://issues.apache.org/jira/browse/SPARK-5949 for related info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps
[
https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984897#comment-14984897
]
Sean Owen commented on SPARK-11016:
---
This was resolved by removing use of roaringbitmaps, in the end
> Spark fails when running with a task that requires a more recent version of
> RoaringBitmaps
> --
>
> Key: SPARK-11016
> URL: https://issues.apache.org/jira/browse/SPARK-11016
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.0
>Reporter: Charles Allen
>Assignee: Liang-Chi Hsieh
> Fix For: 1.6.0
>
>
> The following error appears during Kryo init whenever a more recent version
> (>0.5.0) of Roaring bitmaps is required by a job.
> org/roaringbitmap/RoaringArray$Element was removed in 0.5.0
> {code}
> A needed class was not found. This could be due to an error in your runpath.
> Missing class: org/roaringbitmap/RoaringArray$Element
> java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringArray$Element
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala:338)
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala)
> at
> org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:93)
> at
> org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:237)
> at
> org.apache.spark.serializer.KryoSerializerInstance.(KryoSerializer.scala:222)
> at
> org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:138)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201)
> at
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102)
> at
> org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:85)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
> at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1318)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1006)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1003)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:1003)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:818)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:816)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.textFile(SparkContext.scala:816)
> {code}
> See https://issues.apache.org/jira/browse/SPARK-5949 for related info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps
[
https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14971027#comment-14971027
]
Apache Spark commented on SPARK-11016:
--
User 'viirya' has created a pull request for this issue:
https://github.com/apache/spark/pull/9243
> Spark fails when running with a task that requires a more recent version of
> RoaringBitmaps
> --
>
> Key: SPARK-11016
> URL: https://issues.apache.org/jira/browse/SPARK-11016
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.0
>Reporter: Charles Allen
>
> The following error appears during Kryo init whenever a more recent version
> (>0.5.0) of Roaring bitmaps is required by a job.
> org/roaringbitmap/RoaringArray$Element was removed in 0.5.0
> {code}
> A needed class was not found. This could be due to an error in your runpath.
> Missing class: org/roaringbitmap/RoaringArray$Element
> java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringArray$Element
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala:338)
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala)
> at
> org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:93)
> at
> org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:237)
> at
> org.apache.spark.serializer.KryoSerializerInstance.(KryoSerializer.scala:222)
> at
> org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:138)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201)
> at
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102)
> at
> org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:85)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
> at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1318)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1006)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1003)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:1003)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:818)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:816)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.textFile(SparkContext.scala:816)
> {code}
> See https://issues.apache.org/jira/browse/SPARK-5949 for related info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps
[
https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970783#comment-14970783
]
Sean Owen commented on SPARK-11016:
---
NB: the resolution here may be to simply remove usage of roaringbitmaps:
https://github.com/apache/spark/pull/9243
> Spark fails when running with a task that requires a more recent version of
> RoaringBitmaps
> --
>
> Key: SPARK-11016
> URL: https://issues.apache.org/jira/browse/SPARK-11016
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.0
>Reporter: Charles Allen
>
> The following error appears during Kryo init whenever a more recent version
> (>0.5.0) of Roaring bitmaps is required by a job.
> org/roaringbitmap/RoaringArray$Element was removed in 0.5.0
> {code}
> A needed class was not found. This could be due to an error in your runpath.
> Missing class: org/roaringbitmap/RoaringArray$Element
> java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringArray$Element
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala:338)
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala)
> at
> org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:93)
> at
> org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:237)
> at
> org.apache.spark.serializer.KryoSerializerInstance.(KryoSerializer.scala:222)
> at
> org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:138)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201)
> at
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102)
> at
> org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:85)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
> at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1318)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1006)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1003)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:1003)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:818)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:816)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.textFile(SparkContext.scala:816)
> {code}
> See https://issues.apache.org/jira/browse/SPARK-5949 for related info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps
[
https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964762#comment-14964762
]
Sean Owen commented on SPARK-11016:
---
Yeah that's a good start. It's bridging Kryo with the particular serialization
methods exposed by this class, though I think so far you are only supporting
{{RoaringBitmap}}? because these methods aren't defined by an interface I think
you'd have to write a glue class like you have here for each one.
However because they implement {{Externalizable}} which just delegates to these
custom methods, it seems like you could do something quite similar with
{{ObjectInput}} and {{ObjectOutput}} to support any {{Externalizable}} object
and trivially support the other RoaringBitmap classes, which will be necessary
anyway. How about going that way? I'd support that.
> Spark fails when running with a task that requires a more recent version of
> RoaringBitmaps
> --
>
> Key: SPARK-11016
> URL: https://issues.apache.org/jira/browse/SPARK-11016
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.0
>Reporter: Charles Allen
>
> The following error appears during Kryo init whenever a more recent version
> (>0.5.0) of Roaring bitmaps is required by a job.
> org/roaringbitmap/RoaringArray$Element was removed in 0.5.0
> {code}
> A needed class was not found. This could be due to an error in your runpath.
> Missing class: org/roaringbitmap/RoaringArray$Element
> java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringArray$Element
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala:338)
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala)
> at
> org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:93)
> at
> org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:237)
> at
> org.apache.spark.serializer.KryoSerializerInstance.(KryoSerializer.scala:222)
> at
> org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:138)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201)
> at
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102)
> at
> org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:85)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
> at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1318)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1006)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1003)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:1003)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:818)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:816)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.textFile(SparkContext.scala:816)
> {code}
> See https://issues.apache.org/jira/browse/SPARK-5949 for related info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps
[
https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964043#comment-14964043
]
Charles Allen commented on SPARK-11016:
---
[~srowen] I confirmed locally that https://github.com/metamx/spark/pull/1
prevents this error, but as per your prior comment a "more correct"
implementation would probably provide a Kryo Externalizable bridge of some
kind.
> Spark fails when running with a task that requires a more recent version of
> RoaringBitmaps
> --
>
> Key: SPARK-11016
> URL: https://issues.apache.org/jira/browse/SPARK-11016
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.0
>Reporter: Charles Allen
>
> The following error appears during Kryo init whenever a more recent version
> (>0.5.0) of Roaring bitmaps is required by a job.
> org/roaringbitmap/RoaringArray$Element was removed in 0.5.0
> {code}
> A needed class was not found. This could be due to an error in your runpath.
> Missing class: org/roaringbitmap/RoaringArray$Element
> java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringArray$Element
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala:338)
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala)
> at
> org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:93)
> at
> org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:237)
> at
> org.apache.spark.serializer.KryoSerializerInstance.(KryoSerializer.scala:222)
> at
> org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:138)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201)
> at
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102)
> at
> org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:85)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
> at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1318)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1006)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1003)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:1003)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:818)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:816)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.textFile(SparkContext.scala:816)
> {code}
> See https://issues.apache.org/jira/browse/SPARK-5949 for related info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps
[
https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951772#comment-14951772
]
Sean Owen commented on SPARK-11016:
---
Yes, I get that roaringbitmap has a particular serialization mechanism that
Kryo has to be taught to use. I think the answer to my dumb question was: yes
Spark still uses roaringbitmap so it has to ensure Kryo knows how to serialize
it, including registering serializers. Yes you're doing the right thing then.
These classes implement Externalizable but not Serializable; I think the
KryoJavaSerializer could be registered to delegate serialization to the
correct, custom Java serialization these classes define. But they're not
Serializable. I wonder if we can build a KryoJavaExternalizableSerializer to do
something similar automatically? that's tidy. Then it just needs to register
roaringbitmaps classes to use this.
Otherwise, if it has to be bridged by hand, then since DataOutputStream is a
DataOutput that can wrap an OutputStream, and you can get an OutputStream from
KryoOutput, it seems possible.
> Spark fails when running with a task that requires a more recent version of
> RoaringBitmaps
> --
>
> Key: SPARK-11016
> URL: https://issues.apache.org/jira/browse/SPARK-11016
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.0
>Reporter: Charles Allen
>
> The following error appears during Kryo init whenever a more recent version
> (>0.5.0) of Roaring bitmaps is required by a job.
> org/roaringbitmap/RoaringArray$Element was removed in 0.5.0
> {code}
> A needed class was not found. This could be due to an error in your runpath.
> Missing class: org/roaringbitmap/RoaringArray$Element
> java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringArray$Element
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala:338)
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala)
> at
> org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:93)
> at
> org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:237)
> at
> org.apache.spark.serializer.KryoSerializerInstance.(KryoSerializer.scala:222)
> at
> org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:138)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201)
> at
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102)
> at
> org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:85)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
> at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1318)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1006)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1003)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:1003)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:818)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:816)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.textFile(SparkContext.scala:816)
> {code}
> See https://issues.apache.org/jira/browse/SPARK-5949 for related info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps
[
https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950152#comment-14950152
]
Sean Owen commented on SPARK-11016:
---
This is my ignorance, but is a proper serializer registered for roaringbitmaps
classes in your app (or somehow by kryo by default)? Otherwise, relying on the
default serialization may not work, indeed. This isn't a spark problem though.
> Spark fails when running with a task that requires a more recent version of
> RoaringBitmaps
> --
>
> Key: SPARK-11016
> URL: https://issues.apache.org/jira/browse/SPARK-11016
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.0
>Reporter: Charles Allen
>
> The following error appears during Kryo init whenever a more recent version
> (>0.5.0) of Roaring bitmaps is required by a job.
> org/roaringbitmap/RoaringArray$Element was removed in 0.5.0
> {code}
> A needed class was not found. This could be due to an error in your runpath.
> Missing class: org/roaringbitmap/RoaringArray$Element
> java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringArray$Element
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala:338)
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala)
> at
> org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:93)
> at
> org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:237)
> at
> org.apache.spark.serializer.KryoSerializerInstance.(KryoSerializer.scala:222)
> at
> org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:138)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201)
> at
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102)
> at
> org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:85)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
> at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1318)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1006)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1003)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:1003)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:818)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:816)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.textFile(SparkContext.scala:816)
> {code}
> See https://issues.apache.org/jira/browse/SPARK-5949 for related info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps
[
https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950651#comment-14950651
]
Charles Allen commented on SPARK-11016:
---
[~srowen] As mentioned in
https://issues.apache.org/jira/browse/SPARK-5949?focusedCommentId=14949819=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14949819
spark is relying on native Kryo serde for RoaringBitmap stuff in
KryoSerializer:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala#L368
including the protected Element class:
https://github.com/lemire/RoaringBitmap/blob/RoaringBitmap-0.4.5/src/main/java/org/roaringbitmap/RoaringArray.java#L361
which was removed in 0.5.0 and later (Spark is on 0.4.5 currently)
The SerDe method sanctioned by the RoaringBitmap library is to use the
serialize and deserialize methods provided by the RoaringBitmap or RoaringArray
object. Access to a protected class causes conflicts if a 0.5.0 or later
version of the RoaringBitmap library is used because Spark will unavoidably
fail when it tries to register everything in
org.apache.spark.serializer.KryoSerializer#toRegister , including the
no-longer-existing protected inner static class
I did a quick jab at a patch locally by registering RoaringBitmap and
RoaringArray with a com.esotericsoftware.kryo.Serializer, but it is not clear
how close KryoInput and KryoOutput are to DataInput / DataOutput, which means a
bridging approach might violate the contract of one or the other.
> Spark fails when running with a task that requires a more recent version of
> RoaringBitmaps
> --
>
> Key: SPARK-11016
> URL: https://issues.apache.org/jira/browse/SPARK-11016
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 1.4.0
>Reporter: Charles Allen
>
> The following error appears during Kryo init whenever a more recent version
> (>0.5.0) of Roaring bitmaps is required by a job.
> org/roaringbitmap/RoaringArray$Element was removed in 0.5.0
> {code}
> A needed class was not found. This could be due to an error in your runpath.
> Missing class: org/roaringbitmap/RoaringArray$Element
> java.lang.NoClassDefFoundError: org/roaringbitmap/RoaringArray$Element
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala:338)
> at
> org.apache.spark.serializer.KryoSerializer$.(KryoSerializer.scala)
> at
> org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:93)
> at
> org.apache.spark.serializer.KryoSerializerInstance.borrowKryo(KryoSerializer.scala:237)
> at
> org.apache.spark.serializer.KryoSerializerInstance.(KryoSerializer.scala:222)
> at
> org.apache.spark.serializer.KryoSerializer.newInstance(KryoSerializer.scala:138)
> at
> org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:201)
> at
> org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:102)
> at
> org.apache.spark.broadcast.TorrentBroadcast.(TorrentBroadcast.scala:85)
> at
> org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
> at
> org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63)
> at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1318)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1006)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1.apply(SparkContext.scala:1003)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.hadoopFile(SparkContext.scala:1003)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:818)
> at
> org.apache.spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:816)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
> at org.apache.spark.SparkContext.withScope(SparkContext.scala:700)
> at org.apache.spark.SparkContext.textFile(SparkContext.scala:816)
> {code}
> See https://issues.apache.org/jira/browse/SPARK-5949 for related info
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
