[
https://issues.apache.org/jira/browse/TOREE-464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luciano Resende updated TOREE-464:
----------------------------------
Comment: was deleted
(was: !image-2021-05-31-15-08-28-903.png!)
> Failing actions on RDDs with non-primitive objects
> --------------------------------------------------
>
> Key: TOREE-464
> URL: https://issues.apache.org/jira/browse/TOREE-464
> Project: TOREE
> Issue Type: Bug
> Affects Versions: 0.2.0
> Reporter: Alexander Anokhin
> Priority: Major
> Fix For: 0.5.0
>
> Attachments: test_case.ipynb, toree-464.png
>
>
> Running in non-local mode actions on RDDs fail when RDDs hold custom
> non-primitive objects. It affects current master version
> [0.3.0|https://github.com/apache/incubator-toree] as well as released
> versions
> [Rc1|https://github.com/apache/incubator-toree/releases/tag/v0.2.0-incubating-rc1],
>
> [Rc2|https://github.com/apache/incubator-toree/releases/tag/v0.2.0-incubating-rc2]
> and
> [Rc3|https://github.com/apache/incubator-toree/releases/tag/v0.2.0-incubating-rc3].
> Example:
> {code:scala}
> Cell 1: case class A(i: Int)
> Cell 2: val events = sc.parallelize((1 to 5).toSeq).map { i => A(i) }
> Cell 3: println(events.count())
> Results: "java.lang.NoClassDefFoundError: Could not initialize class ..."
> {code}
> However, it does work if code from the cells 1 and 2 is combined into one
> cell. In that case actions on such RDDs work correctly, but case class
> definition should be prepended to every cell with problem RDD.
> Example:
> {code:scala}
> Cell 1: case class A(i: Int); val events = sc.parallelize((1 to 5).toSeq).map
> { i => A(i) }
> Cell 2: println(events.count())
> Results: "5"
> {code}
>
> Test notebook is in the attachment [^test_case.ipynb] .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
