[
https://issues.apache.org/jira/browse/SPARK-15964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15332224#comment-15332224
]
Michael Armbrust commented on SPARK-15964:
------------------------------------------
Thanks for reporting this, but I believe this is actually specific to the
databricks environment (i.e. it works in the spark shell). The issue here is
that there is a scala compiler bug and as far as we know, you have two choices:
- path dependent types work (i.e. you can refer to a type from another cell in
the next cell)
- multi line (:paste mode in the spark shell) commands work with SQL implicits.
Many more workloads in notebooks depend on the latter, while the former is more
common in the command line REPL. This is why the behavior differs. I'm hoping
the scala 2.11 will give us the best of both worlds if we can fix
https://issues.scala-lang.org/browse/SI-9799
> Assignment to RDD-typed val fails
> ---------------------------------
>
> Key: SPARK-15964
> URL: https://issues.apache.org/jira/browse/SPARK-15964
> Project: Spark
> Issue Type: Bug
> Affects Versions: 2.0.0
> Environment: Notebook on Databricks Community-Edition
> Spark-2.0 preview
> Google Chrome Browser
> Linux Ubuntu 14.04 LTS
> Reporter: Sanjay Dasgupta
>
> Unusual assignment error, giving the following error message:
> found : org.apache.spark.rdd.RDD[Name]
> required : org.apache.spark.rdd.RDD[Name]
> This occurs when the assignment is attempted in a cell that is different from
> the cell in which the item on the right-hand-side is defined. As in the
> following example:
> // CELL-1
> import org.apache.spark.sql.Dataset
> import org.apache.spark.rdd.RDD
> case class Name(number: Int, name: String)
> val names = Seq(Name(1, "one"), Name(2, "two"), Name(3, "three"), Name(4,
> "four"))
> val dataset: Dataset[Name] =
> spark.sparkContext.parallelize(names).toDF.as[Name]
> // CELL-2
> // Error reported here ...
> val dataRdd: RDD[Name] = dataset.rdd
> The error is reported in CELL-2
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
