Hi,
I can't seem to use Dataset using case classes (or tuples) to select per field:
scala> final case class Text(id: Int, text: String)
warning: there was one unchecked warning; re-run with -unchecked for details
defined class Text
scala> val ds = Seq(Text(0, "hello"), Text(1, "world")).toDF.as[Text]
ds: org.apache.spark.sql.Dataset[Text] = [id: int, text: string]
// query per field as symbol works fine
scala> ds.select('id).show
+---+
| id|
+---+
| 0|
| 1|
+---+
// but not per field as Scala attribute
scala> ds.select(_.id).show
<console>:40: error: missing parameter type for expanded function
((x$1) => x$1.id)
ds.select(_.id).show
^
Is this supposed to work in Spark 2.0 (today's build)?
BTW, Why is Seq(Text(0, "hello"), Text(1, "world")).as[Text] not possible?
Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
