oh the map in DataFrame is actually using a RowEncoder. i left it out
because it wasn't important:
so this doesn't compile:
def f[T]: Dataset[T] => Dataset[T] = dataset => {
val df = dataset.toDF
df.map(row => row)(RowEncoder(df.schema)).as[T]
}
now this does compile. but i don't like it, s
Hi Koert,
map will take the value that has an implicit Encoder to any value that
may or may not have an encoder in scope. That's why I'm asking about
the map function to see what it does.
Pozdrawiam,
Jacek Laskowski
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.l
the map operation works on DataFrame so it doesn't need an encoder. It
could have been any operation on DataFrame. the issue is at the end going
back to Dataset[T] using as[T]. this requires an encoder for T which i know
i already have since i started with a Dataset[T].
i could add an implicit enc
Hi,
Can you show the code from map to reproduce the issue? You can create
encoders using Encoders object (I'm using it all over the place for schema
generation).
Jacek
On 25 Jan 2017 10:19 p.m., "Koert Kuipers" wrote:
> i often run into problems like this:
>
> i need to write a Dataset[T] => D
i often run into problems like this:
i need to write a Dataset[T] => Dataset[T], and inside i need to switch to
DataFrame for a particular operation.
but if i do:
dataset.toDF.map(...).as[T] i get error:
Unable to find encoder for type stored in a Dataset.
i know it has an encoder, because i sta