if you are using the kryo encoder, you can only use it to to map to/from
kryo encoded binary data. This is because spark does not understand kryo's
encoding, its just using it as an opaque blob of bytes.
On Mon, May 23, 2016 at 1:28 AM, Han JU wrote:
> Just one more question: does Dataset suppo
Just one more question: does Dataset suppose to be able to cast data to an
avro type? For a very simple format (a string and a long), I can cast it to
a tuple or case class, but not an avro type (also contains only a string
and a long).
The error is like this for this very simple type:
=== Result
What is the error? I would definitely expect it to work with kryo at least.
On Fri, May 20, 2016 at 2:37 AM, Han JU wrote:
> Hello,
>
> I'm looking at the Dataset API in 1.6.1 and also in upcoming 2.0. However
> it does not seems to work with Avro data types:
>
>
> object Datasets extends App {
Hello,
I'm looking at the Dataset API in 1.6.1 and also in upcoming 2.0. However
it does not seems to work with Avro data types:
object Datasets extends App {
val conf = new SparkConf()
conf.setAppName("Dataset")
conf.setMaster("local[2]")
conf.setIfMissing("spark.serializer", classOf[Kr