Thanks for your quick reply. I've tried with this encoder:
implicit def RowEncoder: org.apache.spark.sql.Encoder[Row] = org.apache.spark.sql.Encoders.kryo[Row] Using a suggestion from http://stackoverflow.com/questions/36648128/how-to-store-custom-objects-in-a-dataset-in-spark-1-6 How did you setup your encoder? ----- Mail original ----- De: "Sun Rui" <sunrise_...@163.com> À: "Julien Nauroy" <julien.nau...@u-psud.fr> Cc: user@spark.apache.org Envoyé: Samedi 23 Juillet 2016 15:55:21 Objet: Re: Using flatMap on Dataframes with Spark 2.0 I did a try. the schema after flatMap is the same, which is expected. What’s your Row encoder? On Jul 23, 2016, at 20:36, Julien Nauroy < julien.nau...@u-psud.fr > wrote: Hi, I'm trying to call flatMap on a Dataframe with Spark 2.0 (rc5). The code is the following: var data = spark.read.parquet(fileName).flatMap(x => List(x)) Of course it's an overly simplified example, but the result is the same. The dataframe schema goes from this: root |-- field1: double (nullable = true) |-- field2: integer (nullable = true) (etc) to this: root |-- value: binary (nullable = true) Plus I have to provide an encoder for Row. I expect to get the same schema after calling flatMap. Any idea what I could be doing wrong? Best regards, Julien
