2.0.2 with scala 2.11
On Tue, May 9, 2017 at 11:30 AM, Michael Armbrust <mich...@databricks.com>
wrote:
> Which version of Spark?
>
> On Tue, May 9, 2017 at 11:28 AM, Yang <teddyyyy...@gmail.com> wrote:
>
>> actually with var it's the same:
>>
>>
>> scala> class Person4 {
>> |
>> | @scala.beans.BeanProperty var X:Int = 1
>> | }
>> defined class Person4
>>
>> scala> val personEncoder = Encoders.bean[Person4](classOf[Person4])
>> personEncoder: org.apache.spark.sql.Encoder[Person4] = class[x[0]: int]
>>
>> scala> val person_rdd =sc.parallelize(Array( (new Person4(), 1), (new
>> Person4(), 2) ))
>> person_rdd: org.apache.spark.rdd.RDD[(Person4, Int)] =
>> ParallelCollectionRDD[3] at parallelize at <console>:39
>>
>> scala> sqlContext.createDataFrame(person_rdd)
>> java.lang.UnsupportedOperationException: Schema for type Person4 is not
>> supported
>> at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(Sca
>> laReflection.scala:716)
>> at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schem
>> aFor$2.apply(ScalaReflection.scala:712)
>> at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schem
>> aFor$2.apply(ScalaReflection.scala:711)
>> at scala.collection.TraversableLike$$anonfun$map$1.apply(
>> TraversableLike.scala:234)
>> at scala.collection.TraversableLike$$anonfun$map$1.apply(
>> TraversableLike.scala:234)
>> at scala.collection.immutable.List.foreach(List.scala:381)
>> at scala.collection.TraversableLike$class.map(TraversableLike.
>> scala:234)
>> at scala.collection.immutable.List.map(List.scala:285)
>> at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(Sca
>> laReflection.scala:711)
>> at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(Sca
>> laReflection.scala:654)
>> at org.apache.spark.sql.SparkSession.createDataFrame(SparkSessi
>> on.scala:251)
>> at org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.
>> scala:278)
>> ... 54 elided
>>
>> On Tue, May 9, 2017 at 11:19 AM, Michael Armbrust <mich...@databricks.com
>> > wrote:
>>
>>> I think you are supposed to set BeanProperty on a var as they do here
>>> <https://github.com/apache/spark/blob/f830bb9170f6b853565d9dd30ca7418b93a54fe3/mllib/src/main/scala/org/apache/spark/mllib/tree/configuration/Strategy.scala#L71-L83>.
>>> If you are using scala though I'd consider using the case class encoders.
>>>
>>> On Tue, May 9, 2017 at 12:21 AM, Yang <teddyyyy...@gmail.com> wrote:
>>>
>>>> I'm trying to use Encoders.bean() to create an encoder for my custom
>>>> class, but it fails complaining about can't find the schema:
>>>>
>>>>
>>>> class Person4 { @scala.beans.BeanProperty def setX(x:Int): Unit = {}
>>>> @scala.beans.BeanProperty def getX():Int = {1} } val personEncoder =
>>>> Encoders.bean[Person4](classOf[Person4]) scala> val person_rdd =sc.
>>>> parallelize(Array( (new Person4(), 1), (new Person4(), 2) )) person_rdd
>>>> : org.apache.spark.rdd.RDD[(Person4, Int)] = ParallelCollectionRDD[1]
>>>> at parallelize at <con sole>:31 scala> sqlcontext.createDataFrame(per
>>>> son_rdd) java.lang.UnsupportedOperationException: Schema for type
>>>> Person4 is not supported at org.apache.spark.sql.catalyst.
>>>> ScalaReflection$.schemaFor(ScalaReflection.scala:716) at org.apache.
>>>> spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$2.apply(
>>>> ScalaReflection.scala:71 2) at org.apache.spark.sql.catalyst.
>>>> ScalaReflection$$anonfun$schemaFor$2.apply(ScalaReflection.scala:71 1)
>>>> at scala.collection.TraversableLike$$anonfun$map$1.apply(Traver
>>>> sableLike.scala:234) at
>>>>
>>>>
>>>> but if u look at the encoder's schema, it does know it:
>>>> but the system does seem to understand the schema for "Person4":
>>>>
>>>>
>>>> scala> personEncoder.schema
>>>> res38: org.apache.spark.sql.types.StructType =
>>>> StructType(StructField(x,IntegerType,false))
>>>>
>>>>
>>>
>>
>
