mazeboard opened a new pull request #24367: [SPARK-27457][SQL] modify bean encoder to support avro objects URL: https://github.com/apache/spark/pull/24367 ## What changes were proposed in this pull request? Currently we modified JavaTypeInference to be able to create encoders for Avro objects; we have now two solutions, the one in the PR https://github.com/apache/spark/pull/24299 and this PR (fewer code changes); which one is better? ## How was this patch tested? We added one test in ExpressionencoderSuite and used the following program to test it locally: implicit val avroExampleEncoder = Encoders.bean[AvroExample1](classOf[AvroExample1]).asInstanceOf[ExpressionEncoder[AvroExample1]] val input: AvroExample1 = AvroExample1.newBuilder() .setMyarray(List("Foo", "Bar").asJava) .setMyboolean(true) .setMybytes(java.nio.ByteBuffer.wrap("MyBytes".getBytes())) .setMydouble(2.5) .setMyfixed(new Magic("magic".getBytes)) .setMyfloat(25.0F) .setMyint(100) .setMylong(10L) .setMystring("hello") .setMymap(Map( "foo" -> new java.lang.Integer(1), "bar" -> new java.lang.Integer(2)).asJava) .setMymoney(Money.newBuilder().setAmount(100.0F).setCurrency(Currency.EUR).build()) .build() val row: InternalRow = avroExampleEncoder.toRow(input) val output: AvroExample1 = avroExampleEncoder.resolveAndBind().fromRow(row) val ds: Dataset[AvroExample1] = List(input).toDS() println(ds.schema) println(ds.collect().toList) ds.write.format("avro").save("example1") val fooDF = spark.read.format("avro").load("example1") val fooDS = fooDF.as[AvroExample1] println(fooDS.collect().toList)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
