mazeboard edited a comment on issue #24367: [SPARK-27457][SQL] modify bean encoder to support avro objects URL: https://github.com/apache/spark/pull/24367#issuecomment-493660597 The following example shows why I prefer the solution that modifies ScalaReflection (https://github.com/apache/spark/pull/24299) In the example below, If we use bean encoder we must declare a tuple encoder as shown: ``` implicit val barcodeEncoder = Encoders.bean[Barcode](classOf[Barcode]).asInstanceOf[ExpressionEncoder[Barcode]] implicit val tuplEncoder = Encoders.tuple[String, Money, CrpAddDesc]( Encoders.STRING, Encoders.bean[Money](classOf[Money]).asInstanceOf[ExpressionEncoder[Money]], Encoders.bean[CrpAddDesc](classOf[CrpAddDesc]).asInstanceOf[ExpressionEncoder[CrpAddDesc]]) ... ... val ds: Dataset[Barcode] = List(barcode).toDS() val x: Dataset[(String, Money, CrpAddDesc)] = ds.map(a => { ( a.getBarcode, a.getPrdTaxVal, a.getCrpAttributes.get(0).getCrpAddDesc.get(0)) }) ``` if we do not declare the tuple encoder we get an error `No Encoder found for common.lib.v1.Money` But with the ScalaReflection (https://github.com/apache/spark/pull/24299) modification we just need to declare an encoder for Barcode: ``` implicit val barcodeEncoder = ExpressionEncoder[Barcode]() ... ... val ds: Dataset[Barcode] = List(barcode).toDS() val x: Dataset[(String, Money, CrpAddDesc)] = ds.map(a => { ( a.getBarcode, a.getPrdTaxVal, a.getCrpAttributes.get(0).getCrpAddDesc.get(0)) }) ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
