dongjoon-hyun commented on a change in pull request #31771:
URL: https://github.com/apache/spark/pull/31771#discussion_r590667749
##########
File path:
external/avro/src/main/scala/org/apache/spark/sql/avro/functions.scala
##########
@@ -51,18 +54,30 @@ object functions {
* To deserialize the data with a compatible and evolved schema, the
expected Avro schema can be
* set via the option avroSchema.
*
+ * Since [SPARK-34652] also its possible to get the latest schema of the
topic
+ * from SchemaRegistry, it needs to pass the `schemaRegistryUrl` to the
options
+ * and set `t-value` or `t-key` on the jsonFormatSchema filed for topic t.
+ *
* @param data the binary column.
* @param jsonFormatSchema the avro schema in JSON string format.
* @param options options to control how the Avro record is parsed.
*
* @since 3.0.0
*/
+ @throws(classOf[java.io.IOException])
@Experimental
def from_avro(
data: Column,
jsonFormatSchema: String,
options: java.util.Map[String, String]): Column = {
- new Column(AvroDataToCatalyst(data.expr, jsonFormatSchema,
options.asScala.toMap))
+ if (options.containsKey("schemaRegistryUrl")) {
Review comment:
indentation looks wrong. Could you check once more?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
