cloud-fan commented on a change in pull request #25419: [SPARK-28698][SQL]
Allow user-specified output schema in function `to_avro`
URL: https://github.com/apache/spark/pull/25419#discussion_r312925637
##########
File path: python/pyspark/sql/avro/functions.py
##########
@@ -69,26 +69,35 @@ def from_avro(data, jsonFormatSchema, options={}):
@ignore_unicode_prefix
@since(3.0)
-def to_avro(data):
+def to_avro(data, jsonFormatSchema=""):
"""
Converts a column into binary of avro format.
Note: Avro is built-in but external data source module since Spark 2.4.
Please deploy the
application as per the deployment section of "Apache Avro Data Source
Guide".
:param data: the data column.
+ :param jsonFormatSchema: user-specified output avro schema in JSON string
format.
>>> from pyspark.sql import Row
>>> from pyspark.sql.avro.functions import to_avro
- >>> data = [(1, Row(name='Alice', age=2))]
- >>> df = spark.createDataFrame(data, ("key", "value"))
- >>> df.select(to_avro(df.value).alias("avro")).collect()
Review comment:
can we keep the original example? Users don't have to specify the json
schema string.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
