Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22121#discussion_r211127696
--- Diff: docs/avro-data-source-guide.md ---
@@ -0,0 +1,260 @@
+---
+layout: global
+title: Apache Avro Data Source Guide
+---
+
+* This will become a table of contents (this text will be scraped).
+{:toc}
+
+Since Spark 2.4 release, [Spark
SQL](https://spark.apache.org/docs/latest/sql-programming-guide.html) provides
built-in support for reading and writing Apache Avro data.
+
+## Deploying
+The `spark-avro` module is external and not included in `spark-submit` or
`spark-shell` by default.
+
+As with any Spark applications, `spark-submit` is used to launch your
application. `spark-avro_{{site.SCALA_BINARY_VERSION}}`
+and its dependencies can be directly added to `spark-submit` using
`--packages`, such as,
+
+ ./bin/spark-submit --packages
org.apache.spark:spark-avro_{{site.SCALA_BINARY_VERSION}}:{{site.SPARK_VERSION_SHORT}}
...
+
+For experimenting on `spark-shell`, you can also use `--packages` to add
`org.apache.spark:spark-avro_{{site.SCALA_BINARY_VERSION}}` and its
dependencies directly,
+
+ ./bin/spark-shell --packages
org.apache.spark:spark-avro_{{site.SCALA_BINARY_VERSION}}:{{site.SPARK_VERSION_SHORT}}
...
+
+See [Application Submission Guide](submitting-applications.html) for more
details about submitting applications with external dependencies.
+
+## Load/Save Functions
+
+Since `spark-avro` module is external, there is not such API as `.avro` in
+`DataFrameReader` or `DataFrameWriter`.
+To load/save data in Avro format, you need to specify the data source
option `format` as short name `avro` or full name `org.apache.spark.sql.avro`.
+<div class="codetabs">
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+
+val usersDF =
spark.read.format("avro").load("examples/src/main/resources/users.avro")
+usersDF.select("name",
"favorite_color").write.format("avro").save("namesAndFavColors.avro")
+
+{% endhighlight %}
+</div>
+<div data-lang="java" markdown="1">
+{% highlight java %}
+
+Dataset<Row> usersDF =
spark.read().format("avro").load("examples/src/main/resources/users.avro");
+usersDF.select("name",
"favorite_color").write().format("avro").save("namesAndFavColors.avro");
+
+{% endhighlight %}
+</div>
+<div data-lang="python" markdown="1">
+{% highlight python %}
+
+df =
spark.read.format("avro").load("examples/src/main/resources/users.avro")
+df.select("name",
"favorite_color").write.format("avro").save("namesAndFavColors.avro")
+
+{% endhighlight %}
+</div>
+<div data-lang="r" markdown="1">
+{% highlight r %}
+
+df <- read.df("examples/src/main/resources/users.avro", "avro")
+write.df(select(df, "name", "favorite_color"), "namesAndFavColors.avro",
"avro")
+
+{% endhighlight %}
+</div>
+</div>
+
+## Data Source Options
+
+Data source options of Avro can be set using the `.option` method on
`DataFrameReader` or `DataFrameWriter`.
+<table class="table">
+ <tr><th><b>Property
Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr>
+ <tr>
+ <td><code>avroSchema</code></td>
+ <td>None</td>
+ <td>Optional Avro schema provided by an user in JSON format.</td>
+ <td>read and write</td>
+ </tr>
+ <tr>
+ <td><code>recordName</code></td>
+ <td>topLevelRecord</td>
+ <td>Top level record name in write result, which is required in Avro
spec.</td>
+ <td>write</td>
+ </tr>
+ <tr>
+ <td><code>recordNamespace</code></td>
+ <td>""</td>
+ <td>Record namespace in write result.</td>
+ <td>write</td>
+ </tr>
+ <tr>
+ <td><code>ignoreExtension</code></td>
+ <td>true</td>
+ <td>The option controls ignoring of files without <code>.avro</code>
extensions in read. If the option is enabled, all files (with and without
<code>.avro</code> extension) are loaded.</td>
+ <td>read</td>
+ </tr>
+ <tr>
+ <td><code>compression</code></td>
+ <td>snappy</td>
+ <td>The <code>compression</code> option allows to specify a
compression codec used in write. Currently supported codecs are
<code>uncompressed</code>, <code>snappy</code>, <code>deflate</code>,
<code>bzip2</code> and <code>xz</code>. If the option is not set, the
configuration <code>spark.sql.avro.compression.codec</code> config is taken
into account.</td>
--- End diff --
For data source options, yes.
For SQL configuration, I think the only one matters is the one in
https://github.com/apache/spark/pull/22133. I am thinking of a better name for
that configuration.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
