This is an automated email from the ASF dual-hosted git repository. gengliang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new c6afd6e [SPARK-35951][DOCS] Add since versions for Avro options in Documentation c6afd6e is described below commit c6afd6ed5296980e81160e441a4e9bea98c74196 Author: Gengliang Wang <gengli...@apache.org> AuthorDate: Wed Jun 30 17:24:48 2021 +0800 [SPARK-35951][DOCS] Add since versions for Avro options in Documentation ### What changes were proposed in this pull request? There are two new Avro options `datetimeRebaseMode` and `positionalFieldMatching` after Spark 3.2. We should document the since version so that users can know whether the option works in their Spark version. ### Why are the changes needed? Better documentation. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manual preview on local setup. <img width="828" alt="Screen Shot 2021-06-30 at 5 05 54 PM" src="https://user-images.githubusercontent.com/1097932/123934000-ba833b00-d947-11eb-9ca5-ce8ff8add74b.png"> <img width="711" alt="Screen Shot 2021-06-30 at 5 06 34 PM" src="https://user-images.githubusercontent.com/1097932/123934126-d4bd1900-d947-11eb-8d80-69df8f3d9900.png"> Closes #33153 from gengliangwang/version. Authored-by: Gengliang Wang <gengli...@apache.org> Signed-off-by: Gengliang Wang <gengli...@apache.org> --- docs/sql-data-sources-avro.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/docs/sql-data-sources-avro.md b/docs/sql-data-sources-avro.md index 7fb0ef5..94dd7e1 100644 --- a/docs/sql-data-sources-avro.md +++ b/docs/sql-data-sources-avro.md @@ -224,7 +224,7 @@ Data source options of Avro can be set via: * the `options` parameter in function `from_avro`. <table class="table"> - <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr> + <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th><th><b>Since Version</b></th></tr> <tr> <td><code>avroSchema</code></td> <td>None</td> @@ -244,24 +244,28 @@ Data source options of Avro can be set via: </ul> </td> <td> read, write and function <code>from_avro</code></td> + <td>2.4.0</td> </tr> <tr> <td><code>recordName</code></td> <td>topLevelRecord</td> <td>Top level record name in write result, which is required in Avro spec.</td> <td>write</td> + <td>2.4.0</td> </tr> <tr> <td><code>recordNamespace</code></td> <td>""</td> <td>Record namespace in write result.</td> <td>write</td> + <td>2.4.0</td> </tr> <tr> <td><code>ignoreExtension</code></td> <td>true</td> <td>The option controls ignoring of files without <code>.avro</code> extensions in read.<br> If the option is enabled, all files (with and without <code>.avro</code> extension) are loaded.<br> The option has been deprecated, and it will be removed in the future releases. Please use the general data source option <a href="./sql-data-sources-generic-options.html#path-global-filter">pathGlobFilter</a> for filtering file names.</td> <td>read</td> + <td>2.4.0</td> </tr> <tr> <td><code>compression</code></td> @@ -269,6 +273,7 @@ Data source options of Avro can be set via: <td>The <code>compression</code> option allows to specify a compression codec used in write.<br> Currently supported codecs are <code>uncompressed</code>, <code>snappy</code>, <code>deflate</code>, <code>bzip2</code> and <code>xz</code>.<br> If the option is not set, the configuration <code>spark.sql.avro.compression.codec</code> config is taken into account.</td> <td>write</td> + <td>2.4.0</td> </tr> <tr> <td><code>mode</code></td> @@ -282,6 +287,7 @@ Data source options of Avro can be set via: </ul> </td> <td>function <code>from_avro</code></td> + <td>2.4.0</td> </tr> <tr> <td><code>datetimeRebaseMode</code></td> @@ -295,12 +301,14 @@ Data source options of Avro can be set via: </ul> </td> <td>read and function <code>from_avro</code></td> + <td>3.2.0</td> </tr> <tr> <td><code>positionalFieldMatching</code></td> <td>false</td> <td>This can be used in tandem with the `avroSchema` option to adjust the behavior for matching the fields in the provided Avro schema with those in the SQL schema. By default, the matching will be performed using field names, ignoring their positions. If this option is set to "true", the matching will be based on the position of the fields.</td> <td>read and write</td> + <td>3.2.0</td> </tr> </table> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org