HyukjinKwon commented on a change in pull request #32546: URL: https://github.com/apache/spark/pull/32546#discussion_r635973928
########## File path: docs/sql-data-sources-orc.md ########## @@ -172,3 +172,26 @@ When reading from Hive metastore ORC tables and inserting to Hive metastore ORC <td>2.0.0</td> </tr> </table> + +## Data Source Option + +Data source options of ORC can be set via: +* the `.option`/`.options` methods of `DataFrameReader` or `DataFrameWriter` +* the `.option`/`.options` methods of `DataStreamReader` or `DataStreamWriter` + +<table class="table"> + <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr> + <tr> + <td><code>mergeSchema</code></td> + <td>None</td> + <td>sets whether we should merge schemas collected from all ORC part-files. This will override <code>spark.sql.orc.mergeSchema</code>. The default value is specified in <code>spark.sql.orc.mergeSchema</code>.</td> + <td>read</td> + </tr> + <tr> + <td><code>compression</code></td> + <td>None</td> + <td>compression codec to use when saving to file. This can be one of the known case-insensitive shorten names (none, snappy, zlib, lzo, and zstd). This will override <code>orc.compress</code> and <code>spark.sql.orc.compression.codec</code>. If None is set, it uses the value specified in <code>spark.sql.orc.compression.codec</code>.</td> + <td>write</td> + </tr> +</table> +Other generic options can be found in <a href="https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html";> Generic File Source Options</a>. Review comment: If you're going to link generic options here, let's don't mention generic options in the API documentation in the codes as it duplicates. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
