itholic commented on a change in pull request #32546: URL: https://github.com/apache/spark/pull/32546#discussion_r636625094
########## File path: docs/sql-data-sources-orc.md ########## @@ -172,3 +172,29 @@ When reading from Hive metastore ORC tables and inserting to Hive metastore ORC <td>2.0.0</td> </tr> </table> + +## Data Source Option + +Data source options of ORC can be set via: +* the `.option`/`.options` methods of + * `DataFrameReader` + * `DataFrameWriter` + * `DataStreamReader` + * `DataStreamWriter` + +<table class="table"> + <tr><th><b>Property Name</b></th><th><b>Default</b></th><th><b>Meaning</b></th><th><b>Scope</b></th></tr> + <tr> + <td><code>mergeSchema</code></td> + <td>None</td> + <td>sets whether we should merge schemas collected from all ORC part-files. This will override <code>spark.sql.orc.mergeSchema</code>. The default value is specified in <code>spark.sql.orc.mergeSchema</code>.</td> + <td>read</td> + </tr> + <tr> + <td><code>compression</code></td> + <td>None</td> + <td>compression codec to use when saving to file. This can be one of the known case-insensitive shorten names (none, snappy, zlib, lzo, and zstd). This will override <code>orc.compress</code> and <code>spark.sql.orc.compression.codec</code>. If None is set, it uses the value specified in <code>spark.sql.orc.compression.codec</code>.</td> + <td>write</td> + </tr> +</table> +Other generic options can be found in <a href="https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html";> Generic File Source Options</a>. Review comment: Thanks, @dongjoon-hyun . I took a look for that but seems tricky to create a link for each release in Scaladoc .. I created a JIRA to track it separately here: SPARK-35481. I will take a separate look if that's fine to you too! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
