dongjoon-hyun commented on a change in pull request #31030:
URL: https://github.com/apache/spark/pull/31030#discussion_r552318233
##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveUtils.scala
##########
@@ -130,7 +130,10 @@ private[spark] object HiveUtils extends Logging {
val CONVERT_METASTORE_ORC = buildConf("spark.sql.hive.convertMetastoreOrc")
.doc("When set to true, the built-in ORC reader and writer are used to
process " +
- "ORC tables created by using the HiveQL syntax, instead of Hive serde.")
+ "ORC tables created by using the HiveQL syntax, instead of Hive serde." +
+ "Enabling this parameter will cause insert overwrite when reading the
table," +
+ "and the file format of this table is ORC This error will be thrown[], "
+
Review comment:
@dh20 . It seems that you want to describe one of the differences
between Spark data source tables and Hive serde tables. May I ask a few
questions?
- Is the situation different from `Parquet`? Parquet also has the same conf
`spark.sql.hive.convertMetastoreParquet`.
- Is the added statement correct always? Specifically, the following
statement looks wrong in some situations. For example, did you try to use
`PartitionOverwriteMode.DYNAMIC`?
> When I query an orc format hive table and overwrite the query result into
this orc table, sparksql will throw this error ->
[org.apache.spark.sql.AnalysisException: Cannot overwrite a path that is also
being read from], when I set this parameter to false, Hql will execute normally
- Which Apache Spark version are you using? The situation will be changed
according to the Spark versions. For example,
- SPARK-30112 implemented `Allow insert overwrite same table if using
dynamic partition overwrite` at Apache Spark 3.0.0.
- SPARK-33887 is also trying to implement `Allow insert overwrite same
table with static partition if using dynamic partition overwrite mode` at
Apache Spark 3.2.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
