sunchao commented on a change in pull request #33382:
URL: https://github.com/apache/spark/pull/33382#discussion_r670992179
##########
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
##########
@@ -876,29 +876,21 @@ private[client] class Shim_v0_13 extends Shim_v0_12 {
} else {
logDebug(s"Hive metastore filter is '$filter'.")
val tryDirectSqlConfVar = HiveConf.ConfVars.METASTORE_TRY_DIRECT_SQL
- // We should get this config value from the metaStore. otherwise hit
SPARK-18681.
- // To be compatible with hive-0.12 and hive-0.13, In the future we can
achieve this by:
- // val tryDirectSql =
hive.getMetaConf(tryDirectSqlConfVar.varname).toBoolean
- val tryDirectSql =
hive.getMSC.getConfigValue(tryDirectSqlConfVar.varname,
- tryDirectSqlConfVar.defaultBoolVal.toString).toBoolean
try {
// Hive may throw an exception when calling this method in some
circumstances, such as
// when filtering on a non-string partition column when the hive
config key
// hive.metastore.try.direct.sql is false
getPartitionsByFilterMethod.invoke(hive, table, filter)
.asInstanceOf[JArrayList[Partition]]
} catch {
- case ex: InvocationTargetException if
ex.getCause.isInstanceOf[MetaException] &&
- !tryDirectSql =>
+ case ex: InvocationTargetException if
ex.getCause.isInstanceOf[MetaException] =>
logWarning("Caught Hive MetaException attempting to get partition
metadata by " +
"filter from Hive. Falling back to fetching all partition
metadata, which will " +
"degrade performance. Modifying your Hive metastore
configuration to set " +
- s"${tryDirectSqlConfVar.varname} to true may resolve this
problem.", ex)
+ s"${tryDirectSqlConfVar.varname} to true (if it is not true
already) may resolve " +
+ "this problem.", ex)
// HiveShim clients are expected to handle a superset of the
requested partitions
getAllPartitionsMethod.invoke(hive,
table).asInstanceOf[JSet[Partition]]
- case ex: InvocationTargetException if
ex.getCause.isInstanceOf[MetaException] &&
- tryDirectSql =>
- throw QueryExecutionErrors.getPartitionMetadataByFilterError(ex)
Review comment:
Yes, before there are two cases:
1. (this branch) when `tryDirectSql` is true, we'd throw exception and fail
the query
2. when `tryDirectSql` is false, we'd fallback to `getAllPartitionsMethod`.
Now we merge the case 1) into 2) and always fallback to
`getAllPartitionsMethod`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
