Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/15673#discussion_r216037341
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala ---
@@ -586,17 +587,31 @@ private[client] class Shim_v0_13 extends Shim_v0_12 {
getAllPartitionsMethod.invoke(hive,
table).asInstanceOf[JSet[Partition]]
} else {
logDebug(s"Hive metastore filter is '$filter'.")
+ val tryDirectSqlConfVar =
HiveConf.ConfVars.METASTORE_TRY_DIRECT_SQL
+ val tryDirectSql =
+ hive.getConf.getBoolean(tryDirectSqlConfVar.varname,
tryDirectSqlConfVar.defaultBoolVal)
try {
+ // Hive may throw an exception when calling this method in some
circumstances, such as
+ // when filtering on a non-string partition column when the hive
config key
+ // hive.metastore.try.direct.sql is false
getPartitionsByFilterMethod.invoke(hive, table, filter)
.asInstanceOf[JArrayList[Partition]]
} catch {
- case e: InvocationTargetException =>
- // SPARK-18167 retry to investigate the flaky test. This
should be reverted before
- // the release is cut.
- val retry = Try(getPartitionsByFilterMethod.invoke(hive,
table, filter))
- logError("getPartitionsByFilter failed, retry success = " +
retry.isSuccess)
- logError("all partitions: " + getAllPartitions(hive, table))
- throw e
+ case ex: InvocationTargetException if
ex.getCause.isInstanceOf[MetaException] &&
+ !tryDirectSql =>
+ logWarning("Caught Hive MetaException attempting to get
partition metadata by " +
+ "filter from Hive. Falling back to fetching all partition
metadata, which will " +
+ "degrade performance. Modifying your Hive metastore
configuration to set " +
+ s"${tryDirectSqlConfVar.varname} to true may resolve this
problem.", ex)
+ // HiveShim clients are expected to handle a superset of the
requested partitions
+ getAllPartitionsMethod.invoke(hive,
table).asInstanceOf[JSet[Partition]]
+ case ex: InvocationTargetException if
ex.getCause.isInstanceOf[MetaException] &&
+ tryDirectSql =>
+ throw new RuntimeException("Caught Hive MetaException
attempting to get partition " +
--- End diff --
Hi @rezasafi
I believe the reasoning is if the user has disabled direct sql, we will try
to fetch the partitions for the requested partition predicate anyway. However,
since we don't expect that call to succeed, we just log a warning and fallback
to the legacy behavior.
On the other hand, if the user has enabled direct sql, then we expect the
call to Hive to succeed. If it fails, we consider that an error and throw an
exception.
I hope that helps clarify things.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
