This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new 1b5b2a15dec3 [SPARK-45389][SQL][3.5] Correct MetaException matching
rule on getting partition metadata
1b5b2a15dec3 is described below
commit 1b5b2a15dec3d17fa1e7be9a78b37dc600eeb8ab
Author: Cheng Pan <[email protected]>
AuthorDate: Sat Oct 7 15:26:17 2023 -0700
[SPARK-45389][SQL][3.5] Correct MetaException matching rule on getting
partition metadata
This is the backport of https://github.com/apache/spark/pull/43191 for
`branch-3.5`, it should also be applicable for `branch-3.3` and `branch-3.4`
### What changes were proposed in this pull request?
This PR aims to fix the HMS call fallback logic introduced in SPARK-35437.
```patch
try {
...
hive.getPartitionNames
...
hive.getPartitionsByNames
} catch {
- case ex: InvocationTargetException if
ex.getCause.isInstanceOf[MetaException] =>
+ case ex: HiveException if ex.getCause.isInstanceOf[MetaException] =>
...
}
```
### Why are the changes needed?
Directly method call won't throw `InvocationTargetException`, and check the
code of `hive.getPartitionNames` and `hive.getPartitionsByNames`, both of them
will wrap a `HiveException` if `MetaException` throws.
### Does this PR introduce _any_ user-facing change?
Yes, it should be a bug fix.
### How was this patch tested?
Pass GA and code review. (I'm not sure how to construct/simulate a
MetaException during the HMS thrift call with the current HMS testing
infrastructure)
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #43260 from pan3793/SPARK-45389-3.5.
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
index 63f672b22bad..60ff9ec42f29 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
@@ -33,7 +33,7 @@ import org.apache.hadoop.hive.metastore.TableType
import org.apache.hadoop.hive.metastore.api.{Database, EnvironmentContext,
Function => HiveFunction, FunctionType, Index, MetaException, PrincipalType,
ResourceType, ResourceUri}
import org.apache.hadoop.hive.ql.Driver
import org.apache.hadoop.hive.ql.io.AcidUtils
-import org.apache.hadoop.hive.ql.metadata.{Hive, Partition, Table}
+import org.apache.hadoop.hive.ql.metadata.{Hive, HiveException, Partition,
Table}
import org.apache.hadoop.hive.ql.plan.AddPartitionDesc
import org.apache.hadoop.hive.ql.processors.{CommandProcessor,
CommandProcessorFactory}
import org.apache.hadoop.hive.ql.session.SessionState
@@ -1190,7 +1190,7 @@ private[client] class Shim_v0_13 extends Shim_v0_12 {
recordHiveCall()
hive.getPartitionsByNames(table, partNames.asJava)
} catch {
- case ex: InvocationTargetException if
ex.getCause.isInstanceOf[MetaException] =>
+ case ex: HiveException if ex.getCause.isInstanceOf[MetaException] =>
logWarning("Caught Hive MetaException attempting to get partition
metadata by " +
"filter from client side. Falling back to fetching all partition
metadata", ex)
recordHiveCall()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
