[
https://issues.apache.org/jira/browse/SPARK-45040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-45040.
----------------------------------
Resolution: Not A Problem
Resolving as Not A Problem: this 'Caught Hive MetaException attempting to get
partition metadata by filter from Hive' error is a metastore-integration issue,
and the exception message itself documents the workaround — set
spark.sql.hive.manageFilesourcePartitions=false (or ensure the external
metastore, e.g. AWS Glue, supports partition filter pushdown). No Spark code
change is actionable here. Please reopen with a reproducer against a supported
version if you believe this is a Spark bug.
> Caught Hive MetaException attempting to get partition metadata by filter from
> Hive
> ----------------------------------------------------------------------------------
>
> Key: SPARK-45040
> URL: https://issues.apache.org/jira/browse/SPARK-45040
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 2.4.0
> Environment: AWS Glue
> Reporter: Yangbo Yu
> Priority: Critical
> Attachments: Screenshot Capture - 2023-08-31 - 19-02-31.png
>
>
> We are integrating Spark 2.4 with our AWS Glue ETL job.
> And we recently realized that a lot of our jobs are failed because of the
> below error:
> {{Exception in User Class: java.lang.RuntimeException : Caught Hive
> MetaException attempting to get partition metadata by filter from Hive. You
> can set the Spark configuration setting
> spark.sql.hive.manageFilesourcePartitions to false to work around this
> problem, however this will result in degraded performance. Please report a
> bug: }}{{https://issues.apache.org/jira/browse/SPARK
> }}
> This error first happens from Aug 30th, and it occurs from time to time. It
> is gone for several hours and occurs again. During the time that the error
> occurs, most of the jobs fail, only few succeed.
> And we tried to set {{spark.sql.hive.manageFilesourcePartitions}} to false
> but it did not work out. Some other issue is coming out.
> Can you look into the error and let me know if there is any work around to
> mitigate the issue?
> Let me know if you need anything from my end.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
