Yangbo Yu created SPARK-45040:
---------------------------------
Summary: Caught Hive MetaException attempting to get partition
metadata by filter from Hive
Key: SPARK-45040
URL: https://issues.apache.org/jira/browse/SPARK-45040
Project: Spark
Issue Type: Bug
Components: Java API
Affects Versions: 2.4.0
Environment: AWS Glue
Reporter: Yangbo Yu
We are integrating Spark 2.4 with our AWS Glue ETL job.
And we recently realized that a lot of our jobs are failed because of the below
error:
{{Exception in User Class: java.lang.RuntimeException : Caught Hive
MetaException attempting to get partition metadata by filter from Hive. You can
set the Spark configuration setting spark.sql.hive.manageFilesourcePartitions
to false to work around this problem, however this will result in degraded
performance. Please report a bug:
}}{{https://issues.apache.org/jira/browse/SPARK
}}
This error first happens from Aug 30th, and it occurs from time to time. It is
gone for several hours and occurs again. During the time that the error occurs,
most of the jobs fail, only few succeed.
And we tried to set {{spark.sql.hive.manageFilesourcePartitions}} to false but
it did not work out. Some other issue is coming out.
Can you look into the error and let me know if there is any work around to
mitigate the issue?
Let me know if you need anything from my end.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
