Stijn De Haes created SPARK-43357:
-------------------------------------
Summary: Spark AWS Glue date partition push down broken
Key: SPARK-43357
URL: https://issues.apache.org/jira/browse/SPARK-43357
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.3.2, 3.2.4, 3.2.3, 3.3.1, 3.2.2, 3.3.0, 3.2.1, 3.1.3,
3.2.0, 3.1.2, 3.1.1, 3.1.0
Reporter: Stijn De Haes
When using the following project:
[https://github.com/awslabs/aws-glue-data-catalog-client-for-apache-hive-metastore]
To have glue supported as as a hive metastore for spark there is an issue when
reading a date-partitioned data set. Writing is fine.
You get the following error:
{quote}org.apache.hadoop.hive.metastore.api.InvalidObjectException: Unsupported
expression '2023 - 05 - 03' (Service: AWSGlue; Status Code: 400; Error Code:
InvalidInputException; Request ID: beed68c6-b228-442e-8783-52c25b9d2243; Proxy:
null)
{quote}
A fix for this is making sure the date passed to glue is quoted
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
