Date handling was tightened up in Spark 3. I think you need to compare to a date literal, not a string literal.
On Mon, Aug 23, 2021 at 5:12 AM Gourav Sengupta < gourav.sengupta.develo...@gmail.com> wrote: > Hi, > > while I am running in EMR 6.3.0 (SPARK 3.1.1) a simple query as "SELECT * > FROM <table_name> WHERE <date parition field> > '2021-03-01'" the query is > failing with error: > --------------------------------------------------------------------------- > pyspark.sql.utils.AnalysisException: > org.apache.hadoop.hive.metastore.api.InvalidObjectException: Unsupported > expression '2021 - 03 - 01' (Service: AWSGlue; Status Code: 400; Error > Code: InvalidInputException; Request ID: > dd3549c2-2eeb-4616-8dc5-5887ba43dd22; Proxy: null) > --------------------------------------------------------------------------- > > The above query works fine in all previous versions of SPARK. > > Is this the expected behaviour in SPARK 3.1.1? If so can someone please > let me know how to write this query. > > Also if this is the expected behaviour I think that a lot of users will > have to make these changes in their existing code making transition to > SPARK 3.1.1 expensive I think. > > Regards, > Gourav Sengupta >
