[
https://issues.apache.org/jira/browse/SPARK-45854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-45854:
-----------------------------------
Labels: pull-request-available (was: )
> spark.catalog.listTables fails with ParseException after upgrading to Spark
> 3.4.1 from 3.3.1
> --------------------------------------------------------------------------------------------
>
> Key: SPARK-45854
> URL: https://issues.apache.org/jira/browse/SPARK-45854
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Spark Core, Spark Submit
> Affects Versions: 3.4.0, 3.4.1
> Reporter: Andrej Zachar
> Priority: Major
> Labels: pull-request-available
>
> After upgrading to Spark 3.4.1, the listTables() method in PySpark now throws
> a ParseException with the message "Syntax error at or near end of input.".
> This did not occur in previous versions of Spark, such as 3.3.1.
> Install Spark version 3.4.1.
>
> Run pyspark
> ```bash
> {{pyspark --packages io.delta:delta-core_2.12:2.4.0 --conf
> "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf
> "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"}}
> ```
>
> Attempt to list tables using
> ```console
> {{spark.range(1).createTempView("test_view")}}
> {{spark.catalog.listTables()}}
> ```
> Expected result: The listTables() method should return a list of tables
> without throwing any exceptions.
> Actual result:
> {{Traceback (most recent call last):}}
> {{File "<stdin>", line 1, in <module>}}
> {{File ".venv/lib/python3.10/site-packages/pyspark/sql/catalog.py", line 302,
> in listTables}}
> {{iter = self._jcatalog.listTables(dbName).toLocalIterator()}}
> {{File
> ".venv/lib/python3.10/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py",
> line 1322, in _{_}call{_}_}}
> {{File
> ".venv/lib/python3.10/site-packages/pyspark/errors/exceptions/captured.py",
> line 175, in deco}}
> {{raise converted from None}}
> {{pyspark.errors.exceptions.captured.ParseException:}}
> {{[PARSE_SYNTAX_ERROR] Syntax error at or near end of input.(line 1, pos 0)}}
> == SQL ==
> ^^^
> >>>
> The same code worked correctly in Spark version 3.3.1.
> No changes were made to the code aside from upgrading Spark.
> Thank you for considering this issue! Any assistance in resolving it would be
> greatly appreciated.
> Best regards,
> Andrej
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
