Andrej Zachar created SPARK-45854:
-------------------------------------
Summary: spark.catalog.listTables fails with ParseException after
upgrading to Spark 3.4.1 from 3.3.1
Key: SPARK-45854
URL: https://issues.apache.org/jira/browse/SPARK-45854
Project: Spark
Issue Type: Bug
Components: PySpark, Spark Core, Spark Submit
Affects Versions: 3.4.1, 3.4.0
Reporter: Andrej Zachar
After upgrading to Spark 3.4.1, the listTables() method in PySpark now throws a
ParseException with the message "Syntax error at or near end of input.". This
did not occur in previous versions of Spark, such as 3.3.1.
Install Spark version 3.4.1.
Run pyspark
```bash
{{pyspark --packages io.delta:delta-core_2.12:2.4.0 --conf
"spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf
"spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"}}
```
Attempt to list tables using
```console
{{spark.range(1).createTempView("test_view")}}
{{spark.catalog.listTables()}}
```
Expected result: The listTables() method should return a list of tables without
throwing any exceptions.
Actual result:
{{Traceback (most recent call last):}}
{{File "<stdin>", line 1, in <module>}}
{{File ".venv/lib/python3.10/site-packages/pyspark/sql/catalog.py", line 302,
in listTables}}
{{iter = self._jcatalog.listTables(dbName).toLocalIterator()}}
{{File
".venv/lib/python3.10/site-packages/pyspark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py",
line 1322, in _{_}call{_}_}}
{{File
".venv/lib/python3.10/site-packages/pyspark/errors/exceptions/captured.py",
line 175, in deco}}
{{raise converted from None}}
{{pyspark.errors.exceptions.captured.ParseException:}}
{{[PARSE_SYNTAX_ERROR] Syntax error at or near end of input.(line 1, pos 0)}}
== SQL ==
^^^
>>>
The same code worked correctly in Spark version 3.3.1.
No changes were made to the code aside from upgrading Spark.
Thank you for considering this issue! Any assistance in resolving it would be
greatly appreciated.
Best regards,
Andrej
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
