JyunSheng Kao created SPARK-61213:
-------------------------------------
Summary: Catalog.createDatabase should convert non-string property
values to strings like CREATE NAMESPACE
Key: SPARK-61213
URL: https://issues.apache.org/jira/browse/SPARK-61213
Project: Spark
Issue Type: Bug
Components: Connect, PySpark
Affects Versions: 4.2.1, 4.3.0, 4.2.0
Reporter: JyunSheng Kao
h3. Problem
{{Catalog.createDatabase(dbName, ifNotExists, properties)}} (added in 4.2.0 by
SPARK-56221) in PySpark classic (in-memory) does no type checking on the dict.
{{Catalog.createDatabase}} (python/pyspark/sql/catalog.py) calls {{m.put(k,
v)}} on a Java HashMap, so any Python object goes through. However, the Scala
side assumes
{{{}Map[String, String]{}}}.
It silently creates a poisoned catalog and gives out exceptions when users call
{{Catalog.listDatabases()}}
h3. How to reproduce
{code:python}
pyspark> spark.catalog.createDatabase("some_db", properties={"k": 1})
# succeeds silently
pyspark> spark.catalog.databaseExists("some_db")
# java.lang.ClassCastException: Integer cannot be cast to String
pyspark> spark.catalog.getDatabase("some_db")
# ClassCastException
pyspark> spark.catalog.listDatabases()
# ClassCastException, broken for ALL databases
pyspark> spark.sql("DESCRIBE NAMESPACE EXTENDED some_db")
# ClassCastException
pyspark> spark.catalog.dropDatabase("some_db", cascade=True)
# workaround: still works
{code}
h3. Impact
The same user mistake gives three different outcomes, none of them a clear
PySpark error
||Mode / catalog||Behaviour||
|Classic, Memory|Silent success, then the catalog is poisoned:
{{listDatabases()}} fails for every database until the bad one is dropped|
|Classic, Hive|AnalysisException from the metastore wrapping a
ClassCastException|
|Spark Connect|Immediate but opaque protobuf TypeError: bad argument type for
built-in operation|
- Python-only. Scala and Java callers cannot put an Integer into Map[String,
String]
h3. Expected behavior
Non-string property values are converted to strings the same way the SQL path
does, so {{{}properties={"k": 1, "b": True{}}}} stores {{k=1}} and {{b=true}}
in all three modes.
This also matches {{DataFrameReader.option}} / {{{}DataFrameWriter.option{}}},
which already convert values with {{{}pyspark.sql.utils.to_str{}}}.
h3. Cause
python module
* Classic: {{{}python/pyspark/sql/catalog.py{}}}, {{Catalog.createDatabase}}
puts each value into a {{java.util.HashMap}} with {{m.put(k, v)}} and no
conversion.
* Connect: {{{}python/pyspark/sql/connect/plan.py{}}}, {{CreateDatabase.plan}}
assigns {{cmd.properties[k] = v}} into a {{map<string, string>}} proto field.
h3. Proposed fix
Apply {{to_str}} to the values at both call sites and add a test in
{{CatalogTestsMixin}} that runs in classic and Connect mode. Only Python
callers are affected; the Scala and Java APIs take {{{}Map[String, String]{}}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
