[
https://issues.apache.org/jira/browse/SPARK-61213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61213:
-----------------------------------
Labels: pull-request-available (was: )
> Catalog.createDatabase should convert non-string property values to strings
> like CREATE NAMESPACE
> -------------------------------------------------------------------------------------------------
>
> Key: SPARK-61213
> URL: https://issues.apache.org/jira/browse/SPARK-61213
> Project: Spark
> Issue Type: Bug
> Components: Connect, PySpark
> Affects Versions: 4.2.0, 4.3.0, 4.2.1
> Reporter: JyunSheng Kao
> Priority: Minor
> Labels: pull-request-available
>
> h3. Problem
> {{Catalog.createDatabase(dbName, ifNotExists, properties)}} (added in 4.2.0
> by SPARK-56221) in PySpark classic (in-memory) does no type checking on the
> dict.
>
> {{Catalog.createDatabase}} (python/pyspark/sql/catalog.py) calls {{m.put(k,
> v)}} on a Java HashMap, so any Python object goes through. However, the Scala
> side assumes
> {{{}Map[String, String]{}}}.
> It silently creates a poisoned catalog and gives out exceptions when users
> call {{Catalog.listDatabases()}}
> h3. How to reproduce
> {code:python}
> pyspark> spark.catalog.createDatabase("some_db", properties={"k": 1})
> # succeeds silently
> pyspark> spark.catalog.databaseExists("some_db")
> # java.lang.ClassCastException: Integer cannot be cast to String
> pyspark> spark.catalog.getDatabase("some_db")
> # ClassCastException
> pyspark> spark.catalog.listDatabases()
> # ClassCastException, broken for ALL databases
> pyspark> spark.sql("DESCRIBE NAMESPACE EXTENDED some_db")
> # ClassCastException
> pyspark> spark.catalog.dropDatabase("some_db", cascade=True)
> # workaround: still works
> {code}
> h3. Impact
> The same user mistake gives three different outcomes, none of them a clear
> PySpark error
> ||Mode / catalog||Behaviour||
> |Classic, Memory|Silent success, then the catalog is poisoned:
> {{listDatabases()}} fails for every database until the bad one is dropped|
> |Classic, Hive|AnalysisException from the metastore wrapping a
> ClassCastException|
> |Spark Connect|Immediate but opaque protobuf TypeError: bad argument type for
> built-in operation|
>
> - Python-only. Scala and Java callers cannot put an Integer into Map[String,
> String]
> h3. Expected behavior
> Non-string property values are converted to strings the same way the SQL path
> does, so {{{}properties={"k": 1, "b": True{}}}} stores {{k=1}} and {{b=true}}
> in all three modes.
> This also matches {{DataFrameReader.option}} /
> {{{}DataFrameWriter.option{}}}, which already convert values with
> {{{}pyspark.sql.utils.to_str{}}}.
> h3. Cause
> python module
> * Classic: {{{}python/pyspark/sql/catalog.py{}}}, {{Catalog.createDatabase}}
> puts each value into a {{java.util.HashMap}} with {{m.put(k, v)}} and no
> conversion.
> * Connect: {{{}python/pyspark/sql/connect/plan.py{}}},
> {{CreateDatabase.plan}} assigns {{cmd.properties[k] = v}} into a
> {{map<string, string>}} proto field.
> h3. Proposed fix
> Apply {{to_str}} to the values at both call sites and add a test in
> {{CatalogTestsMixin}} that runs in classic and Connect mode. Only Python
> callers are affected; the Scala and Java APIs take {{{}Map[String,
> String]{}}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
