Hyukjin Kwon created SPARK-61613:
------------------------------------
Summary: Take bytes, as PySpark does, in the table names and paths
of the catalog, the readers and the writers, in the Python client
Key: SPARK-61613
URL: https://issues.apache.org/jira/browse/SPARK-61613
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark 4.3.0's Connect client puts the table names, view names, database
names, patterns and paths of the catalog, the readers and the writers straight
into proto string fields, which protobuf's default (upb) backend takes as str
or as UTF-8 bytes. The Python client of spark-connect-rust took most of them as
str only, and refused bytes with PyO3's TypeError:
Catalog.createTable/createExternalTable(path=...) and the other catalog
methods, DataFrameWriter.insertInto/saveAsTable, DataFrame.writeTo/mergeInto,
the temp-view methods, DataFrameReader.table/changes and
DataStreamReader.table/changes, and DataStreamWriter.toTable. SPARK-61535 fixed
only the writer paths.
Also, DataStreamWriter.start(path) and toTable(tableName) set the path or table
only if it is true, as PySpark does (`if path:`), where the client raised a
TypeError for a false value such as 0.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]