Hyukjin Kwon created SPARK-61613:
------------------------------------

             Summary: Take bytes, as PySpark does, in the table names and paths 
of the catalog, the readers and the writers, in the Python client
                 Key: SPARK-61613
                 URL: https://issues.apache.org/jira/browse/SPARK-61613
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


PySpark 4.3.0's Connect client puts the table names, view names, database 
names, patterns and paths of the catalog, the readers and the writers straight 
into proto string fields, which protobuf's default (upb) backend takes as str 
or as UTF-8 bytes. The Python client of spark-connect-rust took most of them as 
str only, and refused bytes with PyO3's TypeError: 
Catalog.createTable/createExternalTable(path=...) and the other catalog 
methods, DataFrameWriter.insertInto/saveAsTable, DataFrame.writeTo/mergeInto, 
the temp-view methods, DataFrameReader.table/changes and 
DataStreamReader.table/changes, and DataStreamWriter.toTable. SPARK-61535 fixed 
only the writer paths.

Also, DataStreamWriter.start(path) and toTable(tableName) set the path or table 
only if it is true, as PySpark does (`if path:`), where the client raised a 
TypeError for a false value such as 0.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to