Hyukjin Kwon created SPARK-61771:
------------------------------------

             Summary: Raise the errors of Column.alias when the plan is built, 
as PySpark's ColumnAlias raises them
                 Key: SPARK-61771
                 URL: https://issues.apache.org/jira/browse/SPARK-61771
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Column.alias and Column.name of the drop-in raised every error of their names 
and metadata at the call, where PySpark's ColumnAlias raises them when a 
request builds the plan: a name that protobuf refuses (TypeError, or UTF-8's 
UnicodeEncodeError and UnicodeDecodeError), metadata of one name that 
json.dumps refuses, and metadata of no names (CANNOT_PROVIDE_METADATA). So a 
Column that PySpark builds and never uses raised here, and the Column of such 
an alias did not copy or pickle. Its __repr__ also differed: PySpark's joins 
the names, which raises a TypeError of a name that is not a str (bytes too).

This raises them as PySpark does: the core expression of such an alias carries 
its errors (ExpressionNode::Unbuildable), which a request raises in the order 
in which PySpark builds the plan, and the __repr__ of the Column raises 
PySpark's TypeError of the join. A pickled or copied Column keeps the errors, 
also on a Python worker of the server.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to