[ 
https://issues.apache.org/jira/browse/SPARK-61771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61771:
---------------------------------
    Fix Version/s: connect-rust-4.3.0

> Raise the errors of Column.alias when the plan is built, as PySpark's 
> ColumnAlias raises them
> ---------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61771
>                 URL: https://issues.apache.org/jira/browse/SPARK-61771
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.3.0
>
>
> Column.alias and Column.name of the drop-in raised every error of their names 
> and metadata at the call, where PySpark's ColumnAlias raises them when a 
> request builds the plan: a name that protobuf refuses (TypeError, or UTF-8's 
> UnicodeEncodeError and UnicodeDecodeError), metadata of one name that 
> json.dumps refuses, and metadata of no names (CANNOT_PROVIDE_METADATA). So a 
> Column that PySpark builds and never uses raised here, and the Column of such 
> an alias did not copy or pickle. Its __repr__ also differed: PySpark's joins 
> the names, which raises a TypeError of a name that is not a str (bytes too).
> This raises them as PySpark does: the core expression of such an alias 
> carries its errors (ExpressionNode::Unbuildable), which a request raises in 
> the order in which PySpark builds the plan, and the __repr__ of the Column 
> raises PySpark's TypeError of the join. A pickled or copied Column keeps the 
> errors, also on a Python worker of the server.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to