[
https://issues.apache.org/jira/browse/SPARK-24347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16622687#comment-16622687
]
Axel Magnuson commented on SPARK-24347:
---------------------------------------
I'd like to take a look at this. I'm working with [~holdenkarau].
> df.alias() in python API should not clear metadata by default
> -------------------------------------------------------------
>
> Key: SPARK-24347
> URL: https://issues.apache.org/jira/browse/SPARK-24347
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.3.0
> Reporter: Tomasz Bartczak
> Priority: Minor
>
> currently when doing an alias on a column in pyspark I lose metadata:
> {code:java}
> print("just select = ", df.select(col("v")).schema.fields[0].metadata.keys())
> print("select alias= ",
> df.select(col("v").alias("vv")).schema.fields[0].metadata.keys()){code}
> gives:
> {code:java}
> just select = dict_keys(['ml_attr'])
> select alias= dict_keys([]){code}
> After looking at alias() documentation I see that metadata is an optional
> param. But it should not clear the metadata when it is not set. A default
> solution should be to keep it as-is.
> Otherwise - it generates problems in a later part of the processing pipeline
> when someone is depending on the metadata.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
