[
https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996901#comment-15996901
]
Maciej Szymkiewicz commented on SPARK-12467:
--------------------------------------------
[~hyukjin.kwon] Personally I like {{namedtuple}} because it is static type
checker friendly. This is a huge advantage over {{Row}}. But it is just a
preference.
Regarding this JIRA my opinion is the same as for the other one - it is simply
won't fix. Considering we are still committed to supporting Python 2.7,
dropping support for <= 3.5 is at least decade away. Any other attempt to
"fix" this will break backward compatibility and I've seen user code depending
on sorting behavior. Finally as you said it is documented.
> Get rid of sorting in Row's constructor in pyspark
> --------------------------------------------------
>
> Key: SPARK-12467
> URL: https://issues.apache.org/jira/browse/SPARK-12467
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 1.5.2, 2.2.0
> Reporter: Irakli Machabeli
> Priority: Minor
>
> Current implementation of Row's __new__ sorts columns by name
> First of all there is no obvious reason to sort, second, if one converts
> dataframe to rdd and than back to dataframe, order of column changes. While
> this is not a bug, nevetheless it makes looking at the data really
> inconvenient.
> def __new__(self, *args, **kwargs):
> if args and kwargs:
> raise ValueError("Can not use both args "
> "and kwargs to create Row")
> if args:
> # create row class or objects
> return tuple.__new__(self, args)
> elif kwargs:
> # create row objects
> names = sorted(kwargs.keys()) # just get rid of sorting here!!!
> row = tuple.__new__(self, [kwargs[n] for n in names])
> row.__fields__ = names
> return row
> else:
> raise ValueError("No args or kwargs")
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
