[
https://issues.apache.org/jira/browse/SPARK-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996702#comment-15996702
]
Maciej Szymkiewicz commented on SPARK-12467:
--------------------------------------------
??Row has named fields, so it shouldn't depend upon the ordering in order to
make a match.???
Unfortunately this is not so simple. There is no requirement for schema names
to match the input.
Moreover this:
{code}
schema = spark.sql('SELECT number, letters, some_date FROM
test_trash.thingy').schema # C-works
{code}
doesn't work. It just fails silently by casting data to incorrect types.
Finally:
??If you can't write data into it's own implied schema??
is a good point, but it is not it's own schema. It's "own implied schema" is:
{code}
spark.table('test_trash.thingy').schema
{code}
Maybe the best solution here is to deprecate and remove {{**kwargs}} variant?
It is not really necessary, and given language limitations, it is more
confusing than useful. Or at least remove it from examples and encourage users
to use "long form":
{code}
Row("numers", "letters", "some_date")(1, "real1", datetime(2017,12,1,3,15))
{code}
or {{namedtuple}}.
> Get rid of sorting in Row's constructor in pyspark
> --------------------------------------------------
>
> Key: SPARK-12467
> URL: https://issues.apache.org/jira/browse/SPARK-12467
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 1.5.2
> Reporter: Irakli Machabeli
> Priority: Minor
>
> Current implementation of Row's __new__ sorts columns by name
> First of all there is no obvious reason to sort, second, if one converts
> dataframe to rdd and than back to dataframe, order of column changes. While
> this is not a bug, nevetheless it makes looking at the data really
> inconvenient.
> def __new__(self, *args, **kwargs):
> if args and kwargs:
> raise ValueError("Can not use both args "
> "and kwargs to create Row")
> if args:
> # create row class or objects
> return tuple.__new__(self, args)
> elif kwargs:
> # create row objects
> names = sorted(kwargs.keys()) # just get rid of sorting here!!!
> row = tuple.__new__(self, [kwargs[n] for n in names])
> row.__fields__ = names
> return row
> else:
> raise ValueError("No args or kwargs")
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
