[
https://issues.apache.org/jira/browse/SPARK-13802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15853008#comment-15853008
]
Maciej Szymkiewicz edited comment on SPARK-13802 at 2/5/17 12:45 AM:
---------------------------------------------------------------------
[~szymonm] Realistically it is rather unlikely to change:
- Possible fix is completely meaningless when using Python < 3.6.0. In any
earlier version Python does not preserve the order of the keyword arguments
([PEP 468|https://www.python.org/dev/peps/pep-0468/]) so sorting by name is
effectively the only option to enforce reproducible behavior.
- Any modification of this behavior can break existing user code, especially
when using Python 2.7 / 3.5 (probably majority of users in the foreseeable
future) and there possible replacements / workarounds.
It is not my call but "Won't fix" looks like the only reasonable resolution at
this moment.
CC [~rxin], [~holdenk]
was (Author: zero323):
[~szymonm] Realistically it is rather unlikely to change:
- Possible fix is completely meaningless when using Python < 3.6.0. In any
earlier version Python does not preserve the order of the keyword arguments
([PEP 468|https://www.python.org/dev/peps/pep-0468/]) so sorting by name is
effectively the only option to enforce reproducible behavior.
- Any modification of this behavior can break existing user code, especially
when using Python 2.7 / 3.5 (probably majority of users in the foreseeable
future) and there possible replacements / workarounds.
It is not my call but "Won't fix" looks like the only reasonable resolution at
this moment.
CC [~rxin] [~holdenk]
> Fields order in Row(**kwargs) is not consistent with Schema.toInternal method
> -----------------------------------------------------------------------------
>
> Key: SPARK-13802
> URL: https://issues.apache.org/jira/browse/SPARK-13802
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.6.0
> Reporter: Szymon Matejczyk
>
> When using Row constructor from kwargs, fields in the tuple underneath are
> sorted by name. When Schema is reading the row, it is not using the fields in
> this order.
> {code}
> from pyspark.sql import Row
> from pyspark.sql.types import *
> schema = StructType([
> StructField("id", StringType()),
> StructField("first_name", StringType())])
> row = Row(id="39", first_name="Szymon")
> schema.toInternal(row)
> Out[5]: ('Szymon', '39')
> {code}
> {code}
> df = sqlContext.createDataFrame([row], schema)
> df.show(1)
> +------+----------+
> | id|first_name|
> +------+----------+
> |Szymon| 39|
> +------+----------+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
