[
https://issues.apache.org/jira/browse/SPARK-7505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicholas Chammas updated SPARK-7505:
------------------------------------
Component/s: SQL
> Update PySpark DataFrame docs: encourage __getitem__, mark as experimental,
> etc.
> --------------------------------------------------------------------------------
>
> Key: SPARK-7505
> URL: https://issues.apache.org/jira/browse/SPARK-7505
> Project: Spark
> Issue Type: Improvement
> Components: Documentation, PySpark, SQL
> Affects Versions: 1.3.1
> Reporter: Nicholas Chammas
> Priority: Minor
>
> The PySpark docs for DataFrame need the following fixes and improvements:
> # Per [SPARK-7035], we should encourage the use of {{\_\_getitem\_\_}} over
> {{\_\_getattr\_\_}} and change all our examples accordingly.
> # We should say clearly that the API is experimental. (That is currently not
> the case for the PySpark docs.)
> # We should provide an example of how to join and select from 2 DataFrames
> that have identically named columns, because it is not obvious:
> {code}
> >>> df1 = sqlContext.jsonRDD(sc.parallelize(['{"a": 4, "other": "I know"}']))
> >>> df2 = sqlContext.jsonRDD(sc.parallelize(['{"a": 4, "other": "I dunno"}']))
> >>> df12 = df1.join(df2, df1['a'] == df2['a'])
> >>> df12.select(df1['a'], df2['other']).show()
> a other
>
> 4 I dunno {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
