viirya commented on issue #24614: [SPARK-27712][PySpark][SQL] Returns correct schema even under different column order when creating dataframe URL: https://github.com/apache/spark/pull/24614#issuecomment-492672560 This is more interesting, as we allow something like: ```python data = [Row(key=i, value=str(i)) for i in range(100)] rdd = spark.sparkContext.parallelize(data, 5) # field names can differ. df = rdd.toDF(" a: int, b: string ") ``` So, the question is, in `createDataFrame`, should we respect original Row's schema in the RDD? Currently, * In case creating dataframe from local list of Row, we respect the Row's schema. * In case from RDD of Row, we don't respect it, as shown in the example in the PR description. It is inconsistent in two cases, obviously. This difference is also seen in following case. Field names can't differ, if from local list of Row. ```python >>> spark.createDataFrame([Row(A="1", B="2")], "B string, a string").first() Traceback (most recent call last): File "/Users/viirya/repos/spark-1/python/pyspark/sql/types.py", line 1527, in __getitem__ idx = self.__fields__.index(item) ValueError: 'a' is not in list ``` cc @HyukjinKwon @cloud-fan
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
