spark git commit: [SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles

2017-09-17 Thread gurwls223
Repository: spark Updated Branches: refs/heads/branch-2.2 309c401a5 -> a86831d61 [SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles ## What changes were proposed in this pull request? This PR proposes to improve error message from: ``` >>> sc.show_profiles()

spark git commit: [SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles

2017-09-17 Thread gurwls223
Repository: spark Updated Branches: refs/heads/branch-2.1 99de4b8f5 -> b35136a9e [SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles ## What changes were proposed in this pull request? This PR proposes to improve error message from: ``` >>> sc.show_profiles()

spark git commit: [SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles

2017-09-17 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 6308c65f0 -> 7c7266208 [SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles ## What changes were proposed in this pull request? This PR proposes to improve error message from: ``` >>> sc.show_profiles()

spark git commit: [SPARK-21953] Show both memory and disk bytes spilled if either is present

2017-09-17 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.1 3ae7ab8e8 -> 99de4b8f5 [SPARK-21953] Show both memory and disk bytes spilled if either is present As written now, there must be both memory and disk bytes spilled to show either of them. If there is only one of those types of spill

spark git commit: [SPARK-21953] Show both memory and disk bytes spilled if either is present

2017-09-17 Thread wenchen
Repository: spark Updated Branches: refs/heads/branch-2.2 42852bb17 -> 309c401a5 [SPARK-21953] Show both memory and disk bytes spilled if either is present As written now, there must be both memory and disk bytes spilled to show either of them. If there is only one of those types of spill

spark git commit: [SPARK-21985][PYSPARK] PairDeserializer is broken for double-zipped RDDs

2017-09-17 Thread gurwls223
Repository: spark Updated Branches: refs/heads/branch-2.1 e49c997fe -> 3ae7ab8e8 [SPARK-21985][PYSPARK] PairDeserializer is broken for double-zipped RDDs ## What changes were proposed in this pull request? (edited) Fixes a bug introduced in #16121 In PairDeserializer convert each batch of

spark git commit: [SPARK-21985][PYSPARK] PairDeserializer is broken for double-zipped RDDs

2017-09-17 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master f4073020a -> 6adf67dd1 [SPARK-21985][PYSPARK] PairDeserializer is broken for double-zipped RDDs ## What changes were proposed in this pull request? (edited) Fixes a bug introduced in #16121 In PairDeserializer convert each batch of keys

spark git commit: [SPARK-21985][PYSPARK] PairDeserializer is broken for double-zipped RDDs

2017-09-17 Thread gurwls223
Repository: spark Updated Branches: refs/heads/branch-2.2 51e5a821d -> 42852bb17 [SPARK-21985][PYSPARK] PairDeserializer is broken for double-zipped RDDs ## What changes were proposed in this pull request? (edited) Fixes a bug introduced in #16121 In PairDeserializer convert each batch of

spark git commit: [SPARK-22032][PYSPARK] Speed up StructType conversion

2017-09-17 Thread gurwls223
Repository: spark Updated Branches: refs/heads/master 73d906722 -> f4073020a [SPARK-22032][PYSPARK] Speed up StructType conversion ## What changes were proposed in this pull request? StructType.fromInternal is calling f.fromInternal(v) for every field. We can use precalculated information