[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19249 We can split two `needConversion` for key and value only and save key conversion or value conversion call though? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user maver1ck commented on the issue: https://github.com/apache/spark/pull/19249 @ueshin I think that for Maptype this is not a solution because every key / value of MapType is the same type so we need conversion for all entries or for nothing --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19249 Thanks for double checking @ueshin. Yes, I noticed that too while reviewing it. I just decided to merge it as is because I am quite sure of this one given struct type is the root type and this case looks quite common, and regarding that it looks the first contribution. Even though this one has a downside, practically the improvement looked better. I am also fine with doing this for others too (I am +0 for other types). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/19249 A late LGTM. Btw, can we use the same idea for `MapType`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19249 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19249 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19249 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81854/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19249 **[Test build #81854 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81854/testReport)** for PR 19249 at commit [`e9b7798`](https://github.com/apache/spark/commit/e9b779863dd84bda7a75af2f3fe3cbde892092a1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19249 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19249 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81853/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19249 **[Test build #81853 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81853/testReport)** for PR 19249 at commit [`8708a9d`](https://github.com/apache/spark/commit/8708a9db9ccbdf2d54ac4c68a562ecaa7cec88de). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19249 **[Test build #81854 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81854/testReport)** for PR 19249 at commit [`e9b7798`](https://github.com/apache/spark/commit/e9b779863dd84bda7a75af2f3fe3cbde892092a1). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user maver1ck commented on the issue: https://github.com/apache/spark/pull/19249 Done. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19249 LGTM too but hey @maver1ck could you add some comments around the codes and move the worst case benchmarks into the PR description? I guess this wouldn't be too demanding. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19249 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19249 **[Test build #81853 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81853/testReport)** for PR 19249 at commit [`8708a9d`](https://github.com/apache/spark/commit/8708a9db9ccbdf2d54ac4c68a562ecaa7cec88de). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19249 **[Test build #81852 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81852/testReport)** for PR 19249 at commit [`b1800ac`](https://github.com/apache/spark/commit/b1800acce09c43bdea96932f3f50e862d53677a0). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19249 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81852/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19249 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #19249: [SPARK-22032][PySpark] Speed up StructType conversion
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19249 **[Test build #81852 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81852/testReport)** for PR 19249 at commit [`b1800ac`](https://github.com/apache/spark/commit/b1800acce09c43bdea96932f3f50e862d53677a0). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
