[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/18378 @edlee123 a Spark `DoubleType` will produce a `float64` dtype in Pandas and `FloatType` will be `float32`. `DateType` will be Python datetime.date objects. Also keep in mind that if you have integer data with null values, then Pandas will treat it as floats and represent the null values as `NaN`s. In this case, Spark will not change the dtype. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user edlee123 commented on the issue: https://github.com/apache/spark/pull/18378 I see the rationale now, thank you everyone --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18378 It's pretty natural to convert integer type to int32. Although Spark tries its best to avoid behavior changes, it's allowed to fix some wrong behaviors in new releases, and I believe it's well documented in the Spark 2.3 release notes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user edlee123 commented on the issue: https://github.com/apache/spark/pull/18378 Ok I see, I can see part of the rationale is performance (from discussion of astype above) and consistency with pyarrow https://arrow.apache.org/docs/python/pandas.html I guess without knowing much about the work with Arrow I was expecting it to be consistent with how pandas converts python types e.g in Spark 2.2 What happens with Double and DateType? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/18378 Looks good, I'll update #15821 with this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78448/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78448 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78448/testReport)** for PR 18378 at commit [`d8ba545`](https://github.com/apache/spark/commit/d8ba5452539c5fd5b650b7f5e51e467aabc33739). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18378 merged, thanks for your review! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78448 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78448/testReport)** for PR 18378 at commit [`d8ba545`](https://github.com/apache/spark/commit/d8ba5452539c5fd5b650b7f5e51e467aabc33739). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18378 the last commit just fixes a typo in comment, and the python style check passed locally, I'm going to merge this PR to unblock #15821 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18378 LGTM except for the nit ^. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18378 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78443 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78443/testReport)** for PR 18378 at commit [`357a798`](https://github.com/apache/spark/commit/357a79800f966fcdadaaf9729b191dc3c58327ea). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78443/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78443 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78443/testReport)** for PR 18378 at commit [`357a798`](https://github.com/apache/spark/commit/357a79800f966fcdadaaf9729b191dc3c58327ea). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18378 I sent a PR to your branch - https://github.com/cloud-fan/spark/pull/7 @cloud-fan. I will double check as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18378 (I will try to find a workaround ...) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18378 It sounds `astype` with the dict added from 0.19.0 - https://github.com/pandas-dev/pandas/commit/63a1e5c58af8ddc8dec192f39a0999aad74acaf9#diff-fb14ed747473b618d0c021fdef7ee85b. Mine was lower then that and I assume Jenkins one is the same case too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78432 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78432/testReport)** for PR 18378 at commit [`dfaa392`](https://github.com/apache/spark/commit/dfaa392c6d64a6e906c8d383b56fca9bb5c40327). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78432/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18378 My pleasure. I will give a shot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18378 @HyukjinKwon can you give me a hand for this? I can't reproduce this locally... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18378 Hm.. actually. this failure looks legitimate. I can reproduce this in my local too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78432/testReport)** for PR 18378 at commit [`dfaa392`](https://github.com/apache/spark/commit/dfaa392c6d64a6e906c8d383b56fca9bb5c40327). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78429 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78429/testReport)** for PR 18378 at commit [`1e98c49`](https://github.com/apache/spark/commit/1e98c494e0c414ca218b029bfc1a9d9faf3c2960). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78429/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78429/testReport)** for PR 18378 at commit [`1e98c49`](https://github.com/apache/spark/commit/1e98c494e0c414ca218b029bfc1a9d9faf3c2960). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18378 It sounds ok to me just except missing `_have_pandas = False` above `try:` . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78427/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78427 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78427/testReport)** for PR 18378 at commit [`36dc5e7`](https://github.com/apache/spark/commit/36dc5e7df4549270e66b33d4d171898e8b21faae). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78427 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78427/testReport)** for PR 18378 at commit [`36dc5e7`](https://github.com/apache/spark/commit/36dc5e7df4549270e66b33d4d171898e8b21faae). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78426/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78426 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78426/testReport)** for PR 18378 at commit [`36f9cb6`](https://github.com/apache/spark/commit/36f9cb63f21600db4a95ce05d370a72245649100). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78426 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78426/testReport)** for PR 18378 at commit [`36f9cb6`](https://github.com/apache/spark/commit/36f9cb63f21600db4a95ce05d370a72245649100). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/18378 > How about applying astype only for primitive types? Yeah, that might work since `astype` takes a dict you probably don't need to specify all the columns. It does seem like it makes a deep copy of the data that is being casted, so still might have an impact on performance. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/18378 How about applying `astype` only for primitive types? I guess the problem here is up-convert from `Byte/Short/IntegerType` to `int64`, `FloatType` to `float64`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78398 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78398/testReport)** for PR 18378 at commit [`afa74ab`](https://github.com/apache/spark/commit/afa74abce240c1e7536f1f25cfe48420fff58d42). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78398/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78398/testReport)** for PR 18378 at commit [`afa74ab`](https://github.com/apache/spark/commit/afa74abce240c1e7536f1f25cfe48420fff58d42). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78395/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78395 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78395/testReport)** for PR 18378 at commit [`e352817`](https://github.com/apache/spark/commit/e3528171db58acdecde287a04dc700d57cda91ff). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user ueshin commented on the issue: https://github.com/apache/spark/pull/18378 LGTM, pending Jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78395 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78395/testReport)** for PR 18378 at commit [`e352817`](https://github.com/apache/spark/commit/e3528171db58acdecde287a04dc700d57cda91ff). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78392/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18378 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78392 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78392/testReport)** for PR 18378 at commit [`8a033fb`](https://github.com/apache/spark/commit/8a033fb9ad6da0e0d69b90c9e4b00392d8e65ad2). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18378 **[Test build #78392 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78392/testReport)** for PR 18378 at commit [`8a033fb`](https://github.com/apache/spark/commit/8a033fb9ad6da0e0d69b90c9e4b00392d8e65ad2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18378: [SPARK-21163][SQL] DataFrame.toPandas should respect the...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18378 cc @ueshin @BryanCutler --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
