[GitHub] spark issue #17406: [SPARK-20009][SQL] Use DDL strings for defining schema i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17406 **[Test build #75207 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75207/testReport)** for PR 17406 at commit [`ec5452f`](https://github.com/apache/spark/commit/ec5452fb6a3c97a05d29c10b2f843fc96570e092). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17425: [HOTFIX] [SQL] Fix the failed test cases in Gener...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17425 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17425: [HOTFIX] [SQL] Fix the failed test cases in GeneratorFun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17425 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17425: [HOTFIX] [SQL] Fix the failed test cases in GeneratorFun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17425 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75206/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17425: [HOTFIX] [SQL] Fix the failed test cases in GeneratorFun...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17425 **[Test build #75206 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75206/testReport)** for PR 17425 at commit [`206370d`](https://github.com/apache/spark/commit/206370de02e45553c399048117086ba456856cee). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17425: [HOTFIX] [SQL] Fix the failed test cases in GeneratorFun...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17425 Merged to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17425: [HOTFIX] [SQL] Fix the failed test cases in GeneratorFun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17425 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17425: [HOTFIX] [SQL] Fix the failed test cases in GeneratorFun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17425 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75204/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17425: [HOTFIX] [SQL] Fix the failed test cases in GeneratorFun...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17425 **[Test build #75204 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75204/testReport)** for PR 17425 at commit [`48b73db`](https://github.com/apache/spark/commit/48b73dbaf8c7aaa567b3fec7b89b7d19396bfb5c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17427: [SPARK-20092][R][PROJECT INFRA] Add the detection for Sc...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17427 **[Test build #75208 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75208/testReport)** for PR 17427 at commit [`cfb3a88`](https://github.com/apache/spark/commit/cfb3a88d81ca2db4ff90eeaf08b4403154199d79). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17427: [SPARK-20092][R][PROJECT INFRA] Add the detection for Sc...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17427 cc @shivaram and @felixcheung. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17427: [SPARK-20092][R][PROJECT INFRA] Add the detection for Sc...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17427 These PR above were made against `detect-tests` which has a diff - https://github.com/spark-test/spark/compare/master...spark-test:detect-tests and therefore the PR against branch `detect-tests` enabled the tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17427: [SPARK-20092][R][PROJECT INFRA] Add the detection...
GitHub user HyukjinKwon opened a pull request: https://github.com/apache/spark/pull/17427 [SPARK-20092][R][PROJECT INFRA] Add the detection for Scala codes dedicated for R in AppVeyor tests ## What changes were proposed in this pull request? We are currently detecting the changes in `R/` directory only and then trigger AppVeyor tests. It seems we need to tests when there are Scala codes dedicated for R in `core/src/main/scala/org/apache/spark/r/` and `sql/core/src/main/scala/org/apache/spark/sql/api/r/` too. This will enables the changes, for example, for SPARK-20088. ## How was this patch tested? Tests with manually created PRs. - Changes in `sql/core/src/main/scala/org/apache/spark/sql/api/r/SQLUtils.scala` https://github.com/spark-test/spark/pull/13 - Changes in `core/src/main/scala/org/apache/spark/api/r/SerDe.scala` https://github.com/spark-test/spark/pull/12 - Changes in `README.md` https://github.com/spark-test/spark/pull/14 You can merge this pull request into a Git repository by running: $ git pull https://github.com/HyukjinKwon/spark SPARK-20092 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/17427.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17427 commit cfb3a88d81ca2db4ff90eeaf08b4403154199d79 Author: hyukjinkwon Date: 2017-03-25T06:02:48Z Add Scala codes that are related with R --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17343: [SPARK-20014] Optimize mergeSpillsWithFileStream method
Github user mridulm commented on the issue: https://github.com/apache/spark/pull/17343 LGTM will wait a bit to allow for others to comment. @zsxwing can you also take a look ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17425: [HOTFIX] [SQL] Fix the failed test cases in GeneratorFun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17425 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75203/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17425: [HOTFIX] [SQL] Fix the failed test cases in GeneratorFun...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17425 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17425: [HOTFIX] [SQL] Fix the failed test cases in GeneratorFun...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17425 **[Test build #75203 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75203/testReport)** for PR 17425 at commit [`f9d9082`](https://github.com/apache/spark/commit/f9d9082d78f669e7f19ceeac2ee47bd7faf7208a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17422: Attach accumulators / metrics to 'TaskKilled' end reason
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17422 Hi @noodle-fb, it seems not a trivial change that does not need a JIRA. Could we create a JIRA and put this in the title (see http://spark.apache.org/contributing.html)? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17406: [SPARK-20009][SQL] Use DDL strings for defining schema i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17406 **[Test build #75207 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75207/testReport)** for PR 17406 at commit [`ec5452f`](https://github.com/apache/spark/commit/ec5452fb6a3c97a05d29c10b2f843fc96570e092). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17406: [SPARK-20009][SQL] Use DDL strings for defining schema i...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/17406 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17426: [SPARK-17137][ML][WIP] Compress logistic regression coef...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17426 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17426: [SPARK-17137][ML][WIP] Compress logistic regression coef...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17426 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75205/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17426: [SPARK-17137][ML][WIP] Compress logistic regression coef...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17426 **[Test build #75205 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75205/testReport)** for PR 17426 at commit [`c677696`](https://github.com/apache/spark/commit/c677696fe714be04df614877a0ee0d4f76254812). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17406: [SPARK-20009][SQL] Use DDL strings for defining schema i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17406 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17406: [SPARK-20009][SQL] Use DDL strings for defining schema i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17406 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75202/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17406: [SPARK-20009][SQL] Use DDL strings for defining schema i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17406 **[Test build #75202 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75202/testReport)** for PR 17406 at commit [`ec5452f`](https://github.com/apache/spark/commit/ec5452fb6a3c97a05d29c10b2f843fc96570e092). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17324: [SPARK-19969] [ML] Imputer doc and example
Github user hhbyyh commented on the issue: https://github.com/apache/spark/pull/17324 Updated with python example. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17421: [SPARK-20040][ML][python] pyspark wrapper for ChiSquareT...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17421 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75199/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17421: [SPARK-20040][ML][python] pyspark wrapper for ChiSquareT...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17421 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17421: [SPARK-20040][ML][python] pyspark wrapper for ChiSquareT...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17421 **[Test build #75199 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75199/testReport)** for PR 17421 at commit [`32a0b0c`](https://github.com/apache/spark/commit/32a0b0c93338f08effb72059759a8baea514fa7c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17425: [HOTFIX] [SQL] Fix the failed test cases in GeneratorFun...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17425 **[Test build #75206 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75206/testReport)** for PR 17425 at commit [`206370d`](https://github.com/apache/spark/commit/206370de02e45553c399048117086ba456856cee). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17394: [SPARK-20067] [SQL] Use treeString to print out the tabl...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17394 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75200/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17426: [SPARK-17137][ML][WIP] Compress logistic regression coef...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17426 **[Test build #75205 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75205/testReport)** for PR 17426 at commit [`c677696`](https://github.com/apache/spark/commit/c677696fe714be04df614877a0ee0d4f76254812). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17394: [SPARK-20067] [SQL] Use treeString to print out the tabl...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17394 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17394: [SPARK-20067] [SQL] Use treeString to print out the tabl...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17394 **[Test build #75200 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75200/testReport)** for PR 17394 at commit [`1d72079`](https://github.com/apache/spark/commit/1d720792b4bb2619bb9273b4bfb06f850b182896). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17424: [SPARK-20089] [SQL] [TEST] Added DESC FUNCTION and DESC ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17424 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75201/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17424: [SPARK-20089] [SQL] [TEST] Added DESC FUNCTION and DESC ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17424 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17426: [SPARK-17137][ML][WIP] Compress logistic regressi...
GitHub user sethah opened a pull request: https://github.com/apache/spark/pull/17426 [SPARK-17137][ML][WIP] Compress logistic regression coefficients ## What changes were proposed in this pull request? Use the new `compressed` method on matrices to store the logistic regression coefficients as sparse or dense - whichever is requires less memory. Marked as WIP so we can add some performance test results. Basically, we should see if prediction is slower because of using a sparse matrix over a dense one. This can happen since sparse matrices do not use native BLAS operations when computing the margins. ## How was this patch tested? Unit tests added. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sethah/spark SPARK-17137 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/17426.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17426 commit c677696fe714be04df614877a0ee0d4f76254812 Author: sethah Date: 2017-03-25T04:04:55Z compress log reg coefficients --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17424: [SPARK-20089] [SQL] [TEST] Added DESC FUNCTION and DESC ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17424 **[Test build #75201 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75201/testReport)** for PR 17424 at commit [`3ef9a06`](https://github.com/apache/spark/commit/3ef9a06daf2e79d9282c108243b194a5c54bed62). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17425: [HOTFIX] [SQL] Disable the failed test cases in Generato...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17425 **[Test build #75204 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75204/testReport)** for PR 17425 at commit [`48b73db`](https://github.com/apache/spark/commit/48b73dbaf8c7aaa567b3fec7b89b7d19396bfb5c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17425: [HOTFIX] [SQL] Revert changes on supportCodegen of Gener...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17425 It sounds like the codegen version has a bug. It did not respect the following logics: ``` * @param outer when true, each input row will be output at least once, even if the output of the * given `generator` is empty. `outer` has no effect when `join` is false. ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17329: [SPARK-19991]FileSegmentManagedBuffer performance...
Github user witgo commented on a diff in the pull request:
https://github.com/apache/spark/pull/17329#discussion_r108027203
--- Diff:
common/network-common/src/main/java/org/apache/spark/network/buffer/FileSegmentManagedBuffer.java
---
@@ -37,13 +37,24 @@
* A {@link ManagedBuffer} backed by a segment in a file.
*/
public final class FileSegmentManagedBuffer extends ManagedBuffer {
- private final TransportConf conf;
+ private final boolean lazyFileDescriptor;
+ private final int memoryMapBytes;
private final File file;
private final long offset;
private final long length;
public FileSegmentManagedBuffer(TransportConf conf, File file, long
offset, long length) {
-this.conf = conf;
+this(conf.lazyFileDescriptor(), conf.memoryMapBytes(), file, offset,
length);
+ }
+
+ public FileSegmentManagedBuffer(
--- End diff --
This branch [SPARK-19991_try2
](https://github.com/witgo/spark/commits/SPARK-19991_try2) needs `244.45` s in
my test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17425: [HOTFIX] [SQL] Revert changes on supportCodegen of Gener...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17425 **[Test build #75203 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75203/testReport)** for PR 17425 at commit [`f9d9082`](https://github.com/apache/spark/commit/f9d9082d78f669e7f19ceeac2ee47bd7faf7208a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17406: [SPARK-20009][SQL] Use DDL strings for defining schema i...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17406 Just submitted a fix. NVM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17425: [HOTFIX] [SQL] Revert changes on supportCodegen of Gener...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17425 cc @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17425: [HOTFIX] [BUILD] Revert changes on supportCodegen...
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/17425 [HOTFIX] [BUILD] Revert changes on supportCodegen of GenerateExec ### What changes were proposed in this pull request? Multiple tests failed. Revert the changes on `supportCodegen` of `GenerateExec`. For example, - https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75194/testReport/ ### How was this patch tested? N/A You can merge this pull request into a Git repository by running: $ git pull https://github.com/gatorsmile/spark turnOnCodeGenGenerateExec Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/17425.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17425 commit f9d9082d78f669e7f19ceeac2ee47bd7faf7208a Author: Xiao Li Date: 2017-03-25T03:20:27Z revert --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17411: logging improvements
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17411 FWIW, I am not positive about proposing trivial changes for the same reason in https://github.com/apache/spark/pull/17411#issuecomment-289071694. > ... going forward given the overhead of reviewing and testing and merging (I am worried of swarming trivial small PRs. I kind of have been feeling that a first small typo PR merged as a reference tends to make other people open similar PRs.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17421: [SPARK-20040][ML][python] pyspark wrapper for Chi...
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17421#discussion_r108026117 --- Diff: python/pyspark/ml/stat.py --- @@ -0,0 +1,87 @@ +from pyspark import since, SparkContext +from pyspark.ml.common import _java2py, _py2java +from pyspark.ml.wrapper import _jvm + + +class ChiSquareTest(object): --- End diff -- Also, we put the triple-quotes on their own line elsewhere in pyspark --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17421: [SPARK-20040][ML][python] pyspark wrapper for Chi...
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/17421#discussion_r108023140
--- Diff: python/pyspark/ml/tests.py ---
@@ -1692,6 +1692,23 @@ def test_new_java_array(self):
self.assertEqual(_java2py(self.sc, java_array), [])
+class ChiSquareTestTests(SparkSessionTestCase):
+
+def test_ChiSquareTest(self):
+labels = [1, 2, 0]
+vectors = [_convert_to_vector([0, 1, 2]),
+ _convert_to_vector([1, 1, 1]),
+ _convert_to_vector([2, 1, 0])]
+data = zip(labels, vectors)
+df = self.spark.createDataFrame(data, ['label', 'feat'])
+res = ChiSquareTest.test(df, 'feat', 'label')
+# pValues = res.select("pValues").collect())
--- End diff --
(Noting that this can be updated once the Spark SQL bug is fixed)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17421: [SPARK-20040][ML][python] pyspark wrapper for Chi...
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17421#discussion_r108026690 --- Diff: python/pyspark/ml/tests.py --- @@ -1692,6 +1692,23 @@ def test_new_java_array(self): self.assertEqual(_java2py(self.sc, java_array), []) +class ChiSquareTestTests(SparkSessionTestCase): + +def test_ChiSquareTest(self): +labels = [1, 2, 0] +vectors = [_convert_to_vector([0, 1, 2]), + _convert_to_vector([1, 1, 1]), + _convert_to_vector([2, 1, 0])] +data = zip(labels, vectors) --- End diff -- Same for the doc test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17421: [SPARK-20040][ML][python] pyspark wrapper for Chi...
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17421#discussion_r108022929 --- Diff: python/pyspark/ml/stat.py --- @@ -0,0 +1,87 @@ +from pyspark import since, SparkContext +from pyspark.ml.common import _java2py, _py2java +from pyspark.ml.wrapper import _jvm + + +class ChiSquareTest(object): --- End diff -- Mark as Experimental (Search for other examples to see how this is marked) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17421: [SPARK-20040][ML][python] pyspark wrapper for Chi...
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17421#discussion_r108022935 --- Diff: python/pyspark/ml/tests.py --- @@ -1692,6 +1692,23 @@ def test_new_java_array(self): self.assertEqual(_java2py(self.sc, java_array), []) +class ChiSquareTestTests(SparkSessionTestCase): + +def test_ChiSquareTest(self): --- End diff -- This is a little arbitrary, but to follow other examples, write this as: ```test_chisquaretest``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17421: [SPARK-20040][ML][python] pyspark wrapper for Chi...
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17421#discussion_r108023008 --- Diff: python/pyspark/ml/tests.py --- @@ -1692,6 +1692,23 @@ def test_new_java_array(self): self.assertEqual(_java2py(self.sc, java_array), []) +class ChiSquareTestTests(SparkSessionTestCase): + +def test_ChiSquareTest(self): +labels = [1, 2, 0] +vectors = [_convert_to_vector([0, 1, 2]), --- End diff -- Use DenseVector, not _convert_to_vector. (use public APIs wherever possible) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17421: [SPARK-20040][ML][python] pyspark wrapper for Chi...
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17421#discussion_r108026677 --- Diff: python/pyspark/ml/stat.py --- @@ -0,0 +1,102 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from pyspark import since, SparkContext +from pyspark.ml.common import _java2py, _py2java +from pyspark.ml.wrapper import _jvm + + +class ChiSquareTest(object): +""" Conduct Pearson's independence test for every feature against the label. For each feature, +the (feature, label) pairs are converted into a contingency matrix for which the Chi-squared +statistic is computed. All label and feature values must be categorical. + +The null hypothesis is that the occurrence of the outcomes is statistically independent. + +:param dataset: --- End diff -- Same for the return value text --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17421: [SPARK-20040][ML][python] pyspark wrapper for Chi...
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17421#discussion_r108022984 --- Diff: python/pyspark/ml/stat.py --- @@ -0,0 +1,87 @@ +from pyspark import since, SparkContext +from pyspark.ml.common import _java2py, _py2java +from pyspark.ml.wrapper import _jvm + + +class ChiSquareTest(object): --- End diff -- Mark as Experimental (Search for other example of this) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17421: [SPARK-20040][ML][python] pyspark wrapper for Chi...
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17421#discussion_r108026673 --- Diff: python/pyspark/ml/stat.py --- @@ -0,0 +1,102 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from pyspark import since, SparkContext +from pyspark.ml.common import _java2py, _py2java +from pyspark.ml.wrapper import _jvm + + +class ChiSquareTest(object): +""" Conduct Pearson's independence test for every feature against the label. For each feature, +the (feature, label) pairs are converted into a contingency matrix for which the Chi-squared +statistic is computed. All label and feature values must be categorical. + +The null hypothesis is that the occurrence of the outcomes is statistically independent. + +:param dataset: --- End diff -- Copy param text from the Scala doc, unless there's a need to customize it for Python --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17421: [SPARK-20040][ML][python] pyspark wrapper for Chi...
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17421#discussion_r108023069 --- Diff: python/pyspark/ml/tests.py --- @@ -1692,6 +1692,23 @@ def test_new_java_array(self): self.assertEqual(_java2py(self.sc, java_array), []) +class ChiSquareTestTests(SparkSessionTestCase): + +def test_ChiSquareTest(self): +labels = [1, 2, 0] +vectors = [_convert_to_vector([0, 1, 2]), + _convert_to_vector([1, 1, 1]), + _convert_to_vector([2, 1, 0])] +data = zip(labels, vectors) --- End diff -- It can also be nicer to write this in a per-row format, rather than zipping labels and vectors which are defined separately. See other examples of createDataFrame in this file. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17421: [SPARK-20040][ML][python] pyspark wrapper for Chi...
Github user jkbradley commented on a diff in the pull request: https://github.com/apache/spark/pull/17421#discussion_r108026186 --- Diff: python/pyspark/ml/stat.py --- @@ -0,0 +1,104 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from pyspark import since, SparkContext +from pyspark.ml.common import _java2py, _py2java +from pyspark.ml.wrapper import _jvm + + +class ChiSquareTest(object): +""" Conduct Pearson's independence test for every feature against the label. For each feature, --- End diff -- I just saw you changed this from the Scala doc b/c I left "RDD" there. Would you mind correcting the Scala doc too? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17406: [SPARK-20009][SQL] Use DDL strings for defining schema i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17406 **[Test build #75202 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75202/testReport)** for PR 17406 at commit [`ec5452f`](https://github.com/apache/spark/commit/ec5452fb6a3c97a05d29c10b2f843fc96570e092). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17406: [SPARK-20009][SQL] Use DDL strings for defining schema i...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/17406 oh, it seems we hit weird errors... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17406: [SPARK-20009][SQL] Use DDL strings for defining schema i...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17406 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17406: [SPARK-20009][SQL] Use DDL strings for defining schema i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17406 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75196/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17406: [SPARK-20009][SQL] Use DDL strings for defining schema i...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17406 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17406: [SPARK-20009][SQL] Use DDL strings for defining schema i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17406 **[Test build #75196 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75196/testReport)** for PR 17406 at commit [`ec5452f`](https://github.com/apache/spark/commit/ec5452fb6a3c97a05d29c10b2f843fc96570e092). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17324: [SPARK-19969] [ML] Imputer doc and example
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17324 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17324: [SPARK-19969] [ML] Imputer doc and example
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17324 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75197/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17324: [SPARK-19969] [ML] Imputer doc and example
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17324 **[Test build #75197 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75197/testReport)** for PR 17324 at commit [`a2e24c0`](https://github.com/apache/spark/commit/a2e24c0b1bd1e640a44e6da2d97c58fd1cbd0ddd). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17424: [SPARK-20089] [SQL] [TEST] Added DESC FUNCTION and DESC ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17424 **[Test build #75201 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75201/testReport)** for PR 17424 at commit [`3ef9a06`](https://github.com/apache/spark/commit/3ef9a06daf2e79d9282c108243b194a5c54bed62). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17424: [SPARK-20089] [SQL] [TEST] Added DESC FUNCTION and DESC ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17424 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17394: [SPARK-20067] [SQL] Use treeString to print out the tabl...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17394 **[Test build #75200 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75200/testReport)** for PR 17394 at commit [`1d72079`](https://github.com/apache/spark/commit/1d720792b4bb2619bb9273b4bfb06f850b182896). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17423: [SPARK-20088] Do not create new SparkContext in SparkR c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17423 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17423: [SPARK-20088] Do not create new SparkContext in SparkR c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17423 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75193/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17423: [SPARK-20088] Do not create new SparkContext in SparkR c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17423 **[Test build #75193 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75193/testReport)** for PR 17423 at commit [`f07e3a9`](https://github.com/apache/spark/commit/f07e3a945c3a3eecd448a8525b86542e999a2f8a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17424: [SPARK-20089] [SQL] [TEST] Added DESC FUNCTION and DESC ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17424 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75194/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17424: [SPARK-20089] [SQL] [TEST] Added DESC FUNCTION and DESC ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17424 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17424: [SPARK-20089] [SQL] [TEST] Added DESC FUNCTION and DESC ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17424 **[Test build #75194 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75194/testReport)** for PR 17424 at commit [`3ef9a06`](https://github.com/apache/spark/commit/3ef9a06daf2e79d9282c108243b194a5c54bed62). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17421: [SPARK-20040][ML][python] pyspark wrapper for ChiSquareT...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17421 **[Test build #75199 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75199/testReport)** for PR 17421 at commit [`32a0b0c`](https://github.com/apache/spark/commit/32a0b0c93338f08effb72059759a8baea514fa7c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17421: [SPARK-20040][ML][python] pyspark wrapper for ChiSquareT...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17421 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17421: [SPARK-20040][ML][python] pyspark wrapper for ChiSquareT...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17421 **[Test build #75198 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75198/testReport)** for PR 17421 at commit [`b71caef`](https://github.com/apache/spark/commit/b71caef69a21f9a9a515e42ed9cc045d058ef80c). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17421: [SPARK-20040][ML][python] pyspark wrapper for ChiSquareT...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17421 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75198/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17421: [SPARK-20040][ML][python] pyspark wrapper for ChiSquareT...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17421 **[Test build #75198 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75198/testReport)** for PR 17421 at commit [`b71caef`](https://github.com/apache/spark/commit/b71caef69a21f9a9a515e42ed9cc045d058ef80c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17324: [SPARK-19969] [ML] Imputer doc and example
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17324 **[Test build #75197 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75197/testReport)** for PR 17324 at commit [`a2e24c0`](https://github.com/apache/spark/commit/a2e24c0b1bd1e640a44e6da2d97c58fd1cbd0ddd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17420: [SPARK-20070][SQL] Fix 2.10 build
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17420 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17297: [SPARK-14649][CORE] DagScheduler should not run duplicat...
Github user sitalkedia commented on the issue: https://github.com/apache/spark/pull/17297 @kayousterhout - It seems like the test timeout might be related to the change. But I am not able to find the culprit test case from the build log. Any idea what is wrong? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17420: [SPARK-20070][SQL] Fix 2.10 build
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17420 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75191/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17420: [SPARK-20070][SQL] Fix 2.10 build
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17420 **[Test build #75191 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75191/testReport)** for PR 17420 at commit [`f4c3097`](https://github.com/apache/spark/commit/f4c3097ed6df3dbbdde11ebd8a207224cf304df7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17088: [SPARK-19753][CORE] Un-register all shuffle output on a ...
Github user sitalkedia commented on the issue: https://github.com/apache/spark/pull/17088 Filed a JIRA SPARK-20091 to allow running multiple concurrent attempts of a stage. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17335: [SPARK-19995][YARN] Register tokens to current UGI to av...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/17335 Sorry @vanzin about it. Just update the description, please review again. Thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17406: [SPARK-20009][SQL] Use DDL strings for defining schema i...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17406 **[Test build #75196 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75196/testReport)** for PR 17406 at commit [`ec5452f`](https://github.com/apache/spark/commit/ec5452fb6a3c97a05d29c10b2f843fc96570e092). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17088: [SPARK-19753][CORE] Un-register all shuffle output on a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17088 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75184/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17088: [SPARK-19753][CORE] Un-register all shuffle output on a ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17088 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17088: [SPARK-19753][CORE] Un-register all shuffle output on a ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17088 **[Test build #75184 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75184/testReport)** for PR 17088 at commit [`be3b3db`](https://github.com/apache/spark/commit/be3b3dbd2d813a3d1d164d9b7f8127d09b752880). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #17406: [SPARK-20009][SQL] Use DDL strings for defining s...
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17406#discussion_r108024156
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/sources/SimpleTextRelation.scala
---
@@ -36,7 +38,13 @@ class SimpleTextSource extends TextBasedFileFormat with
DataSourceRegister {
sparkSession: SparkSession,
options: Map[String, String],
files: Seq[FileStatus]): Option[StructType] = {
-Some(DataType.fromJson(options("dataSchema")).asInstanceOf[StructType])
+val schemaAsString = options("dataSchema")
+val schema = try {
+ DataType.fromJson(schemaAsString)
+} catch {
+ case NonFatal(_) => DataType.fromDdl(schemaAsString)
+}
+Some(schema.asInstanceOf[StructType])
--- End diff --
Aha, I missed the point. okay, I'll revert this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17419: [SPARK-19634][ML] Multivariate summarizer - dataframes A...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17419 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75190/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17419: [SPARK-19634][ML] Multivariate summarizer - dataframes A...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17419 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17419: [SPARK-19634][ML] Multivariate summarizer - dataframes A...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17419 **[Test build #75190 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75190/testReport)** for PR 17419 at commit [`ba200bb`](https://github.com/apache/spark/commit/ba200bb1d52d468acebc9fe44f6727c570a35335). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17421: [SPARK-20040][ML][python] pyspark wrapper for ChiSquareT...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17421 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75195/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17421: [SPARK-20040][ML][python] pyspark wrapper for ChiSquareT...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17421 **[Test build #75195 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75195/testReport)** for PR 17421 at commit [`37e187b`](https://github.com/apache/spark/commit/37e187b26bcb32a5d341ec96a9da8ac7196741ad). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17421: [SPARK-20040][ML][python] pyspark wrapper for ChiSquareT...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17421 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17406: [SPARK-20009][SQL] Use DDL strings for defining schema i...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/17406 okay, I'll update soon! Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
