[GitHub] spark pull request: [SPARK-3394] [SQL] Fix crash in TakeOrdered wh...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2264#issuecomment-54703741 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3394] [SQL] Fix crash in TakeOrdered wh...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2264#issuecomment-54703751 LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Decrease partitions when testing
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2164#issuecomment-54703762 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3349] [SQL] Output partitioning of limi...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2262#issuecomment-54703774 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3367] Remove spark.shuffle.spill.compre...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2247#issuecomment-54703802 Ok I'm going to close this for now given the debate. @mridulm it would be great to hear more about the performance difference. It is hard for me to imagine a scenario in which you would want different settings for the two. @andrewor14 can you fix the bug? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3367] Remove spark.shuffle.spill.compre...
Github user rxin closed the pull request at: https://github.com/apache/spark/pull/2247 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL]Add Dynamic Partition suppo...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2226#issuecomment-54703827 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3353] parent stage should have lower st...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2273#issuecomment-54703841 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: MetadataCleaner - fine control cleanup documen...
Github user puravaggarwal closed the pull request at: https://github.com/apache/spark/pull/89 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3361] Expand PEP 8 checks to include EC...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2297 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: MetadataCleaner - fine control cleanup documen...
Github user puravaggarwal commented on the pull request: https://github.com/apache/spark/pull/89#issuecomment-54703862 As per discussions above - closing the pull request --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [BUILD] Fix for false positives reported by mi...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2285#issuecomment-54703906 Can you change an old API to make sure this is reporting correctly? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3430] [PySpark] [Doc] generate PySpark ...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/2292#issuecomment-54704106 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3030] [PySpark] Reuse Python worker
Github user davies commented on the pull request: https://github.com/apache/spark/pull/2259#issuecomment-54704111 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3417] -Use new-style classes in PySpark
Github user davies commented on the pull request: https://github.com/apache/spark/pull/2288#issuecomment-54704149 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3418] Sparse Matrix support (CCS) and a...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2294#issuecomment-54704308 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19896/consoleFull) for PR 2294 at commit [`848406c`](https://github.com/apache/spark/commit/848406c1b6779eea9bbcf1dd582b541db46d7dad). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class SparseMatrix(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3394] [SQL] Fix crash in TakeOrdered wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2264#issuecomment-54704437 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19902/consoleFull) for PR 2264 at commit [`c87355b`](https://github.com/apache/spark/commit/c87355bf0c3e99dbff08c5a7708901471c86f45b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Decrease partitions when testing
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2164#issuecomment-54704448 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19905/consoleFull) for PR 2164 at commit [`b035325`](https://github.com/apache/spark/commit/b035325d380f750f6a63922cdeabba7a8b2ed47c). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3349] [SQL] Output partitioning of limi...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2262#issuecomment-54704440 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19903/consoleFull) for PR 2262 at commit [`3e1b05c`](https://github.com/apache/spark/commit/3e1b05c95958076901992e17e9df4ea60f2ad24c). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3030] [PySpark] Reuse Python worker
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2259#issuecomment-54704441 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19904/consoleFull) for PR 2259 at commit [`583716e`](https://github.com/apache/spark/commit/583716ee156d0d7eb87220295a624bfc1427032a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3353] parent stage should have lower st...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2273#issuecomment-54704438 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19901/consoleFull) for PR 2273 at commit [`0e02379`](https://github.com/apache/spark/commit/0e023797e63fea8abc1ad7f8b4edb12921d8adbd). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-938][doc] Add OpenStack Swift support
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/2298 [SPARK-938][doc] Add OpenStack Swift support See compiled doc at http://people.apache.org/~rxin/tmp/openstack-swift/_site/storage-openstack-swift.html This is based on #1010. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rxin/spark openstack-swift Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2298.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2298 commit b6c37ef5a28dfa0ed07f6ab8f154fcd43a2131af Author: Gil Vernik g...@il.ibm.com Date: 2014-06-08T07:23:41Z Openstack Swift support commit ce483d76a1d524800859764b967c8b5a98fbd9ea Author: Gil Vernik g...@il.ibm.com Date: 2014-06-08T07:34:04Z SPARK-938 - Openstack Swift object storage support This is initial documentation describing how to integrate Spark with Swift. This commit contains documentation for stand alone cluster. Next patches will contain details how to integrate Swift in other deployment of Spark. commit eff538dd8fb7e306c84874e9b4c7da68fa0fe5d0 Author: Gil Vernik g...@il.ibm.com Date: 2014-06-08T07:34:04Z SPARK-938 - Openstack Swift object storage support Documentation how to integrate Spark with Openstack Swift. commit 9b625b569e3fc0024edd38b8fc33825c53be8151 Author: Gil Vernik g...@il.ibm.com Date: 2014-06-10T13:15:01Z Merge branch 'master' of https://github.com/gilv/spark Conflicts: docs/openstack-integration.md commit 2aba7631c98d881c031c233a39342b936749e161 Author: Gil Vernik g...@il.ibm.com Date: 2014-06-10T13:15:01Z Fix to docs/openstack-integration.md commit c977658580663c8ff0d003938ae833e4d9a19631 Author: Gil Vernik g...@il.ibm.com Date: 2014-06-10T15:14:05Z Merge branch 'master' of https://github.com/gilv/spark Conflicts: docs/openstack-integration.md commit 39a9737e16b27435f448030f1f7f7a6c506e08dc Author: Gil Vernik g...@il.ibm.com Date: 2014-06-12T09:13:29Z Spark integration with Openstack Swift commit eb22295f1c3813d330abf1a4509cc2e8b62abcff Author: Reynold Xin r...@apache.org Date: 2014-06-13T06:55:18Z Merge pull request #1010 from gilv/master SPARK-938 - Openstack Swift object storage support commit 99f095d9577802912fa715495bd9aec3e3867d54 Author: Reynold Xin r...@apache.org Date: 2014-06-14T20:13:17Z Pending openstack changes. commit cca719227c828e790f7b2e3c94ef83f5fb55ceb3 Author: Gil Vernik g...@il.ibm.com Date: 2014-06-16T08:39:51Z Removed white spases from pom.xml commit 47ce99d10715e1097feba8690a10a38c09f69fd9 Author: Reynold Xin r...@apache.org Date: 2014-06-17T01:34:12Z Merge branch 'master' into openstack commit ac0679eb81389cef45c3b604581fb53274023f1c Author: Reynold Xin r...@apache.org Date: 2014-06-17T01:46:14Z Fixed an unclosed tr. commit 69948278581a090cd635ead6672272809858d608 Author: Gil Vernik g...@il.ibm.com Date: 2014-06-17T06:16:17Z Merge pull request #1 from rxin/openstack Bring the branch up to date and fixed some documentation typos. commit 9233fef3450846fc6ff1e7e7e3c75191a543a573 Author: Gil Vernik g...@il.ibm.com Date: 2014-06-18T05:19:48Z Fixed typos commit e9c376186e5ab4791a15fe3244687909a999b9bb Author: Reynold Xin r...@apache.org Date: 2014-09-06T06:14:16Z Merge pull request #1010 from gilv/master SPARK-938 - Openstack Swift object storage support commit 0447c9fc563ef4bc02f937bfea63ea1d62f252cf Author: Reynold Xin r...@apache.org Date: 2014-09-06T06:59:34Z Removed sample code. commit 846f5cbbb605421646587cb2e065f070d83143ae Author: Reynold Xin r...@apache.org Date: 2014-09-06T07:05:18Z Added a link from overview page. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3408] Fixed Limit operator so it works ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2281#issuecomment-54705097 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19898/consoleFull) for PR 2281 at commit [`1ef7780`](https://github.com/apache/spark/commit/1ef7780c167cd22dfecdd00bef49113eb58d617c). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3409][SQL] Avoid pulling in Exchange op...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2282#issuecomment-54705275 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19897/consoleFull) for PR 2282 at commit [`1de3f88`](https://github.com/apache/spark/commit/1de3fd0cf5a3d35c9ae99450ebe9e5f8c356). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Decrease partitions when testing
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2164#issuecomment-54705322 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19905/consoleFull) for PR 2164 at commit [`b035325`](https://github.com/apache/spark/commit/b035325d380f750f6a63922cdeabba7a8b2ed47c). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3409][SQL] Avoid pulling in Exchange op...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2282#issuecomment-54705320 Merging this in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3409][SQL] Avoid pulling in Exchange op...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2282 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3408] Fixed Limit operator so it works ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2281#issuecomment-54705341 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3394] [SQL] Fix crash in TakeOrdered wh...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2264#issuecomment-54705417 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19902/consoleFull) for PR 2264 at commit [`c87355b`](https://github.com/apache/spark/commit/c87355bf0c3e99dbff08c5a7708901471c86f45b). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3030] [PySpark] Reuse Python worker
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2259#issuecomment-54705444 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19904/consoleFull) for PR 2259 at commit [`583716e`](https://github.com/apache/spark/commit/583716ee156d0d7eb87220295a624bfc1427032a). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3408] Fixed Limit operator so it works ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2281#issuecomment-54705491 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19907/consoleFull) for PR 2281 at commit [`1ef7780`](https://github.com/apache/spark/commit/1ef7780c167cd22dfecdd00bef49113eb58d617c). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3353] parent stage should have lower st...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2273#issuecomment-54705508 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19901/consoleFull) for PR 2273 at commit [`0e02379`](https://github.com/apache/spark/commit/0e023797e63fea8abc1ad7f8b4edb12921d8adbd). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3349] [SQL] Output partitioning of limi...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2262#issuecomment-54706276 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19903/consoleFull) for PR 2262 at commit [`3e1b05c`](https://github.com/apache/spark/commit/3e1b05c95958076901992e17e9df4ea60f2ad24c). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3353] parent stage should have lower st...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2273#issuecomment-54706606 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19908/consoleFull) for PR 2273 at commit [`abbb4c6`](https://github.com/apache/spark/commit/abbb4c638b46096438a1923e916b7f249f9606f8). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3415] [PySpark] removes SerializingAdap...
Github user wardviaene commented on the pull request: https://github.com/apache/spark/pull/2287#issuecomment-54707160 Hi @JoshRosen The bug would only be triggered in a Class, that's why I initially wrote it like that. My last commit removes the references to SerializingAdapter and adds a test script that calls the save function directly. The test fails before the patch, but succeeds after the patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3408] Fixed Limit operator so it works ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2281#issuecomment-54707290 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19907/consoleFull) for PR 2281 at commit [`1ef7780`](https://github.com/apache/spark/commit/1ef7780c167cd22dfecdd00bef49113eb58d617c). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3415] [PySpark] removes SerializingAdap...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2287#issuecomment-54707788 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19909/consoleFull) for PR 2287 at commit [`aaf10b7`](https://github.com/apache/spark/commit/aaf10b7a933df556c5165e4319df941475132d72). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3415] [PySpark] removes SerializingAdap...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2287#issuecomment-54707812 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19909/consoleFull) for PR 2287 at commit [`aaf10b7`](https://github.com/apache/spark/commit/aaf10b7a933df556c5165e4319df941475132d72). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3353] parent stage should have lower st...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2273#issuecomment-54707818 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19908/consoleFull) for PR 2273 at commit [`abbb4c6`](https://github.com/apache/spark/commit/abbb4c638b46096438a1923e916b7f249f9606f8). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] Clarify slice vs partition in the...
GitHub user mattf opened a pull request: https://github.com/apache/spark/pull/2299 [SPARK-1701] Clarify slice vs partition in the programming guide This is a partial solution to SPARK-1701, only addressing the documentation confusion. Additional work can be to actually change the numSlices parameter name across languages, with care required for scala python to maintain backward compatibility for named parameters. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattf/spark SPARK-1701-programming-guide Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2299.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2299 commit 7b045e0845b70ccb9b6e197fe36546dccec167c6 Author: Matthew Farrellee m...@redhat.com Date: 2014-09-06T12:55:12Z [SPARK-1701] Clarify slice vs partition in the programming guide This is a partial solution to SPARK-1701, only addressing the documentation confusion. Additional work can be to actually change the numSlices parameter name across languages, with care required for scala python to maintain backward compatibility for named parameters. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Provide a default PYSPARK_PYTHON for python/ru...
GitHub user mattf opened a pull request: https://github.com/apache/spark/pull/2300 Provide a default PYSPARK_PYTHON for python/run_tests Without this the version of python used in the test is not recorded. The error is, Testing with Python version: ./run-tests: line 57: --version: command not found You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattf/spark master-fix-python-run-tests Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2300.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2300 commit 65a09f5a95a3636bc8a2081fa02a15faa6a52ed4 Author: Matthew Farrellee m...@redhat.com Date: 2014-09-06T13:26:49Z Provide a default PYSPARK_PYTHON for python/run_tests Without this the version of python used in the test is not recorded. The error is, Testing with Python version: ./run-tests: line 57: --version: command not found --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [EC2] don't duplicate default values
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/2290#issuecomment-54712456 Failed test seems unrelated. ```log [error] Failed tests: [error] org.apache.spark.deploy.SparkSubmitSuite ``` Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Provide a default PYSPARK_PYTHON for python/ru...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2300#issuecomment-54712597 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19910/consoleFull) for PR 2300 at commit [`65a09f5`](https://github.com/apache/spark/commit/65a09f5a95a3636bc8a2081fa02a15faa6a52ed4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] Clarify slice vs partition in the...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2299#issuecomment-54712599 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19912/consoleFull) for PR 2299 at commit [`7b045e0`](https://github.com/apache/spark/commit/7b045e0845b70ccb9b6e197fe36546dccec167c6). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3415] [PySpark] removes SerializingAdap...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2287#issuecomment-54712596 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19911/consoleFull) for PR 2287 at commit [`afc4a9a`](https://github.com/apache/spark/commit/afc4a9a358923bab96dcc145f30b4d6b5b2991be). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [EC2] don't duplicate default values
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/2290#issuecomment-54712738 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] [PySpark] deprecated numSlices fo...
GitHub user mattf opened a pull request: https://github.com/apache/spark/pull/2302 [SPARK-1701] [PySpark] deprecated numSlices for numPartitions You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattf/spark SPARK-1701-pyspark Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2302.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2302 commit 207a62ef3f2917fad47cc05e86ce832193d2b5b0 Author: Matthew Farrellee m...@redhat.com Date: 2014-09-06T14:20:20Z [SPARK-1701] [PySpark] deprecated numSlices for numPartitions --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] remove unnecessary use of numSlic...
GitHub user mattf opened a pull request: https://github.com/apache/spark/pull/2303 [SPARK-1701] remove unnecessary use of numSlices from pyspark tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattf/spark SPARK-1701-pyspark-remove-numSlices Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2303.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2303 commit 0883e681429c07ea94858573b2ca3370657c82bb Author: Matthew Farrellee m...@redhat.com Date: 2014-09-06T14:25:50Z [SPARK-1701] remove unnecessary use of numSlices from pyspark tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] [PySpark] remove slice terminolog...
GitHub user mattf opened a pull request: https://github.com/apache/spark/pull/2304 [SPARK-1701] [PySpark] remove slice terminology from python examples You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattf/spark SPARK-1701-partition-over-slice-for-python-examples Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2304.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2304 commit 928a581d34eeaa816543341c672eadde2ff4b39c Author: Matthew Farrellee m...@redhat.com Date: 2014-09-06T14:33:04Z [SPARK-1701] [PySpark] remove slice terminology from python examples --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] Clarify slice vs partition in the...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2299#issuecomment-54714137 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19912/consoleFull) for PR 2299 at commit [`7b045e0`](https://github.com/apache/spark/commit/7b045e0845b70ccb9b6e197fe36546dccec167c6). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [EC2] don't duplicate default values
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2290#issuecomment-54714148 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19917/consoleFull) for PR 2290 at commit [`0c6d3bb`](https://github.com/apache/spark/commit/0c6d3bbe90b81dc433791a82d26ddc695cacf1d7). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] [PySpark] remove slice terminolog...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2304#issuecomment-54714149 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19915/consoleFull) for PR 2304 at commit [`928a581`](https://github.com/apache/spark/commit/928a581d34eeaa816543341c672eadde2ff4b39c). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Provide a default PYSPARK_PYTHON for python/ru...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2300#issuecomment-54714156 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19910/consoleFull) for PR 2300 at commit [`65a09f5`](https://github.com/apache/spark/commit/65a09f5a95a3636bc8a2081fa02a15faa6a52ed4). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] [PySpark] deprecated numSlices fo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2302#issuecomment-54714150 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19913/consoleFull) for PR 2302 at commit [`207a62e`](https://github.com/apache/spark/commit/207a62ef3f2917fad47cc05e86ce832193d2b5b0). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3425] do not set MaxPermSize for OpenJD...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2301#issuecomment-54714152 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19916/consoleFull) for PR 2301 at commit [`2cfafd8`](https://github.com/apache/spark/commit/2cfafd899e0b77f2039d51cefeabb1f51e5ff169). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] remove unnecessary use of numSlic...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2303#issuecomment-54714147 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19914/consoleFull) for PR 2303 at commit [`0883e68`](https://github.com/apache/spark/commit/0883e681429c07ea94858573b2ca3370657c82bb). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3415] [PySpark] removes SerializingAdap...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2287#issuecomment-54714249 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19911/consoleFull) for PR 2287 at commit [`afc4a9a`](https://github.com/apache/spark/commit/afc4a9a358923bab96dcc145f30b4d6b5b2991be). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] Clarify slice vs partition in the...
Github user mattf commented on the pull request: https://github.com/apache/spark/pull/2299#issuecomment-54714295 This patch fails unit tests. the failures are in the kafka tests and unrelated to this patch --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] remove unnecessary use of numSlic...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2303#issuecomment-54715939 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19914/consoleFull) for PR 2303 at commit [`0883e68`](https://github.com/apache/spark/commit/0883e681429c07ea94858573b2ca3370657c82bb). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] [PySpark] remove slice terminolog...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2304#issuecomment-54716023 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19915/consoleFull) for PR 2304 at commit [`928a581`](https://github.com/apache/spark/commit/928a581d34eeaa816543341c672eadde2ff4b39c). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3425] do not set MaxPermSize for OpenJD...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2301#issuecomment-54716032 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19916/consoleFull) for PR 2301 at commit [`2cfafd8`](https://github.com/apache/spark/commit/2cfafd899e0b77f2039d51cefeabb1f51e5ff169). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] [PySpark] deprecated numSlices fo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2302#issuecomment-54716164 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19913/consoleFull) for PR 2302 at commit [`207a62e`](https://github.com/apache/spark/commit/207a62ef3f2917fad47cc05e86ce832193d2b5b0). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [EC2] don't duplicate default values
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2290#issuecomment-54716161 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19917/consoleFull) for PR 2290 at commit [`0c6d3bb`](https://github.com/apache/spark/commit/0c6d3bbe90b81dc433791a82d26ddc695cacf1d7). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] remove unnecessary use of numSlic...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2303#issuecomment-54716695 Hi @mattf, It looks like you've opened four separate PRS (#2299, #2303, #2302, #2304) for SPARK-1701. Since these all deal with the same JIRA, can you please close all of these and re-open a single PR that contains all of these commits? Four separate PRs for such a small feature makes this much harder for us to review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [EC2] don't duplicate default values
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/2290#issuecomment-54716854 @JoshRosen This PR is good for review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] remove unnecessary use of numSlic...
Github user mattf commented on the pull request: https://github.com/apache/spark/pull/2303#issuecomment-54716879 @JoshRosen only #2299 is truly for SPARK-1701, the others are tangentially related so i tagged them along with 1701, but they can all stand alone. my hope is that the reviews can move forward independently as distinct units. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3178 setting SPARK_WORKER_MEMORY to a va...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2227#issuecomment-54717377 If you want to change the branch used for the pr you will need to close and open a new one Sent from my phone On Sep 5, 2014, at 7:24 PM, Bill Bejeck notificati...@github.com wrote: I've made the changes for mocking out the environment variable in the test. I'd like to move the code from forked master branch to a topic branch. If I rebase and move the code to a topic branch will the PR be automatically updated or will I have to close the PR, do the rebase and re-submit? â Reply to this email directly or view it on GitHub. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3030] [PySpark] Reuse Python worker
Github user davies commented on the pull request: https://github.com/apache/spark/pull/2259#issuecomment-54718008 Jenkins, retest this please. On Sat, Sep 6, 2014 at 12:40 AM, Apache Spark QA notificati...@github.com wrote: QA tests have finished https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19904/consoleFull for PR 2259 at commit 583716e https://github.com/apache/spark/commit/583716ee156d0d7eb87220295a624bfc1427032a . - This patch *fails* unit tests. - This patch merges cleanly. - This patch adds no public classes. -- Reply to this email directly or view it on GitHub https://github.com/apache/spark/pull/2259#issuecomment-54705444. -- - Davies --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] remove unnecessary use of numSlic...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2303#issuecomment-54718542 I'd still like for these to be combined into a single PR. Each additional PR means another run of Jenkins tests, another commit message, another round of emails / entry in our [PR tracker](http://spark-prs.appspot.com/); this adds up to be a lot more work for us (there's a large backlog of pull requests that we're making good progress in reviewing and merging). Sometimes there are very good reasons to split up a PR. For example, the recent TorrentBroadcast improvements are being performed as a series of multiple PRs, but I that's okay because those PRs are much larger units of functionality (e.g. hundreds of lines, instead of 10) which we might want to merge and backport separately. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] remove unnecessary use of numSlic...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2303#issuecomment-54719377 Was your intent for us to merge / backport only the documentation fixes? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-787] Add S3 configuration parameters to...
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/1120#issuecomment-54719617 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1983#issuecomment-54720047 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19918/consoleFull) for PR 1983 at commit [`cec7658`](https://github.com/apache/spark/commit/cec7658964c4295262d04055e7948f4cf5f4c53e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-787] Add S3 configuration parameters to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1120#issuecomment-54720048 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19919/consoleFull) for PR 1120 at commit [`41fd938`](https://github.com/apache/spark/commit/41fd9388ed10fe65a50edf44ed5a4cf1364707f1). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-787] Add S3 configuration parameters to...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1120#issuecomment-54721820 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19919/consoleFull) for PR 1120 at commit [`41fd938`](https://github.com/apache/spark/commit/41fd9388ed10fe65a50edf44ed5a4cf1364707f1). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1983#issuecomment-54722195 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19918/consoleFull) for PR 1983 at commit [`cec7658`](https://github.com/apache/spark/commit/cec7658964c4295262d04055e7948f4cf5f4c53e). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class Document(docId: Int, content: Array[Int]) ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3415] [PySpark] removes SerializingAdap...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/2287#issuecomment-54722820 LGTM, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2788] [STREAMING] Add location filterin...
Github user sjbrunst commented on a diff in the pull request:
https://github.com/apache/spark/pull/1717#discussion_r17210353
--- Diff:
external/twitter/src/main/scala/org/apache/spark/streaming/twitter/TwitterInputDStream.scala
---
@@ -42,9 +44,19 @@ class TwitterInputDStream(
@transient ssc_ : StreamingContext,
twitterAuth: Option[Authorization],
filters: Seq[String],
+locations: Seq[Seq[Double]],
storageLevel: StorageLevel
) extends ReceiverInputDStream[Status](ssc_) {
+ def this(
--- End diff --
Sounds good. I'll take that out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Provide a default PYSPARK_PYTHON for python/ru...
Github user davies commented on the pull request: https://github.com/apache/spark/pull/2300#issuecomment-54723033 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2788] [STREAMING] Add location filterin...
Github user sjbrunst commented on a diff in the pull request:
https://github.com/apache/spark/pull/1717#discussion_r17210396
--- Diff:
external/twitter/src/main/scala/org/apache/spark/streaming/twitter/TwitterUtils.scala
---
@@ -33,15 +33,20 @@ object TwitterUtils {
*twitter4j.oauth.consumerSecret, twitter4j.oauth.accessToken and
*twitter4j.oauth.accessTokenSecret
* @param filters Set of filter strings to get only those tweets that
match them
+ * @param locations Set of longitude and latitude coordinates to get
only those tweets within the
+bounding box defined by those points. Example:
Seq(Seq(-180.0,-90.0),Seq(180.0,90.0))
+gives any geotagged tweet. If locations and filters are both
nonempty, then any tweet
+matching either condition may be returned.
* @param storageLevel Storage level to use for storing the received
objects
*/
def createStream(
ssc: StreamingContext,
twitterAuth: Option[Authorization],
filters: Seq[String] = Nil,
+ locations: Seq[Seq[Double]] = Nil,
--- End diff --
Ah, I didn't realize overloaded functions with default params would be an
issue (I'm fairly new to Scala). It seems to be working now, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3367] Remove spark.shuffle.spill.compre...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2247#issuecomment-54723134 Ok. I have filed a JIRA at https://issues.apache.org/jira/browse/SPARK-3426 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1825] Fixes cross-platform submit probl...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/899#issuecomment-54723174 No worries. We can build on top of your patch to make this work for hadoop versions 2.4. Thanks for digging through this code. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] remove unnecessary use of numSlic...
Github user mattf closed the pull request at: https://github.com/apache/spark/pull/2303 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] [PySpark] deprecated numSlices fo...
Github user mattf closed the pull request at: https://github.com/apache/spark/pull/2302 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] [PySpark] remove slice terminolog...
Github user mattf closed the pull request at: https://github.com/apache/spark/pull/2304 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] Clarify slice vs partition in the...
Github user mattf closed the pull request at: https://github.com/apache/spark/pull/2299 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] Clarify slice vs partition in the...
GitHub user mattf opened a pull request: https://github.com/apache/spark/pull/2305 [SPARK-1701] Clarify slice vs partition in the programming guide This is a partial solution to SPARK-1701, only addressing the documentation confusion. Additional work can be to actually change the numSlices parameter name across languages, with care required for scala python to maintain backward compatibility for named parameters. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mattf/spark SPARK-1701 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2305.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2305 commit 7b045e0845b70ccb9b6e197fe36546dccec167c6 Author: Matthew Farrellee m...@redhat.com Date: 2014-09-06T12:55:12Z [SPARK-1701] Clarify slice vs partition in the programming guide This is a partial solution to SPARK-1701, only addressing the documentation confusion. Additional work can be to actually change the numSlices parameter name across languages, with care required for scala python to maintain backward compatibility for named parameters. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] Clarify slice vs partition in the...
Github user mattf commented on the pull request: https://github.com/apache/spark/pull/2299#issuecomment-54723659 oops, i didn't realize renaming my branch would close this PR. i'll open another. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] Clarify slice vs partition in the...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2305#issuecomment-54724156 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19920/consoleFull) for PR 2305 at commit [`7b045e0`](https://github.com/apache/spark/commit/7b045e0845b70ccb9b6e197fe36546dccec167c6). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-787] Add S3 configuration parameters to...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1120#issuecomment-54724412 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Spark 2017
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1682#issuecomment-54726062 @ash211 This weekend, I'm actually working on writing a design document for web UI improvements in Spark 1.2. SSL encryption, authentication, and ACLs are all features that I'm planning to put on the roadmap. Do you have SSH to your EC2 machines? One option is to use a SSH proxy to view the full web UI in your browser. Once you've set up the proxy, you can use a browser plugin like [FoxyProxy](http://getfoxyproxy.org/) to seamlessly proxy requests for the UI. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1701] Clarify slice vs partition in the...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2305#issuecomment-54726088 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19920/consoleFull) for PR 2305 at commit [`7b045e0`](https://github.com/apache/spark/commit/7b045e0845b70ccb9b6e197fe36546dccec167c6). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3415] [PySpark] removes SerializingAdap...
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2287#discussion_r17210951 --- Diff: python/pyspark/tests.py --- @@ -181,6 +181,18 @@ def tearDown(self): sys.path = self._old_sys_path +# Regression test for SPARK-3415 +class CloudPickleTestCase(PySparkTestCase): --- End diff -- One final naming nit: so far, the classes named `*TestCase` are used for factoring out common setup / teardown code (such as `unittest.TestCase`, `PySparkTestCase`, etc.), while the classes with the actual tests have been called `[ComponentName]Tests` or `Test[ComponentName]`. Therefore, I'd prefer to call this `CloudPickleTests`. Also, since this is just testing CloudPickle without using any PySpark features, it should extend `unittest.TestCase` instead of `PySparkTestCase` so that we don't run a setup / teardown method for a SparkContext that we never use. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3415] [PySpark] removes SerializingAdap...
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2287#discussion_r17210968 --- Diff: python/pyspark/tests.py --- @@ -181,6 +181,18 @@ def tearDown(self): sys.path = self._old_sys_path +# Regression test for SPARK-3415 +class CloudPickleTestCase(PySparkTestCase): +def test_pickling_file_handles(self): +from pyspark.cloudpickle import CloudPickler +from StringIO import StringIO --- End diff -- Do we need the StringIO? Can we just use `pyspark.cloudpickle.dumps`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3415] [PySpark] removes SerializingAdap...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2287#issuecomment-54726495 This looks fine to me, too, although I have two minor naming nits. Sorry to be so nitpicky on the names and test code, but I'd like this code to be really clean so that it serves as an example for future CloudPickle tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3417] -Use new-style classes in PySpark
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2288#issuecomment-54726511 Jenkins, this is ok to test. Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2706][SQL] Enable Spark to support Hive...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2241#issuecomment-54726521 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Spark-3406 add a default storage level to pyth...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2280#issuecomment-54726570 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1981][Streaming] Updated kinesis docs a...
GitHub user cfregly opened a pull request: https://github.com/apache/spark/pull/2306 [SPARK-1981][Streaming] Updated kinesis docs and added architecture diagram You can merge this pull request into a Git repository by running: $ git pull https://github.com/cfregly/spark kinesis-doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2306.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2306 commit b3b0ff118cac3c0a5a10f9912b383bb0665c9a1b Author: Chris Fregly ch...@fregly.com Date: 2014-07-16T07:03:04Z [SPARK-1981] Add AWS Kinesis streaming support commit d18e680670dfed3039477941e953b5811adde4ab Author: Chris Fregly ch...@fregly.com Date: 2014-07-16T22:23:21Z Merge remote-tracking branch 'upstream/master' Resolved conflict: project/SparkBuild.scala commit cd68c0d7bb0c1ef38e7c92d0cd6eb4a7ccf2ce27 Author: Chris Fregly ch...@fregly.com Date: 2014-07-19T03:16:13Z fixed typos and backward compatibility commit e7c8978482e1092cf599a31b402ab1f4ec11e36c Author: Chris Fregly ch...@fregly.com Date: 2014-07-22T03:09:42Z Merge remote-tracking branch 'upstream/master' commit 828f8aeb1081cf7ad9e5386e1cce933ece9c3d62 Author: Chris Fregly ch...@fregly.com Date: 2014-07-22T05:20:42Z more cleanup commit 338997e6e750c206bfb50a654b725be5f33beb07 Author: Chris Fregly ch...@fregly.com Date: 2014-07-22T15:54:35Z improve build docs for kinesis commit 6c395619dde93a9b8e9137b1150de4ae5129cf4b Author: Chris Fregly ch...@fregly.com Date: 2014-07-24T03:55:55Z parameterized the versions of the aws java sdk and kinesis client commit 21de67fbc5298ce73dd36fe8372dbeee27f69854 Author: Chris Fregly ch...@fregly.com Date: 2014-07-31T00:43:29Z Merge remote-tracking branch 'upstream/master' Incorporated feedback from TD. Renamed project to kinesis-asl, simplified examples, improved docs, fixed bug with AWS regions outside of us-east-1 commit db3eefd0b845ade8251235698c74f8bdc5d35e5a Author: Chris Fregly ch...@fregly.com Date: 2014-07-31T00:43:40Z Merge remote-tracking branch 'upstream/master' commit 912640cb344c77102e4ca4d884b8b0d0206ed627 Author: Chris Fregly ch...@fregly.com Date: 2014-07-31T01:03:27Z changed the foundKinesis class to be a publically-avail class commit d17ca6d6a36ddf0a3030eacae0eace3fdd758cc5 Author: Chris Fregly ch...@fregly.com Date: 2014-07-31T17:00:09Z per TD's feedback: updated docs, simplified the KinesisUtils api commit bf614e9ed870a3c23670d3783d574b1e4280bd81 Author: Chris Fregly ch...@fregly.com Date: 2014-07-31T17:33:20Z per matei's feedback: moved the kinesis examples into the examples/ dir commit e33cbeb0238be90a878b71bc7354f957dfe45891 Author: Chris Fregly ch...@fregly.com Date: 2014-08-01T06:07:46Z Merge remote-tracking branch 'upstream/master' commit 74e5c7c3ce99f5cd30d269d62aca31d2b275288c Author: Chris Fregly ch...@fregly.com Date: 2014-08-01T21:14:42Z updated per TD's feedback. simplified examples, updated docs commit 0e1c67b079b87b12f58557922ea7d4b257ee571b Author: Chris Fregly ch...@fregly.com Date: 2014-08-01T21:15:41Z Merge remote-tracking branch 'upstream/master' commit 691a6be900015358d55a03c046f93d6336297ea2 Author: Chris Fregly ch...@fregly.com Date: 2014-08-01T21:47:00Z fixed tests and formatting, fixed a bug with JavaKinesisWordCount during union of streams commit 0393795b53c2789973c081dba6f7651fd8678adc Author: Chris Fregly ch...@fregly.com Date: 2014-08-02T02:23:04Z moved Kinesis examples out of examples/ and back into extras/kinesis-asl updated the build to only include kinesis-asl inside the examples jar when -Pkinesis-asl is specified commit 47745816b21d7d2255a98283e3055a5a2a397a27 Author: Chris Fregly ch...@fregly.com Date: 2014-08-02T07:10:07Z updated docs, renamed retry to retryRandom to be more clear, removed retries around store() method commit 8e1ae2e2174e78bac1b73cb72ceec45adad1c35a Author: Chris Fregly ch...@fregly.com Date: 2014-08-03T02:02:40Z Merge remote-tracking branch 'upstream/master' commit 862df67eddc234e86a047a1ddae7a438d794c280 Author: Chris Fregly ch...@fregly.com Date: 2014-08-03T23:38:56Z Merge remote-tracking branch 'upstream/master' commit 0f37061e2e5ac4a73213d2dccc848aa8bc57b56f Author: Chris Fregly ch...@fregly.com Date: 2014-08-03T23:40:45Z SPARK-1981: (Kinesis streaming support) updated streaming-kinesis.md commit 9b1c71afb83856c4f0f68cbf92d2b6b444a0bb73 Author: Chris Fregly ch...@fregly.com Date: 2014-08-03T23:56:06Z better explained why spark checkpoints are disabled in the example (due to no stateful operations being used) commit baefa30c8acd8ad2d26e4a7f54b071cdaab09b58 Author: Chris Fregly ch...@fregly.com Date: 2014-09-06T20:14:18Z
[GitHub] spark pull request: [SPARK-1981][Streaming] Updated kinesis docs a...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2306#issuecomment-54727680 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3417] -Use new-style classes in PySpark
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2288#issuecomment-54727745 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19922/consoleFull) for PR 2288 at commit [`4aadab6`](https://github.com/apache/spark/commit/4aadab60028dec0a88f15aec4c57399919e362e6). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
