[GitHub] spark pull request: fixed spelling errors in documentation
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3691#issuecomment-66905844 LGTM; thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: fixed spelling errors in documentation
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3691 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: fixed spelling errors in documentation
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/3691#issuecomment-66905888 Merged into master and all of the maintenance branches. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Spark Core - [SPARK-3620] - Refactor of SparkS...
Github user tigerquoll commented on the pull request: https://github.com/apache/spark/pull/2516#issuecomment-66909268 No probs, it was actually a nice way of starting to poke through the code to figure out how things are put together. I'll stick to smaller jobs from now on in. Regards,Dale. Date: Tue, 9 Dec 2014 19:11:25 -0800 From: notificati...@github.com To: sp...@noreply.github.com CC: tigerqu...@outlook.com Subject: Re: [spark] Spark Core - [SPARK-3620] - Refactor of SparkSubmit Argument parsing code (#2516) Hey @tigerquoll usually for large patches like this we require a design doc on the JIRA. Especially because the priority of this is not super important, I would recommend that we close this issue for now, and maybe open a new one later once there is a consensus on how we should restructure Spark submit. Thanks for your work so far. â Reply to this email directly or view it on GitHub. = --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3640] [Streaming] [Kinesis] Allow users...
Github user aniketbhatnagar closed the pull request at: https://github.com/apache/spark/pull/3092 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3640] [Streaming] [Kinesis] Allow users...
Github user aniketbhatnagar commented on the pull request: https://github.com/apache/spark/pull/3092#issuecomment-66912820 Tested and Chris's suggestion to use EC2 profiles works. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4527][SQl]Add BroadcastNestedLoopJoin o...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/3395#issuecomment-66915681 LGTM, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-66916125 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-66916267 [Test build #24440 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24440/consoleFull) for PR 3672 at commit [`31a6520`](https://github.com/apache/spark/commit/31a6520aea59f12d1a6896db6079c5cea62659a3). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-66919317 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24440/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-66919314 [Test build #24440 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24440/consoleFull) for PR 3672 at commit [`31a6520`](https://github.com/apache/spark/commit/31a6520aea59f12d1a6896db6079c5cea62659a3). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Boolean)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Add a Note on jsonFile having separate JSON ob...
Github user petervandenabeele commented on the pull request: https://github.com/apache/spark/pull/3517#issuecomment-66922392 Bump ... I suggest we revert to something close to my original proposal: * no change in filenames (too complex for now) * add a small(er) note in the doc about the non-standard format In our DataScienceBe project, I just got this message from a new Spark user: to reitarate (and make sure I understand correctly), the `jsonFile`function does not read valid JSON files, but rather special files containing a valid JSON object on each line. Just making this clear to the users will already avoid some frustration. Could you please confirm that I can make this proposal (or a different path to resolve this). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Add a Note on jsonFile having separate JSON ob...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3517#issuecomment-66924151 Sure, I'm happy with clarifications to the documentation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4837] NettyBlockTransferService should ...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/3688#issuecomment-66925510 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4837] NettyBlockTransferService should ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3688#issuecomment-66925608 [Test build #24441 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24441/consoleFull) for PR 3688 at commit [`ebd2007`](https://github.com/apache/spark/commit/ebd2007d82ea909b1d060114148e4abc4f749b25). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4812][SQL] Fix the initialization issue...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/3660#issuecomment-66926121 This is a good catch, but perhaps the real bug here is that we are overriding the `sqlContext` in `InMemoryTableScan`. Instead, what do you think about just removing that? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4837] NettyBlockTransferService should ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3688#issuecomment-66928132 [Test build #24441 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24441/consoleFull) for PR 3688 at commit [`ebd2007`](https://github.com/apache/spark/commit/ebd2007d82ea909b1d060114148e4abc4f749b25). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4837] NettyBlockTransferService should ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3688#issuecomment-66928133 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24441/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-785 [CORE] ClosureCleaner not invoked on...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/3690#issuecomment-66928843 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-785 [CORE] ClosureCleaner not invoked on...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3690#issuecomment-66928921 [Test build #24442 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24442/consoleFull) for PR 3690 at commit [`8df68fe`](https://github.com/apache/spark/commit/8df68fed84cbbcc328d9ccce5df930f1c76c6b07). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4814 [CORE] Enable assertions in SBT, Ma...
GitHub user srowen opened a pull request: https://github.com/apache/spark/pull/3692 SPARK-4814 [CORE] Enable assertions in SBT, Maven tests / AssertionError from Hive's LazyBinaryInteger This enables assertions for the Maven and SBT build, but overrides the Hive module to not enable assertions. You can merge this pull request into a Git repository by running: $ git pull https://github.com/srowen/spark SPARK-4814 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3692.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3692 commit f71e783b38cf709cb1bf6dcfea45237d820c7bd1 Author: Sean Owen so...@cloudera.com Date: 2014-12-14T18:29:40Z Enable assertions for SBT and Maven build commit caca7047a6bcd672ae5e9657f4b2d5a61ba97cb7 Author: Sean Owen so...@cloudera.com Date: 2014-12-14T21:27:31Z Disable assertions just for Hive --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4814 [CORE] Enable assertions in SBT, Ma...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3692#issuecomment-66930746 [Test build #24443 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24443/consoleFull) for PR 3692 at commit [`caca704`](https://github.com/apache/spark/commit/caca7047a6bcd672ae5e9657f4b2d5a61ba97cb7). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-785 [CORE] ClosureCleaner not invoked on...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3690#issuecomment-66931556 [Test build #24442 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24442/consoleFull) for PR 3690 at commit [`8df68fe`](https://github.com/apache/spark/commit/8df68fed84cbbcc328d9ccce5df930f1c76c6b07). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-785 [CORE] ClosureCleaner not invoked on...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3690#issuecomment-66931560 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24442/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-66933007 [Test build #2 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/2/consoleFull) for PR 3672 at commit [`526315d`](https://github.com/apache/spark/commit/526315d0cd3dbf34f7e6b5a3ea6d5187bf96cccb). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Added setMinCount to Word2Vec.scala
GitHub user ganonp opened a pull request: https://github.com/apache/spark/pull/3693 Added setMinCount to Word2Vec.scala Wanted to customize the private minCount variable in the Word2Vec class. Added a method to do so. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ganonp/spark my-custom-spark Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3693.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3693 commit 5eb91000cd74ddd7704c79ca69259ee48c5840f9 Author: ganonp gan...@gmail.com Date: 2014-12-14T21:56:19Z Added setMinCount to Word2Vec.scala Wanted to customize the minCount variable in the Word2Vec class. Added a method to do so. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Added setMinCount to Word2Vec.scala
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3693#issuecomment-66934430 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4814 [CORE] Enable assertions in SBT, Ma...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3692#issuecomment-66934745 [Test build #24443 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24443/consoleFull) for PR 3692 at commit [`caca704`](https://github.com/apache/spark/commit/caca7047a6bcd672ae5e9657f4b2d5a61ba97cb7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4814 [CORE] Enable assertions in SBT, Ma...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3692#issuecomment-66934748 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24443/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-66936450 [Test build #2 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/2/consoleFull) for PR 3672 at commit [`526315d`](https://github.com/apache/spark/commit/526315d0cd3dbf34f7e6b5a3ea6d5187bf96cccb). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class Analyzer(catalog: Catalog, registry: FunctionRegistry, caseSensitive: Boolean)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-66936452 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/2/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1507][YARN]specify num of cores for AM
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/3686#issuecomment-66937888 @tgravescs In all other places we've managed to avoid referring to the yarn-client AM as the driver and I think blurring this distinction would be pretty confusing. It's not difficult to imagine someone wanting to set a driver-cores property for yarn-cluster AMs in their defaults conf file. Even if someone did want to set the yarn-client AM cores, I can't think of a situation where they'd want the former number to apply to it --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Core] Adding a parallelismRatio to control th...
GitHub user scwf opened a pull request: https://github.com/apache/spark/pull/3694 [Core] Adding a parallelismRatio to control the partitions num of shuffledRDD Adding parallelismRatio to control the partitions num of shuffledRDD, the rule is: Math.max(1, parallelismRatio * number of partitions of the largest upstream RDD) The ratio is 1.0 by default to make it compatible with the old version. When we have a good experience on it, we can change this. You can merge this pull request into a Git repository by running: $ git pull https://github.com/scwf/spark parallismRatio Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3694.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3694 commit e6c43ecdf4e49ea73befea8b87fb2a47eec2fb37 Author: wangfei wangf...@huawei.com Date: 2014-12-14T09:25:17Z adding spark.default.parallelismRatio commit 63826ae63bb1f912a6000f0cd958c44579960c1e Author: wangfei wangf...@huawei.com Date: 2014-12-14T09:31:58Z minor fix commit a71ce3b92a3f49f8035fa14b4249775087203af5 Author: wangfei wangf...@huawei.com Date: 2014-12-15T01:01:13Z minor fix commit f21bfd4904fa340099d190bd3963fefc79f0faa4 Author: wangfei wangf...@huawei.com Date: 2014-12-15T01:11:15Z minor fix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Core] Adding a parallelismRatio to control th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3694#issuecomment-66940757 [Test build #24445 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24445/consoleFull) for PR 3694 at commit [`f21bfd4`](https://github.com/apache/spark/commit/f21bfd4904fa340099d190bd3963fefc79f0faa4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4812][SQL] Fix the initialization issue...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/3660#issuecomment-66941524 I heard from @liancheng about some plan of using two SchemaRDDs from different sqlContext. Is the override sqlContext designed to support it? @liancheng since you added it in https://github.com/liancheng/spark/commit/378410559a1759b3218be15f1fda3a90240c143d --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Core] Adding a parallelismRatio to control th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3694#issuecomment-66943835 [Test build #24445 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24445/consoleFull) for PR 3694 at commit [`f21bfd4`](https://github.com/apache/spark/commit/f21bfd4904fa340099d190bd3963fefc79f0faa4). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Core] Adding a parallelismRatio to control th...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3694#issuecomment-66943841 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24445/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-66944216 The most recent test failures should be caused by other components. Let's wait for a moment. This PR now LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4826][Streaming] - Create unique file n...
GitHub user harishreedharan opened a pull request: https://github.com/apache/spark/pull/3695 [SPARK-4826][Streaming] - Create unique file names for each test in WALS... ...uite to avoid conflicts. You can merge this pull request into a Git repository by running: $ git pull https://github.com/harishreedharan/spark WALSuite-Fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3695.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3695 commit ae63f35d0289255a5d4b71ffb31a5d54c7069269 Author: Hari Shreedharan hshreedha...@apache.org Date: 2014-12-15T03:32:26Z [SPARK-4826][Streaming] - Create unique file names for each test in WALSuite to avoid conflicts. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4826][Streaming] - Create unique file n...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3695#issuecomment-66946543 [Test build #24446 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24446/consoleFull) for PR 3695 at commit [`ae63f35`](https://github.com/apache/spark/commit/ae63f35d0289255a5d4b71ffb31a5d54c7069269). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4826][Streaming] - Create unique file n...
Github user harishreedharan commented on the pull request: https://github.com/apache/spark/pull/3695#issuecomment-66946690 @tdas @JoshRosen Since the directories are created atomically each time the suite is run, conflicts reported in SPARK-4826 was likely caused by multiple tests running at the same time and using the same filename (since the writeLogSegments method is not thread-safe). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4826][Streaming] - Create unique file n...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3695#issuecomment-66946766 [Test build #24447 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24447/consoleFull) for PR 3695 at commit [`2948408`](https://github.com/apache/spark/commit/2948408b79a25afd8131eaa79fa6a7f66130f829). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4826][Streaming] - Create unique file n...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3695#issuecomment-66946801 [Test build #24447 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24447/consoleFull) for PR 3695 at commit [`2948408`](https://github.com/apache/spark/commit/2948408b79a25afd8131eaa79fa6a7f66130f829). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4826][Streaming] - Create unique file n...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3695#issuecomment-66946803 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24447/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4826][Streaming] - Create unique file n...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3695#issuecomment-66947024 [Test build #24448 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24448/consoleFull) for PR 3695 at commit [`2e2bf28`](https://github.com/apache/spark/commit/2e2bf28754cb93c4caeb882c6d7b115e4a86b39a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4843 [YARN] Squash ExecutorRunnableUtil ...
GitHub user ksakellis opened a pull request: https://github.com/apache/spark/pull/3696 SPARK-4843 [YARN] Squash ExecutorRunnableUtil and ExecutorRunnable ExecutorRunnableUtil is a parent of ExecutorRunnable because of the yarn-alpha and yarn-stable split. Now that yarn-alpha is gone, this commit squashes the unnecessary hierarchy. The methods from ExecutorRunnableUtil are added as private. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ksakellis/spark kostas-spark-4843 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3696.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3696 commit 155f308651bde79ec67091986269e1b13c796fae Author: Kostas Sakellis kos...@cloudera.com Date: 2014-12-15T04:44:19Z SPARK-4843 [YARN] Squash ExecutorRunnableUtil and ExecutorRunnable ExecutorRunnableUtil is a parent of ExecutorRunnable because of the yarn-alpha and yarn-stable split. Now that yarn-alpha is gone, this commit squashes the unnecessary hierarchy. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4843 [YARN] Squash ExecutorRunnableUtil ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3696#issuecomment-66949811 [Test build #24449 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24449/consoleFull) for PR 3696 at commit [`155f308`](https://github.com/apache/spark/commit/155f308651bde79ec67091986269e1b13c796fae). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] SPARK-4700: Add HTTP protocol spark thri...
Github user judynash commented on the pull request: https://github.com/apache/spark/pull/3672#issuecomment-66950036 Thank you Cheng for the update. From: Cheng Lian [mailto:notificati...@github.com] Sent: Sunday, December 14, 2014 6:57 PM To: apache/spark Cc: Judy Nash Subject: Re: [spark] [SQL] SPARK-4700: Add HTTP protocol spark thrift server (#3672) The most recent test failures should be caused by other components. Let's wait for a moment. This PR now LGTM. â Reply to this email directly or view it on GitHubhttps://github.com/apache/spark/pull/3672#issuecomment-66944216. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4826][Streaming] - Create unique file n...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3695#issuecomment-66950408 [Test build #24446 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24446/consoleFull) for PR 3695 at commit [`ae63f35`](https://github.com/apache/spark/commit/ae63f35d0289255a5d4b71ffb31a5d54c7069269). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4826][Streaming] - Create unique file n...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3695#issuecomment-66950410 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24446/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4826][Streaming] - Create unique file n...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3695#issuecomment-66950933 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24448/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4826][Streaming] - Create unique file n...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3695#issuecomment-66950931 [Test build #24448 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24448/consoleFull) for PR 3695 at commit [`2e2bf28`](https://github.com/apache/spark/commit/2e2bf28754cb93c4caeb882c6d7b115e4a86b39a). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB]SPARK-4846: When the vocabulary size is...
GitHub user jinntrance opened a pull request: https://github.com/apache/spark/pull/3697 [MLLIB]SPARK-4846: When the vocabulary size is large, Word2Vec may yield OOM make variables syn0Global and syn1Global lazy, in order that the high-order function passed to RDD.mapPartitionsWithIndex would not yield OutOfMemoryError(Requested array size exceeds VM limit) when serialized if syn0Globalsyn1Global are too large arrays. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinntrance/spark branch-1.1-word2vec-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3697.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3697 commit b33dfe7a5ccb6c4a2f46f0953d21e45fca2c7c1f Author: Joseph J.C. Tang jinntra...@gmail.com Date: 2014-12-15T02:59:57Z make variables syn0Global and syn1Global lazy, in order that the high-order function passed to RDD.mapPartitionsWithIndex would not yield OutOfMemoryError(Requested array size exceeds VM limit) when serialized if syn0Globalsyn1Global are too large arrays. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [MLLIB]SPARK-4846: When the vocabulary size is...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3697#issuecomment-66952147 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4845][Core] Adding a parallelismRatio t...
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/3694#issuecomment-66952763 Jekins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4843 [YARN] Squash ExecutorRunnableUtil ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3696#issuecomment-66952800 [Test build #24449 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24449/consoleFull) for PR 3696 at commit [`155f308`](https://github.com/apache/spark/commit/155f308651bde79ec67091986269e1b13c796fae). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4843 [YARN] Squash ExecutorRunnableUtil ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3696#issuecomment-66952805 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24449/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4845][Core] Adding a parallelismRatio t...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3694#issuecomment-66952871 [Test build #24450 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24450/consoleFull) for PR 3694 at commit [`f21bfd4`](https://github.com/apache/spark/commit/f21bfd4904fa340099d190bd3963fefc79f0faa4). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4812][SQL] Fix the initialization issue...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/3660#issuecomment-66954133 @zsxwing I guess what you mentioned is the plan to support multiple data sources via the newly introduced external data source API? The commit you mentioned is irrelevant. In that commit I just want to ensure the `SQLContext` we used in `InMemoryColumnarTableScan` is the same one used by its child . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4812][SQL] Fix the initialization issue...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/3660#issuecomment-66954374 I guess what you mentioned is the plan to support multiple data sources via the newly introduced external data source API? Yes. In that commit I just want to ensure the SQLContext we used in InMemoryColumnarTableScan is the same one used by its child . In which case, it won't be `protected[spark] val sqlContext = SparkPlan.currentContext.get()`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4812][SQL] Fix the initialization issue...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/3660#issuecomment-66955023 Ah I see you point, so here we referenced a field overriden by subclass in the constructor of the parent class. Then I think it's generally not safe to allow `sqlContext` to be overriden. How about marking `SparkPlan.sqlContext` as `final` to prevent this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4812][SQL] Fix the initialization issue...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/3660#issuecomment-66955060 Yes, we should remove it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4812][SQL] Fix the initialization issue...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3660#issuecomment-66955512 [Test build #24451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24451/consoleFull) for PR 3660 at commit [`1cbb623`](https://github.com/apache/spark/commit/1cbb623d5f20471ee116cd54137f00a30c9592ff). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4812][SQL] Fix the initialization issue...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/3660#issuecomment-66955549 Done. Also updated the description of this PR and JIRA. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4847][SQL]Fix extraStrategies cannot t...
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/3698 [SPARK-4847][SQL]Fix extraStrategies cannot take effect in SQLContext issue You can merge this pull request into a Git repository by running: $ git pull https://github.com/jerryshao/apache-spark SPARK-4847 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3698.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3698 commit 4741130819ca02ad6a426a3aeb0f6ef1f972f36e Author: jerryshao saisai.s...@intel.com Date: 2014-12-15T06:52:56Z Make later added extraStrategies effect when calling strategies --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4845][Core] Adding a parallelismRatio t...
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/3694#issuecomment-66956061 Hmm, seems there are some problems with ```org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDDSuite```, and i noticed that other PRs also failed there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4845][Core] Adding a parallelismRatio t...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3694#issuecomment-66956286 [Test build #24450 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24450/consoleFull) for PR 3694 at commit [`f21bfd4`](https://github.com/apache/spark/commit/f21bfd4904fa340099d190bd3963fefc79f0faa4). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4845][Core] Adding a parallelismRatio t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3694#issuecomment-66956294 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24450/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4847][SQL]Fix extraStrategies cannot t...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3698#issuecomment-66956337 [Test build #24452 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24452/consoleFull) for PR 3698 at commit [`4741130`](https://github.com/apache/spark/commit/4741130819ca02ad6a426a3aeb0f6ef1f972f36e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
GitHub user nkronenfeld opened a pull request: https://github.com/apache/spark/pull/3699 [SPARK-4848] Stand-alone cluster: Allow differences between workers with multiple instances I've changed the stand-alone cluster run scripts to allow different workers to have different numbers of instances, and base webui ports. I did this by moving the loop over instances from start-slaves to start-slave. In order to stop things properly, I had to make similar changes in stop-slaves (and introduce stop-slave). While I was at it, I changed SPARK_WORKER_PORT to work the same way as SPARK_WORKER_UI_PORT, since the new methods works fine for both. You can merge this pull request into a Git repository by running: $ git pull https://github.com/nkronenfeld/spark-1 startup-scripts Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3699.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3699 commit 479c31c9d3e580879d76146e2a687b5235c87b33 Author: Nathan Kronenfeld nkronenf...@oculusinfo.com Date: 2014-12-15T06:58:08Z Move looping through instances from the master to the workers, so that each worker respects its own number of instances and web-ui port. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Github user nkronenfeld commented on the pull request: https://github.com/apache/spark/pull/3699#issuecomment-66957137 I'm not sure how mesos and yarn clusters are started/stopped (nor do I have such clusters on which to test), so I'm not sure how this will affect them. I think the way I did this should be safe - it's mostly just moving code around - but I could use a knowledgeable set of eyes to be sure. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3699#issuecomment-66957276 [Test build #24453 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24453/consoleFull) for PR 3699 at commit [`479c31c`](https://github.com/apache/spark/commit/479c31c9d3e580879d76146e2a687b5235c87b33). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-4843 [YARN] Squash ExecutorRunnableUtil ...
Github user ksakellis commented on the pull request: https://github.com/apache/spark/pull/3696#issuecomment-66958311 Hmm.. tests failed but I'm not sure they are related to this change. Am I missing something? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4812][SQL] Fix the initialization issue...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3660#issuecomment-66959907 [Test build #24451 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24451/consoleFull) for PR 3660 at commit [`1cbb623`](https://github.com/apache/spark/commit/1cbb623d5f20471ee116cd54137f00a30c9592ff). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4812][SQL] Fix the initialization issue...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3660#issuecomment-66959913 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24451/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
