[GitHub] spark pull request: SPARK-1830 Deploy failover, Make Persistence e...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/771#issuecomment-54020889 Hey @aarondav, Do you think its worth having in its current condition ? I can rebase it ofcourse. I was actually unsure of changing it further. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3330 [BUILD] Successive test runs with d...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2221#issuecomment-54022209 Hey @srowen Thanks for fixing this. I feel your argument is plausible, so I am not verifying it. The change looks reasonable too. Looks good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3331 [BUILD] PEP8 tests fail because the...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/#issuecomment-54022844 In conclusion, this is a good change ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Renamed ColumnStat to ColumnMetrics to a...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2189#issuecomment-54023734 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2973][SQL] Lightweight SQL commands wit...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2215#discussion_r16942016
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala ---
@@ -90,10 +90,9 @@ case class SetCommand(
throw new IllegalArgumentException()
}
- def execute(): RDD[Row] = {
-val rows = sideEffectResult.map { line = new
GenericRow(Array[Any](line)) }
-context.sparkContext.parallelize(rows, 1)
- }
+ def execute(): RDD[Row] =
context.sparkContext.parallelize(executeCollect(), 1)
+
+ override def executeCollect(): Array[Row] =
sideEffectResult.map(Row(_)).toArray
--- End diff --
Good idea. Refactored a bit, now `Command.sideEffectResult` return
`Seq[Row]` and `Command.executeCollect()` simply returns
`sideEffectResult.toArray`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2973][SQL] Lightweight SQL commands wit...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2215#issuecomment-54024310 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19550/consoleFull) for PR 2215 at commit [`e0e12e9`](https://github.com/apache/spark/commit/e0e12e958988fec45285068858dea08accba8c32). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2558][DOCS] Add spark.yarn.queue descri...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2218#issuecomment-54025736 I just checked, `--queue` is a valid option in spark submit. And thanks for updating the docs. Looks good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-54025962 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19552/consoleFull) for PR 2198 at commit [`c081f8e`](https://github.com/apache/spark/commit/c081f8e8e81f72643870ff850e19ed7adb17e4e0). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3325] Add a parameter to the method pri...
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/2216#discussion_r16942761
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala ---
@@ -603,14 +603,14 @@ abstract class DStream[T: ClassTag] (
* Print the first ten elements of each RDD generated in this DStream.
This is an output
* operator, so this DStream will be registered as an output stream and
there materialized.
*/
- def print() {
+ def print(num: Int = 10) {
def foreachFunc = (rdd: RDD[T], time: Time) = {
- val first11 = rdd.take(11)
+ val firstNum = rdd.take(num + 1)
--- End diff --
Why +1 ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3325] Add a parameter to the method pri...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2216#issuecomment-54028823 This change is okay to have, but to print N elements from Dstream you can do something like dstream.foreachRDD(rdd = println(rdd.take(N).mkString)). I will let @tdas take a quick look. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3332] Revert spark-ec2 patch that ident...
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/2225#issuecomment-54030562 FYI: It looks like branch-1.1 has an out-of-date version of `run-tests-jenkins`. Might want to update that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3303][core] fix SparkContextSchedulerCr...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2199#issuecomment-54031780 I was in an impression that mesos native libs are set up on Jenkins. But anyway this seems to be a good change. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3304] [YARN] ApplicationMaster's Finish...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2198#issuecomment-54031723 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19552/consoleFull) for PR 2198 at commit [`c081f8e`](https://github.com/apache/spark/commit/c081f8e8e81f72643870ff850e19ed7adb17e4e0). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `logInfo(Interrupting user class to stop.)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL]Add Dynamic Partition suppo...
GitHub user baishuo opened a pull request: https://github.com/apache/spark/pull/2226 [SPARK-3007][SQL]Add Dynamic Partition support to Spark Sql hive a new PR base on new master. changes are the same as https://github.com/apache/spark/pull/1919 You can merge this pull request into a Git repository by running: $ git pull https://github.com/baishuo/spark patch-3007 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2226.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2226 commit d3e206e1a2fadc271e365462bd93730e31a094eb Author: baishuo(ç½ç¡) vc_j...@hotmail.com Date: 2014-08-12T17:27:54Z Update HiveQl.scala commit b22857a365925a428c41dd3e93d0da3613053071 Author: baishuo(ç½ç¡) vc_j...@hotmail.com Date: 2014-08-12T17:29:36Z Update SparkHadoopWriter.scala commit bade51d4726b8c55de83fef5c3e42c48f5af8f59 Author: baishuo(ç½ç¡) vc_j...@hotmail.com Date: 2014-08-12T17:31:01Z Update InsertIntoHiveTable.scala commit d211d330550260d93752349682e7c8447691a9e5 Author: baishuo(ç½ç¡) vc_j...@hotmail.com Date: 2014-08-12T17:53:04Z Update InsertIntoHiveTable.scala commit f0f620d277ecc7e342c42d88e5b12062eecd8261 Author: baishuo(ç½ç¡) vc_j...@hotmail.com Date: 2014-08-18T06:29:21Z Update HiveCompatibilitySuite.scala commit 412a48b185785dafb7a0ff450018e65dde7c4189 Author: baishuo(ç½ç¡) vc_j...@hotmail.com Date: 2014-08-18T06:34:53Z Update InsertIntoHiveTable.scala commit 567972c2c4ff85e9d09b2c75fbffe5891b438b1c Author: baishuo(ç½ç¡) vc_j...@hotmail.com Date: 2014-08-18T06:36:58Z Update HiveQuerySuite.scala commit 8e51a4bc47a1f5517e99dd1ebb456ae95376d8c2 Author: baishuo(ç½ç¡) vc_j...@hotmail.com Date: 2014-08-18T07:18:07Z Update Cast.scala commit b80f2021eca650b29a7baad35ba61ece90a7fc54 Author: baishuo(ç½ç¡) vc_j...@hotmail.com Date: 2014-08-18T07:44:07Z Update InsertIntoHiveTable.scala commit 924042c3118337bb6a944e0d4e3ece46ec65dd83 Author: baishuo(ç½ç¡) vc_j...@hotmail.com Date: 2014-08-18T07:57:20Z Update Cast.scala commit af8411aeefeae90fb5c79b88b38a5d299b11ddff Author: baishuo vc_j...@hotmail.com Date: 2014-08-19T16:01:49Z update file after test commit 0c324beaa38abfd089257466a0a0ddd6e57c5fad Author: baishuo vc_j...@hotmail.com Date: 2014-08-19T17:14:53Z do a little modify commit 2a0e0b82cacf50552de60aead7b25e04323cd0f9 Author: baishuo vc_j...@hotmail.com Date: 2014-09-01T06:28:17Z for dynamic partition erge branch 'patch-1' into patch-3007 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL]Add Dynamic Partition suppo...
Github user baishuo commented on the pull request: https://github.com/apache/spark/pull/1919#issuecomment-54032088 Hi @ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3007][SQL]Add Dynamic Partition suppo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2226#issuecomment-54032115 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2973][SQL] Lightweight SQL commands wit...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2215#issuecomment-54033083 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19550/consoleFull) for PR 2215 at commit [`e0e12e9`](https://github.com/apache/spark/commit/e0e12e958988fec45285068858dea08accba8c32). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Renamed ColumnStat to ColumnMetrics to a...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2189#issuecomment-54033709 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19551/consoleFull) for PR 2189 at commit [`8bb3b21`](https://github.com/apache/spark/commit/8bb3b21a5b0544dbfc3873eb621ad064935cb198). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class ColumnMetrics(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3233] Executor never stop its SparnEnv,...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2138#discussion_r16945208
--- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala ---
@@ -285,7 +286,8 @@ object SparkEnv extends Logging {
sparkFilesDir,
metricsSystem,
shuffleMemoryManager,
- conf)
+ conf,
+ isDriver)
--- End diff --
You don't need to past this to SparkEnv itself. Just pass `isDriver` to the
`BlockManagerMaster`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1912] Lazily initialize buffers for loc...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/2179#issuecomment-54034960 It's much neater and simpler :+1: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3177 (on Master Branch)
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/2204#discussion_r16945967
--- Diff:
yarn/common/src/test/scala/org/apache/spark/deploy/yarn/ClientBaseSuite.scala
---
@@ -232,6 +233,15 @@ class ClientBaseSuite extends FunSuite with Matchers {
def getFieldValue[A, B](clazz: Class[_], field: String, defaults: =
B)(mapTo: A = B): B =
Try(clazz.getField(field)).map(_.get(null).asInstanceOf[A]).toOption.map(mapTo).getOrElse(defaults)
+ def getFieldValue2[A: ClassTag, A1: ClassTag, B](clazz: Class[_], field:
String,
--- End diff --
Why do you need `ClassTag` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3301]The spark version in the welcome m...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2196#issuecomment-54037509 This is okay, but I did not understand the need for a separate PR and JIRA for same issue. Issue simply applies to both pyspark and spark-shell. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3325] Add a parameter to the method pri...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2216#discussion_r16946191
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala ---
@@ -603,14 +603,14 @@ abstract class DStream[T: ClassTag] (
* Print the first ten elements of each RDD generated in this DStream.
This is an output
* operator, so this DStream will be registered as an output stream and
there materialized.
*/
- def print() {
+ def print(num: Int = 10) {
def foreachFunc = (rdd: RDD[T], time: Time) = {
- val first11 = rdd.take(11)
+ val firstNum = rdd.take(num + 1)
--- End diff --
It matches the original logic, which takes 11 in order to print 10. If
there are 11 elements, it prints ... at the end to indicate there is at least
one more that is not printed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2895: Add mapPartitionsWithContext relat...
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/2194#discussion_r16946230
--- Diff: core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala
---
@@ -186,6 +186,56 @@ trait JavaRDDLike[T, This : JavaRDDLike[T, This]]
extends Serializable {
}
/**
+ * :: DeveloperApi ::
+ * Return a new RDD by applying a function to each partition of this
RDD. This is a variant of
+ * mapPartitions that also passes the TaskContext into the closure.
+ *
+ * `preservesPartitioning` indicates whether the input function
preserves the partitioner, which
+ * should be `false` unless this is a pair RDD and the input function
doesn't modify the keys.
+ */
+ @DeveloperApi
+ def mapPartitionsWithContext[R](
+ f: JFunction2[TaskContext,
java.util.Iterator[T], java.util.Iterator[R]],
--- End diff --
Wrong indentation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3330 [BUILD] Successive test runs with d...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2221#issuecomment-54037958 Yes, PS, I did verify that this was the cause, by changing the code to print the stderr from the command that fails in SparkSubmitSuite. It was due to multiple assemblies, and I could reproduce making multiple assemblies of course locally by running what Jenkins does. I can't think of a reason that a top-level mvn clean *shouldn't* remove assemblies. So yeah 99% sure this is the fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3325] Add a parameter to the method pri...
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/2216#discussion_r16946373
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala ---
@@ -603,14 +603,14 @@ abstract class DStream[T: ClassTag] (
* Print the first ten elements of each RDD generated in this DStream.
This is an output
* operator, so this DStream will be registered as an output stream and
there materialized.
*/
- def print() {
+ def print(num: Int = 10) {
def foreachFunc = (rdd: RDD[T], time: Time) = {
- val first11 = rdd.take(11)
+ val firstNum = rdd.take(num + 1)
--- End diff --
Ah, I see.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3200][REPL] Classes defined with refere...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2158#issuecomment-54038679 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19553/consoleFull) for PR 2158 at commit [`8e039f0`](https://github.com/apache/spark/commit/8e039f0f4eeea3bc4475fb252147986ff64efad6). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3251][MLLIB]: Clarify learning interfac...
Github user BigCrunsh commented on the pull request: https://github.com/apache/spark/pull/2137#issuecomment-54040903 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3251][MLLIB]: Clarify learning interfac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2137#issuecomment-54041405 QA tests have started for PR 2137. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19554/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2895: Add mapPartitionsWithContext relat...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2194#issuecomment-54043344 It might be good to add a test suite for this in `JavaAPISuite.java`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3200][REPL] Classes defined with refere...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2158#issuecomment-54044103 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19553/consoleFull) for PR 2158 at commit [`8e039f0`](https://github.com/apache/spark/commit/8e039f0f4eeea3bc4475fb252147986ff64efad6). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Add SSDs to block device mapping
Github user darabos commented on a diff in the pull request:
https://github.com/apache/spark/pull/2081#discussion_r16948878
--- Diff: ec2/spark_ec2.py ---
@@ -342,6 +343,15 @@ def launch_cluster(conn, opts, cluster_name):
device.delete_on_termination = True
block_map[/dev/sdv] = device
+# AWS ignores the AMI-specified block device mapping for M3.
+if opts.instance_type.startswith('m3.'):
+for i in range(get_num_disks(opts.instance_type)):
+dev = BlockDeviceType()
+dev.ephemeral_name = 'ephemeral{}'.format(i)
--- End diff --
This format syntax won't work in Python 2.6 unfortunately; use
ephemeral%d % i
Done.
The script has a good number of `.format()` string interpolations though.
Does it really support Python 2.6?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Add SSDs to block device mapping
Github user darabos commented on the pull request: https://github.com/apache/spark/pull/2081#issuecomment-54044479 I've tested this now with `ec2/spark-ec2 -s 1 --instance-type m3.2xlarge --region=us-east-1 launch` and the machines have mounted the SSDs. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Add SSDs to block device mapping
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2081#issuecomment-54044709 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19555/consoleFull) for PR 2081 at commit [`1ceb2c8`](https://github.com/apache/spark/commit/1ceb2c8a8fc3527cb3d7389f009ad2a1709f4ac1). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2159: Add support for stopping SparkCont...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/1230#issuecomment-54045207 @adamosloizou Incase we agree on `:quit` is the correct way. Can you close this PR ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2159: Add support for stopping SparkCont...
Github user adamosloizou commented on the pull request: https://github.com/apache/spark/pull/1230#issuecomment-54045637 Fair enough. Closing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2159: Add support for stopping SparkCont...
Github user adamosloizou closed the pull request at: https://github.com/apache/spark/pull/1230 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3251][MLLIB]: Clarify learning interfac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2137#issuecomment-54048245 QA results for PR 2137:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19554/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Add SSDs to block device mapping
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2081#issuecomment-54049490 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19555/consoleFull) for PR 2081 at commit [`1ceb2c8`](https://github.com/apache/spark/commit/1ceb2c8a8fc3527cb3d7389f009ad2a1709f4ac1). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3000][CORE] drop old blocks to disk in ...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2134#issuecomment-54050427 There is something simillar in #791 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3233] Executor never stop its SparnEnv,...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2138#issuecomment-54050485 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19556/consoleFull) for PR 2138 at commit [`889e2d1`](https://github.com/apache/spark/commit/889e2d16bd7db75e286fde5661c145c8f4bea284). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3233] Executor never stop its SparnEnv,...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2138#issuecomment-54050554 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19556/consoleFull) for PR 2138 at commit [`889e2d1`](https://github.com/apache/spark/commit/889e2d16bd7db75e286fde5661c145c8f4bea284). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class BlockManagerMaster(var driverActor: ActorRef, conf: SparkConf, isDriver: Boolean) extends Logging ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3000][CORE] drop old blocks to disk in ...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2134#issuecomment-54050560 And some of the comment there applies to this patch as well.. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [CORE] SPARK-3178 setting SPARK_WORKER_MEMORY ...
GitHub user bbejeck opened a pull request: https://github.com/apache/spark/pull/2227 [CORE] SPARK-3178 setting SPARK_WORKER_MEMORY to a value without a label (m or g) sets the worker memory limit to zero Now the worker will fail fast if the memory is set to zero by leaving off the the label (m or g) either in the SPARK_WORKER_MEMORY environment variable or from the command line. You can merge this pull request into a Git repository by running: $ git pull https://github.com/bbejeck/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2227.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2227 commit 8abf665afef9b76e9886e72e596e687bd7516475 Author: Bill Bejeck bbej...@gmail.com Date: 2014-09-01T02:54:29Z SPARK-3178 - Validate the memory is greater than zero when set from the SPARK_WORKER_MEMORY environment variable or command line without a g or m label. Added unit tests. If memory is 0 an IllegalStateException is thrown. commit e9bcf2cf6fe5b3aa1649c80c82f8ff240662f413 Author: Bill Bejeck bbej...@gmail.com Date: 2014-09-01T02:57:56Z Merge remote-tracking branch 'upstream/master' Merging updates from Spark upstream on 8/31/14 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3233] Executor never stop its SparnEnv,...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2138#issuecomment-54051264 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19557/consoleFull) for PR 2138 at commit [`039b747`](https://github.com/apache/spark/commit/039b7479cf97753432f73bd4e0b3fd8d6315f9b3). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [CORE] SPARK-3178 setting SPARK_WORKER_MEMORY ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2227#issuecomment-54051575 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3328 fixed make-distribution script --wi...
GitHub user prudhvije opened a pull request: https://github.com/apache/spark/pull/2228 SPARK-3328 fixed make-distribution script --with-tachyon option. Directory path for dependencies jar and resources in Tachyon 0.5.0 has been changed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/prudhvije/spark SPARK-3328/make-dist-fix Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2228.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2228 commit d1d2c2229796d7cdcf621305c2db27538a106419 Author: Prudhvi Krishna prudhvi...@gmail.com Date: 2014-09-01T11:58:22Z SPARK-3328 fixed make-distribution script --with-tachyon option. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3328 fixed make-distribution script --wi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2228#issuecomment-54051959 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3328 fixed make-distribution script --wi...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2228#issuecomment-54052034 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3328 fixed make-distribution script --wi...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2228#issuecomment-54052144 Actually running jenkins test on this is wasteful. @pwendell This looks like a good fix to me, may be this should go in 1.1.0 too ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...
GitHub user ScrapCodes opened a pull request: https://github.com/apache/spark/pull/2229 SPARK-3337 Paranoid quoting in shell to allow install dirs with spaces within. ... Still testing it out, with spark install dir name having spaces in it. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ScrapCodes/spark-1 SPARK-3337/quoting-shell-scripts Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2229.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2229 commit 4bcd7e8c219dec2475a5ad1844853e2f7b0e4ee6 Author: Prashant Sharma prashan...@imaginea.com Date: 2014-09-01T12:23:39Z SPARK-3337 Paranoid quoting in shell to allow install dirs with spaces within. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2229#issuecomment-54053703 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19558/consoleFull) for PR 2229 at commit [`4bcd7e8`](https://github.com/apache/spark/commit/4bcd7e8c219dec2475a5ad1844853e2f7b0e4ee6). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `$FWDIR/bin/spark-submit --class org.apache.spark.repl.Main $` * `$FWDIR/bin/spark-submit --class org.apache.spark.repl.Main $` * `exec $SPARK_HOME/bin/spark-class org.apache.spark.deploy.SparkSubmit $` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3337 Paranoid quoting in shell to allow ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2229#issuecomment-54053690 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19558/consoleFull) for PR 2229 at commit [`4bcd7e8`](https://github.com/apache/spark/commit/4bcd7e8c219dec2475a5ad1844853e2f7b0e4ee6). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2096 Correctly parse dot notations
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/2230
SPARK-2096 Correctly parse dot notations
First let me write down the current `projections` grammar of spark sql:
expression: orExpression
orExpression : andExpression {or andExpression}
andExpression : comparisonExpression {and
comparisonExpression}
comparisonExpression : termExpression | termExpression =
termExpression | termExpression termExpression | ...
termExpression: productExpression {+|-
productExpression}
productExpression : baseExpression {*|/|% baseExpression}
baseExpression: expression [ expression ] | (
expression ) | ident | ...
ident : identChar {identChar | digit} | delimiters
| ...
identChar : letter | _ | .
delimiters: , | ; | ( | ) | [ | ] | ...
projection: expression [[AS] ident]
projections : projection { , projection}
For something like `a.b.c[1]`, it will be parsed as:
img src=http://img51.imgspice.com/i/03008/4iltjsnqgmtt_t.jpg; border=0
But for something like `a[1].b`, the current grammar can't parse it
correctly.
A simple solution is written in `ParquetQuerySuite#NestedSqlParser`,
changed grammars are:
identChar : letter | _
baseExpression: expression [ expression ] | expression
. ident | ( expression ) | ident | ...
This works well, but can't cover some corner case like `select t.a.b from
table as t`:
img src=http://img51.imgspice.com/i/03008/v2iau3hoxoxg_t.jpg; border=0
`t.a.b` parsed as `GetField(GetField(UnResolved(t), a), b)` instead
of `GetField(UnResolved(t.a), b)` using this new grammar.
However, we can't resolve `t` as it's not a filed, but the whole table.(if
we could do this, then `select t from table as t` is legal, which is unexpected)
My solution is:
dotExpressionHeader : ident . ident
baseExpression: expression [ expression ] |
dotExpressionHeader | expression . ident | ( expression ) | ident | ...
I passed all test cases under sql locally and add a more complex case.
I'm not familiar with the latter optimize phase, please correct me if I
missed something.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark dot
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2230.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2230
commit de630829028d9d4a7ef55b8a0ff31e09f0b549d9
Author: Wenchen Fan cloud0...@163.com
Date: 2014-09-01T11:10:02Z
SPARK-2096 Correctly parse dot notations
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3251][MLLIB]: Clarify learning interfac...
Github user BigCrunsh commented on the pull request: https://github.com/apache/spark/pull/2137#issuecomment-54055010 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3251][MLLIB]: Clarify learning interfac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2137#issuecomment-54055383 QA tests have started for PR 2137. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19559/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3251][MLLIB]: Clarify learning interfac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2137#issuecomment-54055470 QA results for PR 2137:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19559/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Update building-with-maven.md
Github user loachli commented on the pull request: https://github.com/apache/spark/pull/2102#issuecomment-54056189 I have created SPARK-3191(https://issues.apache.org/jira/browse/SPARK-3191) for it. Do you think it is enough for this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3233] Executor never stop its SparnEnv,...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2138#issuecomment-54056581 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19557/consoleFull) for PR 2138 at commit [`039b747`](https://github.com/apache/spark/commit/039b7479cf97753432f73bd4e0b3fd8d6315f9b3). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class BlockManagerMaster(var driverActor: ActorRef,` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3251][MLLIB]: Clarify learning interfac...
Github user BigCrunsh commented on the pull request: https://github.com/apache/spark/pull/2137#issuecomment-54057841 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3251][MLLIB]: Clarify learning interfac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2137#issuecomment-54058061 QA tests have started for PR 2137. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19560/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3251][MLLIB]: Clarify learning interfac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2137#issuecomment-54058346 QA results for PR 2137:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19560/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3171] Don't print meaningless informati...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2078#issuecomment-54059387 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19561/consoleFull) for PR 2078 at commit [`000b941`](https://github.com/apache/spark/commit/000b941db18f462c06d1bf3106898e8cc5f4383b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2435] Add shutdown hook to pyspark
Github user mattf commented on the pull request: https://github.com/apache/spark/pull/2183#issuecomment-54061965 What's the problem without this patch? I remember that the JVM will shutdown itself after shell exited. davies, i went back and tried to reproduce the shell issues (while scripting the shell) and couldn't reproduce w/ or w/o this patch. so i don't have a bug to justify the patch. apologies. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2435] Add shutdown hook to pyspark
Github user mattf commented on the pull request: https://github.com/apache/spark/pull/2183#issuecomment-54062440 Is it better to put atexit.register() in context.py? So all the pyspark jobs can have this. i think it's a question of who owns the context. the owner is whomever constructed it. they should be responsible for stopping it, and given the open not to stop it if they so desire. in the pyspark shell, it's the shell.py that creates it and should therefore stop it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1983#issuecomment-54064930 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19562/consoleFull) for PR 1983 at commit [`b21b9c1`](https://github.com/apache/spark/commit/b21b9c1ef5de53f933445c4db64e82362d440db7). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1983#issuecomment-54065121 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19562/consoleFull) for PR 1983 at commit [`b21b9c1`](https://github.com/apache/spark/commit/b21b9c1ef5de53f933445c4db64e82362d440db7). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class Document(docId: Int, content: Iterable[Int], var topics: Iterable[Int] = null,` * `class TopicModel(val topicCounts_ : BDV[Double],` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1983#issuecomment-54065708 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1983#issuecomment-54066312 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19563/consoleFull) for PR 1983 at commit [`b21b9c1`](https://github.com/apache/spark/commit/b21b9c1ef5de53f933445c4db64e82362d440db7). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3171] Don't print meaningless informati...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2078#issuecomment-54066507 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19561/consoleFull) for PR 2078 at commit [`000b941`](https://github.com/apache/spark/commit/000b941db18f462c06d1bf3106898e8cc5f4383b). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2634: Change MapOutputTrackerWorker.mapS...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/1541#issuecomment-54067191 ping @JoshRosen, could you help take a look at this one? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3301]The spark version in the welcome m...
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/2196#issuecomment-54067786 Yeah, that's what I suggested on the linked PR. It makes the set of related changes atomic, too. But I guess it's not a big deal. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3251][MLLIB]: Clarify learning interfac...
Github user BigCrunsh commented on the pull request: https://github.com/apache/spark/pull/2137#issuecomment-54067988 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3098]In some cases, the result of ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2083#issuecomment-54068545 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19565/consoleFull) for PR 2083 at commit [`df59bea`](https://github.com/apache/spark/commit/df59bea54691ec68b4cde1603f6f1a0db15efb06). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3251][MLLIB]: Clarify learning interfac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2137#issuecomment-54068540 QA tests have started for PR 2137. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19564/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3303][core] fix SparkContextSchedulerCr...
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/2199#issuecomment-54071586 yep, this error produced when mesos native lib is on --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1405][MLLIB]Collapsed Gibbs sampli...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1983#issuecomment-54072638 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19563/consoleFull) for PR 1983 at commit [`b21b9c1`](https://github.com/apache/spark/commit/b21b9c1ef5de53f933445c4db64e82362d440db7). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class Document(docId: Int, content: Iterable[Int], var topics: Iterable[Int] = null,` * `class TopicModel(val topicCounts_ : BDV[Double],` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-3098]In some cases, the result of ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2083#issuecomment-54074910 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19565/consoleFull) for PR 2083 at commit [`df59bea`](https://github.com/apache/spark/commit/df59bea54691ec68b4cde1603f6f1a0db15efb06). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1477]: Add the lifecycle interface
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/991#issuecomment-54077338 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19566/consoleFull) for PR 991 at commit [`79f48dc`](https://github.com/apache/spark/commit/79f48dcff976de99f06057da48951fa16e6d68d8). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3251][MLLIB]: Clarify learning interfac...
Github user BigCrunsh commented on the pull request: https://github.com/apache/spark/pull/2137#issuecomment-54079294 Jenkins, retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3251][MLLIB]: Clarify learning interfac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2137#issuecomment-54079494 QA tests have started for PR 2137. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19567/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1477]: Add the lifecycle interface
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/991#issuecomment-54081074 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19566/consoleFull) for PR 991 at commit [`79f48dc`](https://github.com/apache/spark/commit/79f48dcff976de99f06057da48951fa16e6d68d8). * This patch **fails** unit tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait Lifecycle extends Service ` * `trait Service extends java.io.Closeable ` * `class SparkContext(config: SparkConf) extends Logging with Lifecycle ` * `class JavaStreamingContext(val ssc: StreamingContext) extends Lifecycle ` * `class JobGenerator(jobScheduler: JobScheduler) extends Logging with Lifecycle ` * `class JobScheduler(val ssc: StreamingContext) extends Logging with Lifecycle ` * `class ReceiverTracker(ssc: StreamingContext) extends Logging with Lifecycle ` * ` class ReceiverLauncher extends Lifecycle ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3251][MLLIB]: Clarify learning interfac...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2137#issuecomment-54084205 QA results for PR 2137:br- This patch PASSES unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19567/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Use SquaredL2Updater in LogisticRegressionWith...
GitHub user BigCrunsh opened a pull request: https://github.com/apache/spark/pull/2231 Use SquaredL2Updater in LogisticRegressionWithSGD SimpleUpdater ignores the regularizer, which leads to an unregularized LogReg. To enable the common L2 regularizer (and the corresponding regularization parameter) for logistic regression the SquaredL2Updater has to be used in SGD (see, e.g., [SVMWithSGD]) You can merge this pull request into a Git repository by running: $ git pull https://github.com/soundcloud/spark fix-regparam-logreg Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2231.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2231 commit 78d9e90f09eea751ca5bccfd4afbccb7eab608f0 Author: Christoph Sawade christ...@sawade.me Date: 2014-09-01T21:17:44Z Use SquaredL2Updater in LogisticRegressionWithSGD SimpleUpdater ignores the regularizer. To enable the common L2 regularizer (and the corresponding regularization parameter) for logistic regression the SquaredL2Updater has to be used in SGD (see, e.g., [SVMWithSGD]) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3251][MLLIB]: Clarify learning interfac...
Github user BigCrunsh commented on the pull request: https://github.com/apache/spark/pull/2137#issuecomment-54092745 @mengxr, do you agree with this modification? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3177 (on Master Branch)
Github user chesterxgchen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2204#discussion_r16964789
--- Diff:
yarn/common/src/test/scala/org/apache/spark/deploy/yarn/ClientBaseSuite.scala
---
@@ -232,6 +233,15 @@ class ClientBaseSuite extends FunSuite with Matchers {
def getFieldValue[A, B](clazz: Class[_], field: String, defaults: =
B)(mapTo: A = B): B =
Try(clazz.getField(field)).map(_.get(null).asInstanceOf[A]).toOption.map(mapTo).getOrElse(defaults)
+ def getFieldValue2[A: ClassTag, A1: ClassTag, B](clazz: Class[_], field:
String,
--- End diff --
without ClassTag, you will get the following warnings:
abstract type pattern A is unchecked since it is eliminated by erasure
abstract type pattern A1 is unchecked since it is eliminated by erasure
so that
case v: A = mapTo(v)
case v1: A1 = mapTo1(v1)
will not match
with ClassTag, the runtime type are returned and above match will work as
expected
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3319 / 3338] Resolve Spark submit confi...
GitHub user andrewor14 opened a pull request: https://github.com/apache/spark/pull/2232 [SPARK-3319 / 3338] Resolve Spark submit config paths **SPARK-3319.** There is currently a divergence in behavior when the user passes in additional jars through `--jars` and through setting `spark.jars` in the default properties file. The former will happily resolve the paths (e.g. convert `my.jar` to `file:/absolute/path/to/my.jar`), while the latter does not. We should resolve paths consistently in both cases. This also applies to the following pairs of command line arguments and Spark configs: - `--jars` ~ `spark.jars` - `--files` ~ `spark.files` / `spark.yarn.dist.files` - `--archives` ~ `spark.yarn.dist.archives` - `--py-files` ~ `spark.submit.pyFiles` **SPARK-3338.** This PR also fixes the following bug: if the user sets `spark.submit.pyFiles` in his/her properties file, it does not actually get picked up, even if `--py-files` is not set. This is because it is overridden by an empty string. You can merge this pull request into a Git repository by running: $ git pull https://github.com/andrewor14/spark resolve-config-paths Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2232.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2232 commit fe039d35855b4af1635ff7046adccbe639884ed1 Author: Andrew Or andrewo...@gmail.com Date: 2014-08-30T01:58:07Z Beef up tests to test fixed-pointed-ness of Utils.resolveURI(s) commit 460117e9744b173f4209f28a6c7dde2fcc519efd Author: Andrew Or andrewo...@gmail.com Date: 2014-09-01T22:58:16Z Resolve config paths properly commit 05e03d649980d5025a1ba53c4c9c27b51e5224cc Author: Andrew Or andrewo...@gmail.com Date: 2014-09-01T22:58:51Z Add tests for resolving both command line and config paths --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3319 / 3338] Resolve Spark submit confi...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2232#issuecomment-54096247 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19569/consoleFull) for PR 2232 at commit [`05e03d6`](https://github.com/apache/spark/commit/05e03d649980d5025a1ba53c4c9c27b51e5224cc). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2895: Add mapPartitionsWithContext relat...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2194#issuecomment-54097265 @ChengXiangLi could you describe a bit more what the context is being used for? This is an unstable API so I'm a bit hesitant to expose this in its current form. It would be better to look at exactly what Hive needs from this interface and see if we can come up with a stable interface for it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1919] Fix Windows spark-shell --jars
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/2211#discussion_r16966248
--- Diff: repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala ---
@@ -965,11 +966,9 @@ class SparkILoop(in0: Option[BufferedReader],
protected val out: JPrintWriter,
def createSparkContext(): SparkContext = {
val execUri = System.getenv(SPARK_EXECUTOR_URI)
-val jars = SparkILoop.getAddedJars
val conf = new SparkConf()
.setMaster(getMaster())
.setAppName(Spark shell)
- .setJars(jars)
--- End diff --
Hm yeah. Actually for this PR I think I'll just modify only the jar paths
added to the classpath of the shell, but not worry about changing the value of
`spark.jars` when different things are set.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1919] Fix Windows spark-shell --jars
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2211#issuecomment-54097710 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19570/consoleFull) for PR 2211 at commit [`0d5a0c1`](https://github.com/apache/spark/commit/0d5a0c133ea3995c26c81d06dd55c36afe853865). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1919] Fix Windows spark-shell --jars
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2211#issuecomment-54097997 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19571/consoleFull) for PR 2211 at commit [`262c6a2`](https://github.com/apache/spark/commit/262c6a2cff8438b568e90fae5a1dc04baf656f0c). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3319 / 3338] Resolve Spark submit confi...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2232#issuecomment-54098222 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19569/consoleFull) for PR 2232 at commit [`05e03d6`](https://github.com/apache/spark/commit/05e03d649980d5025a1ba53c4c9c27b51e5224cc). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2634: Change MapOutputTrackerWorker.mapS...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1541#issuecomment-54098544 Thanks for the reminder. @kayousterhout I looked over @zsxwing's example and I agree that there's a thread-safety issue here.We can definitely have multiple concurrent block fetches that could race when accessing mapStatuses. There's a lot of other state in MapOutputTracker that's guarded with `synchronized`, which implies that this instances of MapOutputTracker will be accessed from multiple threads. In fact, there's even a `statuses.synchronized` at the end of `getServerStatuses` that's guarding a `MapOutputTracker.convertMapStatuses` call, but for some reason the other branch of the `if` guards it using `fetchedStatuses.synchronized` (which doesn't even make sense, since `fetchedStatuses` is a local variable defined inside of `getServerStatuses`). Since the synchronization logic here seems kind of messy / confusing and mapStatuses is only accessed from MapOutputTracker, maybe it would be better to just add proper synchronization around reads/writes to mapStatuses rather than converting it to a ConcurrentHashMap. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2634: Change MapOutputTrackerWorker.mapS...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1541#issuecomment-54098674 Actually, it looks like the `fetchedStatuses` vs `statuses` synchronization is correct, since it's guarding against modification to that statuses array while reading it in `convertMapStatuses`. This needs a closer look, but I'm not sure whether we need this synchronization, since the output status for a particular map task should be immutable once written. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1820][tools] Make GenerateMimaIgnore @D...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/845#issuecomment-54099382 @nikhils05 Can you close this PR ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1919] Fix Windows spark-shell --jars
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2211#issuecomment-54099820 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19570/consoleFull) for PR 2211 at commit [`0d5a0c1`](https://github.com/apache/spark/commit/0d5a0c133ea3995c26c81d06dd55c36afe853865). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1919] Fix Windows spark-shell --jars
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2211#issuecomment-54100241 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19571/consoleFull) for PR 2211 at commit [`262c6a2`](https://github.com/apache/spark/commit/262c6a2cff8438b568e90fae5a1dc04baf656f0c). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Use optional third argument as edge attribute.
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/901#issuecomment-54100620 @ankurdave @rxin could you guys come to a decision on this one way or the other? Also @npanj mind adding `[GRAPHX]` to the title here? Right now this is getting sorted with the Spark core patches, which might be why people haven't seen it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-1174] Adding port configuration for Htt...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/81#issuecomment-54100664 I believed this was fixed by the larger change in #1777 so we can close this issue for now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
