[GitHub] spark pull request: [SPARK-11184] [MLLIB] Declare most of .mllib c...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/9169#issuecomment-151396799 Thanks! On `@Experimental`, I assume it simply means that it is intended for end users (i.e. not a `@DeveloperApi`) but may be removed or changed at any time (without even deprecation). While it seems nice to try to keep these stable or use deprecation, as you say, it tends to make them act like stable methods. This can be a small problem if an API is added under the theory that it's just `@Experimental` and is thus low-risk, but is then treated like it can't be changed. I imagine that most users don't pay a lot of attention to the tag, and therefore might generally be surprised if such a method went away. I think all of this simply argues for more rapidly reflecting reality: lots of `@Experimental` methods are long since really stable and should be untagged. This is a good first crack at that. I'll put down a to-do to do the same for core and streaming, as I think most of those tags can probably go. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43089007
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/TrackStateSpec.scala ---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{HashPartitioner, Partitioner}
+import org.apache.spark.api.java.JavaPairRDD
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * Abstract class having all the specifications of
DStream.trackStateByKey().
+ * Use the `TrackStateSpec.create()` or `TrackStateSpec.create()` to
create instances of this class.
+ *
+ * {{{
+ *TrackStateSpec(trackingFunction)// in Scala
+ *TrackStateSpec.create(trackingFunction) // in Java
+ * }}}
+ */
+sealed abstract class TrackStateSpec[K: ClassTag, V: ClassTag, S:
ClassTag, T: ClassTag]
+ extends Serializable {
+
+ def initialState(rdd: RDD[(K, S)]): this.type
+ def initialState(javaPairRDD: JavaPairRDD[K, S]): this.type
+
+ def numPartitions(numPartitions: Int): this.type
+ def partitioner(partitioner: Partitioner): this.type
+
+ def timeout(interval: Duration): this.type
+}
+
+
+/** Builder object for creating instances of TrackStateSpec */
+object TrackStateSpec {
+
+ def apply[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ trackingFunction: (K, Option[V], State[S]) => Option[T]):
TrackStateSpec[K, V, S, T] = {
+new TrackStateSpecImpl[K, V, S, T](trackingFunction)
+ }
+
+ def create[K: ClassTag, V: ClassTag, S: ClassTag, T: ClassTag](
+ trackingFunction: (K, Option[V], State[S]) => Option[T]):
TrackStateSpec[K, V, S, T] = {
+apply(trackingFunction)
+ }
--- End diff --
I think here Java friendly constructor is necessary, `create` might not be
directly used in Java code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11297] Add new code tags
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9265 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SAPRK-8546] Add PMML export for Naive Bayes
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/9057#issuecomment-151396091 @yinxusen Could you update the PR title? `SAPRK` is a typo. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11297] Add new code tags
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/9265#issuecomment-151396021 LGTM. Merged into master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/8669#discussion_r43088776
--- Diff: bin/beeline ---
@@ -23,8 +23,10 @@
# Enter posix mode for bash
set -o posix
-# Figure out where Spark is installed
-FWDIR="$(cd "`dirname "$0"`"/..; pwd)"
+# Figure out if SPARK_HOME is set
+if [ -z "${SPARK_HOME}" ]; then
+export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
--- End diff --
Thanks @srowen for your comments, I will change to the 2-space indent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43088697
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala
---
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+/**
+ * Abstract class for getting and updating the tracked state in the
`trackStateByKey` operation of
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair
DStream]] and
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream]].
+ * {{{
+ *
+ * }}}
+ */
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise wise it will throw an exception.
+ * Check with `exists()` whether the state exists or not before calling
`get()`.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value. Note that you cannot update the
state if the state is
+ * timing out (that is, `isTimingOut() return true`, or if the state has
already been removed by
+ * `remove()`.
+ */
+ def update(newState: S): Unit
+
+ /** Remove the state if it exists. */
+ def remove(): Unit
+
+ /** Is the state going to be timed out by the system after this batch
interval */
+ def isTimingOut(): Boolean
+
+ @inline final def getOption(): Option[S] = Option(get())
+
+ /** Get the state if it exists, otherwise return the default value */
+ @inline final def getOrElse[S1 >: S](default: => S1): S1 = {
+if (exists) this.get else default
+ }
+
+ @inline final override def toString() = getOption.map { _.toString
}.getOrElse("")
+}
+
+/** Internal implementation of the [[State]] interface */
+private[streaming] class StateImpl[S] extends State[S] {
+
+ private var state: S = null.asInstanceOf[S]
+ private var defined: Boolean = true
+ private var timingOut: Boolean = false
+ private var updated: Boolean = false
+ private var removed: Boolean = false
+
+ // = Public API =
+ def exists(): Boolean = {
+defined
+ }
+
+ def get(): S = {
+state
+ }
+
+ def update(newState: S): Unit = {
+require(!removed, "Cannot update the state after it has been removed")
+require(!timingOut, "Cannot update the state that is timing out")
+state = newState
--- End diff --
Is this required for defensive guard `require(!updated, "cannot update the
state this is already updated")`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/8669#discussion_r43088465
--- Diff: bin/beeline ---
@@ -23,8 +23,10 @@
# Enter posix mode for bash
set -o posix
-# Figure out where Spark is installed
-FWDIR="$(cd "`dirname "$0"`"/..; pwd)"
+# Figure out if SPARK_HOME is set
+if [ -z "${SPARK_HOME}" ]; then
+export SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"
--- End diff --
Yes, I very much like how this has standardized everything to use
`SPARK_HOME` instead of `FWDIR`, and doesn't overwrite the value if already
set. (Nit: all of the occurrences of this line have a 4-space indent instead of
2)
LGTM; does anyone see a reason this isn't a good idea? I suppose now
`SPARK_HOME`, if set, has an effect everywhere, but it looks like the desired
effect. Docs also make reference to `SPARK_HOME` as if it has this effect.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2629][STREAMING] Basic implementation o...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9256#discussion_r43088193
--- Diff: streaming/src/main/scala/org/apache/spark/streaming/State.scala
---
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.streaming
+
+/**
+ * Abstract class for getting and updating the tracked state in the
`trackStateByKey` operation of
+ * [[org.apache.spark.streaming.dstream.PairDStreamFunctions pair
DStream]] and
+ * [[org.apache.spark.streaming.api.java.JavaPairDStream]].
+ * {{{
+ *
+ * }}}
+ */
+sealed abstract class State[S] {
+
+ /** Whether the state already exists */
+ def exists(): Boolean
+
+ /**
+ * Get the state if it exists, otherwise wise it will throw an exception.
+ * Check with `exists()` whether the state exists or not before calling
`get()`.
+ */
+ def get(): S
+
+ /**
+ * Update the state with a new value. Note that you cannot update the
state if the state is
+ * timing out (that is, `isTimingOut() return true`, or if the state has
already been removed by
+ * `remove()`.
+ */
+ def update(newState: S): Unit
+
+ /** Remove the state if it exists. */
+ def remove(): Unit
+
+ /** Is the state going to be timed out by the system after this batch
interval */
+ def isTimingOut(): Boolean
+
+ @inline final def getOption(): Option[S] = Option(get())
+
+ /** Get the state if it exists, otherwise return the default value */
+ @inline final def getOrElse[S1 >: S](default: => S1): S1 = {
--- End diff --
Not sure is this "call-by-name" parameter Java friendly? Assuming this
`State` should also be used in Java code :).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151393309 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44400/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151393307 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151393236 **[Test build #44400 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44400/consoleFull)** for PR 8669 at commit [`c796397`](https://github.com/apache/spark/commit/c796397da5159335de3c11e62ecedc427a9c5a64). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * ` \"$`\n * `\"$`\n * `exec \"$`\n * `exec \"$`\n * ` nohup nice -n \"$SPARK_NICENESS\" \"$`\n * ` nohup nice -n \"$SPARK_NICENESS\" \"$`\n * ` \"$`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10484][SQL] Optimize the cartesian join...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8652#issuecomment-151391381 **[Test build #44410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44410/consoleFull)** for PR 8652 at commit [`7fda511`](https://github.com/apache/spark/commit/7fda51170c1c994c608be9e362f5464990b3204f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10484][SQL] Optimize the cartesian join...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8652#issuecomment-151389719 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10484][SQL] Optimize the cartesian join...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8652#issuecomment-151389533 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11334] numRunningTasks can't be less th...
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/9288#issuecomment-151385860 IMHO, would it be better to fix this unexpected ordering of events, from my understanding, `SparkListenerTaskEnd` should be triggered before `SparkListenerStageCompleted `, right? I think that's root cause should be fixed, rather than adding some guard codes in dynamic allocation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11225]Prevent generate empty file
Github user viper-kun commented on the pull request: https://github.com/apache/spark/pull/9191#issuecomment-151385025 @JoshRosen Is it ok? If it doesn't work, I will close this pr. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11340][SPARKR] Support setting driver p...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9290#issuecomment-151383876 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11340][SPARKR] Support setting driver p...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9290#issuecomment-151383877 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44409/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11340][SPARKR] Support setting driver p...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9290#issuecomment-151383814 **[Test build #44409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44409/consoleFull)** for PR 9290 at commit [`5ecc9e0`](https://github.com/apache/spark/commit/5ecc9e09874c1bd63d4712412777aef4f2eb5afb). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11340][SPARKR] Support setting driver p...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9290#issuecomment-151382025 **[Test build #44409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44409/consoleFull)** for PR 9290 at commit [`5ecc9e0`](https://github.com/apache/spark/commit/5ecc9e09874c1bd63d4712412777aef4f2eb5afb). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11210][SPARKR] Add window functions int...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/9196#issuecomment-151381136 looks good! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11340][SPARKR] Support setting driver p...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9290#issuecomment-151381066 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11340][SPARKR] Support setting driver p...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9290#issuecomment-151381067 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44405/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11340][SPARKR] Support setting driver p...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9290#issuecomment-151380796 **[Test build #44405 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44405/consoleFull)** for PR 9290 at commit [`d3f8d28`](https://github.com/apache/spark/commit/d3f8d280098f42615c4d63d64d8797c8c76a8970). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11340][SPARKR] Support setting driver p...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9290#issuecomment-151379993 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11340][SPARKR] Support setting driver p...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9290#issuecomment-151380063 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11210][SPARKR] Add window functions int...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9196#issuecomment-151379732 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11210][SPARKR] Add window functions int...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9196#issuecomment-151379733 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44404/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11210][SPARKR] Add window functions int...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9196#issuecomment-151379670 **[Test build #44404 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44404/consoleFull)** for PR 9196 at commit [`2e84e2d`](https://github.com/apache/spark/commit/2e84e2d7e04de1a86c3eb1a24944a3b5b12a97e1). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `public class JavaCountVectorizerExample `\n * `public class JavaTfIdfExample `\n * `public class JavaWord2VecExample `\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9319][SPARKR] Add support for setting c...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/9218#issuecomment-151379530 @sun-rui That's a great point, `coltypes()` as its signature is defined, would only return a list of simple types. But how would one create a DataFrame with complex type from R? I tried a bit and couldn't get it to work. Either I get `Unsupported type for DataFrame: factor` or `unexpected type: environment` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11297] Add new code tags
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9265#issuecomment-151379573 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11297] Add new code tags
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9265#issuecomment-151379574 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44406/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11297] Add new code tags
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9265#issuecomment-151379513 **[Test build #44406 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44406/consoleFull)** for PR 9265 at commit [`7352bc4`](https://github.com/apache/spark/commit/7352bc406282038802d5560a6aafad79b36301c1). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `public class JavaCountVectorizerExample `\n * `public class JavaTfIdfExample `\n * `public class JavaWord2VecExample `\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11341][SQL] Given non-zero ordinal toRo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9292#issuecomment-151379305 **[Test build #44407 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44407/consoleFull)** for PR 9292 at commit [`0506fa7`](https://github.com/apache/spark/commit/0506fa738089573cd3fd97629b24add26118e178). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11340][SPARKR] Support setting driver p...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/9290#issuecomment-151378899 I checked, the user could also set SPARK_DRIVER_MEMORY before running `sparkR.init()` https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L157 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11334] numRunningTasks can't be less th...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9288#issuecomment-151378713 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44397/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151378760 **[Test build #44408 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44408/consoleFull)** for PR 8669 at commit [`41511b4`](https://github.com/apache/spark/commit/41511b417b1009bfda3f3cf8104ef2025f50227d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11334] numRunningTasks can't be less th...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9288#issuecomment-151378709 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11334] numRunningTasks can't be less th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9288#issuecomment-151378473 **[Test build #44397 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44397/consoleFull)** for PR 9288 at commit [`e32e684`](https://github.com/apache/spark/commit/e32e68485eaf0ed9eed7d88478154aff8650da62). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11341][SQL] Given non-zero ordinal toRo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9292#issuecomment-151378248 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11341][SQL] Given non-zero ordinal toRo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9292#issuecomment-151378260 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151378254 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11340][SPARKR] Support setting driver p...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9290#issuecomment-151378199 **[Test build #44405 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44405/consoleFull)** for PR 9290 at commit [`d3f8d28`](https://github.com/apache/spark/commit/d3f8d280098f42615c4d63d64d8797c8c76a8970). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151378261 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11338][WebUI] Prepend app links on Hist...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9291#issuecomment-151378251 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11341][SQL] Given non-zero ordinal toRo...
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/9292 [SPARK-11341][SQL] Given non-zero ordinal toRow in the encoders of primitive types will cause problem JIRA: https://issues.apache.org/jira/browse/SPARK-11341 The toRow in LongEncoder, IntEncoder writes given ordinal of an unsafe row with only one field. Since the ordinal is parametric. If given non-zero ordinal, it will cause problem. You can merge this pull request into a Git repository by running: $ git pull https://github.com/viirya/spark-1 fix-ordinal Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9292.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9292 commit 0506fa738089573cd3fd97629b24add26118e178 Author: Liang-Chi Hsieh Date: 2015-10-27T05:13:19Z Given non-zero ordinal will cause problem. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11341][SQL] Given non-zero ordinal toRo...
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/9292#issuecomment-151378144 ping @marmbrus --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11297] Add new code tags
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9265#issuecomment-151378091 **[Test build #44406 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44406/consoleFull)** for PR 9265 at commit [`7352bc4`](https://github.com/apache/spark/commit/7352bc406282038802d5560a6aafad79b36301c1). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2960][Deploy] Support executing Spark f...
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/8669#issuecomment-151377956 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11338][WebUI] Prepend app links on Hist...
GitHub user ckadner opened a pull request: https://github.com/apache/spark/pull/9291 [SPARK-11338][WebUI] Prepend app links on HistoryPage with uiRoot path - `HistoryPage.scala` ...prepending all page links with the web proxy (`uiRoot`) path - `HistoryServerSuite.scala` ...adding a test case to verify all site-relative links are prefixed when the environment variable `APPLICATION_WEB_PROXY_BASE` (or System property `spark.ui.proxyBase`) is set You can merge this pull request into a Git repository by running: $ git pull https://github.com/ckadner/spark SPARK-11338 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9291.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9291 commit 8bcb3dc16dd07916ef829bceced46f1d436d1b10 Author: Christian Kadner Date: 2015-10-27T05:07:33Z [SPARK-11338][WebUI] Prepend application links on HistoryPage with uiRoot path --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11210][SPARKR] Add window functions int...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9196#issuecomment-151377883 **[Test build #44404 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44404/consoleFull)** for PR 9196 at commit [`2e84e2d`](https://github.com/apache/spark/commit/2e84e2d7e04de1a86c3eb1a24944a3b5b12a97e1). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core][wip] first successful shuff...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9214#issuecomment-151377803 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44399/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core][wip] first successful shuff...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9214#issuecomment-151377802 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8029][core][wip] first successful shuff...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9214#issuecomment-151377734 **[Test build #44399 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44399/consoleFull)** for PR 9214 at commit [`4b7c71a`](https://github.com/apache/spark/commit/4b7c71a938d69be93baecb8ce320a2151b7a4658). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `case class ShuffleMapStatusBlockId(shuffleId: Int, mapId: Int) extends BlockId `\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11297] Add new code tags
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9265#issuecomment-151377428 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11340][SPARKR] Support setting driver p...
Github user felixcheung commented on the pull request: https://github.com/apache/spark/pull/9290#issuecomment-151377437 Manual testing with: ``` library(SparkR, lib.loc='/opt/spark-1.6.0-bin-hadoop2.6/R/lib') sc <- sparkR.init(master = "local[*]", sparkEnvir = list(spark.driver.memory = "2g")) ``` before  after  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11340][SPARKR] Support setting driver p...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9290#issuecomment-151377429 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11340][SPARKR] Support setting driver p...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9290#issuecomment-151377420 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11297] Add new code tags
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9265#issuecomment-151377421 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11340][SPARKR] Support setting driver p...
GitHub user felixcheung opened a pull request: https://github.com/apache/spark/pull/9290 [SPARK-11340][SPARKR] Support setting driver properties when starting Spark from R programmatically or from RStudio Mapping spark.driver.memory from sparkEnvir to spark-submit commandline arguments. @shivaram suggested that we possibly add other spark.driver.* properties - do we want to add all of those? I thought those could be set in SparkConf? @sun-rui You can merge this pull request into a Git repository by running: $ git pull https://github.com/felixcheung/spark rdrivermem Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9290.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9290 commit d3f8d280098f42615c4d63d64d8797c8c76a8970 Author: felixcheung Date: 2015-10-27T05:07:49Z Support setting spark.driver.memory from sparkEnvir when launching JVM backend --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11210][SPARKR] Add window functions int...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9196#issuecomment-151377012 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11210][SPARKR] Add window functions int...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9196#issuecomment-151377023 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11210][SPARKR][WIP] Add window function...
Github user sun-rui commented on the pull request: https://github.com/apache/spark/pull/9196#issuecomment-151376631 Rebased to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11297] Add new code tags
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/9265#issuecomment-151376585
@mengxr The simplest way to change the view is adding a css font constrain:
```css
div .highlight pre {
font-size: 12px;
}
```
It controls the elements inslide `xx`. I also check the font sizes of
other code examples, and they are 12px, which is defined in bootstrap.min.css.
Instead of changing the code tag, I think modifing the css is better.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10654][MLlib] Add columnSimilarities to...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8792 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10654][MLlib] Add columnSimilarities to...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/8792#issuecomment-151376315 LGTM. Merged into master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10286][ML][PySpark][Docs] Add @since an...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9275#issuecomment-151376263 **[Test build #44403 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44403/consoleFull)** for PR 9275 at commit [`ccdc82e`](https://github.com/apache/spark/commit/ccdc82e6bcc3df60dc6714f3534a840644929cfe). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10286][ML][PySpark][Docs] Add @since an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9275#issuecomment-151376285 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44403/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10286][ML][PySpark][Docs] Add @since an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9275#issuecomment-151376284 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10263] [ML] Add @Since annotation to ml...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/8935#issuecomment-151376098 @hiross Sorry for my slow response! Please check the public API instead of line changes. The former is more accurate. The PR has merge conflicts with the master branch. Could you merge current master and resolve conflicts? Thanks! @yu-iskw Could you help review this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10263] [ML] Add @Since annotation to ml...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/8935#discussion_r43084418
--- Diff: mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala ---
@@ -82,8 +82,11 @@ abstract class PipelineStage extends Params with Logging
{
* an identity transformer.
*/
@Experimental
-class Pipeline(override val uid: String) extends Estimator[PipelineModel] {
+@Since("1.5.0")
+class Pipeline(
+@Since("1.5.0") override val uid: String) extends
Estimator[PipelineModel] {
--- End diff --
`uid` should be since 1.4.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10263] [ML] Add @Since annotation to ml...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/8935#discussion_r43084419
--- Diff: mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala ---
@@ -82,8 +82,11 @@ abstract class PipelineStage extends Params with Logging
{
* an identity transformer.
*/
@Experimental
-class Pipeline(override val uid: String) extends Estimator[PipelineModel] {
+@Since("1.5.0")
+class Pipeline(
+@Since("1.5.0") override val uid: String) extends
Estimator[PipelineModel] {
+ @Since("1.5.0")
--- End diff --
This is since 1.2 as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10263] [ML] Add @Since annotation to ml...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/8935#discussion_r43084417
--- Diff: mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala ---
@@ -82,8 +82,11 @@ abstract class PipelineStage extends Params with Logging
{
* an identity transformer.
*/
@Experimental
-class Pipeline(override val uid: String) extends Estimator[PipelineModel] {
+@Since("1.5.0")
--- End diff --
`Pipeline` was introduced in 1.2.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8542][MLlib]PMML export for Decision Tr...
Github user yinxusen commented on the pull request: https://github.com/apache/spark/pull/7842#issuecomment-151375671 @mengxr Sure I'll reivew it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SAPRK-8546] Add PMML export for Naive Bayes
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/9057#discussion_r43084336
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala ---
@@ -19,6 +19,8 @@ package org.apache.spark.mllib.classification
import java.lang.{Iterable => JIterable}
+import org.apache.spark.mllib.pmml.PMMLExportable
--- End diff --
organize imports
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10286][ML][PySpark][Docs] Add @since an...
Github user lidinghao commented on the pull request: https://github.com/apache/spark/pull/9275#issuecomment-151375697 @mengxr Ok, I will add the versionadded --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SAPRK-8546] Add PMML export for Naive Bayes
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/9057#discussion_r43084342
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/pmml/export/NaiveBayesPMMLModelExport.scala
---
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.pmml.export
+
+import scala.{Array => SArray}
--- End diff --
It might be useful to include a comment explaining why we need to rename
`Array`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SAPRK-8546] Add PMML export for Naive Bayes
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/9057#issuecomment-151375621 @JasmineGeorge Could you make a pass? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8542][MLlib]PMML export for Decision Tr...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/7842#issuecomment-151375567 @JasmineGeorge Sorry for late response! We are running out of code review bandwidth. How about you and @yinxusen helping review each other's PR? His PR is at https://github.com/apache/spark/pull/9057. @yinxusen Please let me know whether you can make another pass on this PR. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11209][SPARKR] Add window functions int...
Github user sun-rui commented on the pull request: https://github.com/apache/spark/pull/9193#issuecomment-151375443 @shivaram, for your suggestion that "we should a make a list of functions that we mask and are incompatible", I submitted https://issues.apache.org/jira/browse/SPARK-11339 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11184] [MLLIB] Declare most of .mllib c...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9169 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11184] [MLLIB] Declare most of .mllib c...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/9169#issuecomment-151375298 Merged into master. Thanks! We can keep the discussion running:) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11184] [MLLIB] Declare most of .mllib c...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/9169#issuecomment-151374915 This looks good to me. Btw, we haven't yet discussed the exact semantic of `@Experimental`. I think we haven't made any break changes on `@Experimental` APIs without deprecating them first in MLlib since 1.0. Then they are basically the same as stable methods. Does `@Experimental` mean having a better chance to be deprecated? If we don't know the exact semantic, most users/developers would be confused by the `@Experimental` tag. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10286][ML][PySpark][Docs] Add @since an...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9275#issuecomment-151374525 **[Test build #44403 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44403/consoleFull)** for PR 9275 at commit [`ccdc82e`](https://github.com/apache/spark/commit/ccdc82e6bcc3df60dc6714f3534a840644929cfe). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-11295 Add packages to JUnit output for P...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/9263#discussion_r43083874
--- Diff: python/pyspark/mllib/tests.py ---
@@ -76,7 +76,8 @@
pass
ser = PickleSerializer()
-sc = SparkContext('local[4]', "MLlib tests")
+conf = SparkConf().set("spark.driver.allowMultipleContexts", "true")
--- End diff --
Is it relevant to this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10286][ML][PySpark][Docs] Add @since an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9275#issuecomment-151372540 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10286][ML][PySpark][Docs] Add @since an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9275#issuecomment-151372561 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10286][ML][PySpark][Docs] Add @since an...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/9275#issuecomment-151372367 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10286][ML][PySpark][Docs] Add @since an...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/9275#issuecomment-151372364 @lidinghao Thanks for working on this! For classes, could you also add `versionadded`? You can find an example here: https://github.com/apache/spark/blob/f85aa06464a10f5d1563302fd76465dded475a12/python/pyspark/mllib/fpm.py#L45. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10024] [pyspark] Python API RF and GBT ...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/9233#issuecomment-151372045 @vectorijk Could you add the since versions back? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10271][Pyspark][MLLib] Added @since tag...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8627 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10271][Pyspark][MLLib] Added @since tag...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/8627#issuecomment-151371862 Merged into master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11322][PySpark] Keep full stack trace i...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9283#issuecomment-151371826 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11322][PySpark] Keep full stack trace i...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9283#issuecomment-151371829 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44402/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11322][PySpark] Keep full stack trace i...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9283#issuecomment-151371791 **[Test build #44402 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44402/consoleFull)** for PR 9283 at commit [`8f69480`](https://github.com/apache/spark/commit/8f694803194951391663b1d120b90996bb181019). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_:\n * `class CapturedException(Exception):`\n * `class AnalysisException(CapturedException):`\n * `class IllegalArgumentException(CapturedException):`\n --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11284] [ML] ALS produces float predicti...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/9252#issuecomment-151371779 @dahlem If you plan to make a new pull request, could you close this PR manually? We don't have direct access to close Spark PRs. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11289][DOC] Substitute code examples in...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9266 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10562][SQL] support mixed case partitio...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/9226 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11277] [SQL] sort_array throws exceptio...
Github user cloud-fan commented on the pull request: https://github.com/apache/spark/pull/9247#issuecomment-151370701 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
