[GitHub] spark pull request: [SPARK-3654][SQL] Unifies SQL and HiveQL parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2698#issuecomment-58466598 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21525/consoleFull) for PR 2698 at commit [`ba2c121`](https://github.com/apache/spark/commit/ba2c1214c20e75853b7541f1744f9d1761d97a5d). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3861][SQL] Avoid rebuilding hash tables...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/2727#discussion_r18627919
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
---
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.joins
+
+import java.util.{HashMap = JavaHashMap}
+
+import org.apache.spark.sql.catalyst.expressions.{Projection, Row}
+import org.apache.spark.util.collection.CompactBuffer
+
+
+/**
+ * Interface for a hashed relation by some key. Use
[[HashedRelation.apply]] to create a concrete
+ * object.
+ */
+private[joins] sealed trait HashedRelation {
+ def get(key: Row): CompactBuffer[Row]
+}
+
+
+/**
+ * A general [[HashedRelation]] backed by a hash map that maps the key
into a sequence of values.
+ */
+private[joins] final class GeneralHashedRelation(hashTable:
JavaHashMap[Row, CompactBuffer[Row]])
+ extends HashedRelation with Serializable {
+
+ override def get(key: Row) = hashTable.get(key)
+}
+
+
+/**
+ * A specialized [[HashedRelation]] that maps key into a single value.
This implementation
+ * assumes the key is unique.
+ */
+final class UniqueKeyHashedRelation(hashTable: JavaHashMap[Row, Row])
+ extends HashedRelation with Serializable {
+
+ override def get(key: Row) = {
+val v = hashTable.get(key)
+if (v eq null) null else CompactBuffer(v)
--- End diff --
Sorry, I mean for each row in stream side, will create a `CompactBuffer`
instance if it finds a matched row in build side, this probably too heavy.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3325] Add a parameter to the method pri...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2216#discussion_r18627949
--- Diff:
streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaDStream.scala
---
@@ -94,6 +94,14 @@ class JavaDStream[T](val dstream: DStream[T])(implicit
val classTag: ClassTag[T]
* returned DStream has exactly numPartitions partitions.
*/
def repartition(numPartitions: Int): JavaDStream[T] =
dstream.repartition(numPartitions)
+
+ /**
+ * Print the first num elements of each RDD generated in this DStream.
This is an output
+ * operator, so this DStream will be registered as an output stream and
there materialized.
+ */
+ override def print(num: Int): Unit = {
--- End diff --
@watermen Ah OK, my head's on straight now. The issue is really people
extending `JavaDStream`. This doesn't help people extending `JavaDStreamLike`
but that can't be helped.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3654][SQL] Unifies SQL and HiveQL parse...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2698#issuecomment-58467387 This PR conflicts with #2678. It would be good to merge that one first, and then I'll rebase. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3559][SQL] Remove unnecessary columns f...
Github user gvramana commented on the pull request: https://github.com/apache/spark/pull/2713#issuecomment-58467725 I have verified that only required columns are being passed by adding log. This cannot be verified by result test, as TableReader will pick required columns only when converting result to row. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3861][SQL] Avoid rebuilding hash tables...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2727#issuecomment-58468491 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21523/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3861][SQL] Avoid rebuilding hash tables...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2727#issuecomment-58468489 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21523/consoleFull) for PR 2727 at commit [`7fcffb5`](https://github.com/apache/spark/commit/7fcffb5372662d451612d7541194ab43800eda81). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3844][UI] Truncate appName in WebUI if ...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2707#issuecomment-58468535 Ah I see- okay makes sense. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3654][SQL] Unifies SQL and HiveQL parse...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2698#issuecomment-58468683 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21525/consoleFull) for PR 2698 at commit [`ba2c121`](https://github.com/apache/spark/commit/ba2c1214c20e75853b7541f1744f9d1761d97a5d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` protected case class Keyword(str: String)` * `class SqlLexical(val keywords: Seq[String]) extends StdLexical ` * ` case class FloatLit(chars: String) extends Token ` * `class SqlParser extends AbstractSparkSQLParser ` * `case class SetCommand(kv: Option[(String, Option[String])]) extends Command ` * `case class ShellCommand(cmd: String) extends Command` * `case class SourceCommand(filePath: String) extends Command` * `case class SetCommand(kv: Option[(String, Option[String])], output: Seq[Attribute])(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3654][SQL] Unifies SQL and HiveQL parse...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2698#issuecomment-58468687 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21525/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3844][UI] Truncate appName in WebUI if ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2707#issuecomment-58468950 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/298/consoleFull) for PR 2707 at commit [`87834ce`](https://github.com/apache/spark/commit/87834cee5d5eff72a151d82586c9123939525d62). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3868][PySpark] Hard to recognize which ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2724#issuecomment-58468966 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/297/consoleFull) for PR 2724 at commit [`c63d9fa`](https://github.com/apache/spark/commit/c63d9faf3f712327d5e84050097638092c3dced2). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class AbstractParams[T: TypeTag] ` * ` case class Params(inputFile: String = null, threshold: Double = 0.1)` * `class Word2VecModel(object):` * `class Word2Vec(object):` * ` class SparkIMain(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2805] akka 2.3.4
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/1685#discussion_r18629153
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala ---
@@ -144,59 +142,6 @@ class InputStreamsSuite extends TestSuiteBase with
BeforeAndAfter {
conf.set(spark.streaming.clock,
org.apache.spark.streaming.util.ManualClock)
}
- // TODO: This test works in IntelliJ but not through SBT
- ignore(actor input stream) {
--- End diff --
Okay @ScrapCodes is going to add a patch to fix this - I guess we were
using some deprecated API's here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2805] akka 2.3.4
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1685#issuecomment-58469419 Great LGTM - thanks @avati for the help with this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3861][SQL] Avoid rebuilding hash tables...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/2727#discussion_r18629178
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala
---
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.joins
+
+import java.util.{HashMap = JavaHashMap}
+
+import org.apache.spark.sql.catalyst.expressions.{Projection, Row}
+import org.apache.spark.util.collection.CompactBuffer
+
+
+/**
+ * Interface for a hashed relation by some key. Use
[[HashedRelation.apply]] to create a concrete
+ * object.
+ */
+private[joins] sealed trait HashedRelation {
+ def get(key: Row): CompactBuffer[Row]
+}
+
+
+/**
+ * A general [[HashedRelation]] backed by a hash map that maps the key
into a sequence of values.
+ */
+private[joins] final class GeneralHashedRelation(hashTable:
JavaHashMap[Row, CompactBuffer[Row]])
+ extends HashedRelation with Serializable {
+
+ override def get(key: Row) = hashTable.get(key)
+}
+
+
+/**
+ * A specialized [[HashedRelation]] that maps key into a single value.
This implementation
+ * assumes the key is unique.
+ */
+final class UniqueKeyHashedRelation(hashTable: JavaHashMap[Row, Row])
+ extends HashedRelation with Serializable {
+
+ override def get(key: Row) = {
+val v = hashTable.get(key)
+if (v eq null) null else CompactBuffer(v)
--- End diff --
Yea. What I meant was we will add a new operator that specializes for
unique key joins, and that operator would just call getValue, bypassing the
creation of CompactBuffer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2805] akka 2.3.4
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1685 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2805] akka 2.3.4
Github user ScrapCodes commented on a diff in the pull request:
https://github.com/apache/spark/pull/1685#discussion_r18629324
--- Diff:
streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala ---
@@ -144,59 +142,6 @@ class InputStreamsSuite extends TestSuiteBase with
BeforeAndAfter {
conf.set(spark.streaming.clock,
org.apache.spark.streaming.util.ManualClock)
}
- // TODO: This test works in IntelliJ but not through SBT
- ignore(actor input stream) {
--- End diff --
I have created an issue to track this.
https://issues.apache.org/jira/browse/SPARK-3872
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2728#issuecomment-58469995 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21524/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2728#issuecomment-58469990 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21524/consoleFull) for PR 2728 at commit [`8b5a96e`](https://github.com/apache/spark/commit/8b5a96e9ed312ac3936147ce0aea6b37347eedfc). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3844][UI] Truncate appName in WebUI if ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2707#issuecomment-58470529 Ok, I'm merging this into master and 1.1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3529] [SQL] Delete the temp files after...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/2393#issuecomment-58470589 @srowen thank you very much, this is quite informative. I've updated the code with `Utils.registerShutdownDeleteDir`, the code is very clean now! However, the temp directory still there when test exit, seems the register doesn't work on purpose, should I file another jira issue for this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3844][UI] Truncate appName in WebUI if ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2707 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add spark.driver.memory to config docs
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2410#issuecomment-58470754 Ok I merged this into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add spark.driver.memory to config docs
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2410 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3529] [SQL] Delete the temp files after...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2393#issuecomment-58470882 Perhaps you can debug a bit first to see if the shutdown hook is called and it attempts to delete the dir? is there are error while deleting it? this mechanism appears to work for unit test temp files. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3529] [SQL] Delete the temp files after...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2393#issuecomment-58471062 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21526/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3529] [SQL] Delete the temp files after...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/2393#issuecomment-58471183 Actually, I searched the code, seems we haven't set the shutdown hook for `Utils`, you mean I have to do that myself ? I was thinking it should be set properly somewhere in spark core. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3529] [SQL] Delete the temp files after...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2393#issuecomment-58471362 Ah, right. This is the shutdown hook: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L257 ... but it requires you to have used `createTempDir()`. Is this a temp dir? I have an outstanding PR that improves a few things here, including, just directly deleting everything seen by `registerShutdownDeleteDir` in a single shutdown hook: https://github.com/apache/spark/pull/2670 If that PR looks OK then, it would be good to get it committed, since it would make @chenghao-intel 's simpler solution here work. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3158][MLLIB]Avoid 1 extra aggregation f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2708#issuecomment-58471777 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21527/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3158][MLLIB]Avoid 1 extra aggregation f...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/2708#issuecomment-58472333 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3158][MLLIB]Avoid 1 extra aggregation f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2708#issuecomment-58472774 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21528/consoleFull) for PR 2708 at commit [`8e269ea`](https://github.com/apache/spark/commit/8e269ea2902ed8cd6dc1ae30938544009be1e374). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Add spark.tmp.dir to set temp directory for sp...
GitHub user kelepi opened a pull request: https://github.com/apache/spark/pull/2729 Add spark.tmp.dir to set temp directory for spark Related to SPARK-3875. You can merge this pull request into a Git repository by running: $ git pull https://github.com/kelepi/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2729.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2729 commit f14b68a4c98bf01db84afa55505868d42d8e4eb5 Author: yf_liu yf_...@ctrip.com Date: 2014-10-09T07:32:01Z Add spark.tmp.dir to set temp directory for spark --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Add spark.tmp.dir to set temp directory for sp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2729#issuecomment-58473404 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3875] Add TEMP DIRECTORY configuration
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2729#issuecomment-58473598 `Utils.getLocalDir` already does basically this. I agree that files should not have to go to `/tmp` since this is rarely a good place for lots of stuff on servers. But I do wonder whether this local dir is in fact the standard and right place for all of these things rather than another temp dir setting. Distributions already correctly configure where `Utils.getLocalDir` looks so this is going to be much better if possible. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3121] Wrong implementation of implicit ...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/2712#issuecomment-58473731 Hmm, yeah, copyBytes is no good if it doesn't appear in Hadoop 1. My suggestion would be to use from copyOfRange from java.util.Arrays. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3875] Add TEMP DIRECTORY configuration
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2729#discussion_r18630676
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -233,7 +233,11 @@ private[spark] object Utils extends Logging {
}
/** Create a temporary directory inside the given parent directory */
- def createTempDir(root: String = System.getProperty(java.io.tmpdir)):
File = {
+ def createTempDir(): File = {
+val root: String = SparkEnv.get.conf.contains(spark.tmp.dir) match {
--- End diff --
PS this can all be one line if you set the default value to `...
.get(spark.tmp.dir, System.getProperty(java.io.tmpdir))`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3875] Add TEMP DIRECTORY configuration
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/2729#issuecomment-58473957 At least for yarn, this will create issues if overridden from default. Not sure about mesos. Why not use std java property and define it for local and standalone mode where relevant. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3875] Add TEMP DIRECTORY configuration
Github user kelepi commented on a diff in the pull request:
https://github.com/apache/spark/pull/2729#discussion_r18630974
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -233,7 +233,11 @@ private[spark] object Utils extends Logging {
}
/** Create a temporary directory inside the given parent directory */
- def createTempDir(root: String = System.getProperty(java.io.tmpdir)):
File = {
+ def createTempDir(): File = {
+val root: String = SparkEnv.get.conf.contains(spark.tmp.dir) match {
--- End diff --
Thanks @srowen for the comment. That's more clear!
Currently I am running Spark 1.1 in Standalone mode. And I have set the
'SPARK_LOCAL_DIRS' to data disk, which will store the broadcast Files in the
target directory. However, the dependencies for executors will be fetched and
stored in the /tmp/ directory.
And the dependencies will not be removed.
Like the snappy dependency, for example:
snappy-1.0.5.3-f4880c9f-95d9-4ab6-b1c8-8686d0b88f42-libsnappyjava.so
So the /tmp/ directory will grow large.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3569][SQL] Add metadata field to Struct...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2701#issuecomment-58474390 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21529/consoleFull) for PR 2701 at commit [`93518fb`](https://github.com/apache/spark/commit/93518fbfcef06621b81ea33439833a6e2c158bc7). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3875] Add TEMP DIRECTORY configuration
Github user kelepi commented on the pull request: https://github.com/apache/spark/pull/2729#issuecomment-58474600 @mridulm Using std java property is fine. Just add a more specific configuration argument. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3529] [SQL] Delete the temp files after...
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/2393#issuecomment-58475061 `TestHive` creates temporal files/directories by specifying the prefix / suffix, it will be great if `Utils` support that as well, the code will be even more cleaner then. Currently I prefer wait for the #2670 to be merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3366][MLLIB]Compute best splits distrib...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/2595#issuecomment-58476229 @jkbradley Thanks for running the experiments! It is clear that the regression happens when the shuffle size is not large enough to make dist-agg faster than tree-agg, in particular, in the cases of shallow levels, small number of features, or small number of trees. So the question becomes what is the problem scale we really want to solve in practice. If we train a single tree, is depth 5 good enough in most cases (including boosting)? If we use random forest with SQRT, would 5 trees be good enough? It would be really helpful if we can find some references. Then let's decide whether we want to keep both approaches. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3158][MLLIB]Avoid 1 extra aggregation f...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2708#issuecomment-58478349 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21528/consoleFull) for PR 2708 at commit [`8e269ea`](https://github.com/apache/spark/commit/8e269ea2902ed8cd6dc1ae30938544009be1e374). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3158][MLLIB]Avoid 1 extra aggregation f...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2708#issuecomment-58478358 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21528/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3158][MLLIB]Avoid 1 extra aggregation f...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/2708#issuecomment-58478907 Merged into master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3158][MLLIB]Avoid 1 extra aggregation f...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2708 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1720][SPARK-1719] use LD_LIBRARY_P...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2711#issuecomment-58479612 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21530/consoleFull) for PR 2711 at commit [`a69bca1`](https://github.com/apache/spark/commit/a69bca1ee44329f1d05694baa44d9e902ed640ef). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3875] Add TEMP DIRECTORY configuration
Github user mridulm commented on the pull request: https://github.com/apache/spark/pull/2729#issuecomment-58479810 There is a java property which controls this ... java.io.tmpdir On 09-Oct-2014 1:22 pm, åé°å¸ notificati...@github.com wrote: @mridulm https://github.com/mridulm Using std java property is fine. Just add a more specific configuration argument. â Reply to this email directly or view it on GitHub https://github.com/apache/spark/pull/2729#issuecomment-58474600. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3559][SQL] Remove unnecessary columns f...
Github user gvramana commented on the pull request: https://github.com/apache/spark/pull/2713#issuecomment-58480671 @marmbrus, Jenkins Test Failure is due to git fetch failure. Can you please trigger the test again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3569][SQL] Add metadata field to Struct...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2701#issuecomment-58481258 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21529/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3569][SQL] Add metadata field to Struct...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2701#issuecomment-58481254 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21529/consoleFull) for PR 2701 at commit [`93518fb`](https://github.com/apache/spark/commit/93518fbfcef06621b81ea33439833a6e2c158bc7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class AttributeReference(` * `case class StructField(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2675#issuecomment-58481729 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21531/consoleFull) for PR 2675 at commit [`e79b2d7`](https://github.com/apache/spark/commit/e79b2d781fb1aa3260f7d6d4007cef19333f3e9c). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1720][SPARK-1719] use LD_LIBRARY_P...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2711#issuecomment-58482738 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21532/consoleFull) for PR 2711 at commit [`880e06d`](https://github.com/apache/spark/commit/880e06de3b4022f8113251e43c3e28ab2d38a35e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2675#issuecomment-58485309 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21531/consoleFull) for PR 2675 at commit [`e79b2d7`](https://github.com/apache/spark/commit/e79b2d781fb1aa3260f7d6d4007cef19333f3e9c). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2675#issuecomment-58485318 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21531/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1720][SPARK-1719] use LD_LIBRARY_P...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2711#issuecomment-58486265 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21530/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1720][SPARK-1719] use LD_LIBRARY_P...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2711#issuecomment-58486257 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21530/consoleFull) for PR 2711 at commit [`a69bca1`](https://github.com/apache/spark/commit/a69bca1ee44329f1d05694baa44d9e902ed640ef). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Minor] use norm operator after breeze 0.10 up...
GitHub user witgo opened a pull request: https://github.com/apache/spark/pull/2730 [Minor] use norm operator after breeze 0.10 upgrade cc @mengxr You can merge this pull request into a Git repository by running: $ git pull https://github.com/witgo/spark SPARK-3856 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2730.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2730 commit 2cffce14e404751939f72e9c8f129773183f24fa Author: GuoQiang Li wi...@qq.com Date: 2014-10-09T10:07:02Z use norm operator after breeze 0.10 upgrade --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Minor] use norm operator after breeze 0.10 up...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2730#issuecomment-58489133 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21533/consoleFull) for PR 2730 at commit [`2cffce1`](https://github.com/apache/spark/commit/2cffce14e404751939f72e9c8f129773183f24fa). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1720][SPARK-1719] use LD_LIBRARY_P...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2711#issuecomment-58489209 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21532/consoleFull) for PR 2711 at commit [`880e06d`](https://github.com/apache/spark/commit/880e06de3b4022f8113251e43c3e28ab2d38a35e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [WIP][SPARK-1720][SPARK-1719] use LD_LIBRARY_P...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2711#issuecomment-58489217 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21532/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3830] Implement genetic algorithms in M...
GitHub user epahomov opened a pull request: https://github.com/apache/spark/pull/2731 [SPARK-3830] Implement genetic algorithms in MLLib You can merge this pull request into a Git repository by running: $ git pull https://github.com/epahomov/spark SPARK-3830 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2731.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2731 commit bb00e96ed10d4d01d35f4506babe967bc438e877 Author: epahomov pahomov.e...@gmail.com Date: 2014-10-09T10:15:01Z [SPARK-3830] Implement genetic algorithms in MLLib --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3830] Implement genetic algorithms in M...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2731#issuecomment-58489634 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21534/consoleFull) for PR 2731 at commit [`bb00e96`](https://github.com/apache/spark/commit/bb00e96ed10d4d01d35f4506babe967bc438e877). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3877] Throw an exception when applicati...
GitHub user zsxwing opened a pull request: https://github.com/apache/spark/pull/2732 [SPARK-3877] Throw an exception when application is not successful so that the exit code wil be set to 1 When an yarn application fails (yarn-cluster mode), the exit code of spark-submit is still 0. It's hard for people to write some automatic scripts to run spark jobs in yarn because the failure can not be detected in these scripts. This PR added a status checking after `monitorApplication`. If an application is not successful, `run()` will throw an `SparkException`, so that Client.scala will exit with code 1. Therefore, people can use the exit code of `spark-submit` to write some automatic tasks. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zsxwing/spark SPARK-3877 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2732.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2732 commit 6a2c103b9514b085fb5b139f859cde1cbf1a048e Author: zsxwing zsxw...@gmail.com Date: 2014-10-09T09:57:02Z [SPARK-3877] Throw an exception when application is not successful so that the exit code wil be set to 1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: code style format and some litter improvment
GitHub user shijinkui opened a pull request: https://github.com/apache/spark/pull/2733 code style format and some litter improvment You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/spark styleFormat Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2733.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2733 commit bd103b78adebf933d3e33d4628c2394a845dbaa2 Author: shijinkui shijinkui...@163.com Date: 2014-09-29T05:34:02Z code style format commit 0c9d1754f9c8bf36236f915868eb6f62863fea25 Author: shijinkui shijinkui...@163.com Date: 2014-10-03T14:21:44Z code format commit 237bacc3d1c911e6475da4ad08dcd4d0031883ec Author: shijinkui shijinkui...@163.com Date: 2014-10-08T03:05:01Z resolve conflic commit 78f69b9526ea546251dac5eddf4da9c9eb6e20ad Author: shijinkui shijinkui...@163.com Date: 2014-10-08T03:31:24Z code format commit 725eec51fb7d29d0df99b92df3ef62fcec301d90 Author: çç jinkui@alibaba-inc.com Date: 2014-10-08T14:42:29Z resolve test fail commit e54344b33b4c8cb4c1ff0dfb18a08188de464cfc Author: çç jinkui@alibaba-inc.com Date: 2014-10-09T08:55:18Z code format --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: code style format and some litter improvment
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2733#issuecomment-58493480 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3877] Throw an exception when applicati...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2732#issuecomment-58493700 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21535/consoleFull) for PR 2732 at commit [`6a2c103`](https://github.com/apache/spark/commit/6a2c103b9514b085fb5b139f859cde1cbf1a048e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2675#issuecomment-58494555 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21536/consoleFull) for PR 2675 at commit [`e79e1cd`](https://github.com/apache/spark/commit/e79e1cd74b9b478ddff761d957cf495c5417f2cf). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Minor] use norm operator after breeze 0.10 up...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2730#issuecomment-58494871 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21533/consoleFull) for PR 2730 at commit [`2cffce1`](https://github.com/apache/spark/commit/2cffce14e404751939f72e9c8f129773183f24fa). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Minor] use norm operator after breeze 0.10 up...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2730#issuecomment-58494879 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21533/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3830][MLlib] Implement genetic algorith...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2731#issuecomment-58495258 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21534/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3830][MLlib] Implement genetic algorith...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2731#issuecomment-58495248 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21534/consoleFull) for PR 2731 at commit [`bb00e96`](https://github.com/apache/spark/commit/bb00e96ed10d4d01d35f4506babe967bc438e877). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `trait IndividualOperations[I, T] ` * `class ListOperations[E, T](` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2675#issuecomment-58497656 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21536/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2675#issuecomment-58497647 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21536/consoleFull) for PR 2675 at commit [`e79e1cd`](https://github.com/apache/spark/commit/e79e1cd74b9b478ddff761d957cf495c5417f2cf). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2675#issuecomment-58497909 The last build failure should be irrelevant. Observed the same failure in #2701. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3877] Throw an exception when applicati...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2732#issuecomment-58499713 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21535/consoleFull) for PR 2732 at commit [`6a2c103`](https://github.com/apache/spark/commit/6a2c103b9514b085fb5b139f859cde1cbf1a048e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3877] Throw an exception when applicati...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2732#issuecomment-58499718 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21535/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2750] support https in spark web ui
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/1980#discussion_r18641539
--- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala ---
@@ -205,10 +231,74 @@ private[spark] object JettyUtils extends Logging {
ServerInfo(server, boundPort, collection)
}
+ // to generate a new url string scheme://server:port+path
--- End diff --
Hi @vanzin, actually here i just refer to the code of jetty 9
see(https://github.com/eclipse/jetty.project/blob/master/jetty-util/src/main/java/org/eclipse/jetty/util/URIUtil.java#L726-L733)
since there is no ```newURI```method in spark jetty version(spark use jetty
8).
And L238 is for the case to handle IPv6 address, here we can remove it if
unnecessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/2728#issuecomment-58504345 @sarutak thanks for working on this. Did you by chance test this to see if the tracking url is really not being set in the RM UI for all the modes (including using spark-shell)? If its not there might be a bug somewhere else as that used to work. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format and little impr...
GitHub user shijinkui opened a pull request: https://github.com/apache/spark/pull/2734 [SPARK-3781] code Style format and little improvement 1. use scala recommended usage 2. method body's left bracket 3. parameter list format 4. explicit mutable collection, such as new mutable.HashMap 5. others You can merge this pull request into a Git repository by running: $ git pull https://github.com/shijinkui/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2734.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2734 commit bd103b78adebf933d3e33d4628c2394a845dbaa2 Author: shijinkui shijinkui...@163.com Date: 2014-09-29T05:34:02Z code style format commit 0c9d1754f9c8bf36236f915868eb6f62863fea25 Author: shijinkui shijinkui...@163.com Date: 2014-10-03T14:21:44Z code format commit 237bacc3d1c911e6475da4ad08dcd4d0031883ec Author: shijinkui shijinkui...@163.com Date: 2014-10-08T03:05:01Z resolve conflic commit 78f69b9526ea546251dac5eddf4da9c9eb6e20ad Author: shijinkui shijinkui...@163.com Date: 2014-10-08T03:31:24Z code format commit 725eec51fb7d29d0df99b92df3ef62fcec301d90 Author: çç jinkui@alibaba-inc.com Date: 2014-10-08T14:42:29Z resolve test fail commit e54344b33b4c8cb4c1ff0dfb18a08188de464cfc Author: çç jinkui@alibaba-inc.com Date: 2014-10-09T08:55:18Z code format --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3781] code Style format and little impr...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2734#issuecomment-58507800 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3877] Throw an exception when applicati...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/2732#issuecomment-58513769 does this properly handle yarn client mode? For instance I don't see it throwing in YarnClientSchedulerBackend.asyncMonitorApplication. This would be a little different failure if something went wrong on the yarn side. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: code style format and some litter improvment
Github user shijinkui closed the pull request at: https://github.com/apache/spark/pull/2733 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2675#issuecomment-58520636 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/300/consoleFull) for PR 2675 at commit [`e79e1cd`](https://github.com/apache/spark/commit/e79e1cd74b9b478ddff761d957cf495c5417f2cf). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3824][SQL] Sets in-memory table default...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2686#issuecomment-58520732 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/301/consoleFull) for PR 2686 at commit [`35d2ed0`](https://github.com/apache/spark/commit/35d2ed0ce5a1864a78acef4d5f462b6e53077220). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Update RecoverableNetworkWordCount.scala
GitHub user comcmipi opened a pull request: https://github.com/apache/spark/pull/2735 Update RecoverableNetworkWordCount.scala Trying this example, I missed the moment when the checkpoint was iniciated You can merge this pull request into a Git repository by running: $ git pull https://github.com/comcmipi/spark patch-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2735.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2735 commit 96fe274beeffa003df56dac724a6d475641d7383 Author: comcmipi pito...@fns.uniba.sk Date: 2014-10-09T15:00:03Z Update RecoverableNetworkWordCount.scala Trying this example, I missed the moment when the checkpoint was iniciated --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3877] Throw an exception when applicati...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2732#issuecomment-58522681 I think yarn client mode is another story. Driver can detect the failure of executors by itself since it runs in the local. And if the driver crashes, it means SparkSubmit also crashes. So spark-submit will exit with 1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Update RecoverableNetworkWordCount.scala
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2735#issuecomment-58522723 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3606] [yarn] Correctly configure AmIpFi...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2497#issuecomment-58550486 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark] RDD take() method: overestimate too mu...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2648#issuecomment-58551768 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/302/consoleFull) for PR 2648 at commit [`4391d3b`](https://github.com/apache/spark/commit/4391d3bc2f20fda20c0c14f14821ead8338c2435). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3606] [yarn] Correctly configure AmIpFi...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2497#issuecomment-58563675 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21541/Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Update JavaCustomReceiver.java
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2699#issuecomment-58572816 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3809][SQL] Fixes test suites in hive-th...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2675#issuecomment-58572850 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Add type checking debugging functions
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2657#issuecomment-58573284 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21547/Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3855][SQL] Preserve the result attribut...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2717#issuecomment-58574660 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21549/consoleFull) for PR 2717 at commit [`6343bcb`](https://github.com/apache/spark/commit/6343bcb26e0f916c52f66886e61b60ee6ea6a06a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Use optional third argument as edge attribute.
Github user npanj closed the pull request at: https://github.com/apache/spark/pull/901 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3853][SQL] JSON Schema support for Time...
Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/2720#discussion_r18673921
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/json/JsonRDD.scala
---
@@ -361,6 +362,14 @@ private[sql] object JsonRDD extends Logging {
}
}
+ private def toTimestamp(value: Any): Timestamp = {
+value match {
+case value: java.lang.Integer = new
Timestamp(value.asInstanceOf[Int].toLong)
+case value: java.lang.Long = new Timestamp(value)
+case value: java.lang.String = Timestamp.valueOf(value)
--- End diff --
Oh, right. Cool.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2261] Make event logger use a single fi...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1222#issuecomment-58577499 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21548/consoleFull) for PR 1222 at commit [`16fd491`](https://github.com/apache/spark/commit/16fd491fbc473e7a8c848094086429fea97d1793). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3834][SQL] Backticks not correctly hand...
GitHub user ravipesala opened a pull request: https://github.com/apache/spark/pull/2737 [SPARK-3834][SQL] Backticks not correctly handled in subquery aliases The queries like '''SELECT a.key FROM (SELECT key FROM src) `a`''' does not work as backticks in subquery aliases are not handled properly. This PR fixes that. Author : ravipesala ravindra.pes...@huawei.com You can merge this pull request into a Git repository by running: $ git pull https://github.com/ravipesala/spark SPARK-3834 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2737.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2737 commit 0e0ab984cf58374914a4766a69e545c84022e2ed Author: ravipesala ravindra.pes...@huawei.com Date: 2014-10-09T21:23:40Z Fixing issue in backtick handling for subquery aliases --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Spark] RDD take() method: overestimate too mu...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2648#issuecomment-58581751 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/314/consoleFull) for PR 2648 at commit [`a8e74bb`](https://github.com/apache/spark/commit/a8e74bb0015cde80877aa7ce839f1bb4446bce80). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
