[GitHub] spark pull request: SPARK-3874: Provide stable TaskContext API
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2803#issuecomment-59317667 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3874: Provide stable TaskContext API
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2803#issuecomment-59317650 Jenkins, test this please. @liancheng - I am getting a SQL failure here that seems like it might be legitimate but I'm having trouble interpreting why it could possibly be caused by this patch - any idea? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3939][SQL] NPE caused by SessionState.o...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2812#issuecomment-59317882 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21791/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3939][SQL] NPE caused by SessionState.o...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2812#issuecomment-59317878 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21791/consoleFull) for PR 2812 at commit [`e89a7c2`](https://github.com/apache/spark/commit/e89a7c26fd095a1aea380860e0a45f80d5fcbd6b). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3874: Provide stable TaskContext API
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2803#issuecomment-59317948 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21794/consoleFull) for PR 2803 at commit [`56d5b7a`](https://github.com/apache/spark/commit/56d5b7a703afb2529d969ffa0664f6c601186fb0). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Core] Upgrading ScalaStyle version to 0.5 and...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2799#issuecomment-59318015 Thanks Prashant - I merged this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [Core] Upgrading ScalaStyle version to 0.5 and...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2799 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3939][SQL] NPE caused by SessionState.o...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/2812#issuecomment-59318221 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2621. Update task InputMetrics increment...
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2087#discussion_r18939560
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
---
@@ -121,6 +125,31 @@ class SparkHadoopUtil extends Logging {
UserGroupInformation.loginUserFromKeytab(principalName, keytabFilename)
}
+ /**
+ * Returns a function that can be called to find the number of Hadoop
FileSystem bytes read by
+ * this thread so far. Reflection is required because thread-level
FileSystem statistics are only
+ * available as of Hadoop 2.5 (see HADOOP-10688). Returns None if the
required method can't be
+ * found.
+ */
+ def getInputBytesReadCallback(path: Path, conf: Configuration):
Option[() = Long] = {
--- End diff --
I mean that the reflective lookups are themselves expensive, not calling
Method.invoke. However, now I see that this is just returning a new function
that calls Method.invoke so it should be fine performance wise.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3870] EOL character enforcement
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2726#issuecomment-59318575 test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2621. Update task InputMetrics increment...
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2087#discussion_r18939658
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
---
@@ -121,6 +125,31 @@ class SparkHadoopUtil extends Logging {
UserGroupInformation.loginUserFromKeytab(principalName, keytabFilename)
}
+ /**
+ * Returns a function that can be called to find the number of Hadoop
FileSystem bytes read by
+ * this thread so far. Reflection is required because thread-level
FileSystem statistics are only
+ * available as of Hadoop 2.5 (see HADOOP-10688). Returns None if the
required method can't be
+ * found.
+ */
+ def getInputBytesReadCallback(path: Path, conf: Configuration):
Option[() = Long] = {
+val qualifiedPath = path.getFileSystem(conf).makeQualified(path)
+val scheme = qualifiedPath.toUri().getScheme()
+val stats =
FileSystem.getAllStatistics().filter(_.getScheme().equals(scheme))
+try {
+ val threadStats = stats.map(Utils.invoke(classOf[Statistics], _,
getThreadStatistics))
+ val statisticsDataClass =
+
Class.forName(org.apache.hadoop.fs.FileSystem$Statistics$StatisticsData)
+ val getBytesReadMethod =
statisticsDataClass.getDeclaredMethod(getBytesRead)
+ val f = () =
threadStats.map(getBytesReadMethod.invoke(_).asInstanceOf[Long]).sum
+ val start = f()
+ Some(() = f() - start)
--- End diff --
ah I see now - so should we call this function `getThreadLocalBytesRead` or
something? It seems like it only semantically makes sense if the returned
function is called from the same thread as the function was generated in. That
might also be worth documenting in the javadoc. The use of the phrase so far
there also threw me off a bit - maybe there is a better phrase for that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2621. Update task InputMetrics increment...
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2087#discussion_r18939679
--- Diff: core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala ---
@@ -222,12 +221,33 @@ class HadoopRDD[K, V](
case eof: EOFException =
finished = true
}
+
+// Update bytes read metric every 32 records
+if (recordsSinceMetricsUpdate == 32
bytesReadCallback.isDefined) {
+ recordsSinceMetricsUpdate = 0
+ inputMetrics.bytesRead = bytesReadCallback.get()
--- End diff --
One thought is to make it two statements:
```
val bytesReadFn = bytesReadCallback.get()
inputBetrics.bytesRead = bytesReadFn()
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2621. Update task InputMetrics increment...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2087#issuecomment-59318880 Yeah you are totally right - the performance bit was not correct from my end. I added some more comments on this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-2621. Update task InputMetrics increment...
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2087#discussion_r18939723
--- Diff:
core/src/test/scala/org/apache/spark/metrics/InputMetricsSuite.scala ---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.metrics
+
+import org.scalatest.FunSuite
+
+import org.apache.spark.SharedSparkContext
+import org.apache.spark.scheduler.{SparkListenerTaskEnd, SparkListener}
+
+import scala.collection.mutable.ArrayBuffer
+
+import java.io.{FileWriter, PrintWriter, File}
+
+class InputMetricsSuite extends FunSuite with SharedSparkContext {
+ test(input metrics when reading text file) {
+val file = new File(getClass.getSimpleName + .txt)
+val pw = new PrintWriter(new FileWriter(file))
+pw.println(some stuff)
+pw.println(some other stuff)
+pw.println(yet more stuff)
+pw.println(too much stuff)
+pw.close()
+file.deleteOnExit()
+
+val taskBytesRead = new ArrayBuffer[Long]()
+sc.addSparkListener(new SparkListener() {
+ override def onTaskEnd(taskEnd: SparkListenerTaskEnd) {
+taskBytesRead += taskEnd.taskMetrics.inputMetrics.get.bytesRead
+ }
+})
+sc.textFile(file:// + file.getAbsolutePath, 2).count()
+
+// Wait for task end events to come in
+Thread.sleep(100)
--- End diff --
Can you use the utility that waits into the listener bus is empty here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3961] Python API for mllib.feature
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2819#issuecomment-59319145 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21790/consoleFull) for PR 2819 at commit [`a405ae7`](https://github.com/apache/spark/commit/a405ae7b967a1a9398e3cdbb812149be7314f29e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class VectorTransformer(object):` * `class Normalizer(VectorTransformer):` * `class JavaModelWrapper(VectorTransformer):` * `class StandardScalerModel(JavaModelWrapper):` * `class StandardScaler(object):` * `class HashingTF(object):` * `class IDFModel(JavaModelWrapper):` * `class IDF(object):` * `class Word2VecModel(JavaModelWrapper):` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3961] Python API for mllib.feature
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2819#issuecomment-59319148 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21790/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3888] [PySpark] limit the memory used b...
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2743#discussion_r18939808
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -65,9 +66,12 @@ private[spark] class PythonRDD(
val localdir = env.blockManager.diskBlockManager.localDirs.map(
f = f.getPath()).mkString(,)
envVars += (SPARK_LOCAL_DIRS - localdir) // it's also used in
monitor thread
-if (reuse_worker) {
+if (reuseWorker) {
envVars += (SPARK_REUSE_WORKER - 1)
}
+if (memoryLimit 0) {
+ envVars += (SPARK_WORKER_MEMORY_LIMIT - memoryLimit.toString)
--- End diff --
It would be better to name this more narrowly PYSPARK_WORKER_MEMORY_LIMIT
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3888] [PySpark] limit the memory used b...
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2743#discussion_r18939813
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -53,7 +53,8 @@ private[spark] class PythonRDD(
extends RDD[Array[Byte]](parent) {
val bufferSize = conf.getInt(spark.buffer.size, 65536)
- val reuse_worker = conf.getBoolean(spark.python.worker.reuse, true)
+ val reuseWorker = conf.getBoolean(spark.python.worker.reuse, true)
+ val memoryLimit = conf.getInt(spark.executor.python.memory.limit, 0)
--- End diff --
Can you document this in the configuration docs?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3888] [PySpark] limit the memory used b...
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2743#discussion_r18939839
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -53,7 +53,8 @@ private[spark] class PythonRDD(
extends RDD[Array[Byte]](parent) {
val bufferSize = conf.getInt(spark.buffer.size, 65536)
- val reuse_worker = conf.getBoolean(spark.python.worker.reuse, true)
+ val reuseWorker = conf.getBoolean(spark.python.worker.reuse, true)
+ val memoryLimit = conf.getInt(spark.executor.python.memory.limit, 0)
--- End diff --
Also - will it be confusing that we require the user to specify the memory
differently than in every other place? We should probably write a python
function for interpreting memory strings (128m, 15g) etc. As it is now the
unit is unspecified, it's also inconsistent with other configs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3939][SQL] NPE caused by SessionState.o...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2812#issuecomment-59319365 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21795/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3939][SQL] NPE caused by SessionState.o...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/2812#issuecomment-59319438 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3812] [BUILD] Adapt maven build to publ...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2673#issuecomment-59319525 What if we just add the maven-shade plug-in but we configure it in a way where it effectively is doing no operations? I.e. we add a relocation of a class that does not exist. Will it still produce an effective pom? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3812] [BUILD] Adapt maven build to publ...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2673#issuecomment-59319582 I think that a solution that doesn't require us using our own plug-in is good, even if we do something silly like we have to shade a class that is in fact un-used. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [HOTFIX][SQL] missing golden file for SPARK-38...
GitHub user adrian-wang opened a pull request: https://github.com/apache/spark/pull/2821 [HOTFIX][SQL] missing golden file for SPARK-3807 create golden file for SPARK-3807 You can merge this pull request into a Git repository by running: $ git pull https://github.com/adrian-wang/spark golden Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2821.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2821 commit ef07c0f3e160edcca280961f1caf62d33dcdc0c1 Author: Daoyuan Wang daoyuan.w...@intel.com Date: 2014-10-16T06:33:16Z missing golden file --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3939][SQL] NPE caused by SessionState.o...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2812#issuecomment-59319859 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21796/consoleFull) for PR 2812 at commit [`e89a7c2`](https://github.com/apache/spark/commit/e89a7c26fd095a1aea380860e0a45f80d5fcbd6b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [HOTFIX][SQL] missing golden file for SPARK-38...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2821#issuecomment-59320188 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21797/consoleFull) for PR 2821 at commit [`ef07c0f`](https://github.com/apache/spark/commit/ef07c0f3e160edcca280961f1caf62d33dcdc0c1). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3966][SQL] Fix nullabilities of Cast re...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2820#issuecomment-59320277 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21792/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3966][SQL] Fix nullabilities of Cast re...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2820#issuecomment-59320272 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21792/consoleFull) for PR 2820 at commit [`ca4a745`](https://github.com/apache/spark/commit/ca4a745977b7315d69c1fd2e9338c264a8ef1bce). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3965: Ensure that Spark assembly for had...
GitHub user dajac opened a pull request: https://github.com/apache/spark/pull/2822 SPARK-3965: Ensure that Spark assembly for hadoop2 contains avro-mapred for hadoop2 When building current Spark assembly for hadoop2, the wrong version of avro-mapred is picked. This patch adds org.apache.avro:avro-mapred as direct dependency of the assembly to ensure that the correct version is added to it. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dajac/spark SPARK-3965 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2822.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2822 commit be820a27d9efab62b70486062db79ebcbdb41301 Author: David Jacot david.ja...@gmail.com Date: 2014-10-16T05:38:05Z SPARK-3965: Ensure that Spark assembly for hadoop2 contains avro-mapred for hadoop2 when assembly is built. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3965: Ensure that Spark assembly for had...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2822#issuecomment-59321034 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3888] [PySpark] limit the memory used b...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2743#issuecomment-59321587 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21798/consoleFull) for PR 2743 at commit [`7df7966`](https://github.com/apache/spark/commit/7df79666852b12ff025b153dd32af9cafe18ec6c). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3888] [PySpark] limit the memory used b...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2743#issuecomment-59321684 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/383/consoleFull) for PR 2743 at commit [`623c8a7`](https://github.com/apache/spark/commit/623c8a76c2e91bd4f80193a0d7c4813d1cb3bc7a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3569][SQL] Add metadata field to Struct...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2701#issuecomment-59322320 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21793/consoleFull) for PR 2701 at commit [`4266f4d`](https://github.com/apache/spark/commit/4266f4dd4df4b006d3a54144558cb92bf46003a7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class AttributeReference(` * `case class StructField(` * `class MetadataBuilder ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3569][SQL] Add metadata field to Struct...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2701#issuecomment-59322329 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21793/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [HOTFIX][SQL] missing golden file for SPARK-38...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2821#issuecomment-59322559 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21797/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [HOTFIX][SQL] missing golden file for SPARK-38...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2821#issuecomment-59322554 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21797/consoleFull) for PR 2821 at commit [`ef07c0f`](https://github.com/apache/spark/commit/ef07c0f3e160edcca280961f1caf62d33dcdc0c1). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3874: Provide stable TaskContext API
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2803#issuecomment-59322774 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21794/consoleFull) for PR 2803 at commit [`56d5b7a`](https://github.com/apache/spark/commit/56d5b7a703afb2529d969ffa0664f6c601186fb0). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `public abstract class TaskContext implements Serializable ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [HOTFIX][SQL] missing golden file for SPARK-38...
Github user adrian-wang commented on the pull request: https://github.com/apache/spark/pull/2821#issuecomment-59322715 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3874: Provide stable TaskContext API
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2803#issuecomment-59322777 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21794/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [HOTFIX][SQL] missing golden file for SPARK-38...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2821#issuecomment-59323161 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21799/consoleFull) for PR 2821 at commit [`ef07c0f`](https://github.com/apache/spark/commit/ef07c0f3e160edcca280961f1caf62d33dcdc0c1). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3939][SQL] NPE caused by SessionState.o...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2812#issuecomment-59323505 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21796/consoleFull) for PR 2812 at commit [`e89a7c2`](https://github.com/apache/spark/commit/e89a7c26fd095a1aea380860e0a45f80d5fcbd6b). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3939][SQL] NPE caused by SessionState.o...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2812#issuecomment-59323510 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21796/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3944][Core] Code re-factored as suggest...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2810#issuecomment-59326697 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21800/consoleFull) for PR 2810 at commit [`051d82f`](https://github.com/apache/spark/commit/051d82f9835fbcc25a08600a11a8923701d13a3e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3888] [PySpark] limit the memory used b...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2743#issuecomment-59327830 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21798/consoleFull) for PR 2743 at commit [`7df7966`](https://github.com/apache/spark/commit/7df79666852b12ff025b153dd32af9cafe18ec6c). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3888] [PySpark] limit the memory used b...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2743#issuecomment-59327837 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21798/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3888] [PySpark] limit the memory used b...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2743#issuecomment-59327910 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/383/consoleFull) for PR 2743 at commit [`623c8a7`](https://github.com/apache/spark/commit/623c8a76c2e91bd4f80193a0d7c4813d1cb3bc7a). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [HOTFIX][SQL] missing golden file for SPARK-38...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2821#issuecomment-59328172 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21799/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [HOTFIX][SQL] missing golden file for SPARK-38...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2821#issuecomment-59328169 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21799/consoleFull) for PR 2821 at commit [`ef07c0f`](https://github.com/apache/spark/commit/ef07c0f3e160edcca280961f1caf62d33dcdc0c1). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3814][SQL] Bitwise does not work in H...
Github user ravipesala commented on the pull request: https://github.com/apache/spark/pull/2789#issuecomment-59329218 Added support for Bitwise AND(), OR(|) ,XOR(^), NOT(~) in this PR only. Please review it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3812] [BUILD] Adapt maven build to publ...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2673#issuecomment-59331286 Even if we do this trick and shade something in all artifacts, what about spark-parent ? There since we don't build jar shading plugin throws NPEs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2706][SQL] Enable Spark to support Hive...
Github user scwf commented on a diff in the pull request:
https://github.com/apache/spark/pull/2241#discussion_r18944242
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala ---
@@ -288,23 +290,26 @@ class HiveContext(sc: SparkContext) extends
SQLContext(sc) {
val cmd_trimmed: String = cmd.trim()
val tokens: Array[String] = cmd_trimmed.split(\\s+)
val cmd_1: String = cmd_trimmed.substring(tokens(0).length()).trim()
- val proc: CommandProcessor = CommandProcessorFactory.get(tokens(0),
hiveconf)
+ val proc: CommandProcessor =
HiveShim.getCommandProcessor(Array(tokens(0)), hiveconf)
proc match {
case driver: Driver =
- driver.init()
-
- val results = new JArrayList[String]
+ val results = HiveShim.createDriverResultsArray
val response: CommandProcessorResponse = driver.run(cmd)
// Throw an exception if there is an error in query processing.
if (response.getResponseCode != 0) {
-driver.destroy()
+driver.close()
throw new QueryExecutionException(response.getErrorMessage)
}
driver.setMaxRows(maxRows)
driver.getResults(results)
- driver.destroy()
- results
+ driver.close()
+ results.map { r =
+r match {
+ case s: String = s
+ case o = o.toString
--- End diff --
Here ```r``` maybe a ```Array```
type(https://github.com/scwf/hive/blob/branch-0.13/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchFormatter.java#L53-L64),
we should cover that case, otherwise this will lead to console result printed
as follows:
``` result
[Object@5e41108b
```
And on the other hand i suggest that we should do some tests with this PR
merged with #2685 to check the basic functionality
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3944][Core] Code re-factored as suggest...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2810#issuecomment-59333977 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21800/consoleFull) for PR 2810 at commit [`051d82f`](https://github.com/apache/spark/commit/051d82f9835fbcc25a08600a11a8923701d13a3e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3944][Core] Code re-factored as suggest...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2810#issuecomment-59333984 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21800/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3945]Properties of hive-site.xml is inv...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2800#issuecomment-59335440 This LGTM, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3945]Properties of hive-site.xml is inv...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2800#issuecomment-59335748 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/384/consoleFull) for PR 2800 at commit [`3679efc`](https://github.com/apache/spark/commit/3679efc5eb6c037769ed4032f0a93d7e56ed9b8c). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Fixes the race condition that may cause ...
GitHub user liancheng opened a pull request: https://github.com/apache/spark/pull/2823 [SQL] Fixes the race condition that may cause test failure The removed `Future` was used to end the test case as soon as the Spark SQL CLI process exits. When the process exits prematurely, this mechanism prevents the test case to wait until timeout. But it also creates a race condition: when `foundAllExpectedAnswers.tryFailure` is called, there are chances that the last expected output line of the CLI process hasn't been caught by the main logics of the test code, thus fails the test case. Removing this `Future` doesn't affect correctness. You can merge this pull request into a Git repository by running: $ git pull https://github.com/liancheng/spark clean-clisuite Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2823.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2823 commit 489a97c3ab0fb034d4a9bff2ce3cb994756021e8 Author: Cheng Lian lian.cs@gmail.com Date: 2014-10-16T09:00:39Z Fixes the race condition that may cause test failure --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Fixes the race condition that may cause ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2823#issuecomment-59341971 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21801/consoleFull) for PR 2823 at commit [`489a97c`](https://github.com/apache/spark/commit/489a97c3ab0fb034d4a9bff2ce3cb994756021e8). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2816#issuecomment-59342873 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21802/consoleFull) for PR 2816 at commit [`037a948`](https://github.com/apache/spark/commit/037a9488ef36bd8971f90556663b31456e628314). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Fixes the race condition that may cause ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2823#issuecomment-59345187 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21801/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SQL] Fixes the race condition that may cause ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2823#issuecomment-59345184 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21801/consoleFull) for PR 2823 at commit [`489a97c`](https://github.com/apache/spark/commit/489a97c3ab0fb034d4a9bff2ce3cb994756021e8). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2816#issuecomment-59345860 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21802/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2816#issuecomment-59345856 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21802/consoleFull) for PR 2816 at commit [`037a948`](https://github.com/apache/spark/commit/037a9488ef36bd8971f90556663b31456e628314). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2816#issuecomment-59346655 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21803/consoleFull) for PR 2816 at commit [`735c1fe`](https://github.com/apache/spark/commit/735c1fed46c14158698d4c2434d7c5f2e3e24213). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3945]Properties of hive-site.xml is inv...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2800#issuecomment-59347603 **[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/384/consoleFull)** for PR 2800 at commit [`3679efc`](https://github.com/apache/spark/commit/3679efc5eb6c037769ed4032f0a93d7e56ed9b8c) after a configured wait of `120m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3945]Properties of hive-site.xml is inv...
Github user luogankun commented on the pull request: https://github.com/apache/spark/pull/2800#issuecomment-59348084 @liancheng Tests timed out! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-1830 Deploy failover, Make Persistence e...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/771#issuecomment-59350133 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21804/consoleFull) for PR 771 at commit [`f05a910`](https://github.com/apache/spark/commit/f05a910f3571a786e57492d3d213ce1229df7c51). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2816#issuecomment-59351094 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21803/consoleFull) for PR 2816 at commit [`735c1fe`](https://github.com/apache/spark/commit/735c1fed46c14158698d4c2434d7c5f2e3e24213). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2816#issuecomment-59351102 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21803/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3945]Properties of hive-site.xml is inv...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2800#issuecomment-59354966 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/385/consoleFull) for PR 2800 at commit [`3679efc`](https://github.com/apache/spark/commit/3679efc5eb6c037769ed4032f0a93d7e56ed9b8c). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2816#discussion_r18952642
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -301,17 +301,67 @@ class SqlParser extends AbstractSparkSQLParser {
CAST ~ ( ~ expression ~ (AS ~ dataType) ~ ) ^^ { case exp ~ t
= Cast(exp, t) }
protected lazy val literal: Parser[Literal] =
-( numericLit ^^ {
-case i if i.toLong Int.MaxValue = Literal(i.toLong)
-case i = Literal(i.toInt)
- }
+( numericLiteral
| NULL ^^^ Literal(null, NullType)
-| floatLit ^^ {case f = Literal(f.toDouble) }
+| booleanLiteral
--- End diff --
Same as above.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2816#discussion_r18952637
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -301,17 +301,67 @@ class SqlParser extends AbstractSparkSQLParser {
CAST ~ ( ~ expression ~ (AS ~ dataType) ~ ) ^^ { case exp ~ t
= Cast(exp, t) }
protected lazy val literal: Parser[Literal] =
-( numericLit ^^ {
-case i if i.toLong Int.MaxValue = Literal(i.toLong)
-case i = Literal(i.toInt)
- }
+( numericLiteral
--- End diff --
Please keep the naming convention consistent, either use `numericLit` or
rename all `xxxLit` to `xxxLiteral`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2816#discussion_r18952768
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -301,17 +301,67 @@ class SqlParser extends AbstractSparkSQLParser {
CAST ~ ( ~ expression ~ (AS ~ dataType) ~ ) ^^ { case exp ~ t
= Cast(exp, t) }
protected lazy val literal: Parser[Literal] =
-( numericLit ^^ {
-case i if i.toLong Int.MaxValue = Literal(i.toLong)
-case i = Literal(i.toInt)
- }
+( numericLiteral
--- End diff --
Sorry, just found out that `numericLit` is a member of `StdTokenParsers`.
Then this naming is alright.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2816#discussion_r18952827
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -301,17 +301,67 @@ class SqlParser extends AbstractSparkSQLParser {
CAST ~ ( ~ expression ~ (AS ~ dataType) ~ ) ^^ { case exp ~ t
= Cast(exp, t) }
protected lazy val literal: Parser[Literal] =
-( numericLit ^^ {
-case i if i.toLong Int.MaxValue = Literal(i.toLong)
-case i = Literal(i.toInt)
- }
+( numericLiteral
| NULL ^^^ Literal(null, NullType)
-| floatLit ^^ {case f = Literal(f.toDouble) }
+| booleanLiteral
| stringLit ^^ {case s = Literal(s, StringType) }
)
+ protected lazy val booleanLiteral: Parser[Literal] =
+( TRUE ^^^ Literal(true, BooleanType)
+| FALSE ^^^ Literal(false, BooleanType)
+)
+
+ protected lazy val numericLiteral: Parser[Literal] =
+signedNumericLiteral | unsignedNumericLiteral | floatLit ^^ { f =
Literal(f.toDouble) }
+
+ protected lazy val sign: Parser[String] =
++ | -
+
+ protected lazy val signedNumericLiteral: Parser[Literal] =
+sign ~ numericLit ^^ { case s ~ l = Literal(getProperTypeInt(s + l))
}
--- End diff --
Sign should be optional:
```scala
sign.? ~ numericLit ^^
{ case s ~ l = Literal(getProperTypeInt(s.getOrElse() + l)) }
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2816#discussion_r18952854
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -301,17 +301,67 @@ class SqlParser extends AbstractSparkSQLParser {
CAST ~ ( ~ expression ~ (AS ~ dataType) ~ ) ^^ { case exp ~ t
= Cast(exp, t) }
protected lazy val literal: Parser[Literal] =
-( numericLit ^^ {
-case i if i.toLong Int.MaxValue = Literal(i.toLong)
-case i = Literal(i.toInt)
- }
+( numericLiteral
| NULL ^^^ Literal(null, NullType)
-| floatLit ^^ {case f = Literal(f.toDouble) }
+| booleanLiteral
| stringLit ^^ {case s = Literal(s, StringType) }
)
+ protected lazy val booleanLiteral: Parser[Literal] =
+( TRUE ^^^ Literal(true, BooleanType)
+| FALSE ^^^ Literal(false, BooleanType)
+)
+
+ protected lazy val numericLiteral: Parser[Literal] =
+signedNumericLiteral | unsignedNumericLiteral | floatLit ^^ { f =
Literal(f.toDouble) }
+
+ protected lazy val sign: Parser[String] =
++ | -
+
+ protected lazy val signedNumericLiteral: Parser[Literal] =
+sign ~ numericLit ^^ { case s ~ l = Literal(getProperTypeInt(s + l))
}
+
+
+ protected lazy val unsignedNumericLiteral: Parser[Literal] =
+numericLit ^^ { n = Literal(getProperTypeInt(n)) }
+
+
+ private val longMax = BigDecimal(s${Long.MaxValue})
+ private val longMin = BigDecimal(s${Long.MinValue})
+ private val intMax = BigDecimal(s${Int.MaxValue})
+ private val intMin = BigDecimal(s${Int.MinValue})
+
+ private def getProperTypeInt(value: String) = {
+val bigIntValue = BigDecimal(value)
+
+if (value.startsWith(-)) {
+ if (bigIntValue intMin) {
+if (bigIntValue longMin) {
+ bigIntValue
+} else {
+ value.toLong
+}
+ } else {
+value.toInt
+ }
+} else {
+ if (bigIntValue intMax) {
+if (bigIntValue longMax) {
+ bigIntValue
+} else {
+ value.toLong
+}
+ } else {
+value.toInt
+ }
+}
+ }
+
protected lazy val floatLit: Parser[String] =
-elem(decimal, _.isInstanceOf[lexical.FloatLit]) ^^ (_.chars)
+( opt(sign) ~ . ~ unsignedNumericLiteral ^^ {
--- End diff --
I'd prefer the `sign.?` syntax here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2816#discussion_r18953268
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -323,11 +373,11 @@ class SqlParser extends AbstractSparkSQLParser {
| cast
| ( ~ expression ~ )
| function
-| - ~ literal ^^ UnaryMinus
| dotExpressionHeader
| ident ^^ UnresolvedAttribute
| * ^^^ Star(None)
| literal
+| sign ~ expression ^^ { case s ~ e = if (s == -) UnaryMinus(e)
else e }
--- End diff --
Sign should be optional. Also I think it would be better to move this rule
to the last alternative of `comparisonExpression`, namely from:
```scala
| termExpression
```
to
```scala
| sign.? ~ termExpression ^^ {
case Some(-) ~ e = UnaryMinus(e)
case _ = e
}
```
The reason is that the type of `termExpression` is guaranteed to be numeric.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-1830 Deploy failover, Make Persistence e...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/771#issuecomment-59358046 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21804/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2816#discussion_r18953527
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -301,17 +301,67 @@ class SqlParser extends AbstractSparkSQLParser {
CAST ~ ( ~ expression ~ (AS ~ dataType) ~ ) ^^ { case exp ~ t
= Cast(exp, t) }
protected lazy val literal: Parser[Literal] =
-( numericLit ^^ {
-case i if i.toLong Int.MaxValue = Literal(i.toLong)
-case i = Literal(i.toInt)
- }
+( numericLiteral
| NULL ^^^ Literal(null, NullType)
-| floatLit ^^ {case f = Literal(f.toDouble) }
+| booleanLiteral
| stringLit ^^ {case s = Literal(s, StringType) }
)
+ protected lazy val booleanLiteral: Parser[Literal] =
+( TRUE ^^^ Literal(true, BooleanType)
+| FALSE ^^^ Literal(false, BooleanType)
+)
+
+ protected lazy val numericLiteral: Parser[Literal] =
+signedNumericLiteral | unsignedNumericLiteral | floatLit ^^ { f =
Literal(f.toDouble) }
+
+ protected lazy val sign: Parser[String] =
++ | -
+
+ protected lazy val signedNumericLiteral: Parser[Literal] =
+sign ~ numericLit ^^ { case s ~ l = Literal(getProperTypeInt(s + l))
}
+
+
+ protected lazy val unsignedNumericLiteral: Parser[Literal] =
+numericLit ^^ { n = Literal(getProperTypeInt(n)) }
+
+
+ private val longMax = BigDecimal(s${Long.MaxValue})
+ private val longMin = BigDecimal(s${Long.MinValue})
+ private val intMax = BigDecimal(s${Int.MaxValue})
+ private val intMin = BigDecimal(s${Int.MinValue})
+
+ private def getProperTypeInt(value: String) = {
--- End diff --
How about `toNarrowestNumericType`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-1830 Deploy failover, Make Persistence e...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/771#issuecomment-59358035 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21804/consoleFull) for PR 771 at commit [`f05a910`](https://github.com/apache/spark/commit/f05a910f3571a786e57492d3d213ce1229df7c51). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class ApplicationInfo(` * `class DriverInfo(` * `trait LeaderElectionAgent ` * `trait LeaderElectable ` * `trait PersistenceEngine ` * `abstract class StandaloneRecoveryModeFactory(conf: SparkConf) ` * `class WorkerInfo(` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2816#discussion_r18953740
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -301,17 +301,67 @@ class SqlParser extends AbstractSparkSQLParser {
CAST ~ ( ~ expression ~ (AS ~ dataType) ~ ) ^^ { case exp ~ t
= Cast(exp, t) }
protected lazy val literal: Parser[Literal] =
-( numericLit ^^ {
-case i if i.toLong Int.MaxValue = Literal(i.toLong)
-case i = Literal(i.toInt)
- }
+( numericLiteral
| NULL ^^^ Literal(null, NullType)
-| floatLit ^^ {case f = Literal(f.toDouble) }
+| booleanLiteral
| stringLit ^^ {case s = Literal(s, StringType) }
)
+ protected lazy val booleanLiteral: Parser[Literal] =
+( TRUE ^^^ Literal(true, BooleanType)
+| FALSE ^^^ Literal(false, BooleanType)
+)
+
+ protected lazy val numericLiteral: Parser[Literal] =
+signedNumericLiteral | unsignedNumericLiteral | floatLit ^^ { f =
Literal(f.toDouble) }
+
+ protected lazy val sign: Parser[String] =
++ | -
+
+ protected lazy val signedNumericLiteral: Parser[Literal] =
+sign ~ numericLit ^^ { case s ~ l = Literal(getProperTypeInt(s + l))
}
+
+
+ protected lazy val unsignedNumericLiteral: Parser[Literal] =
+numericLit ^^ { n = Literal(getProperTypeInt(n)) }
+
+
+ private val longMax = BigDecimal(s${Long.MaxValue})
+ private val longMin = BigDecimal(s${Long.MinValue})
+ private val intMax = BigDecimal(s${Int.MaxValue})
+ private val intMin = BigDecimal(s${Int.MinValue})
+
+ private def getProperTypeInt(value: String) = {
+val bigIntValue = BigDecimal(value)
+
+if (value.startsWith(-)) {
+ if (bigIntValue intMin) {
+if (bigIntValue longMin) {
+ bigIntValue
+} else {
+ value.toLong
+}
+ } else {
+value.toInt
+ }
+} else {
+ if (bigIntValue intMax) {
+if (bigIntValue longMax) {
+ bigIntValue
+} else {
+ value.toLong
+}
+ } else {
+value.toInt
+ }
+}
--- End diff --
How about this:
```scala
bigIntValue match {
case v if v longMin || v longMax = v
case v if v intMin || v intMax = v.toLong
case v = v.toInt
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/2816#discussion_r18953909
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala ---
@@ -301,17 +301,67 @@ class SqlParser extends AbstractSparkSQLParser {
CAST ~ ( ~ expression ~ (AS ~ dataType) ~ ) ^^ { case exp ~ t
= Cast(exp, t) }
protected lazy val literal: Parser[Literal] =
-( numericLit ^^ {
-case i if i.toLong Int.MaxValue = Literal(i.toLong)
-case i = Literal(i.toInt)
- }
+( numericLiteral
| NULL ^^^ Literal(null, NullType)
-| floatLit ^^ {case f = Literal(f.toDouble) }
+| booleanLiteral
| stringLit ^^ {case s = Literal(s, StringType) }
)
+ protected lazy val booleanLiteral: Parser[Literal] =
+( TRUE ^^^ Literal(true, BooleanType)
+| FALSE ^^^ Literal(false, BooleanType)
+)
+
+ protected lazy val numericLiteral: Parser[Literal] =
+signedNumericLiteral | unsignedNumericLiteral | floatLit ^^ { f =
Literal(f.toDouble) }
+
+ protected lazy val sign: Parser[String] =
++ | -
+
+ protected lazy val signedNumericLiteral: Parser[Literal] =
+sign ~ numericLit ^^ { case s ~ l = Literal(getProperTypeInt(s + l))
}
--- End diff --
Ah, I see, you made signed and unsigned numerics into two branches.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3629][Doc] improve spark on yarn doc
Github user ssjssh commented on the pull request: https://github.com/apache/spark/pull/2813#issuecomment-59359408 can someone verify this patchï¼ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3945]Properties of hive-site.xml is inv...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2800#issuecomment-59361067 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/385/consoleFull) for PR 2800 at commit [`3679efc`](https://github.com/apache/spark/commit/3679efc5eb6c037769ed4032f0a93d7e56ed9b8c). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3948][Shuffle]Fix stream corruption bug...
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/2824 [SPARK-3948][Shuffle]Fix stream corruption bug in sort-based shuffle Kernel 2.6.32 bug will lead to unexpected behavior of transferTo in copyStream, and this will corrupt the shuffle output file in sort-based shuffle, which will somehow introduce PARSING_ERROR(2), deserialization error or offset out of range. Here fix this by adding append flag, also add some position checking code. Details can be seen in [SPARK-3948](https://issues.apache.org/jira/browse/SPARK-3948). You can merge this pull request into a Git repository by running: $ git pull https://github.com/jerryshao/apache-spark SPARK-3948 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2824.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2824 commit b47cc9f8f32e01868288ec82eedc1f421a717a8b Author: jerryshao saisai.s...@intel.com Date: 2014-10-16T08:45:51Z Fix kernel 2.6.32 bug led unexpected behavior of transferTo --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3948][Shuffle]Fix stream corruption bug...
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/2824#discussion_r18956465
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -280,12 +280,29 @@ private[spark] object Utils extends Logging {
// When both streams are File stream, use transferTo to improve
copy performance.
val inChannel = in.asInstanceOf[FileInputStream].getChannel()
val outChannel = out.asInstanceOf[FileOutputStream].getChannel()
+val initialPos = outChannel.position()
val size = inChannel.size()
// In case transferTo method transferred less data than we have
required.
while (count size) {
count += inChannel.transferTo(count, size - count, outChannel)
}
+
+// Check the position after transferTo loop to see if it is in the
right position and
+// give user information if not.
+// Position will not be increased to the expected length after
calling transferTo in
+// kernel version 2.6.32, this issue can be seen in
+// scalastyle:off
+//
https://bugs.openjdk.java.net/browse/JDK-7052359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel)
--- End diff --
en...I guess this line will trigger scalastyle checker error
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-1209 [CORE] SparkHadoop{MapRed,MapReduce...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2814#issuecomment-59365761 I addressed the MIMA warnings, two of which were just caused by the class being renamed. I'm less sure about the warning around `PairRDDFunctions` since `SparkHadoopMapReduceUtil` was included as a superclass (trait). Unfortunately. Right? it really isn't a case for inheritance. So, I am not 100% clear what the implications of a rename are for binary compatibility. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-1209 [CORE] SparkHadoop{MapRed,MapReduce...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2814#issuecomment-59366632 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21806/consoleFull) for PR 2814 at commit [`821d12a`](https://github.com/apache/spark/commit/821d12a0a2dbf66ea3f9fac552bc1a7caab92207). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3948][Shuffle]Fix stream corruption bug...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/2824#discussion_r18957257
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -280,12 +280,29 @@ private[spark] object Utils extends Logging {
// When both streams are File stream, use transferTo to improve
copy performance.
val inChannel = in.asInstanceOf[FileInputStream].getChannel()
val outChannel = out.asInstanceOf[FileOutputStream].getChannel()
+val initialPos = outChannel.position()
val size = inChannel.size()
// In case transferTo method transferred less data than we have
required.
while (count size) {
count += inChannel.transferTo(count, size - count, outChannel)
}
+
+// Check the position after transferTo loop to see if it is in the
right position and
+// give user information if not.
+// Position will not be increased to the expected length after
calling transferTo in
+// kernel version 2.6.32, this issue can be seen in
+// scalastyle:off
+//
https://bugs.openjdk.java.net/browse/JDK-7052359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel)
--- End diff --
I'm not sure, I found some code in KafkaUtils also use this
`scalastyle:off` to turn off scalacheck.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3948][Shuffle]Fix stream corruption bug...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2824#issuecomment-59367830 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21805/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3948][Shuffle]Fix stream corruption bug...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2824#issuecomment-59368192 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21807/consoleFull) for PR 2824 at commit [`e17ada2`](https://github.com/apache/spark/commit/e17ada2629b7ac0cc6e888cd1a0da6827159ea3b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3948][Shuffle]Fix stream corruption bug...
Github user CodingCat commented on a diff in the pull request:
https://github.com/apache/spark/pull/2824#discussion_r18957913
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -280,12 +280,29 @@ private[spark] object Utils extends Logging {
// When both streams are File stream, use transferTo to improve
copy performance.
val inChannel = in.asInstanceOf[FileInputStream].getChannel()
val outChannel = out.asInstanceOf[FileOutputStream].getChannel()
+val initialPos = outChannel.position()
val size = inChannel.size()
// In case transferTo method transferred less data than we have
required.
while (count size) {
count += inChannel.transferTo(count, size - count, outChannel)
}
+
+// Check the position after transferTo loop to see if it is in the
right position and
+// give user information if not.
+// Position will not be increased to the expected length after
calling transferTo in
+// kernel version 2.6.32, this issue can be seen in
+// scalastyle:off
+//
https://bugs.openjdk.java.net/browse/JDK-7052359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel)
--- End diff --
I see, I didn't notice that line, I think that shall be good
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [spark-3907][sql] add truncate table support
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2770#issuecomment-59369912 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/387/consoleFull) for PR 2770 at commit [`e483547`](https://github.com/apache/spark/commit/e48354730645951e2fe2c485cab310df8033bf72). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [spark-3907][sql] add truncate table support
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2770#issuecomment-59375645 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/387/consoleFull) for PR 2770 at commit [`e483547`](https://github.com/apache/spark/commit/e48354730645951e2fe2c485cab310df8033bf72). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3969 Optimizer should have a super class...
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/2825 SPARK-3969 Optimizer should have a super class as an interface. Some developers want to replace `Optimizer` to fit their projects but can't do so because currently `Optimizer` is an `object`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ueshin/apache-spark issues/SPARK-3969 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2825.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2825 commit 9547a23fd5ca0058d1044d19f6a96bdbb1e3b810 Author: Takuya UESHIN ues...@happy-camper.st Date: 2014-10-16T10:52:19Z Extract abstract class from Optimizer for developers to be able to replace Optimizer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: Remove duplicate removal of local dirs
GitHub user viirya opened a pull request: https://github.com/apache/spark/pull/2826 Remove duplicate removal of local dirs The shutdown hook of `DiskBlockManager` would remove localDirs. So do not need to register them with `Utils.registerShutdownDeleteDir`. It causes duplicate removal of these local dirs and corresponding exceptions. You can merge this pull request into a Git repository by running: $ git pull https://github.com/viirya/spark-1 fix_duplicate_localdir_remove Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2826.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2826 commit 2b91a9c6e93c1dec90ee079558cb3a291e3065de Author: Liang-Chi Hsieh vii...@gmail.com Date: 2014-10-16T15:17:38Z remove duplicate removal of local dirs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3969][SQL] Optimizer should have a supe...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2825#issuecomment-59379459 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21808/consoleFull) for PR 2825 at commit [`9547a23`](https://github.com/apache/spark/commit/9547a23fd5ca0058d1044d19f6a96bdbb1e3b810). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3970] Remove duplicate removal of local...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2826#issuecomment-59379911 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3948][Shuffle]Fix stream corruption bug...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2824#issuecomment-59380188 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21807/consoleFull) for PR 2824 at commit [`e17ada2`](https://github.com/apache/spark/commit/e17ada2629b7ac0cc6e888cd1a0da6827159ea3b). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3948][Shuffle]Fix stream corruption bug...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2824#issuecomment-59380201 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21807/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-1209 [CORE] SparkHadoop{MapRed,MapReduce...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2814#issuecomment-59381926 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21806/consoleFull) for PR 2814 at commit [`821d12a`](https://github.com/apache/spark/commit/821d12a0a2dbf66ea3f9fac552bc1a7caab92207). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-1209 [CORE] SparkHadoop{MapRed,MapReduce...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2814#issuecomment-59381939 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21806/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
