[GitHub] spark issue #20002: [SPARK-22465][Core][WIP] Add a safety-check to RDD defau...

[sujithjay]

Github user sujithjay commented on the issue:

https://github.com/apache/spark/pull/20002
  
Thank you, @HyukjinKwon . The tests passed after rebasing.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18812: [SPARK-21606][SQL]HiveThriftServer2 catches OOMs on requ...

[liufengdb]

Github user liufengdb commented on the issue:

https://github.com/apache/spark/pull/18812
  
I actually did not get the motivation of this PR. HiveThriftServer2 can run 
independently or be started with a SQL context: 
https://github.com/apache/spark/pull/18812/files#diff-709404b0d3defeff035ef0c4f5a960e5L57.
 For the later, this OOM should be handled by the upper layer, right?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18812: [SPARK-21606][SQL]HiveThriftServer2 catches OOMs ...

[liufengdb]

Github user liufengdb commented on a diff in the pull request:

https://github.com/apache/spark/pull/18812#discussion_r157361046
  
--- Diff: 
sql/hive-thriftserver/src/main/java/org/apache/hive/service/server/HiveServer2.java
 ---
@@ -39,6 +32,8 @@
 import org.apache.hive.service.cli.thrift.ThriftCLIService;
 import org.apache.hive.service.cli.thrift.ThriftHttpCLIService;
 
+import java.util.Properties;
--- End diff --

java imports should be the first?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19977: [SPARK-22771][SQL] Concatenate binary inputs into...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/19977#discussion_r157361039
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
 ---
@@ -50,15 +51,23 @@ import org.apache.spark.unsafe.types.{ByteArray, 
UTF8String}
   """)
 case class Concat(children: Seq[Expression]) extends Expression with 
ImplicitCastInputTypes {
 
-  override def inputTypes: Seq[AbstractDataType] = 
Seq.fill(children.size)(StringType)
-  override def dataType: DataType = StringType
+  private lazy val isBinaryMode = children.nonEmpty && 
children.forall(_.dataType == BinaryType)
--- End diff --

Thank you for confirming it! Below is the behavior of DB2


https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.1.0/com.ibm.db2.luw.sql.ref.doc/doc/r781.html



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18812: [SPARK-21606][SQL]HiveThriftServer2 catches OOMs ...

[liufengdb]

Github user liufengdb commented on a diff in the pull request:

https://github.com/apache/spark/pull/18812#discussion_r157361025
  
--- Diff: 
sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java
 ---
@@ -37,21 +30,29 @@
 import org.apache.thrift.transport.TServerSocket;
 import org.apache.thrift.transport.TTransportFactory;
 
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.SynchronousQueue;
+import java.util.concurrent.TimeUnit;
--- End diff --

unnecessary change


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19989: [SPARK-22793][SQL]Memory leak in Spark Thrift Server

[liufengdb]

Github user liufengdb commented on the issue:

https://github.com/apache/spark/pull/19989
  
I think this method can take care of resource clean up automatically: 
https://github.com/apache/spark/blob/master/sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/SessionManager.java#L151

Can you really make a heap dump and find out why the sessions are not 
cleaned up? 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19149: [SPARK-21652][SQL][FOLLOW-UP] Fix rule conflict caused b...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19149
  
**[Test build #85015 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85015/testReport)**
 for PR 19149 at commit 
[`9b6fe36`](https://github.com/apache/spark/commit/9b6fe3644c3fc760de9ddf33c16bc1733e7c971d).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19989: [SPARK-22793][SQL]Memory leak in Spark Thrift Server

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/19989
  
cc @liufengdb 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20001: [SPARK-22816][TEST] Basic tests for PromoteString...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/20001#discussion_r157358754
  
--- Diff: 
sql/core/src/test/resources/sql-tests/inputs/typeCoercion/native/inConversion.sql
 ---
@@ -0,0 +1,245 @@
+--
+--   Licensed to the Apache Software Foundation (ASF) under one or more
+--   contributor license agreements.  See the NOTICE file distributed with
+--   this work for additional information regarding copyright ownership.
+--   The ASF licenses this file to You under the Apache License, Version 
2.0
+--   (the "License"); you may not use this file except in compliance with
+--   the License.  You may obtain a copy of the License at
+--
+--  http://www.apache.org/licenses/LICENSE-2.0
+--
+--   Unless required by applicable law or agreed to in writing, software
+--   distributed under the License is distributed on an "AS IS" BASIS,
+--   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 
implied.
+--   See the License for the specific language governing permissions and
+--   limitations under the License.
+--
+
+CREATE TEMPORARY VIEW t AS SELECT 1;
+
+SELECT cast(1 as tinyint) in (cast(1 as tinyint)) FROM t;
+SELECT cast(1 as tinyint) in (cast(1 as smallint)) FROM t;
+SELECT cast(1 as tinyint) in (cast(1 as int)) FROM t;
+SELECT cast(1 as tinyint) in (cast(1 as bigint)) FROM t;
+SELECT cast(1 as tinyint) in (cast(1 as float)) FROM t;
+SELECT cast(1 as tinyint) in (cast(1 as double)) FROM t;
+SELECT cast(1 as tinyint) in (cast(1 as decimal(10, 0))) FROM t;
+SELECT cast(1 as tinyint) in (cast(1 as string)) FROM t;
+-- Hive and Spark SQL doesn't support below operations:
+-- SELECT cast(1 as tinyint) in (cast('1' as binary)) FROM t;
+-- SELECT cast(1 as tinyint) in (cast(1 as boolean)) FROM t;
+-- SELECT cast(1 as tinyint) in (cast('2017-12-11 09:30:00.0' as 
timestamp)) FROM t;
+-- SELECT cast(1 as tinyint) in (cast('2017-12-11 09:30:00' as date)) FROM 
t;
+
+SELECT cast(1 as smallint) in (cast(1 as tinyint)) FROM t;
+SELECT cast(1 as smallint) in (cast(1 as smallint)) FROM t;
+SELECT cast(1 as smallint) in (cast(1 as int)) FROM t;
+SELECT cast(1 as smallint) in (cast(1 as bigint)) FROM t;
+SELECT cast(1 as smallint) in (cast(1 as float)) FROM t;
+SELECT cast(1 as smallint) in (cast(1 as double)) FROM t;
+SELECT cast(1 as smallint) in (cast(1 as decimal(10, 0))) FROM t;
+SELECT cast(1 as smallint) in (cast(1 as string)) FROM t;
+
+SELECT cast(1 as int) in (cast(1 as tinyint)) FROM t;
+SELECT cast(1 as int) in (cast(1 as smallint)) FROM t;
+SELECT cast(1 as int) in (cast(1 as int)) FROM t;
+SELECT cast(1 as int) in (cast(1 as bigint)) FROM t;
+SELECT cast(1 as int) in (cast(1 as float)) FROM t;
+SELECT cast(1 as int) in (cast(1 as double)) FROM t;
+SELECT cast(1 as int) in (cast(1 as decimal(10, 0))) FROM t;
+SELECT cast(1 as int) in (cast(1 as string)) FROM t;
+
+SELECT cast(1 as bigint) in (cast(1 as tinyint)) FROM t;
+SELECT cast(1 as bigint) in (cast(1 as smallint)) FROM t;
+SELECT cast(1 as bigint) in (cast(1 as int)) FROM t;
+SELECT cast(1 as bigint) in (cast(1 as bigint)) FROM t;
+SELECT cast(1 as bigint) in (cast(1 as float)) FROM t;
+SELECT cast(1 as bigint) in (cast(1 as double)) FROM t;
+SELECT cast(1 as bigint) in (cast(1 as decimal(10, 0))) FROM t;
+SELECT cast(1 as bigint) in (cast(1 as string)) FROM t;
+
+SELECT cast(1 as float) in (cast(1 as tinyint)) FROM t;
+SELECT cast(1 as float) in (cast(1 as smallint)) FROM t;
+SELECT cast(1 as float) in (cast(1 as int)) FROM t;
+SELECT cast(1 as float) in (cast(1 as bigint)) FROM t;
+SELECT cast(1 as float) in (cast(1 as float)) FROM t;
+SELECT cast(1 as float) in (cast(1 as double)) FROM t;
+SELECT cast(1 as float) in (cast(1 as decimal(10, 0))) FROM t;
+SELECT cast(1 as float) in (cast(1 as string)) FROM t;
+
+SELECT cast(1 as double) in (cast(1 as tinyint)) FROM t;
+SELECT cast(1 as double) in (cast(1 as smallint)) FROM t;
+SELECT cast(1 as double) in (cast(1 as int)) FROM t;
+SELECT cast(1 as double) in (cast(1 as bigint)) FROM t;
+SELECT cast(1 as double) in (cast(1 as float)) FROM t;
+SELECT cast(1 as double) in (cast(1 as double)) FROM t;
+SELECT cast(1 as double) in (cast(1 as decimal(10, 0))) FROM t;
+SELECT cast(1 as double) in (cast(1 as string)) FROM t;
+
+SELECT cast(1 as decimal(10, 0)) in (cast(1 as tinyint)) FROM t;
+SELECT cast(1 as decimal(10, 0)) in (cast(1 as smallint)) FROM t;
+SELECT cast(1 as decimal(10, 0)) in (cast(1 as int)) FROM t;
+SELECT cast(1 as decimal(10, 0)) in (cast(1 as bigint)) FROM t;
+SELECT cast(1 as decimal(10, 0)) in (cast(1 as float)) FROM t;
+SELECT cast(1 as decimal(10, 0)) in (cast(1 as double)) FROM 

[GitHub] spark issue #20000: [SPARK-22815] [SQL] Keep PromotePrecision in Optimized P...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/2
  
cc @cloud-fan 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20000: [SPARK-22815] [SQL] Keep PromotePrecision in Optimized P...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/2
  
**[Test build #85014 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85014/testReport)**
 for PR 2 at commit 
[`571ed44`](https://github.com/apache/spark/commit/571ed448b1ecb4df3d185e1c4b5390c1b9206de0).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19995: [SPARK-22807] [Scheduler] Remove config that says docker...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19995
  
FYI, AppVeyor failure is fixed separately in #20003.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20003: [SPARK-22817][R] Use fixed testthat version for S...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/20003


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20002: [SPARK-22465][Core][WIP] Add a safety-check to RDD defau...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20002
  
It was fixed in https://github.com/apache/spark/pull/20003. Rebasing should 
make the test apssed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20003: [SPARK-22817][R] Use fixed testthat version for SparkR t...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20003
  
Merged to master and branch-2.2.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20003: [SPARK-22817][R] Use fixed testthat version for SparkR t...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20003
  
I am also backporting this to branch-2.2 and branch-2.1 too. Currently, we 
don't run AppVeyor on those branches so it's fine but we will potentially meet 
this test failure if we start to run.

Also, I remember I manually run this before each release. It should be 
useful to backport.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20003: [SPARK-22817][R] Use fixed testthat version for SparkR t...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20003
  
I am going to merge this to resolve unrelated AppVeyor test failures in 
other PRs, and as CRAN check failure in Jenkins should not be related with it.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19954: [SPARK-22757][Kubernetes] Enable use of remote dependenc...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19954
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19954: [SPARK-22757][Kubernetes] Enable use of remote dependenc...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19954
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85013/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19954: [SPARK-22757][Kubernetes] Enable use of remote dependenc...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19954
  
**[Test build #85013 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85013/testReport)**
 for PR 19954 at commit 
[`197882d`](https://github.com/apache/spark/commit/197882d787b602b80b0a9b500133463a95a9a691).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19992: [SPARK-22805][CORE] Use StorageLevel aliases in event lo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19992
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19992: [SPARK-22805][CORE] Use StorageLevel aliases in event lo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19992
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85012/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19992: [SPARK-22805][CORE] Use StorageLevel aliases in event lo...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19992
  
**[Test build #85012 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85012/testReport)**
 for PR 19992 at commit 
[`e171f03`](https://github.com/apache/spark/commit/e171f03c27a11d0820f2ade22dc081005dc039a0).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19992: [SPARK-22805][CORE] Use StorageLevel aliases in event lo...

[superbobry]

Github user superbobry commented on the issue:

https://github.com/apache/spark/pull/19992
  
Minor update: I've simulated #18162 on one of our 80G event logs and 
(unless there is a bug in the filtering code) the log shrank to 157M. The 
effect of this patch was almost negligible, it brought the size down to 155M. 
It is unclear for now if this generalizes to other workloads.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17146: [SPARK-19806][ML][PySpark] PySpark GeneralizedLinearRegr...

[yanboliang]

Github user yanboliang commented on the issue:

https://github.com/apache/spark/pull/17146
  
@Antoinelypro Sorry for late response. Actually we have default value if 
users don't set _link_ explicitly. Could you show the detail of your error 
case? Thanks.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19954: [SPARK-22757][Kubernetes] Enable use of remote dependenc...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19954
  
**[Test build #85013 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85013/testReport)**
 for PR 19954 at commit 
[`197882d`](https://github.com/apache/spark/commit/197882d787b602b80b0a9b500133463a95a9a691).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19954: [SPARK-22757][Kubernetes] Enable use of remote de...

[liyinan926]

Github user liyinan926 commented on a diff in the pull request:

https://github.com/apache/spark/pull/19954#discussion_r157351657
  
--- Diff: 
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/rest/k8s/KubernetesSparkDependencyDownloadInitContainer.scala
 ---
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.deploy.rest.k8s
+
+import java.io.File
+import java.util.concurrent.TimeUnit
+
+import scala.concurrent.{ExecutionContext, Future}
+import scala.concurrent.duration.Duration
+
+import org.apache.spark.{SecurityManager => SparkSecurityManager, 
SparkConf}
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.deploy.k8s.Config._
+import org.apache.spark.internal.Logging
+import org.apache.spark.util.{ThreadUtils, Utils}
+
+/**
+ * Process that fetches files from a resource staging server and/or 
arbitrary remote locations.
+ *
+ * The init-container can handle fetching files from any of those sources, 
but not all of the
+ * sources need to be specified. This allows for composing multiple 
instances of this container
+ * with different configurations for different download sources, or using 
the same container to
+ * download everything at once.
+ */
+private[spark] class KubernetesSparkDependencyDownloadInitContainer(
+sparkConf: SparkConf,
+fileFetcher: FileFetcher) extends Logging {
+
+  private implicit val downloadExecutor = 
ExecutionContext.fromExecutorService(
+ThreadUtils.newDaemonCachedThreadPool("download-executor"))
+
+  private val jarsDownloadDir = new File(
+sparkConf.get(JARS_DOWNLOAD_LOCATION))
+  private val filesDownloadDir = new File(
+sparkConf.get(FILES_DOWNLOAD_LOCATION))
+
+  private val remoteJars = sparkConf.get(INIT_CONTAINER_REMOTE_JARS)
+  private val remoteFiles = sparkConf.get(INIT_CONTAINER_REMOTE_FILES)
+
+  private val downloadTimeoutMinutes = 
sparkConf.get(INIT_CONTAINER_MOUNT_TIMEOUT)
+
+  def run(): Unit = {
+val remoteJarsDownload = Future[Unit] {
+  logInfo(s"Downloading remote jars: $remoteJars")
+  downloadFiles(
+remoteJars,
+jarsDownloadDir,
+s"Remote jars download directory specified at $jarsDownloadDir 
does not exist " +
+  "or is not a directory.")
+}
+val remoteFilesDownload = Future[Unit] {
+  logInfo(s"Downloading remote files: $remoteFiles")
+  downloadFiles(
+remoteFiles,
+filesDownloadDir,
+s"Remote files download directory specified at $filesDownloadDir 
does not exist " +
+  "or is not a directory.")
+}
+waitForFutures(
--- End diff --

Updated to create one task per file/jar to download. Regarding the type of 
thread pool, we are using a `CachedThreadPool`, which I think makes sense as it 
can be expected that the tasks are not long-lived.  


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19959: [SPARK-22766] Install R linter package in spark l...

[falaki]

Github user falaki commented on a diff in the pull request:

https://github.com/apache/spark/pull/19959#discussion_r157351383
  
--- Diff: dev/lint-r.R ---
@@ -27,10 +27,11 @@ if (! library(SparkR, lib.loc = LOCAL_LIB_LOC, 
logical.return = TRUE)) {
 # Installs lintr from Github in a local directory.
 # NOTE: The CRAN's version is too old to adapt to our rules.
 if ("lintr" %in% row.names(installed.packages()) == FALSE) {
--- End diff --

I can see two possibilities. Unfortunately, I cannot get access to machines 
to verify:
1. There is already an older version so `("lintr" %in% 
row.names(installed.packages())` returns 'TRUE`
2. When installing `lintr` in a new directory some transient dependency 
check behaves different and we end up rejecting existing `Rcpp` version.

Why do you think re-installing `Rcpp` in a non-system directory is an 
issue? This is under `$SPARK_HOME` and will be wiped with every new test.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19985: [SPARK-22791] [SQL] [SS] Redact Output of Explain

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19985
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19985: [SPARK-22791] [SQL] [SS] Redact Output of Explain

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19985
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85011/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19985: [SPARK-22791] [SQL] [SS] Redact Output of Explain

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19985
  
**[Test build #85011 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85011/testReport)**
 for PR 19985 at commit 
[`e246096`](https://github.com/apache/spark/commit/e246096cab0b0905d7952556ba3e8ecd15c0a2a5).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19992: [SPARK-22805][CORE] Use StorageLevel aliases in event lo...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19992
  
**[Test build #85012 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85012/testReport)**
 for PR 19992 at commit 
[`e171f03`](https://github.com/apache/spark/commit/e171f03c27a11d0820f2ade22dc081005dc039a0).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20003: [SPARK-22817][R] Use fixed testthat version for SparkR t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20003
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85010/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20003: [SPARK-22817][R] Use fixed testthat version for SparkR t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20003
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20003: [SPARK-22817][R] Use fixed testthat version for SparkR t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20003
  
**[Test build #85010 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85010/testReport)**
 for PR 20003 at commit 
[`0631168`](https://github.com/apache/spark/commit/0631168ce11c4d0ccd1918c22be17fc4de7f50b4).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20003: [SPARK-22817][R] Use fixed testthat version for SparkR t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20003
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85009/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20003: [SPARK-22817][R] Use fixed testthat version for SparkR t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20003
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20003: [SPARK-22817][R] Use fixed testthat version for SparkR t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20003
  
**[Test build #85009 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85009/testReport)**
 for PR 20003 at commit 
[`887cfe8`](https://github.com/apache/spark/commit/887cfe8ad84d16bb5304012065b8a72e9d6b73b0).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19985: [SPARK-22791] [SQL] [SS] Redact Output of Explain

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19985
  
**[Test build #85011 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85011/testReport)**
 for PR 19985 at commit 
[`e246096`](https://github.com/apache/spark/commit/e246096cab0b0905d7952556ba3e8ecd15c0a2a5).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19985: [SPARK-22791] [SQL] [SS] Redact Output of Explain

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/19985
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20001: [SPARK-22816][TEST] Basic tests for PromoteStrings and I...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20001
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20001: [SPARK-22816][TEST] Basic tests for PromoteStrings and I...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20001
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85008/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20001: [SPARK-22816][TEST] Basic tests for PromoteStrings and I...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20001
  
**[Test build #85008 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85008/testReport)**
 for PR 20001 at commit 
[`9f48bf7`](https://github.com/apache/spark/commit/9f48bf782f7d14a9611a7e8a7a94fb66e021099d).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19998: [SPARK-22813][BUILD] Use lsof or /usr/sbin/lsof in run-t...

[kiszk]

Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/19998
  
I see, thank you very much.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19988: [Spark-22795] [ML] Raise error when line search i...

[mrkm4ntr]

Github user mrkm4ntr commented on a diff in the pull request:

https://github.com/apache/spark/pull/19988#discussion_r157345097
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/optim/NormalEquationSolver.scala ---
@@ -109,6 +109,11 @@ private[optim] class QuasiNewtonSolver(
   state = states.next()
   arrayBuilder += state.adjustedValue
 }
+if (state == null || state.searchFailed) {
+  val msg = s"${optimizer.getClass.getName} failed."
+  logError(msg)
--- End diff --

I got it. Fixed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20003: [SPARK-22817][R] Use fixed testthat version for SparkR t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20003
  
**[Test build #85010 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85010/testReport)**
 for PR 20003 at commit 
[`0631168`](https://github.com/apache/spark/commit/0631168ce11c4d0ccd1918c22be17fc4de7f50b4).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20003: [SPARK-22817][R] Use fixed testthat version for SparkR t...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20003
  
cc @felixcheung, could you take a look please?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20003: [WIP][SPARK-22817][R] Use fixed testthat version for Spa...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20003
  
**[Test build #85009 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85009/testReport)**
 for PR 20003 at commit 
[`887cfe8`](https://github.com/apache/spark/commit/887cfe8ad84d16bb5304012065b8a72e9d6b73b0).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20003: [WIP][SPARK-22817][R] Use fixed testthat version for Spa...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20003
  
Note that this is unrelated with the ongoing CRAN check failure in Jenkins. 
AppVeyor tests are being failed for a separate reason and we don't run CRAN 
check in AppVeyor.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20003: [WIP][SPARK-22817][R] Use fixed testthat version ...

[HyukjinKwon]

GitHub user HyukjinKwon opened a pull request:

https://github.com/apache/spark/pull/20003

[WIP][SPARK-22817][R] Use fixed testthat version for SparkR tests in 
AppVeyor

## What changes were proposed in this pull request?

`testthat` 2.0.0 is released and AppVeyor now started to use it instead of 
1.0.2. And then, we started to have R tests failed in AppVeyor. See - 
https://ci.appveyor.com/project/ApacheSoftwareFoundation/spark/build/1967-master

```
Error in get(name, envir = asNamespace(pkg), inherits = FALSE) : 
  object 'run_tests' not found
Calls: ::: -> get
```

This seems because we rely on internal `testthat:::run_tests` here:

https://github.com/r-lib/testthat/blob/v1.0.2/R/test-package.R#L62-L75


https://github.com/apache/spark/blob/dc4c351837879dab26ad8fb471dc51c06832a9e4/R/pkg/tests/run-all.R#L49-L52

However, seems it was removed out from 2.0.0.  I tried few other exposed 
APIs like `test_dir` but I failed to make a good compatible fix.

Seems we better fix the `testthat` version first to make the build passed.


## How was this patch tested?

Manually tested and AppVeyor tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HyukjinKwon/spark SPARK-22817

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/20003.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20003


commit 887cfe8ad84d16bb5304012065b8a72e9d6b73b0
Author: hyukjinkwon 
Date:   2017-12-16T15:48:52Z

Use fixed testthat version for SparkR tests in AppVeyor




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19998: [SPARK-22813][BUILD] Use lsof or /usr/sbin/lsof in run-t...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19998
  
Let's wait for a bit. Let me restart this once we started to get passed. 
Seems globally failed in R's CRAN check.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20001: [SPARK-22816][TEST] Basic tests for PromoteStrings and I...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20001
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85007/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20001: [SPARK-22816][TEST] Basic tests for PromoteStrings and I...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20001
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19998: [SPARK-22813][BUILD] Use lsof or /usr/sbin/lsof in run-t...

[kiszk]

Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/19998
  
Should we retest this again?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20001: [SPARK-22816][TEST] Basic tests for PromoteStrings and I...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20001
  
**[Test build #85007 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85007/testReport)**
 for PR 20001 at commit 
[`604f16f`](https://github.com/apache/spark/commit/604f16f872f1cf3e008435577d4a4768711c63ed).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19977
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85005/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19977
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19977
  
**[Test build #85005 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85005/testReport)**
 for PR 19977 at commit 
[`2d3926e`](https://github.com/apache/spark/commit/2d3926e546b3aa60c46449151706941ed17e2441).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `  class FunctionArgumentConversion(conf: SQLConf) extends 
TypeCoercionRule `
  * `case class Concat(children: Seq[Expression], isBinaryMode: Boolean = 
false)`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19950: [SPARK-22450][Core][MLLib][FollowUp] safely regis...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/19950#discussion_r157343175
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/mllib/clustering/KMeansSuite.scala ---
@@ -311,6 +312,23 @@ class KMeansSuite extends SparkFunSuite with 
MLlibTestSparkContext {
 assert(returnModel.clusterCenters(1) === 
initialModel.clusterCenters(1))
   }
 
+  test("Kryo class register") {
+val conf = new SparkConf(false)
+conf.set("spark.kryo.registrationRequired", "true")
+
+val ser = new KryoSerializer(conf).newInstance()
+
+def check(v: VectorWithNorm): Unit = {
--- End diff --

Likewise these seem like things you can just write in a loop over several 
objects to test


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19950: [SPARK-22450][Core][MLLib][FollowUp] safely regis...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/19950#discussion_r157343154
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/ml/feature/LabeledPointSuite.scala ---
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.feature
+
+import scala.reflect.ClassTag
+
+import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.apache.spark.ml.linalg.Vectors
+import org.apache.spark.serializer.KryoSerializer
+
+class LabeledPointSuite extends SparkFunSuite{
--- End diff --

Nit: space before brace


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19950: [SPARK-22450][Core][MLLib][FollowUp] safely regis...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/19950#discussion_r157343164
  
--- Diff: 
mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreePointSuite.scala ---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.tree.impl
+
+import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.apache.spark.serializer.KryoSerializer
+
+class TreePointSuite extends SparkFunSuite{
+  test("Kryo class register") {
+val conf = new SparkConf(false)
+conf.set("spark.kryo.registrationRequired", "true")
+
+val ser = new KryoSerializer(conf).newInstance()
+
+def check(p: TreePoint): Unit = {
--- End diff --

Is there much value in defining this method?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20002: [SPARK-22465][Core][WIP] Add a safety-check to RDD defau...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/20002
  
Yup, AppVeyor test seems unrelated. In a quick look, AppVeyor is related 
with the latest `testthat` (1.0.2 -> 2.0.0). Will take a look for this 
separately.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19988: [Spark-22795] [ML] Raise error when line search i...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/19988#discussion_r157342991
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/optim/NormalEquationSolver.scala ---
@@ -109,6 +109,11 @@ private[optim] class QuasiNewtonSolver(
   state = states.next()
   arrayBuilder += state.adjustedValue
 }
+if (state == null || state.searchFailed) {
+  val msg = s"${optimizer.getClass.getName} failed."
+  logError(msg)
--- End diff --

Yeah, I'm thinking logWarning is more appropriate. I know the existing code 
had logError, and could change too.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20002: [SPARK-22465][Core][WIP] Add a safety-check to RDD defau...

[sujithjay]

Github user sujithjay commented on the issue:

https://github.com/apache/spark/pull/20002
  
cc: @tgravescs @codlife Could you please review this?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20002: [SPARK-22465][Core][WIP] Add a safety-check to RDD defau...

[sujithjay]

Github user sujithjay commented on the issue:

https://github.com/apache/spark/pull/20002
  
Hi @HyukjinKwon , can you please help me with these SparkR tests failures? 
They seem unrelated to me.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20002: [SPARK-22465][Core][WIP] Add a safety-check to RDD defau...

[sujithjay]

Github user sujithjay commented on the issue:

https://github.com/apache/spark/pull/20002
  
SparkR test failure seems unrelated to this PR. Any ideas what's wrong?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20001: [SPARK-22816][TEST] Basic tests for PromoteStrings and I...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20001
  
**[Test build #85008 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85008/testReport)**
 for PR 20001 at commit 
[`9f48bf7`](https://github.com/apache/spark/commit/9f48bf782f7d14a9611a7e8a7a94fb66e021099d).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20002: [SPARK-22465][Core][WIP] Add a safety-check to RDD defau...

[sujithjay]

Github user sujithjay commented on the issue:

https://github.com/apache/spark/pull/20002
  
SparkR test failure seems unrelated to this PR. Any ideas what's wrong?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19995: [SPARK-22807] [Scheduler] Remove config that says docker...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19995
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19995: [SPARK-22807] [Scheduler] Remove config that says docker...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19995
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85004/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19995: [SPARK-22807] [Scheduler] Remove config that says docker...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19995
  
**[Test build #85004 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85004/testReport)**
 for PR 19995 at commit 
[`b3e1af3`](https://github.com/apache/spark/commit/b3e1af3b3f4efad820dad9e989c580c74654390f).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20001: [SPARK-22816][TEST] Basic tests for PromoteStrings and I...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/20001
  
**[Test build #85007 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85007/testReport)**
 for PR 20001 at commit 
[`604f16f`](https://github.com/apache/spark/commit/604f16f872f1cf3e008435577d4a4768711c63ed).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20002: [SPARK-22465][Core][WIP] Add a safety-check to RDD defau...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/20002
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20002: [SPARK-22465][Core][WIP] Add a safety-check to RD...

[sujithjay]

GitHub user sujithjay opened a pull request:

https://github.com/apache/spark/pull/20002

[SPARK-22465][Core][WIP] Add a safety-check to RDD defaultPartitioner

## What changes were proposed in this pull request?
In choosing a Partitioner to use for a cogroup-like operation between a 
number of RDDs, the default behaviour was if some of the RDDs already have a 
partitioner, we choose the one amongst them with the maximum number of 
partitions.

This behaviour, in some cases, could hit the 2G limit (SPARK-6235). To 
illustrate one such scenario, consider two RDDs:
rDD1: with smaller data and smaller number of partitions, alongwith a 
Partitioner.
rDD2: with much larger data and a larger number of partitions, without a 
Partitioner.

The cogroup of these two RDDs could hit the 2G limit, as a larger amount of 
data is shuffled into a smaller number of partitions.

This PR introduces a safety-check wherein the Partitioner is chosen only if 
either of the following conditions are met:
1. if the number of partitions of the RDD associated with the Partitioner 
is greater than or equal to the max number of upstream partitions; or 
2. if the number of partitions of the RDD associated with the Partitioner 
is less than and within a single order of magnitude of the max number of 
upstream partitions.

## How was this patch tested?
Unit tests in PartitioningSuite and PairRDDFunctionsSuite


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sujithjay/spark SPARK-22465

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/20002.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20002


commit 176270b3dbddb1f8d1330709dfea2022eebb7a11
Author: sujithjay 
Date:   2017-12-16T12:16:13Z

[SPARK-22465][Core][WIP] Add a safety-check to RDD defaultPartitioner

 that ignores existing Partitioners, if they are more than a single order 
of magnitude smaller than the max number of upstream partitions

commit be391a78db920f944ce2fe1223dd604aae56871a
Author: sujithjay 
Date:   2017-12-16T12:22:41Z

Merge remote-tracking branch 'origin-apache/master' into SPARK-22465




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20001: [SPARK-22762][TEST] Basic tests for PromoteString...

[wangyum]

GitHub user wangyum opened a pull request:

https://github.com/apache/spark/pull/20001

[SPARK-22762][TEST] Basic tests for PromoteStrings and InConversion

## What changes were proposed in this pull request?

Test Coverage for `PromoteStrings` and `InConversion`, this is a Sub-tasks 
for [SPARK-22722](https://issues.apache.org/jira/browse/SPARK-22722).

## How was this patch tested?

N/A

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wangyum/spark SPARK-22816

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/20001.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20001


commit 604f16f872f1cf3e008435577d4a4768711c63ed
Author: Yuming Wang 
Date:   2017-12-16T12:48:44Z

Basic tests for PromoteStrings and InConversion




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19992: [SPARK-22805][CORE] Use StorageLevel aliases in event lo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19992
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19992: [SPARK-22805][CORE] Use StorageLevel aliases in event lo...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19992
  
**[Test build #85006 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85006/testReport)**
 for PR 19992 at commit 
[`13fe385`](https://github.com/apache/spark/commit/13fe385f3139cbdf9c43ed5eec92cddcff7368f5).
 * This patch **fails to generate documentation**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19992: [SPARK-22805][CORE] Use StorageLevel aliases in event lo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19992
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85006/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19992: [SPARK-22805][CORE] Use StorageLevel aliases in event lo...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19992
  
**[Test build #85006 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85006/testReport)**
 for PR 19992 at commit 
[`13fe385`](https://github.com/apache/spark/commit/13fe385f3139cbdf9c43ed5eec92cddcff7368f5).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #19992: [SPARK-22805][CORE] Use StorageLevel aliases in e...

[superbobry]

Github user superbobry commented on a diff in the pull request:

https://github.com/apache/spark/pull/19992#discussion_r157340550
  
--- Diff: core/src/main/scala/org/apache/spark/storage/StorageLevel.scala 
---
@@ -132,6 +132,23 @@ class StorageLevel private(
 
   override def hashCode(): Int = toInt * 41 + replication
 
+  /** Name of the storage level if it is predefined or [[None]] otherwise. 
*/
+  def name: Option[String] = this match {
--- End diff --

This sounds good, will do. A slightly unrelated point: I feel that the name 
`fromString` somehow implies that it's the opposite of `toString`. What do you 
think about renaming it to `fromName` now that we have `name`?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19977
  
**[Test build #85005 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85005/testReport)**
 for PR 19977 at commit 
[`2d3926e`](https://github.com/apache/spark/commit/2d3926e546b3aa60c46449151706941ed17e2441).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[maropu]

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/19977
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[maropu]

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/19977
  
oh...


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19977
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19977
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85000/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19985: [SPARK-22791] [SQL] [SS] Redact Output of Explain

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19985
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85002/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19998: [SPARK-22813][BUILD] Use lsof or /usr/sbin/lsof in run-t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19998
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/84999/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19977: [SPARK-22771][SQL] Concatenate binary inputs into a bina...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19977
  
**[Test build #85000 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85000/testReport)**
 for PR 19977 at commit 
[`2d3926e`](https://github.com/apache/spark/commit/2d3926e546b3aa60c46449151706941ed17e2441).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `  class FunctionArgumentConversion(conf: SQLConf) extends 
TypeCoercionRule `
  * `case class Concat(children: Seq[Expression], isBinaryMode: Boolean = 
false)`


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19985: [SPARK-22791] [SQL] [SS] Redact Output of Explain

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19985
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19985: [SPARK-22791] [SQL] [SS] Redact Output of Explain

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19985
  
**[Test build #85002 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85002/testReport)**
 for PR 19985 at commit 
[`e246096`](https://github.com/apache/spark/commit/e246096cab0b0905d7952556ba3e8ecd15c0a2a5).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19998: [SPARK-22813][BUILD] Use lsof or /usr/sbin/lsof in run-t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19998
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19998: [SPARK-22813][BUILD] Use lsof or /usr/sbin/lsof in run-t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19998
  
**[Test build #84999 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/84999/testReport)**
 for PR 19998 at commit 
[`6c29a11`](https://github.com/apache/spark/commit/6c29a11e6f08a83cd10eaeda3240b49f15aea07b).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20000: [SPARK-22815] [SQL] Keep PromotePrecision in Optimized P...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/2
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85003/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20000: [SPARK-22815] [SQL] Keep PromotePrecision in Optimized P...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/2
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20000: [SPARK-22815] [SQL] Keep PromotePrecision in Optimized P...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/2
  
**[Test build #85003 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85003/testReport)**
 for PR 2 at commit 
[`64d55c4`](https://github.com/apache/spark/commit/64d55c4c86273f397d99fd29fe85735523877539).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19594: [SPARK-21984] [SQL] Join estimation based on equi-height...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19594
  
Merged build finished. Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19594: [SPARK-21984] [SQL] Join estimation based on equi-height...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19594
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85001/
Test FAILed.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19594: [SPARK-21984] [SQL] Join estimation based on equi-height...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19594
  
**[Test build #85001 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85001/testReport)**
 for PR 19594 at commit 
[`2637429`](https://github.com/apache/spark/commit/263742914e21ba607904acb0ad35ced32aad48ab).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19995: [SPARK-22807] [Scheduler] Remove config that says docker...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/19995
  
**[Test build #85004 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85004/testReport)**
 for PR 19995 at commit 
[`b3e1af3`](https://github.com/apache/spark/commit/b3e1af3b3f4efad820dad9e989c580c74654390f).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19995: [SPARK-22807] [Scheduler] Remove config that says docker...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/19995
  
retest this please


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org