[GitHub] spark pull request #17872: [SPARK-20608] allow standby namenodes in spark.ya...

[morenn520]

Github user morenn520 commented on a diff in the pull request:

https://github.com/apache/spark/pull/17872#discussion_r115183402
  
--- Diff: 
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala
 ---
@@ -48,9 +50,16 @@ private[security] class HadoopFSCredentialProvider
 val tmpCreds = new Credentials()
 val tokenRenewer = getTokenRenewer(hadoopConf)
 hadoopFSsToAccess(hadoopConf, sparkConf).foreach { dst =>
-  val dstFs = dst.getFileSystem(hadoopConf)
-  logInfo("getting token for: " + dst)
-  dstFs.addDelegationTokens(tokenRenewer, tmpCreds)
+  try {
+val dstFs = dst.getFileSystem(hadoopConf)
+logInfo("getting token for: " + dst)
+dstFs.addDelegationTokens(tokenRenewer, tmpCreds)
+  } catch {
+case e: StandbyException =>
+  logWarning(s"Namenode ${dst} is in state standby", e)
--- End diff --

hum..Here is actually fetching tokens from hadoopFS, including in 
hadoopFSCredentialProvider, which means it's exactly HDFS?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17894: [SPARK-17134][ML] Use level 2 BLAS operations in ...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/17894#discussion_r115182882
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
 ---
@@ -23,6 +23,7 @@ import scala.collection.mutable
 
 import breeze.linalg.{DenseVector => BDV}
 import breeze.optimize.{CachedDiffFunction, DiffFunction, LBFGS => 
BreezeLBFGS, LBFGSB => BreezeLBFGSB, OWLQN => BreezeOWLQN}
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
--- End diff --

Is it better to use MLlib BLAS interface?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17872: [SPARK-20608] allow standby namenodes in spark.ya...

[jerryshao]

Github user jerryshao commented on a diff in the pull request:

https://github.com/apache/spark/pull/17872#discussion_r115182668
  
--- Diff: 
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HadoopFSCredentialProvider.scala
 ---
@@ -48,9 +50,16 @@ private[security] class HadoopFSCredentialProvider
 val tmpCreds = new Credentials()
 val tokenRenewer = getTokenRenewer(hadoopConf)
 hadoopFSsToAccess(hadoopConf, sparkConf).foreach { dst =>
-  val dstFs = dst.getFileSystem(hadoopConf)
-  logInfo("getting token for: " + dst)
-  dstFs.addDelegationTokens(tokenRenewer, tmpCreds)
+  try {
+val dstFs = dst.getFileSystem(hadoopConf)
+logInfo("getting token for: " + dst)
+dstFs.addDelegationTokens(tokenRenewer, tmpCreds)
+  } catch {
+case e: StandbyException =>
+  logWarning(s"Namenode ${dst} is in state standby", e)
--- End diff --

It's not accurate to say "Namenode" here, because we may configure to other 
non-HDFS. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17895: Branch 2.0

[fjh100456]

Github user fjh100456 commented on the issue:

https://github.com/apache/spark/pull/17895
  
Sorry, I made a mistake. I'll close it by now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17895: Branch 2.0

[fjh100456]

Github user fjh100456 closed the pull request at:

https://github.com/apache/spark/pull/17895


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17866: [SPARK-20605][Core][Yarn][Mesos] Deprecate not used AM a...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17866
  
**[Test build #76562 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76562/testReport)**
 for PR 17866 at commit 
[`ac710c7`](https://github.com/apache/spark/commit/ac710c7020a4fb8b3d684e0eb5daf47befb039ec).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17884: [SparkR][Doc] fix typo in vignettes

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/17884
  
merged to master/2.2
thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17884: [SparkR][Doc] fix typo in vignettes

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/17884


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17649: [SPARK-20380][SQL] Unable to set/unset table comm...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/17649


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17649: [SPARK-20380][SQL] Unable to set/unset table comment pro...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17649
  
Thanks! Merging to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17895: Branch 2.0

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/17895
  
@fjh100456 looks mistakenly open. Could you close this please?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17866: [SPARK-20605][Core][Yarn][Mesos] Deprecate not us...

[jerryshao]

Github user jerryshao commented on a diff in the pull request:

https://github.com/apache/spark/pull/17866#discussion_r115178712
  
--- Diff: 
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
 ---
@@ -429,8 +429,7 @@ private[spark] class ApplicationMaster(
   }
 
   private def runExecutorLauncher(securityMgr: SecurityManager): Unit = {
-val port = sparkConf.get(AM_PORT)
-rpcEnv = RpcEnv.create("sparkYarnAM", Utils.localHostName, port, 
sparkConf, securityMgr,
+rpcEnv = RpcEnv.create("sparkYarnAM", Utils.localHostName, -1, 
sparkConf, securityMgr,
--- End diff --

This will touch a lot of places, I would incline to leave that `create` as 
it was.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17892: [SPARK-20626][SPARKR] address date test warning w...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/17892


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17892: [SPARK-20626][SPARKR] address date test warning with tim...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/17892
  
merged to master/2.2


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17222: [SPARK-19439][PYSPARK][SQL] PySpark's registerJav...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/17222#discussion_r115177349
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala ---
@@ -475,20 +475,42 @@ class UDFRegistration private[sql] (functionRegistry: 
FunctionRegistry) extends
 case 21 => register(name, udf.asInstanceOf[UDF20[_, _, _, _, 
_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _]], returnType)
 case 22 => register(name, udf.asInstanceOf[UDF21[_, _, _, _, 
_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _]], returnType)
 case 23 => register(name, udf.asInstanceOf[UDF22[_, _, _, _, 
_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _]], returnType)
-case n => logError(s"UDF class with ${n} type arguments is not 
supported ")
+case n =>
+  throw new IOException(s"UDF class with ${n} type arguments 
is not supported.")
   }
 } catch {
   case e @ (_: InstantiationException | _: 
IllegalArgumentException) =>
-logError(s"Can not instantiate class ${className}, please make 
sure it has public non argument constructor")
+throw new IOException(s"Can not instantiate class 
${className}, please make sure it has public non argument constructor")
 }
   }
 } catch {
-  case e: ClassNotFoundException => logError(s"Can not load class 
${className}, please make sure it is on the classpath")
+  case e: ClassNotFoundException => throw new IOException(s"Can not 
load class ${className}, please make sure it is on the classpath")
 }
 
   }
 
   /**
+   * Register a Java UDAF class using reflection, for use from pyspark
+   *
+   * @param name UDAF name
+   * @param classNamefully qualified class name of UDAF
+   */
+  private[sql] def registerJavaUDAF(name: String, className: String): Unit 
= {
--- End diff --

`UserDefinedAggregateFunction` already defines its return type. So we don't 
need to specify it when registering a udfa.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17858: [SPARK-20594][SQL]The staging directory should be append...

[zuotingbing]

Github user zuotingbing commented on the issue:

https://github.com/apache/spark/pull/17858
  
yes i tried the same thing in Hive, got the same error:
`2017-05-08T13:48:04,634 ERROR exec.Task (:()) - Failed with exception 
Unable to move source 
hdfs://nameservice/hive/test_table1/test_hive_2017-05-08_13-47-40_660_5235248825413690559-1/-ext-1
 to destination hdfs://nameservice/hive/test_table1
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source 
hdfs://nameservice/hive/test_table1/test_hive_2017-05-08_13-47-40_660_5235248825413690559-1/-ext-1
 to destination hdfs://nameservice/hive/test_table1
at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2959)
at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:3198)
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:1805)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:355)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1917)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1586)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1331)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1092)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1080)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.FileNotFoundException: File 
hdfs://nameservice/hive/test_table1/test_hive_2017-05-08_13-47-40_660_5235248825413690559-1/-ext-1
 does not exist.
at 
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:697)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:105)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:755)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:751)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:751)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1485)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1525)
at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2896)
... 22 more

2017-05-08T13:48:04,635 ERROR ql.Driver (:()) - FAILED: Execution Error, 
return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move 
source 
hdfs://nameservice/hive/test_table1/test_hive_2017-05-08_13-47-40_660_5235248825413690559-1/-ext-1
 to destination hdfs://nameservice/hive/test_table1`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17395: [SPARK-20065][SS][WIP] Avoid to output empty parquet fil...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/17395
  
Yea, I just pinged because I am just interested in this :).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17395: [SPARK-20065][SS][WIP] Avoid to output empty parquet fil...

[uncleGen]

Github user uncleGen commented on the issue:

https://github.com/apache/spark/pull/17395
  
@HyukjinKwon Sorry for the long absence. I will keep online for next period 
of time. Please give me some time.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17896: [SPARK-20373][SQL][SS] Batch queries with 'Dataset/DataF...

[uncleGen]

Github user uncleGen commented on the issue:

https://github.com/apache/spark/pull/17896
  
cc @zsxwing and @tdas 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17896: [SPARK-20373][SQL][SS] Batch queries with 'Dataset/DataF...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17896
  
**[Test build #76561 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76561/testReport)**
 for PR 17896 at commit 
[`5637212`](https://github.com/apache/spark/commit/563721241851751c2bb1736161febe73b8abba3b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17896: [SPARK-20373][SQL][SS] Batch queries with 'Datase...

[uncleGen]

GitHub user uncleGen opened a pull request:

https://github.com/apache/spark/pull/17896

[SPARK-20373][SQL][SS] Batch queries with 
'Dataset/DataFrame.withWatermark()` does not execute

## What changes were proposed in this pull request?

Any Dataset/DataFrame batch query with the operation `withWatermark` does 
not execute because the batch planner does not have any rule to explicitly 
handle the EventTimeWatermark logical plan. 
The right solution is to simply remove the plan node, as the watermark 
should not affect any batch query in any way.

Changes: 
- In this PR, we add a new rule `CheckEventTimeWatermark` to check if we 
need to ignore the event time watermark. We will ignore watermark in any batch 
query. 

Followups:
- Add `CheckEventTimeWatermark` into analyzer rule list. We can not add 
this rule into analyzer directly, because streaming query will be copied to a 
internal batch query in every trigger, and the rule will be applied to this 
internal batch query mistakenly. IIUC, we should refactor related codes to 
better define a query is batch or streaming. Right?

Others:
- A typo fix in example.

## How was this patch tested?

add new unit test.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/uncleGen/spark SPARK-20373

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/17896.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17896


commit 563721241851751c2bb1736161febe73b8abba3b
Author: uncleGen 
Date:   2017-05-08T03:19:35Z

Ignore event time watermark in batch query.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17887: [SPARK-20399][SQL][WIP] Add a config to fallback string ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17887
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17887: [SPARK-20399][SQL][WIP] Add a config to fallback string ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17887
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76559/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17887: [SPARK-20399][SQL][WIP] Add a config to fallback string ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17887
  
**[Test build #76559 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76559/testReport)**
 for PR 17887 at commit 
[`8ae0747`](https://github.com/apache/spark/commit/8ae074784d145dc2298fa81c0c9097aad48fc349).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17887: [SPARK-20399][SQL][WIP] Add a config to fallback string ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17887
  
**[Test build #76560 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76560/testReport)**
 for PR 17887 at commit 
[`ab77de7`](https://github.com/apache/spark/commit/ab77de7d057ed284160fc051bfbfd6afb0a3a923).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #2633: [https://issues.apache.org/jira/browse/SPARK-4392] Event ...

[Mobe91]

Github user Mobe91 commented on the issue:

https://github.com/apache/spark/pull/2633
  
When is this going the be merged? Would be really useful.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17874: [SPARK-20612][SQL] Throw exception when there is unresol...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/17874
  
Maybe another point of view is, we can split `df.select("y").where("x=1")` 
to two different DataFrames:

val onlyY = df.select("y")  // The schema of onlyY is just "y" attribute
onlyY.where("x=1") // Then we can filter on a non-existing attribute






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/17869
  
Test results are fine. I think my question is why this happens and how the 
current change solves the issue above. Another question is, does second run of 
the test cases fail due to this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17874: [SPARK-20612][SQL] Throw exception when there is unresol...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/17874
  
`select a from t where b > 0` works. However, it can be seen logically as:

Project [a]
  Filter [b > 0]
Relation t [a, b]

It seems to me Spark also parses the above SQL query like this way.

There is an order of evaluation in SQL systems. E.g, MySQL:

select a from test where b > 2;   // works. where is evaluated before 
select
select a from test having b > 2;  // not works. having is evaluated 
after select

`df.select("y").where("x=1")` sematically asks a projection of  just `y`  
attribute before filtering. It seems to me that it is different with the SQL 
query.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17796: [SPARK-20519][SQL][CORE]Modify to prevent some possible ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17796
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76557/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17796: [SPARK-20519][SQL][CORE]Modify to prevent some possible ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17796
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17796: [SPARK-20519][SQL][CORE]Modify to prevent some possible ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17796
  
**[Test build #76557 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76557/testReport)**
 for PR 17796 at commit 
[`0e879c8`](https://github.com/apache/spark/commit/0e879c8d920ef53624d3c13f3176e2408d6563ac).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17894: [SPARK-17134][ML] Use level 2 BLAS operations in Logisti...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17894
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76558/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17894: [SPARK-17134][ML] Use level 2 BLAS operations in Logisti...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17894
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17894: [SPARK-17134][ML] Use level 2 BLAS operations in Logisti...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17894
  
**[Test build #76558 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76558/testReport)**
 for PR 17894 at commit 
[`b4fd733`](https://github.com/apache/spark/commit/b4fd733a708feb63254728f3af1fd20ee892d085).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16781: [SPARK-12297][SQL] Hive compatibility for Parquet...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/16781


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17874: [SPARK-20612][SQL] Throw exception when there is unresol...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/17874
  
in postgres, `select a from t where b > 0` can work, I think it's 
reasonable if `df.select("y").where("x=1")` works in spark.

```
Seq(1).toDF("c1").createOrReplaceTempView("onerow")
sql(
  """
| select 1
|from   (select 1 from onerow t2 LIMIT 1)
|where  t2.c1=1""".stripMargin)
```
this one we should not support, we should not add missing attributes though 
subqueries.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16781: [SPARK-12297][SQL] Hive compatibility for Parquet Timest...

[ueshin]

Github user ueshin commented on the issue:

https://github.com/apache/spark/pull/16781
  
Thanks! Merging to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17222: [SPARK-19439][PYSPARK][SQL] PySpark's registerJav...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/17222#discussion_r115166927
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala ---
@@ -475,20 +475,42 @@ class UDFRegistration private[sql] (functionRegistry: 
FunctionRegistry) extends
 case 21 => register(name, udf.asInstanceOf[UDF20[_, _, _, _, 
_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _]], returnType)
 case 22 => register(name, udf.asInstanceOf[UDF21[_, _, _, _, 
_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _]], returnType)
 case 23 => register(name, udf.asInstanceOf[UDF22[_, _, _, _, 
_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _]], returnType)
-case n => logError(s"UDF class with ${n} type arguments is not 
supported ")
+case n =>
+  throw new IOException(s"UDF class with ${n} type arguments 
is not supported.")
   }
 } catch {
   case e @ (_: InstantiationException | _: 
IllegalArgumentException) =>
-logError(s"Can not instantiate class ${className}, please make 
sure it has public non argument constructor")
+throw new IOException(s"Can not instantiate class 
${className}, please make sure it has public non argument constructor")
 }
   }
 } catch {
-  case e: ClassNotFoundException => logError(s"Can not load class 
${className}, please make sure it is on the classpath")
+  case e: ClassNotFoundException => throw new IOException(s"Can not 
load class ${className}, please make sure it is on the classpath")
 }
 
   }
 
   /**
+   * Register a Java UDAF class using reflection, for use from pyspark
+   *
+   * @param name UDAF name
+   * @param classNamefully qualified class name of UDAF
+   */
+  private[sql] def registerJavaUDAF(name: String, className: String): Unit 
= {
--- End diff --

`registerJava` accepts an optional return type, if not given, spark will 
try to infer it via reflection. do we really not need to do this for udaf?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17077: [SPARK-16931][PYTHON][SQL] Add Python wrapper for...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/17077


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17077: [SPARK-16931][PYTHON][SQL] Add Python wrapper for bucket...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/17077
  
thanks, merging to master!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17887: [SPARK-20399][SQL][WIP] Add a config to fallback string ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17887
  
**[Test build #76559 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76559/testReport)**
 for PR 17887 at commit 
[`8ae0747`](https://github.com/apache/spark/commit/8ae074784d145dc2298fa81c0c9097aad48fc349).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17895: Branch 2.0

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17895
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17895: Branch 2.0

[fjh100456]

GitHub user fjh100456 opened a pull request:

https://github.com/apache/spark/pull/17895

Branch 2.0

## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)

Please review http://spark.apache.org/contributing.html before opening a 
pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/spark branch-2.0

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/17895.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17895


commit b57e2acb134d94dafc81686da875c5dd3ea35c74
Author: Jagadeesan 
Date:   2016-10-03T09:46:38Z

[SPARK-17736][DOCUMENTATION][SPARKR] Update R README for rmarkdown,…

## What changes were proposed in this pull request?

To build R docs (which are built when R tests are run), users need to 
install pandoc and rmarkdown. This was done for Jenkins in 
~~[SPARK-17420](https://issues.apache.org/jira/browse/SPARK-17420)~~

… pandoc]

Author: Jagadeesan 

Closes #15309 from jagadeesanas2/SPARK-17736.

(cherry picked from commit a27033c0bbaae8f31db9b91693947ed71738ed11)
Signed-off-by: Sean Owen 

commit 613863b116b6cbc9ac83845c68a2d11b3b02f7cb
Author: zero323 
Date:   2016-10-04T00:57:54Z

[SPARK-17587][PYTHON][MLLIB] SparseVector __getitem__ should follow 
__getitem__ contract

## What changes were proposed in this pull request?

Replaces` ValueError` with `IndexError` when index passed to `ml` / `mllib` 
`SparseVector.__getitem__` is out of range. This ensures correct iteration 
behavior.

Replaces `ValueError` with `IndexError` for `DenseMatrix` and `SparkMatrix` 
in `ml` / `mllib`.

## How was this patch tested?

PySpark `ml` / `mllib` unit tests. Additional unit tests to prove that the 
problem has been resolved.

Author: zero323 

Closes #15144 from zero323/SPARK-17587.

(cherry picked from commit d8399b600cef706c22d381b01fab19c610db439a)
Signed-off-by: Joseph K. Bradley 

commit 5843932021cc8bbe0277943c6c480cfeae1b29e2
Author: Herman van Hovell 
Date:   2016-10-04T02:32:59Z

[SPARK-17753][SQL] Allow a complex expression as the input a value based 
case statement

## What changes were proposed in this pull request?
We currently only allow relatively simple expressions as the input for a 
value based case statement. Expressions like `case (a > 1) or (b = 2) when true 
then 1 when false then 0 end` currently fail. This PR adds support for such 
expressions.

## How was this patch tested?
Added a test to the ExpressionParserSuite.

Author: Herman van Hovell 

Closes #15322 from hvanhovell/SPARK-17753.

(cherry picked from commit 2bbecdec2023143fd144e4242ff70822e0823986)
Signed-off-by: Herman van Hovell 

commit 7429199e5b34d5594e3fcedb57eda789d16e26f3
Author: Dongjoon Hyun 
Date:   2016-10-04T04:28:16Z

[SPARK-17112][SQL] "select null" via JDBC triggers IllegalArgumentException 
in Thriftserver

## What changes were proposed in this pull request?

Currently, Spark Thrift Server raises `IllegalArgumentException` for 
queries whose column types are `NullType`, e.g., `SELECT null` or `SELECT 
if(true,null,null)`. This PR fixes that by returning `void` like Hive 1.2.

**Before**
```sql
$ bin/beeline -u jdbc:hive2://localhost:1 -e "select null"
Connecting to jdbc:hive2://localhost:1
Connected to: Spark SQL (version 2.1.0-SNAPSHOT)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Error: java.lang.IllegalArgumentException: Unrecognized type name: null 
(state=,code=0)
Closing: 0: jdbc:hive2://localhost:1

$ bin/beeline -u jdbc:hive2://localhost:1 -e "select if(true,null,null)"
Connecting to jdbc:hive2://localhost:1
Connected to: Spark SQL (version 2.1.0-SNAPSHOT)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Error: java.lang.IllegalArgumentException: Unrecognized type name: null 
(state=,code=0)
Closing: 0: jdbc:hive2://localhost:1
```

**After**
```sql
$ bin/beeline -u jdbc:hive2://localhost:1 -e "select null"
Connecting to jdbc:hive2://localhost:1
Connected to: Spark SQL (version 2.1.0-SNAPSHOT)
Driver: Hive JDBC (version 1.2.1.spark2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
+---+--+
| NULL  |
+---+--+
| NUL

[GitHub] spark issue #16781: [SPARK-12297][SQL] Hive compatibility for Parquet Timest...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16781
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76556/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16781: [SPARK-12297][SQL] Hive compatibility for Parquet Timest...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16781
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16781: [SPARK-12297][SQL] Hive compatibility for Parquet Timest...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16781
  
**[Test build #76556 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76556/testReport)**
 for PR 16781 at commit 
[`2537437`](https://github.com/apache/spark/commit/2537437a0c1f022acc215518e4728d24f6f6cf97).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17894: [SPARK-17134][ML] Use level 2 BLAS operations in Logisti...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17894
  
**[Test build #76558 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76558/testReport)**
 for PR 17894 at commit 
[`b4fd733`](https://github.com/apache/spark/commit/b4fd733a708feb63254728f3af1fd20ee892d085).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17894: [SPARK-17134][ML] Use level 2 BLAS operations in ...

[VinceShieh]

GitHub user VinceShieh opened a pull request:

https://github.com/apache/spark/pull/17894

[SPARK-17134][ML] Use level 2 BLAS operations in LogisticAggregator

## What changes were proposed in this pull request?

Multinomial logistic regression uses LogisticAggregator class for gradient 
updates.
This PR refactors MLOR to use level 2 BLAS operations for the updates

## How was this patch tested?
Existing test would do

Signed-off-by: VinceShieh 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/VinceShieh/spark SPARK-17134

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/17894.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17894


commit b4fd733a708feb63254728f3af1fd20ee892d085
Author: VinceShieh 
Date:   2017-05-08T01:54:30Z

[SPARK-17134][ML] Use level 2 BLAS operations in LogisticAggregator

Multinomial logistic regression uses LogisticAggregator class for gradient 
updates.
This PR refactors MLOR to use level 2 BLAS operations for the updates.

Signed-off-by: VinceShieh 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17869: [SPARK-20609][CORE]Run the SortShuffleSuite unit tests h...

[heary-cao]

Github user heary-cao commented on the issue:

https://github.com/apache/spark/pull/17869
  
@HyukjinKwon 
this changes before and after deleting files log information:
**Before changes**

> Run the unit test: SortShuffleManager properly cleans up files for 
shuffles that use the serialized path

```

remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\blockmgr-9c553fa3-dfc9-435d-8624-ab48961c8248\0c

remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\blockmgr-9c553fa3-dfc9-435d-8624-ab48961c8248\0d
...

remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\blockmgr-9c553fa3-dfc9-435d-8624-ab48961c8248

remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\spark-c0db2905-02f1-4bbb-811b-3cf910dee7cf\userFiles-d8906c18-17ab-41e7-917e-11cd7c71243e

remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\spark-c0db2905-02f1-4bbb-811b-3cf910dee7cf

remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a
```

> Run the unit test: SortShuffleManager properly cleans up files for 
shuffles that use the deserialized path

```

remove:C:\Users\Default\AppData\Local\Temp\spark-371eb96b-738f-4f6f-97c9-7a694afc34cc\blockmgr-0691abbf-61b8-41eb-9476-92479d9209d2\0c

remove:C:\Users\Default\AppData\Local\Temp\spark-371eb96b-738f-4f6f-97c9-7a694afc34cc\blockmgr-0691abbf-61b8-41eb-9476-92479d9209d2\0d
...

remove:C:\Users\Default\AppData\Local\Temp\spark-371eb96b-738f-4f6f-97c9-7a694afc34cc\blockmgr-0691abbf-61b8-41eb-9476-92479d9209d2

remove:C:\Users\Default\AppData\Local\Temp\spark-371eb96b-738f-4f6f-97c9-7a694afc34cc

remove:C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\spark-c0db2905-02f1-4bbb-811b-3cf910dee7cf\userFiles-76f038fe-4e25-4622-b8c1-74481ffb6eb9
```

_residual system directory is_  
`C:\Users\Default\AppData\Local\Temp\spark-6d46f183-bb26-4f98-a50a-e19b2068c15a\spark-c0db2905-02f1-4bbb-811b-3cf910dee7cf`

**After changes**

> Run the unit test: SortShuffleManager properly cleans up files for 
shuffles that use the serialized path
```

remove:C:\Users\Default\AppData\Local\Temp\spark-c2a131f1-08a5-428c-bed4-e29cfcacbbb9

remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\spark-3a7708e1-56fb-40ed-890f-6a8cc80d7473\userFiles-68974f6c-d4fc-4f8a-b66f-01a25ed12c44
```
> Run the unit test: SortShuffleManager properly cleans up files for 
shuffles that use the deserialized path
```

remove:C:\Users\Default\AppData\Local\Temp\spark-07ef44ee-c165-4637-a52d-daf2f8780a50

remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\spark-3a7708e1-56fb-40ed-890f-6a8cc80d7473\userFiles-19e1cd23-077d-4d05-a676-6d3f8f2cfa39


remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\spark-3a7708e1-56fb-40ed-890f-6a8cc80d7473

remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\blockmgr-73e600e1-0ff7-4d79-8da0-ffac572be0d3\0c
...

remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\blockmgr-73e600e1-0ff7-4d79-8da0-ffac572be0d3


remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\blockmgr-815208b6-c080-4707-8553-a5cec9709e1e\0c
...

remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b\blockmgr-815208b6-c080-4707-8553-a5cec9709e1e


remove:C:\Users\Default\AppData\Local\Temp\spark-bda0e806-6394-4180-8579-05a14c9aa83b
```

So, after changes, remove file sequence has also changed.

thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17770: [SPARK-20392][SQL] Set barrier to prevent re-entering a ...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/17770
  
I've removed the path of `resolveOperators`. After ran the same benchmark, 
I didn't find obvious performance regression. @hvanhovell @marmbrus @cloud-fan 
@rxin @srinathshankar 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17893: FileFormatWriter wrap the FetchFailedException which bre...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17893
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17893: FileFormatWriter wrap the FetchFailedException which bre...

[mridulm]

Github user mridulm commented on the issue:

https://github.com/apache/spark/pull/17893
  
This is not required anymore - post @squito's change.
Though it cant hurt I guess.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17893: FileFormatWriter wrap the FetchFailedException which bre...

[lshmouse]

Github user lshmouse commented on the issue:

https://github.com/apache/spark/pull/17893
  
Pending. I am formatting the pull name~


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17893: FileFormatWriter wrap the FetchFailedException wh...

[lshmouse]

GitHub user lshmouse opened a pull request:

https://github.com/apache/spark/pull/17893

FileFormatWriter wrap the FetchFailedException which breaks job's failover

## What changes were proposed in this pull request?
Handle the fetch failed exception separately in FileFormatWriter.

## How was this patch tested?
manual tests

Please review http://spark.apache.org/contributing.html before opening a 
pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lshmouse/spark FileFormatWriter

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/17893.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17893


commit c1a635e722e36714582ab10ec04a361ff67c3aa5
Author: Liu Shaohui 
Date:   2017-05-05T08:58:23Z

FileFormatWriter wrap the FetchFailedException which breaks the failure 
recovery chain

commit c869d9c7acfe4fe9c43070185cbe303241248f08
Author: Liu Shaohui 
Date:   2017-05-08T01:19:20Z

Fix bugs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17702: [SPARK-20408][SQL] Get the glob path in parallel to redu...

[xuanyuanking]

Github user xuanyuanking commented on the issue:

https://github.com/apache/spark/pull/17702
  
@gatorsmile @cloud-fan, do we need other performance test? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17887: [SPARK-20399][SQL][WIP] Add a config to fallback ...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/17887#discussion_r115162164
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -196,6 +196,14 @@ object SQLConf {
 .booleanConf
 .createWithDefault(true)
 
+  val NO_UNESCAPED_SQL_STRING = 
buildConf("spark.sql.noUnescapedStringLiteral")
--- End diff --

Sure.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17887: [SPARK-20399][SQL][WIP] Add a config to fallback ...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/17887#discussion_r115162152
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -196,6 +196,14 @@ object SQLConf {
 .booleanConf
 .createWithDefault(true)
 
+  val NO_UNESCAPED_SQL_STRING = 
buildConf("spark.sql.noUnescapedStringLiteral")
+.internal()
+.doc("Since Spark 2.0, we use unescaped SQL string for string literals 
including regex. " +
+  "It is different than 1.6 behavior. Enabling this config can use no 
unescaped SQL string " +
+  "literals and mitigate migration problem.")
--- End diff --

Sure.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17887: [SPARK-20399][SQL][WIP] Add a config to fallback ...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/17887#discussion_r115162139
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParserUtils.scala
 ---
@@ -68,6 +68,11 @@ object ParserUtils {
   /** Convert a string node into a string. */
   def string(node: TerminalNode): String = unescapeSQLString(node.getText)
 
+  /** Convert a string node into a string without unescaping. */
+  def stringWithoutUnescape(node: TerminalNode): String = {
+node.getText.slice(1, node.getText.size - 1)
--- End diff --

The string rule in SqlBase.g4 forces that the input has always quotes at 
the starting and ending. I may add a comment here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17077: [SPARK-16931][PYTHON][SQL] Add Python wrapper for bucket...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/17077
  
LGTM too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17713: [SPARK-20417][SQL] Move subquery error handling to check...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17713
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17713: [SPARK-20417][SQL] Move subquery error handling to check...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17713
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76553/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17713: [SPARK-20417][SQL] Move subquery error handling to check...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17713
  
**[Test build #76553 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76553/testReport)**
 for PR 17713 at commit 
[`3c4f38e`](https://github.com/apache/spark/commit/3c4f38e3eddfd71cf82f543c70be7885980951d6).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17796: [SPARK-20519][SQL][CORE]Modify to prevent some possible ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17796
  
**[Test build #76557 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76557/testReport)**
 for PR 17796 at commit 
[`0e879c8`](https://github.com/apache/spark/commit/0e879c8d920ef53624d3c13f3176e2408d6563ac).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16781: [SPARK-12297][SQL] Hive compatibility for Parquet Timest...

[ueshin]

Github user ueshin commented on the issue:

https://github.com/apache/spark/pull/16781
  
LGTM, pending Jenkins.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17892: [SPARK-20626][SPARKR] address date test warning w...

[felixcheung]

GitHub user felixcheung reopened a pull request:

https://github.com/apache/spark/pull/17892

[SPARK-20626][SPARKR] address date test warning with timezone on windows

## What changes were proposed in this pull request?

set timezone on windows

## How was this patch tested?

unit test, AppVeyor

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark rtimestamptest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/17892.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17892


commit 83b48f8e08b86afab9bb127a2495f6eff55c363e
Author: Felix Cheung 
Date:   2017-05-07T23:10:07Z

set timezone in windows




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17892: [SPARK-20626][SPARKR] address date test warning w...

[felixcheung]

Github user felixcheung closed the pull request at:

https://github.com/apache/spark/pull/17892


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17892: [SPARK-20626][SPARKR] address date test warning with tim...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/17892
  
similar flaky test around ml, with this in the log
```
Exception in thread "SparkListenerBus" java.lang.IllegalAccessError: 
Attempted to access garbage collected accumulator 50693
at 
org.apache.spark.util.AccumulatorContext$$anonfun$get$1.apply(AccumulatorV2.scala:265)
at 
org.apache.spark.util.AccumulatorContext$$anonfun$get$1.apply(AccumulatorV2.scala:261)
at scala.Option.map(Option.scala:146)
at 
org.apache.spark.util.AccumulatorContext$.get(AccumulatorV2.scala:261)
at org.apache.spark.util.AccumulatorV2.name(AccumulatorV2.scala:88)
at 
org.apache.spark.sql.execution.metric.SQLMetric.toInfo(SQLMetrics.scala:67)
at 
org.apache.spark.sql.execution.ui.SQLListener$$anonfun$onTaskEnd$1.apply(SQLListener.scala:216)
at 
org.apache.spark.sql.execution.ui.SQLListener$$anonfun$onTaskEnd$1.apply(SQLListener.scala:216)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at 
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at 
org.apache.spark.sql.execution.ui.SQLListener.onTaskEnd(SQLListener.scala:216)
at 
org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:45)
at 
org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
at 
org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36)
at 
org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:63)
at 
org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:36)
at 
org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiveListenerBus.scala:94)
at 
org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
at 
org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
at 
org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:78)
at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1268)
at 
org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:77
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16781: [SPARK-12297][SQL] Hive compatibility for Parquet Timest...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16781
  
**[Test build #76556 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76556/testReport)**
 for PR 16781 at commit 
[`2537437`](https://github.com/apache/spark/commit/2537437a0c1f022acc215518e4728d24f6f6cf97).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16781: [SPARK-12297][SQL] Hive compatibility for Parquet Timest...

[ueshin]

Github user ueshin commented on the issue:

https://github.com/apache/spark/pull/16781
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17884: [SparkR][Doc] fix typo in vignettes

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17884
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76555/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17884: [SparkR][Doc] fix typo in vignettes

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17884
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17884: [SparkR][Doc] fix typo in vignettes

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17884
  
**[Test build #76555 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76555/testReport)**
 for PR 17884 at commit 
[`b0407b5`](https://github.com/apache/spark/commit/b0407b58b3be668e43ec4cdf0e7b81d578e5d0f7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17892: [SPARK-20626][SPARKR] address date test warning with tim...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17892
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76554/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17892: [SPARK-20626][SPARKR] address date test warning with tim...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17892
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17892: [SPARK-20626][SPARKR] address date test warning with tim...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17892
  
**[Test build #76554 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76554/testReport)**
 for PR 17892 at commit 
[`83b48f8`](https://github.com/apache/spark/commit/83b48f8e08b86afab9bb127a2495f6eff55c363e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17649: [SPARK-20380][SQL] Unable to set/unset table comment pro...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17649
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17649: [SPARK-20380][SQL] Unable to set/unset table comment pro...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17649
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76551/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17649: [SPARK-20380][SQL] Unable to set/unset table comment pro...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17649
  
**[Test build #76551 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76551/testReport)**
 for PR 17649 at commit 
[`4f02ada`](https://github.com/apache/spark/commit/4f02ada87ef1137530414b80f819c826f4055712).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17884: [SparkR][Doc] fix typo in vignettes

[felixcheung]

Github user felixcheung commented on a diff in the pull request:

https://github.com/apache/spark/pull/17884#discussion_r115157452
  
--- Diff: R/pkg/vignettes/sparkr-vignettes.Rmd ---
@@ -1079,19 +1079,19 @@ There are three main object classes in SparkR you 
may be working with.
 + `sdf` stores a reference to the corresponding Spark Dataset in the 
Spark JVM backend.
 + `env` saves the meta-information of the object such as `isCached`.
 
-It can be created by data import methods or by transforming an existing 
`SparkDataFrame`. We can manipulate `SparkDataFrame` by numerous data 
processing functions and feed that into machine learning algorithms.
+It can be created by data import methods or by transforming an 
existing `SparkDataFrame`. We can manipulate `SparkDataFrame` by numerous data 
processing functions and feed that into machine learning algorithms.
--- End diff --

cool! thanks


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17825: [SPARK-20550][SPARKR] R wrapper for Dataset.alias

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/17825


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17825: [SPARK-20550][SPARKR] R wrapper for Dataset.alias

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/17825
  
merged to master.
thank you for working on this and hopefully we could really improve a lot 
of the things we have discussed. 👍 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17884: [SparkR][Doc] fix typo in vignettes

[actuaryzhang]

Github user actuaryzhang commented on a diff in the pull request:

https://github.com/apache/spark/pull/17884#discussion_r115157036
  
--- Diff: R/pkg/vignettes/sparkr-vignettes.Rmd ---
@@ -405,7 +405,7 @@ result <- gapply(
 head(arrange(result, "max_mpg", decreasing = TRUE))
 ```
 
-Like gapply, `gapplyCollect` applies a function to each partition of a 
`SparkDataFrame` and collect the result back to R `data.frame`. The output of 
the function should be a `data.frame` but no schema is required in this case. 
Note that `gapplyCollect` can fail if the output of UDF run on all the 
partition cannot be pulled to the driver and fit in driver memory.
+Like gapply, `gapplyCollect` applies a function to each partition of a 
`SparkDataFrame` and collect the result back to R `data.frame`. The output of 
the function should be a `data.frame` but no schema is required in this case. 
Note that `gapplyCollect` can fail if the output of the UDF on all partitions 
cannot be pulled into the driver's memory.
--- End diff --

Done.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17884: [SparkR][Doc] fix typo in vignettes

[actuaryzhang]

Github user actuaryzhang commented on a diff in the pull request:

https://github.com/apache/spark/pull/17884#discussion_r115157013
  
--- Diff: R/pkg/vignettes/sparkr-vignettes.Rmd ---
@@ -1079,19 +1079,19 @@ There are three main object classes in SparkR you 
may be working with.
 + `sdf` stores a reference to the corresponding Spark Dataset in the 
Spark JVM backend.
 + `env` saves the meta-information of the object such as `isCached`.
 
-It can be created by data import methods or by transforming an existing 
`SparkDataFrame`. We can manipulate `SparkDataFrame` by numerous data 
processing functions and feed that into machine learning algorithms.
+It can be created by data import methods or by transforming an 
existing `SparkDataFrame`. We can manipulate `SparkDataFrame` by numerous data 
processing functions and feed that into machine learning algorithms.
--- End diff --

Minor points, but yes, the four spaces indicate that the text following 
should be aligned with the bullet point. Otherwise, it will start as a new 
paragraph and have the wrong indention.  
You will see the difference after compiling the Rmarkdown file. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17884: [SparkR][Doc] fix typo in vignettes

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17884
  
**[Test build #76555 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76555/testReport)**
 for PR 17884 at commit 
[`b0407b5`](https://github.com/apache/spark/commit/b0407b58b3be668e43ec4cdf0e7b81d578e5d0f7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17892: [SPARK-20626][SPARKR] address date test warning with tim...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17892
  
**[Test build #76554 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76554/testReport)**
 for PR 17892 at commit 
[`83b48f8`](https://github.com/apache/spark/commit/83b48f8e08b86afab9bb127a2495f6eff55c363e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17892: [SPARK-20626][SPARKR] address date test warning w...

[felixcheung]

GitHub user felixcheung opened a pull request:

https://github.com/apache/spark/pull/17892

[SPARK-20626][SPARKR] address date test warning with timezone on windows

## What changes were proposed in this pull request?

set timezone on windows

## How was this patch tested?

unit test, AppVeyor

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark rtimestamptest

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/17892.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17892


commit 83b48f8e08b86afab9bb127a2495f6eff55c363e
Author: Felix Cheung 
Date:   2017-05-07T23:10:07Z

set timezone in windows




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17887: [SPARK-20399][SQL][WIP] Add a config to fallback string ...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17887
  
Generally, it looks reasonable to me. Also cc @jodersky who hit this issue 
before. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14963
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/14963
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76550/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17887: [SPARK-20399][SQL][WIP] Add a config to fallback ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17887#discussion_r115156337
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -196,6 +196,14 @@ object SQLConf {
 .booleanConf
 .createWithDefault(true)
 
+  val NO_UNESCAPED_SQL_STRING = 
buildConf("spark.sql.noUnescapedStringLiteral")
+.internal()
+.doc("Since Spark 2.0, we use unescaped SQL string for string literals 
including regex. " +
+  "It is different than 1.6 behavior. Enabling this config can use no 
unescaped SQL string " +
+  "literals and mitigate migration problem.")
--- End diff --

How about
> When true, string literals (including regex patterns) remains escaped in 
our SQL parser. The default is false since Spark 2.0. Setting it to `true` can 
restore the behavior prior to Spark 2.0.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #14963: [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/14963
  
**[Test build #76550 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76550/testReport)**
 for PR 14963 at commit 
[`215b7b3`](https://github.com/apache/spark/commit/215b7b34170f112c4448fba98b02a50dbb19b2a7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17713: [SPARK-20417][SQL] Move subquery error handling to check...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17713
  
**[Test build #76553 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76553/testReport)**
 for PR 17713 at commit 
[`3c4f38e`](https://github.com/apache/spark/commit/3c4f38e3eddfd71cf82f543c70be7885980951d6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17793: [SPARK-20484][MLLIB] Add documentation to ALS code

[danielyli]

Github user danielyli commented on the issue:

https://github.com/apache/spark/pull/17793
  
Thanks all.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17887: [SPARK-20399][SQL][WIP] Add a config to fallback ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17887#discussion_r115155941
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParserUtils.scala
 ---
@@ -68,6 +68,11 @@ object ParserUtils {
   /** Convert a string node into a string. */
   def string(node: TerminalNode): String = unescapeSQLString(node.getText)
 
+  /** Convert a string node into a string without unescaping. */
+  def stringWithoutUnescape(node: TerminalNode): String = {
+node.getText.slice(1, node.getText.size - 1)
--- End diff --

For safety, do we still need to check whether the starting and ending 
characters are quotes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17825: [SPARK-20550][SPARKR] R wrapper for Dataset.alias

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17825
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76552/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17825: [SPARK-20550][SPARKR] R wrapper for Dataset.alias

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17825
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17825: [SPARK-20550][SPARKR] R wrapper for Dataset.alias

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17825
  
**[Test build #76552 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76552/testReport)**
 for PR 17825 at commit 
[`2b8f288`](https://github.com/apache/spark/commit/2b8f288e64ff21d5da22f6c5e9bc863c0a464854).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17887: [SPARK-20399][SQL][WIP] Add a config to fallback ...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/17887#discussion_r115155773
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -196,6 +196,14 @@ object SQLConf {
 .booleanConf
 .createWithDefault(true)
 
+  val NO_UNESCAPED_SQL_STRING = 
buildConf("spark.sql.noUnescapedStringLiteral")
--- End diff --

Double negatives are not encouraged in conf naming. This sounds the first 
parser conf. 

How about `spark.sql.parser.escapeStringLiterals`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17825: [SPARK-20550][SPARKR] R wrapper for Dataset.alias

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17825
  
**[Test build #76552 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76552/testReport)**
 for PR 17825 at commit 
[`2b8f288`](https://github.com/apache/spark/commit/2b8f288e64ff21d5da22f6c5e9bc863c0a464854).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org