[GitHub] spark issue #18494: [SPARK-21272] SortMergeJoin LeftAnti does not update num...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18494
  
**[Test build #79069 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79069/testReport)**
 for PR 18494 at commit 
[`580dc46`](https://github.com/apache/spark/commit/580dc4652783868036c67302ad131afc0ed136d9).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18508: Add a parameter to UnsafeExternalSorter to config...

[heary-cao]

Github user heary-cao closed the pull request at:

https://github.com/apache/spark/pull/18508


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18508: Add a parameter to UnsafeExternalSorter to config...

[heary-cao]

GitHub user heary-cao opened a pull request:

https://github.com/apache/spark/pull/18508

Add a parameter to UnsafeExternalSorter to configure filebuffersize

## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration 
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, 
remove this)

Please review http://spark.apache.org/contributing.html before opening a 
pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/heary-cao/spark UnsafeExternalSorter

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18508.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18508


commit 4a18aac29ffd23c6bce67271653366745edead0a
Author: caoxuewen 
Date:   2017-06-02T02:29:15Z

Add a parameter to UnsafeExternalSorter to configure filebuffersize




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18159: [SPARK-20703][SQL] Associate metrics with data wr...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/18159#discussion_r125213358
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
 ---
@@ -273,12 +271,36 @@ object FileFormatWriter extends Logging {
* automatically trigger task aborts.
*/
   private trait ExecuteWriteTask {
+
 /**
- * Writes data out to files, and then returns the list of partition 
strings written out.
- * The list of partitions is sent back to the driver and used to 
update the catalog.
+ * The data structures used to measure metrics during writing.
  */
-def execute(iterator: Iterator[InternalRow]): Set[String]
+protected val writingTimePerFile: mutable.ArrayBuffer[Long] = 
mutable.ArrayBuffer.empty
--- End diff --

Since we only care about average writing time, why we send back 
`writingTimePerFile`? Can we just send back total writing time and numFiles?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18159: [SPARK-20703][SQL] Associate metrics with data wr...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/18159#discussion_r125213106
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala
 ---
@@ -53,11 +55,22 @@ case class InsertIntoHadoopFsRelationCommand(
 mode: SaveMode,
 catalogTable: Option[CatalogTable],
 fileIndex: Option[FileIndex])
-  extends RunnableCommand {
+  extends RunnableCommand with MetricUpdater {
   import 
org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils.escapePathName
 
   override def children: Seq[LogicalPlan] = query :: Nil
 
+  override lazy val metrics: Map[String, SQLMetric] = {
--- End diff --

can we move this to the parent trait?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17758
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18502: [SPARK-21278][PYSPARK][WIP] Upgrade to Py4J 0.10.5

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18502
  
**[Test build #79076 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79076/testReport)**
 for PR 18502 at commit 
[`f708dde`](https://github.com/apache/spark/commit/f708ddec38917867f9f13c7136ecef28c46af3a1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17758
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79062/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17758
  
**[Test build #79062 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79062/testReport)**
 for PR 17758 at commit 
[`a9f934e`](https://github.com/apache/spark/commit/a9f934ef420991c0a59130dd41f5e6b98a459096).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class PreprocessDDLCommands(sparkSession: SparkSession) extends 
Rule[LogicalPlan] `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18502: [SPARK-21278][PYSPARK][WIP] Upgrade to Py4J 0.10.5

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/18502
  
Retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...

[manku-timma]

Github user manku-timma commented on the issue:

https://github.com/apache/spark/pull/18174
  
Just to understand what is happening.

1. Shuffle records are written to a serialisation buffer (1M) after 
serialisation
2. The serialised buffer is written to in-memory-sorter’s buffer
3. once in-memory sorter’s buffer is full, the data is copied to 
sorter’s disk buffer (1M)
4. the sorter’s disk buffer is written out to a buffered output stream 
(buffer = 32k)

I am guessing reducing the sorter’s disk buffer (in step 3) is helping 
because it triggers fewer writes at the step 4.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18159: [SPARK-20703][SQL] Associate metrics with data wr...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/18159#discussion_r125212516
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -47,10 +56,56 @@ trait RunnableCommand extends logical.Command {
 }
 
 /**
+ * A trait for classes that can update its metrics of data writing 
operation.
+ */
+trait MetricUpdater {
+
+  val metrics: Map[String, SQLMetric]
+
+  /**
+   * Callback function that update metrics collected from the writing 
operation.
+   */
+  protected def callbackMetricsUpdater(writeSummaries: 
Seq[ExecutedWriteSummary]): Unit = {
--- End diff --

how about `updateWritingMetrics`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18159: [SPARK-20703][SQL] Associate metrics with data wr...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/18159#discussion_r125212464
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
---
@@ -47,10 +56,56 @@ trait RunnableCommand extends logical.Command {
 }
 
 /**
+ * A trait for classes that can update its metrics of data writing 
operation.
+ */
+trait MetricUpdater {
--- End diff --

I'd like to call it `trait InsertionCommand extends RunnableCommand`, as we 
are updating `avgTime`, `numFiles` etc, which is specific to insertion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18174
  
**[Test build #79074 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79074/testReport)**
 for PR 18174 at commit 
[`f6d895c`](https://github.com/apache/spark/commit/f6d895c944c514b7e51db19388ef00016671dddb).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17758
  
**[Test build #79075 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79075/testReport)**
 for PR 17758 at commit 
[`12159c4`](https://github.com/apache/spark/commit/12159c403955f54066ed8c532ed991f829edfc1f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...

[maropu]

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/17758
  
Jenkins, retest this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18174: [SPARK-20950][CORE]add a new config to diskWriteBufferSi...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18174
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18441: [SPARK-21137][CORE] Spark reads many small files ...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/18441#discussion_r125212029
  
--- Diff: core/src/main/scala/org/apache/spark/rdd/BinaryFileRDD.scala ---
@@ -35,8 +36,12 @@ private[spark] class BinaryFileRDD[T](
   extends NewHadoopRDD[String, T](sc, inputFormatClass, keyClass, 
valueClass, conf) {
 
   override def getPartitions: Array[Partition] = {
-val inputFormat = inputFormatClass.newInstance
 val conf = getConf
+// setMinPartitions below will call FileInputFormat.listStatus(), 
which can be quite slow when
+// traversing a large number of directories and files. Parallelize it.
+conf.setIfUnset(FileInputFormat.LIST_STATUS_NUM_THREADS,
+  Runtime.getRuntime.availableProcessors().toString)
--- End diff --

shall we use `CPU_CORES_PER_EXECUTOR`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18464: [SPARK-21250][WEB-UI]Add a url in the table of 'R...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18464


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18464: [SPARK-21250][WEB-UI]Add a url in the table of 'Running ...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18464
  
thanks, merging to master!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18506: [SPARK-21282] [TEST] [2.0] Fix test failure in 2.0

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18506
  
LGTM, merging to 2.0!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18445: [Spark-19726][SQL] Faild to insert null timestamp value ...

[shuangshuangwang]

Github user shuangshuangwang commented on the issue:

https://github.com/apache/spark/pull/18445
  
Hi @gatorsmile,
I don't understand "Nit: -> true. Conceptually, they are different.", Or 
what do you mean:
```
val nullable = if (alwaysNullable) {
true
} else {
rsmd.isNullable(i + 1) != ResultSetMetaData.columnNoNulls
}
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18413: [SPARK-21205][SQL] pmod(number, 0) should be null.

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18413
  
**[Test build #79073 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79073/testReport)**
 for PR 18413 at commit 
[`da037c8`](https://github.com/apache/spark/commit/da037c810a8c121d7075b741478419ffb77202d8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17758
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79064/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17758
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17758
  
**[Test build #79064 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79064/testReport)**
 for PR 17758 at commit 
[`12159c4`](https://github.com/apache/spark/commit/12159c403955f54066ed8c532ed991f829edfc1f).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class PreprocessDDLCommands(sparkSession: SparkSession) extends 
Rule[LogicalPlan] `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18506: [SPARK-21282] [TEST] [2.0] Fix test failure in 2.0

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18506
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79061/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18506: [SPARK-21282] [TEST] [2.0] Fix test failure in 2.0

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18506
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18506: [SPARK-21282] [TEST] [2.0] Fix test failure in 2.0

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18506
  
**[Test build #79061 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79061/consoleFull)**
 for PR 18506 at commit 
[`cfc2e7e`](https://github.com/apache/spark/commit/cfc2e7e1904743242a9c38cbd7116fbdd3596da8).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17995: [SPARK-20762][ML]Make String Params Case-Insensitive

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17995
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79068/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17995: [SPARK-20762][ML]Make String Params Case-Insensitive

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17995
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17995: [SPARK-20762][ML]Make String Params Case-Insensitive

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17995
  
**[Test build #79068 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79068/testReport)**
 for PR 17995 at commit 
[`1997cd1`](https://github.com/apache/spark/commit/1997cd13cd5bca8624367ea2e0363c26e5de2d8a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17995: [SPARK-20762][ML]Make String Params Case-Insensitive

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17995
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79065/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17995: [SPARK-20762][ML]Make String Params Case-Insensitive

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17995
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17995: [SPARK-20762][ML]Make String Params Case-Insensitive

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17995
  
**[Test build #79065 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79065/testReport)**
 for PR 17995 at commit 
[`6557b37`](https://github.com/apache/spark/commit/6557b37534779bdedee7f781daecb2140681fd86).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18501: [SPARK-20256][SQL] SessionState should be created more l...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/18501
  
Thank you, @cloud-fan . I'll try like that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18501: [SPARK-20256][SQL] SessionState should be created more l...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18501
  
My commit was reverted because my assumption was wrong. Some configurations 
have to be set before creating `SparkContext`, so we can't just create 
`SparkContext` and then set confs.

So the corrected logic should be:
1. if `SparkContext` is not created, build a `SparkConf` including the 
given options, and create `SparkContext`.
2. if `SparkContext` has been created, set its conf according to the given 
options.

Then we can safely remove the line `options.foreach { case (k, v) => 
session.sessionState.conf.setConfString(k, v) }`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17227: [SPARK-19507][PySpark][SQL] Show field name in _v...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/17227#discussion_r125204915
  
--- Diff: python/pyspark/sql/tests.py ---
@@ -2367,6 +2380,162 @@ def range_frame_match():
 
 importlib.reload(window)
 
+
+class TypesTest(unittest.TestCase):
+
+def test_verify_type_exception_msg(self):
+name = "test_name"
+try:
+_verify_type(None, StringType(), nullable=False, name=name)
+self.fail('Expected _verify_type() to throw so test can check 
exception message')
+except Exception as e:
+self.assertTrue(str(e).startswith(name))
+
+def test_verify_type_ok_nullable(self):
+obj = None
+for data_type in [IntegerType(), FloatType(), StringType(), 
StructType([])]:
+msg = "_verify_type(%s, %s, nullable=True)" % (obj, data_type)
+try:
+_verify_type(obj, data_type, nullable=True)
+except Exception as e:
+traceback.print_exc()
+self.fail(msg)
+
+def test_verify_type_not_nullable(self):
+import array
+import datetime
+import decimal
+
+MyStructType = StructType([
--- End diff --

Could we make the first character this lower-cased? (or maybe just simply 
`schema`?)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17227: [SPARK-19507][PySpark][SQL] Show field name in _v...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/17227#discussion_r125205112
  
--- Diff: python/pyspark/sql/types.py ---
@@ -1249,7 +1249,7 @@ def _infer_schema_type(obj, dataType):
 }
 
 
-def _verify_type(obj, dataType, nullable=True):
+def _verify_type(obj, dataType, nullable=True, name="obj"):
--- End diff --

Could we maybe then `None` and not print?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18444
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18444
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79059/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18444: [SPARK-16542][SQL][PYSPARK] Fix bugs about types that re...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18444
  
**[Test build #79059 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79059/testReport)**
 for PR 18444 at commit 
[`37e28a4`](https://github.com/apache/spark/commit/37e28a4e34a1264118086ef9298c9fab69542a72).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18501: [SPARK-20256][SQL] SessionState should be created more l...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18501
  
**[Test build #79072 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79072/testReport)**
 for PR 18501 at commit 
[`8a1a64f`](https://github.com/apache/spark/commit/8a1a64f1d1c429709799c00087dabfb97f4ca8b7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18445: [Spark-19726][SQL] Faild to insert null timestamp value ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18445
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79057/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18445: [Spark-19726][SQL] Faild to insert null timestamp value ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18445
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17985: Add "full_outer" name to join types

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17985
  
**[Test build #79071 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79071/testReport)**
 for PR 17985 at commit 
[`9fc9a0a`](https://github.com/apache/spark/commit/9fc9a0ad567dfb28d22d94321fcef0ea3b1ae73b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18488: [SPARK-21255][SQL] Fixed NPE when creating encoder for e...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18488
  
**[Test build #79070 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79070/testReport)**
 for PR 18488 at commit 
[`120bb32`](https://github.com/apache/spark/commit/120bb32bbfec13512e032660309bafb273796c32).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18445: [Spark-19726][SQL] Faild to insert null timestamp value ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18445
  
**[Test build #79057 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79057/testReport)**
 for PR 18445 at commit 
[`718e949`](https://github.com/apache/spark/commit/718e9497b060796b46dd0afd00b30ece6adbd188).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18501: [SPARK-20256][SQL] SessionState should be created more l...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/18501
  
@cloud-fan and @gatorsmile .
In this PR, I'll revert to the first commit until your commit will be 
merged.
For #18172 , I'm not sure the reason why it's reverted. But, as @cloud-fan 
's suggestion, I'll retry the reverted commit under @cloud-fan 's name in 
another PR. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17985: Add "full_outer" name to join types

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/17985
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18488: [SPARK-21255][SQL] Fixed NPE when creating encoder for e...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18488
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18505: [MINOR][SPARK SUBMIT] Print out R file usage in spark-su...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18505
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79055/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18505: [MINOR][SPARK SUBMIT] Print out R file usage in spark-su...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18505
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18505: [MINOR][SPARK SUBMIT] Print out R file usage in spark-su...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18505
  
**[Test build #79055 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79055/testReport)**
 for PR 18505 at commit 
[`5b2b8c2`](https://github.com/apache/spark/commit/5b2b8c2eb7778c9866e0b72f4ddb54625b2e5ba8).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18494: [SPARK-21272] SortMergeJoin LeftAnti does not update num...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18494
  
**[Test build #79069 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79069/testReport)**
 for PR 18494 at commit 
[`580dc46`](https://github.com/apache/spark/commit/580dc4652783868036c67302ad131afc0ed136d9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18494: [SPARK-21272] SortMergeJoin LeftAnti does not update num...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18494
  
add to whitelist


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17995: [SPARK-20762][ML]Make String Params Case-Insensitive

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17995
  
**[Test build #79068 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79068/testReport)**
 for PR 17995 at commit 
[`1997cd1`](https://github.com/apache/spark/commit/1997cd13cd5bca8624367ea2e0363c26e5de2d8a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18507: [SPARK-21283][core]FileOutputStream should be created as...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18507
  
**[Test build #79067 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79067/testReport)**
 for PR 18507 at commit 
[`9788b19`](https://github.com/apache/spark/commit/9788b19d06800cce243a79acc189c3424912f393).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18501: [SPARK-20256][SQL] SessionState should be created...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/18501#discussion_r125207506
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
---
@@ -940,7 +940,6 @@ object SparkSession {
 }
 
 session = new SparkSession(sparkContext, None, None, extensions)
-options.foreach { case (k, v) => 
session.sessionState.conf.setConfString(k, v) }
--- End diff --

Thank you for review, @gatorsmile .
I see. Then, @cloud-fan meant the whole #18172 intead of that line.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17995: [SPARK-20762][ML]Make String Params Case-Insensitive

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17995
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79066/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17995: [SPARK-20762][ML]Make String Params Case-Insensitive

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17995
  
**[Test build #79066 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79066/testReport)**
 for PR 17995 at commit 
[`1715131`](https://github.com/apache/spark/commit/1715131718260cf1295a8960e49c20bcda6e1c4f).
 * This patch **fails to build**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17995: [SPARK-20762][ML]Make String Params Case-Insensitive

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17995
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18507: [SPARK-21283][core]FileOutputStream should be cre...

[10110346]

GitHub user 10110346 opened a pull request:

https://github.com/apache/spark/pull/18507

[SPARK-21283][core]FileOutputStream should be created as append mode

## What changes were proposed in this pull request?

`FileAppender` is used to write `stderr` and `stdout` files  in 
`ExecutorRunner`, But before writing `ErrorStream` into the the `stderr` file, 
the header information has been written into ,if  FileOutputStream is  not 
created as append mode, the  header information will be lost

## How was this patch tested?
unit test case

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/10110346/spark wip-lx-0703

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18507.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18507


commit 9788b19d06800cce243a79acc189c3424912f393
Author: liuxian 
Date:   2017-07-03T03:27:09Z

FileOutputStream should be created as append mode




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18469: [SPARK-21256] [SQL] Add withSQLConf to Catalyst T...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18469#discussion_r125207354
  
--- Diff: 
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/PlanTest.scala 
---
@@ -28,8 +29,9 @@ import org.apache.spark.sql.internal.SQLConf
 /**
  * Provides helper methods for comparing plans.
  */
-abstract class PlanTest extends SparkFunSuite with PredicateHelper {
+trait PlanTest extends SparkFunSuite with PredicateHelper {
 
+  // TODO(gatorsmile): remove this from PlanTest and all the 
analyzer/optimizer rules
   protected val conf = new SQLConf().copy(SQLConf.CASE_SENSITIVE -> true)
--- End diff --

This line should not be needed. We can use the global SQLConf for Catalyst 
package.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18506: [SPARK-21282] [TEST] [2.0] Fix test failure in 2.0

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/18506
  
+1, LGTM.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18469: [SPARK-21256] [SQL] Add withSQLConf to Catalyst T...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18469#discussion_r125207314
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala ---
@@ -89,28 +92,11 @@ private[sql] trait SQLTestUtils
 }
   }
 
-  /**
-   * Sets all SQL configurations specified in `pairs`, calls `f`, and then 
restore all SQL
-   * configurations.
-   *
-   * @todo Probably this method should be moved to a more general place
-   */
-  protected def withSQLConf(pairs: (String, String)*)(f: => Unit): Unit = {
-val (keys, values) = pairs.unzip
-val currentValues = keys.map { key =>
-  if (spark.conf.contains(key)) {
-Some(spark.conf.get(key))
-  } else {
-None
-  }
-}
-(keys, values).zipped.foreach(spark.conf.set)
-try f finally {
-  keys.zip(currentValues).foreach {
-case (key, Some(value)) => spark.conf.set(key, value)
-case (key, None) => spark.conf.unset(key)
-  }
-}
+  protected override def withSQLConf(pairs: (String, String)*)(f: => 
Unit): Unit = {
+// ensure spark's session has been initialized and set to the current 
SQLConf.confGetter
+// TODO: fix the multi-session supports for SQLConf.confGetter
+SQLConf.setSQLConfGetter(() => spark.sessionState.conf)
--- End diff --

Ideally, directly calling `withSQLConf` of `PlanTest` should work. That 
means, this line is not needed. However, it does not work in the current 
`SQLConf.confGetter`. Another PR is needed to fix that issue.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18501: [SPARK-20256][SQL] SessionState should be created...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18501#discussion_r125207136
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
---
@@ -940,7 +940,6 @@ object SparkSession {
 }
 
 session = new SparkSession(sparkContext, None, None, extensions)
-options.foreach { case (k, v) => 
session.sessionState.conf.setConfString(k, v) }
--- End diff --

The original @cloud-fan 's fix is to set them to `sparkContext.conf` 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17995: [SPARK-20762][ML]Make String Params Case-Insensitive

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17995
  
**[Test build #79066 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79066/testReport)**
 for PR 17995 at commit 
[`1715131`](https://github.com/apache/spark/commit/1715131718260cf1295a8960e49c20bcda6e1c4f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16056: [SPARK-18623][SQL] Add `returnNullable` to `StaticInvoke...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16056
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16056: [SPARK-18623][SQL] Add `returnNullable` to `StaticInvoke...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16056
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79056/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16056: [SPARK-18623][SQL] Add `returnNullable` to `StaticInvoke...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16056
  
**[Test build #79056 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79056/testReport)**
 for PR 16056 at commit 
[`b849b59`](https://github.com/apache/spark/commit/b849b59f03c824be0530565032154f12e5001c66).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18501: [SPARK-20256][SQL] SessionState should be created...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18501#discussion_r125206933
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
---
@@ -940,7 +940,6 @@ object SparkSession {
 }
 
 session = new SparkSession(sparkContext, None, None, extensions)
-options.foreach { case (k, v) => 
session.sessionState.conf.setConfString(k, v) }
--- End diff --

`options` are not set by `mergeSparkConf`. Removing this line is wrong.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17995: [SPARK-20762][ML]Make String Params Case-Insensitive

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17995
  
**[Test build #79060 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79060/testReport)**
 for PR 17995 at commit 
[`c58614f`](https://github.com/apache/spark/commit/c58614f8e8a08a86d288094def2dd35543b20062).
 * This patch **fails PySpark pip packaging tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17995: [SPARK-20762][ML]Make String Params Case-Insensitive

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17995
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79060/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17995: [SPARK-20762][ML]Make String Params Case-Insensitive

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17995
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17995: [SPARK-20762][ML]Make String Params Case-Insensitive

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17995
  
**[Test build #79065 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79065/testReport)**
 for PR 17995 at commit 
[`6557b37`](https://github.com/apache/spark/commit/6557b37534779bdedee7f781daecb2140681fd86).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18474: [SPARK-21235][TESTS] UTest should clear temp results whe...

[jiangxb1987]

Github user jiangxb1987 commented on the issue:

https://github.com/apache/spark/pull/18474
  
Sorry but I can't repro this on my local environment. Could you provide 
more detail on this? Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...

[maropu]

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/17758
  
BTW (I think this is not related to this pr though), I saw many validation 
checks in `RunnableCommand.run()`. IMHO these checks also should be done in an 
analyzer phase (e.g., 
[here](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala#L193)),
 maybe


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18474: [SPARK-21235][TESTS] UTest should clear temp results whe...

[wangjiaochun]

Github user wangjiaochun commented on the issue:

https://github.com/apache/spark/pull/18474
  
I have run this case many times,the memoryStore temp file will be 
cleared,but the disk blocks is really not clear. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17758: [SPARK-20460][SPARK-21144][SQL] Make it more cons...

[maropu]

Github user maropu commented on a diff in the pull request:

https://github.com/apache/spark/pull/17758#discussion_r125205974
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -154,14 +144,11 @@ case class CreateViewCommand(
   } else if (tableMetadata.tableType != CatalogTableType.VIEW) {
 throw new AnalysisException(s"$name is not a view")
   } else if (replace) {
-// Detect cyclic view reference on CREATE OR REPLACE VIEW.
-val viewIdent = tableMetadata.identifier
-checkCyclicViewReference(analyzedPlan, Seq(viewIdent), viewIdent)
--- End diff --

To pass the existing tests, I moved `checkCyclicViewReference` into 
`rules`. Since the duplication checks also catch the cyclic cases, I think we 
need to check the cyclic cases first, and then check the name duplication.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18464: [SPARK-21250][WEB-UI]Add a url in the table of 'Running ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18464
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79054/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18464: [SPARK-21250][WEB-UI]Add a url in the table of 'Running ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18464
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18464: [SPARK-21250][WEB-UI]Add a url in the table of 'Running ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18464
  
**[Test build #79054 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79054/testReport)**
 for PR 18464 at commit 
[`1411ed9`](https://github.com/apache/spark/commit/1411ed90741c4086f55477097aae719d47f7c3de).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18182: [SPARK-20959][CORE]Add a parameter to UnsafeExter...

[heary-cao]

Github user heary-cao closed the pull request at:

https://github.com/apache/spark/pull/18182


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #17758: [SPARK-20460][SPARK-21144][SQL] Make it more cons...

[maropu]

Github user maropu commented on a diff in the pull request:

https://github.com/apache/spark/pull/17758#discussion_r125205813
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala ---
@@ -123,28 +122,19 @@ case class CreateViewCommand(
   }
 
   override def run(sparkSession: SparkSession): Seq[Row] = {
-// If the plan cannot be analyzed, throw an exception and don't 
proceed.
-val qe = sparkSession.sessionState.executePlan(child)
-qe.assertAnalyzed()
-val analyzedPlan = qe.analyzed
-
 if (userSpecifiedColumns.nonEmpty &&
-userSpecifiedColumns.length != analyzedPlan.output.length) {
+userSpecifiedColumns.length != child.output.length) {
   throw new AnalysisException(s"The number of columns produced by the 
SELECT clause " +
-s"(num: `${analyzedPlan.output.length}`) does not match the number 
of column names " +
+s"(num: `${child.output.length}`) does not match the number of 
column names " +
 s"specified by CREATE VIEW (num: 
`${userSpecifiedColumns.length}`).")
 }
 
-// When creating a permanent view, not allowed to reference temporary 
objects.
-// This should be called after `qe.assertAnalyzed()` (i.e., `child` 
can be resolved)
-verifyTemporaryObjectsNotExists(sparkSession)
--- End diff --

I moved `verifyTemporaryObjectsNotExists` to `rules` because 
`qe.assertAnalyzed()` is called in `rules` and a resolved plan is passed here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18441: [SPARK-21137][CORE] Spark reads many small files slowly

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18441
  
**[Test build #79063 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79063/testReport)**
 for PR 18441 at commit 
[`2fc2d9a`](https://github.com/apache/spark/commit/2fc2d9a3b66407666c57484cd20d74a49f62df27).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17758
  
**[Test build #79064 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79064/testReport)**
 for PR 17758 at commit 
[`12159c4`](https://github.com/apache/spark/commit/12159c403955f54066ed8c532ed991f829edfc1f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18441: [SPARK-21137][CORE] Spark reads many small files slowly

[jiangxb1987]

Github user jiangxb1987 commented on the issue:

https://github.com/apache/spark/pull/18441
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18308: [SPARK-21099][Spark Core] INFO Log Message Using Incorre...

[jiangxb1987]

Github user jiangxb1987 commented on the issue:

https://github.com/apache/spark/pull/18308
  
Can you update the title to:
```
[SPARK-21099][Core] Log cachedExecutorIdleTimeoutS instead of 
executorIdleTimeoutS  if the executor has cached blocks
```
?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18469: [SPARK-21256] [SQL] Add withSQLConf to Catalyst Test

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18469
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79052/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18469: [SPARK-21256] [SQL] Add withSQLConf to Catalyst Test

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18469
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18469: [SPARK-21256] [SQL] Add withSQLConf to Catalyst Test

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18469
  
**[Test build #79052 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79052/testReport)**
 for PR 18469 at commit 
[`414d642`](https://github.com/apache/spark/commit/414d64228554669012363326226d51ebe5c61ded).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class CastSuite extends SparkFunSuite with ExpressionEvalHelper `
  * `class DateExpressionsSuite extends SparkFunSuite with 
ExpressionEvalHelper `
  * `class JsonExpressionsSuite extends SparkFunSuite with 
ExpressionEvalHelper `
  * `class InferFiltersFromConstraintsSuite extends PlanTest `
  * `class OuterJoinEliminationSuite extends PlanTest `
  * `class PruneFiltersSuite extends PlanTest `
  * `class ConstraintPropagationSuite extends SparkFunSuite with PlanTest `
  * `trait PlanTest extends SparkFunSuite with PredicateHelper `
  * `class AggregateEstimationSuite extends StatsEstimationTestBase with 
PlanTest `
  * `class BasicStatsEstimationSuite extends PlanTest with 
StatsEstimationTestBase `
  * `class DateTimeUtilsSuite extends SparkFunSuite `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17227: [SPARK-19507][PySpark][SQL] Show field name in _verify_t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17227
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17227: [SPARK-19507][PySpark][SQL] Show field name in _verify_t...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/17227
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79058/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17227: [SPARK-19507][PySpark][SQL] Show field name in _verify_t...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17227
  
**[Test build #79058 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79058/testReport)**
 for PR 17227 at commit 
[`6c1e0b6`](https://github.com/apache/spark/commit/6c1e0b690bdd1914b5056c8b2934614534c622cb).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18506: [SPARK-21282] [TEST] [2.0] Fix test failure in 2.0

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18506
  
**[Test build #79061 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79061/consoleFull)**
 for PR 18506 at commit 
[`cfc2e7e`](https://github.com/apache/spark/commit/cfc2e7e1904743242a9c38cbd7116fbdd3596da8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/17758
  
**[Test build #79062 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79062/testReport)**
 for PR 17758 at commit 
[`a9f934e`](https://github.com/apache/spark/commit/a9f934ef420991c0a59130dd41f5e6b98a459096).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18368: [SPARK-21102][SQL] Make refresh resource command less ag...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18368
  
@aokolnychyi Could you please fix the PR title?
```
[SPARK-21102][SQL] Refresh command is too aggressive in parsing
```

Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17758: [SPARK-20460][SPARK-21144][SQL] Make it more consistent ...

[maropu]

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/17758
  
Based on the suggestion @cloud-fan  did, I brushed up this code again and 
fixed the policy to check the duplication;

The check for the SQL(DDL) case:
 - This check should be done in `PreprocessDDLCommands` (that is, in the 
analyzer). So, I moved the existing checks into there.

The check for the datasource case:
 - The check for a user-defined data/paritiotn schema should be done in the 
DataSource constructor.
 - In the inferred case via `FileFormat` and `FileIndex`, the check sould 
be done in `getOrInferFileFormatSchema` (So, if we add a new format in 
datasources, we need not check for the format).

Since the original target in this pr was to make the existing duplication 
check more explicit, I didn't touch the existing behaviour as much as possible. 
For example;
```
scala> Seq((1, 1)).toDF("a", "a").createOrReplaceTempView("t")

scala> sql("SELECT * FROM t").show
+---+---+
|  a|  a|
+---+---+
|  1|  1|
+---+---+
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18506: [SPARK-21282] [TEST] [2.0] Fix test failure in 2.0

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18506
  
cc @cloud-fan @srowen 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org