[GitHub] spark pull request #16068: [SPARK-18637][SQL]Stateful UDF should be consider...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16068#discussion_r90752814
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDFSuite.scala 
---
@@ -487,6 +488,29 @@ class HiveUDFSuite extends QueryTest with 
TestHiveSingleton with SQLTestUtils {
 assert(count4 == 1)
 sql("DROP TABLE parquet_tmp")
   }
+
+  test("Hive Stateful UDF") {
+sql(s"CREATE TEMPORARY FUNCTION statefulUDF AS 
'${classOf[StatefulUDF].getName}'")
+sql(s"CREATE TEMPORARY FUNCTION statelessUDF AS 
'${classOf[StatelessUDF].getName}'")
+val testData = spark.sparkContext.parallelize(
+  (0 until 10) map(x => IntegerCaseClass(1)), 2).toDF()
+testData.createOrReplaceTempView("inputTable")
+val max1 =
+  sql("SELECT MAX(s) FROM (" +
+"SELECT statefulUDF() as s FROM (SELECT i from inputTable 
DISTRIBUTE by i) a" +
+") b").head().getLong(0)
--- End diff --

How about using multiline string syntax here?
```Scala
sql(
  """
|SELECT MAX(s) FROM
|  (SELECT statefulUDF() as s FROM
|(SELECT i from inputTable DISTRIBUTE BY i) a
|  ) b
  """.stripMargin)
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16128: [WIP][SPARK-18671][SS] Added tests to ensure stability o...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16128
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69609/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16128: [WIP][SPARK-18671][SS] Added tests to ensure stability o...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16128
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16128: [WIP][SPARK-18671][SS] Added tests to ensure stability o...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16128
  
**[Test build #69609 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69609/consoleFull)**
 for PR 16128 at commit 
[`d9be1c5`](https://github.com/apache/spark/commit/d9be1c5b78ca7ac9f42f69bb267feea36ab3fb50).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15795: [SPARK-18081][ML][DOCS] Add user guide for Locality Sens...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15795
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15795: [SPARK-18081][ML][DOCS] Add user guide for Locality Sens...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15795
  
**[Test build #69613 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69613/consoleFull)**
 for PR 15795 at commit 
[`7c09f9a`](https://github.com/apache/spark/commit/7c09f9a4705cc47e01b91a3d8ae85d44c2790f0d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15795: [SPARK-18081][ML][DOCS] Add user guide for Locality Sens...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15795
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69613/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16068: [SPARK-18637][SQL]Stateful UDF should be considered as n...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16068
  
We have a rule 
[`PullOutNondeterministic`](https://github.com/apache/spark/blob/2f8776ccad532fbed17381ff97d302007918b8d8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala#L1868
) to pull out the expressions that extend `Nondeterministic`. That does not 
include `HiveSimpleUDF`. Does this have an impact your case?

Also cc @hvanhovell @cloud-fan 





---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15795: [SPARK-18081][ML][DOCS] Add user guide for Locality Sens...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15795
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69612/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16098: [SPARK-18672][CORE] Close recordwriter in SparkHadoopMap...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16098
  
**[Test build #69614 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69614/consoleFull)**
 for PR 16098 at commit 
[`4804862`](https://github.com/apache/spark/commit/48048622067f092ed247bc555e5461c073894a9c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15795: [SPARK-18081][ML][DOCS] Add user guide for Locality Sens...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15795
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15795: [SPARK-18081][ML][DOCS] Add user guide for Locality Sens...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15795
  
**[Test build #69612 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69612/consoleFull)**
 for PR 15795 at commit 
[`7922117`](https://github.com/apache/spark/commit/79221170907075d447e238624362393882ca5541).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `* Locality Sensitive Hashing (LSH): This class of algorithms combines 
aspects of feature transformation with other algorithms.`
  * `[Locality Sensitive Hashing 
(LSH)](https://en.wikipedia.org/wiki/Locality-sensitive_hashing) is an 
important class of hashing techniques, which is commonly used in clustering, 
approximate nearest neighbor search and outlier detection with large datasets.`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16098: [SPARK-18672][CORE] Close recordwriter in SparkHadoopMap...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/16098
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15795: [SPARK-18081][ML][DOCS] Add user guide for Locality Sens...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15795
  
**[Test build #69613 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69613/consoleFull)**
 for PR 15795 at commit 
[`7c09f9a`](https://github.com/apache/spark/commit/7c09f9a4705cc47e01b91a3d8ae85d44c2790f0d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15795: [SPARK-18081][ML][DOCS] Add user guide for Locality Sens...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15795
  
**[Test build #69612 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69612/consoleFull)**
 for PR 15795 at commit 
[`7922117`](https://github.com/apache/spark/commit/79221170907075d447e238624362393882ca5541).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

[brkyvz]

Github user brkyvz commented on the issue:

https://github.com/apache/spark/pull/16030
  
Looks great! Thanks so much @maropu . @tdas or @liancheng would you mind 
taking a last look and possibly merging to 2.1?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16030
  
**[Test build #69611 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69611/consoleFull)**
 for PR 16030 at commit 
[`1ab3363`](https://github.com/apache/spark/commit/1ab3363746d9c53fdcdf24564020fe3a784be06a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16043: [SPARK-18601][SQL] Simplify Create/Get complex expressio...

[eyalfa]

Github user eyalfa commented on the issue:

https://github.com/apache/spark/pull/16043
  
Will do.
I'm currently focusing on a functional issue with simplifying operations on
createMap, I'll take care of these while on it.

On Dec 2, 2016 11:09 PM, "Xiao Li"  wrote:

> I just find some typical style issues. Can you please correct the
> remaining ones? Thanks!
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16030: [SPARK-18108][SQL] Fix a bug to fail partition sc...

[maropu]

Github user maropu commented on a diff in the pull request:

https://github.com/apache/spark/pull/16030#discussion_r90752283
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
 ---
@@ -189,7 +189,15 @@ case class DataSource(
   throw new AnalysisException(
 s"Unable to infer schema for $format. It must be specified 
manually.")
 }
-(dataSchema, partitionSchema)
+
+// Override the fields of the partition schema if the data schema has 
the same field
+val resolvedPartitionSchema = partitionSchema.map { partitionField =>
--- End diff --

yea, removed


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16128: [SPARK-18671][SS] Added tests to ensure stability...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/16128#discussion_r90752217
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
 ---
@@ -60,4 +60,39 @@ class OffsetSeqLogSuite extends SparkFunSuite with 
SharedSQLContext {
 Array(0 -> batch0Serialized, 1 -> batch1Serialized))
 }
   }
+
+  test("read Spark 2.1.0 log format") {
+val (batchId, offsetSeq) = readFromResource("offset-log-version-2.1.0")
+assert(batchId === 16)
+assert(offsetSeq ===
+  OffsetSeq(
+offsets = Seq(Some(SerializedOffset(
+  """
+
|{"kafka-topic":{"23":0,"8":1,"17":1,"11":1,"20":0,"2":6,"5":2,"14":0,"4":4,"13":1,
--- End diff --

This needs to be revised. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16125: [SPARK-18694][SS]Add StreamingQuery.explain and exceptio...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16125
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69607/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16125: [SPARK-18694][SS]Add StreamingQuery.explain and exceptio...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16125
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16125: [SPARK-18694][SS]Add StreamingQuery.explain and exceptio...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16125
  
**[Test build #69607 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69607/consoleFull)**
 for PR 16125 at commit 
[`169cd20`](https://github.com/apache/spark/commit/169cd209343d255299b5fa20b407c0a9fefc3c9c).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

[maropu]

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/16030
  
@brkyvz yea, I think so, but the issue of column ordering seems to be out 
of scope in this original ticket. So, why don't we open a new jira and discuss 
there?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16030: [SPARK-18108][SQL] Fix a bug to fail partition schema in...

[brkyvz]

Github user brkyvz commented on the issue:

https://github.com/apache/spark/pull/16030
  
That is something I tried to standardize. I think we should standardize it,
otherwise it is very confusing. I think it's the right thing to do. I just
don't want people's production jobs failing in weird ways.

On Dec 2, 2016 10:18 PM, "Takeshi Yamamuro" 
wrote:

> *@maropu* commented on this pull request.
> --
>
> In sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/
> OrcPartitionDiscoverySuite.scala
> :
>
> > @@ -146,7 +146,7 @@ class OrcPartitionDiscoverySuite extends QueryTest 
with TestHiveSingleton with B
>  i <- 1 to 10
>  pi <- Seq(1, 2)
>  ps <- Seq("foo", "bar")
> -  } yield Row(i, pi, i.toString, ps))
> +  } yield Row(i, i.toString, pi, ps))
>
> It seems moving partition columns into the end is a correct behaviour by
> referring comments here: https://github.com/apache/
> spark/blame/master/sql/core/src/test/scala/org/apache/spark/sql/test/
> DataFrameReaderWriterSuite.scala#L590. Anything I miss?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or mute the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16120: [SPARK-18634][PySpark][SQL][WIP] Corruption and Correctn...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16120
  
**[Test build #69610 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69610/consoleFull)**
 for PR 16120 at commit 
[`a5594f7`](https://github.com/apache/spark/commit/a5594f7ffcbdc9ab2e83008a99d5878fa9fae2b8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16120: [SPARK-18634][PySpark][SQL][WIP] Corruption and Correctn...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16120
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69608/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16120: [SPARK-18634][PySpark][SQL][WIP] Corruption and Correctn...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16120
  
**[Test build #69608 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69608/consoleFull)**
 for PR 16120 at commit 
[`44aaf39`](https://github.com/apache/spark/commit/44aaf3979770f09e6e5626db870e0b36a5c33927).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16120: [SPARK-18634][PySpark][SQL][WIP] Corruption and Correctn...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16120
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15795: [SPARK-18081][ML][DOCS] Add user guide for Locality Sens...

[jkbradley]

Github user jkbradley commented on the issue:

https://github.com/apache/spark/pull/15795
  
I found myself wanting to make a number of tiny comments, so I thought it'd 
be easier to send a PR.  Could you please take a look at this one?
Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16113: [SPARK-18657][SPARK-18668] Make StreamingQuery.id...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/16113#discussion_r90751494
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala
 ---
@@ -54,6 +61,26 @@ object OffsetSeq {
* `nulls` in the sequence are converted to `None`s.
*/
   def fill(metadata: Option[String], offsets: Offset*): OffsetSeq = {
-OffsetSeq(offsets.map(Option(_)), metadata)
+OffsetSeq(offsets.map(Option(_)), 
metadata.map(OffsetSeqMetadata.apply))
   }
 }
+
+
+/**
+ * Contains metadata associated with a [[OffsetSeq]]. This information is
+ * persisted to the offset log in the checkpoint location via the 
[[OffsetSeq]] metadata field.
+ *
+ * @param batchWatermarkMs: The current eventTime watermark, used to
+ * bound the lateness of data that will processed. Time unit: milliseconds
+ * @param batchTimestampMs: The current batch processing timestamp.
+ * Time unit: milliseconds
+ */
+case class OffsetSeqMetadata(var batchWatermarkMs: Long = 0, var 
batchTimestampMs: Long = 0) {
--- End diff --

Not worth moving these 6 lines of code in a new file.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15813: [SPARK-18362][SQL] Use TextFileFormat in implemen...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/15813#discussion_r90751405
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala
 ---
@@ -173,51 +179,37 @@ class CSVFileFormat extends TextBasedFileFormat with 
DataSourceRegister {
 }
   }
 
-  private def baseRdd(
-  sparkSession: SparkSession,
-  options: CSVOptions,
-  inputPaths: Seq[String]): RDD[String] = {
-readText(sparkSession, options, inputPaths.mkString(","))
-  }
-
-  private def tokenRdd(
-  sparkSession: SparkSession,
-  options: CSVOptions,
-  header: Array[String],
-  inputPaths: Seq[String]): RDD[Array[String]] = {
-val rdd = baseRdd(sparkSession, options, inputPaths)
-// Make sure firstLine is materialized before sending to executors
-val firstLine = if (options.headerFlag) findFirstLine(options, rdd) 
else null
-CSVRelation.univocityTokenizer(rdd, firstLine, options)
-  }
-
   /**
* Returns the first line of the first non-empty file in path
*/
-  private def findFirstLine(options: CSVOptions, rdd: RDD[String]): String 
= {
+  private def findFirstLine(options: CSVOptions, lines: Dataset[String]): 
String = {
+import lines.sqlContext.implicits._
+val nonEmptyLines = lines.filter(length(trim($"value")) > 0)
 if (options.isCommentSet) {
-  val comment = options.comment.toString
-  rdd.filter { line =>
-line.trim.nonEmpty && !line.startsWith(comment)
-  }.first()
+  
nonEmptyLines.filter(!$"value".startsWith(options.comment.toString)).first()
 } else {
-  rdd.filter { line =>
-line.trim.nonEmpty
-  }.first()
+  nonEmptyLines.first()
 }
   }
 
   private def readText(
   sparkSession: SparkSession,
   options: CSVOptions,
-  location: String): RDD[String] = {
+  inputPaths: Seq[String]): Dataset[String] = {
 if (Charset.forName(options.charset) == StandardCharsets.UTF_8) {
-  sparkSession.sparkContext.textFile(location)
+  sparkSession.baseRelationToDataFrame(
+DataSource.apply(
+  sparkSession,
+  paths = inputPaths,
+  className = classOf[TextFileFormat].getName
+).resolveRelation(checkFilesExist = false))
+.select("value").as[String](Encoders.STRING)
 } else {
   val charset = options.charset
-  sparkSession.sparkContext
-.hadoopFile[LongWritable, Text, TextInputFormat](location)
+  val rdd = sparkSession.sparkContext
--- End diff --

@JoshRosen do you know why the special handling for non-utf8 encoding is 
needed? I would think TextFileFormat itself already supports that since it is 
reading it in from Hadoop Text.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16128: [SPARK-18671][SS] Added tests to ensure stability of tha...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16128
  
**[Test build #69609 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69609/consoleFull)**
 for PR 16128 at commit 
[`d9be1c5`](https://github.com/apache/spark/commit/d9be1c5b78ca7ac9f42f69bb267feea36ab3fb50).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16098: [SPARK-18672][CORE] Close recordwriter in SparkHadoopMap...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16098
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69606/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16098: [SPARK-18672][CORE] Close recordwriter in SparkHadoopMap...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16098
  
**[Test build #69606 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69606/consoleFull)**
 for PR 16098 at commit 
[`4804862`](https://github.com/apache/spark/commit/48048622067f092ed247bc555e5461c073894a9c).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16098: [SPARK-18672][CORE] Close recordwriter in SparkHadoopMap...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16098
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15923: [SPARK-4105] retry the fetch or stage if shuffle block i...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/15923
  
cc @zsxwing @JoshRosen does this look good?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15813: [SPARK-18362][SQL] Use TextFileFormat in implemen...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/15813


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16021: [SPARK-18593][SQL] JDBCRDD returns incorrect resu...

[dongjoon-hyun]

Github user dongjoon-hyun closed the pull request at:

https://github.com/apache/spark/pull/16021


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16021: [SPARK-18593][SQL] JDBCRDD returns incorrect results for...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/16021
  
Thank you all so much for giving your attentions and opinions here.

I asked the one which is not allowed here. Please refer #16127 for more 
details, too.

I'm closing this now.

Sorry for all your inconvenience.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15813: [SPARK-18362][SQL] Use TextFileFormat in implementation ...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/15813
  
Merging in master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16127: [SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1....

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/16127
  
I will close this now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16127: [SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACK...

[dongjoon-hyun]

Github user dongjoon-hyun closed the pull request at:

https://github.com/apache/spark/pull/16127


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16127: [SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1....

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/16127
  
I see. If then, it's my fault.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16127: [SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1....

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/16127
  
I didn't actually look at the content. I thought they were bug fixes and as 
a result said we should backport them. The title and description of the 
original pr implied they were bug fixes.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16126: [SPARK-18695] Bump master branch version to 2.2.0...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/16126


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16126: [SPARK-18695] Bump master branch version to 2.2.0-SNAPSH...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/16126
  
Merging in master.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16126: [SPARK-18695] Bump master branch version to 2.2.0-SNAPSH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16126
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69599/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16126: [SPARK-18695] Bump master branch version to 2.2.0-SNAPSH...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16126
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16126: [SPARK-18695] Bump master branch version to 2.2.0-SNAPSH...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16126
  
**[Test build #69599 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69599/consoleFull)**
 for PR 16126 at commit 
[`5362113`](https://github.com/apache/spark/commit/5362113d9b844e66d660b93501b3414e27f3d542).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16090: [SPARK-18661] [SQL] Creating a partitioned datasource ta...

[ericl]

Github user ericl commented on the issue:

https://github.com/apache/spark/pull/16090
  
Seems like the caching broke a bunch of tests. I'll take a look at this
again tomorrow.

On Fri, Dec 2, 2016, 7:49 PM UCB AMPLab  wrote:

> Test FAILed.
>
>
> Refer to this link for build results (access rights to CI server needed):
>
> https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69600/
> Test FAILed.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16092: [SPARK-18662] Move resource managers to separate directo...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16092
  
**[Test build #3465 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3465/consoleFull)**
 for PR 16092 at commit 
[`6763a80`](https://github.com/apache/spark/commit/6763a807de12ee0335d8211ef7bd058cd08b46c0).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16120: [SPARK-18634][PySpark][SQL][WIP] Corruption and Correctn...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16120
  
**[Test build #69608 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69608/consoleFull)**
 for PR 16120 at commit 
[`44aaf39`](https://github.com/apache/spark/commit/44aaf3979770f09e6e5626db870e0b36a5c33927).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15620: [SPARK-18091] [SQL] Deep if expressions cause Generated ...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/15620
  
LGTM and cc @cloud-fan for another check.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16125: [SPARK-18694][SS]Add StreamingQuery.explain and exceptio...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16125
  
**[Test build #69607 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69607/consoleFull)**
 for PR 16125 at commit 
[`169cd20`](https://github.com/apache/spark/commit/169cd209343d255299b5fa20b407c0a9fefc3c9c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16090: [SPARK-18661] [SQL] Creating a partitioned datasource ta...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16090
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16090: [SPARK-18661] [SQL] Creating a partitioned datasource ta...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16090
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69600/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16090: [SPARK-18661] [SQL] Creating a partitioned datasource ta...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16090
  
**[Test build #69600 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69600/consoleFull)**
 for PR 16090 at commit 
[`5a250ad`](https://github.com/apache/spark/commit/5a250adbc76448ccdd5b9158c1193e3ada8af8ef).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16109: [SPARK-18588][Tests]Fix flaky test: KafkaSourceStressFor...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16109
  
**[Test build #3462 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3462/consoleFull)**
 for PR 16109 at commit 
[`8c5ae2d`](https://github.com/apache/spark/commit/8c5ae2d4b123f7872365cc2e27f2c93cdd74af4e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16109: [SPARK-18588][Tests]Fix flaky test: KafkaSourceStressFor...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16109
  
**[Test build #3464 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3464/consoleFull)**
 for PR 16109 at commit 
[`8c5ae2d`](https://github.com/apache/spark/commit/8c5ae2d4b123f7872365cc2e27f2c93cdd74af4e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16109: [SPARK-18588][Tests]Fix flaky test: KafkaSourceStressFor...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16109
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16109: [SPARK-18588][Tests]Fix flaky test: KafkaSourceStressFor...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16109
  
**[Test build #3463 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3463/consoleFull)**
 for PR 16109 at commit 
[`8c5ae2d`](https://github.com/apache/spark/commit/8c5ae2d4b123f7872365cc2e27f2c93cdd74af4e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16109: [SPARK-18588][Tests]Fix flaky test: KafkaSourceStressFor...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16109
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69598/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16109: [SPARK-18588][Tests]Fix flaky test: KafkaSourceStressFor...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16109
  
**[Test build #69598 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69598/consoleFull)**
 for PR 16109 at commit 
[`8c5ae2d`](https://github.com/apache/spark/commit/8c5ae2d4b123f7872365cc2e27f2c93cdd74af4e).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16127: [SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1....

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/16127
  
Oh, my bad.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16127: [SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1....

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/16127
  
Just let me know if this is not what you expect. I will close this and 
#16021 immediately as I mentioned before.
We just make a mistake one commit hopefully.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16127: [SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1....

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16127
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16127: [SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1....

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16127
  
**[Test build #69604 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69604/consoleFull)**
 for PR 16127 at commit 
[`21fbe74`](https://github.com/apache/spark/commit/21fbe7456667541638b64e123b235acc03d06a4f).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16127: [SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1....

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16127
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69604/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16127: [SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1....

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/16127
  
The my comment was
> Hi, @rxin , @gatorsmile , @hvanhovell , @srowen .
> Could you give some opinion? What I hoped here is having these 6 commits 
in branch-1.6.
> But, if backports are not proper for this issue by the Apache Spark 
policy, I will just mark SPARK-18593 as Resolved with a fix version 2.0.0 and 
close this PR.
> I know that It's not a good time for you to see this. Sorry for asking 
this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16094: [SPARK-18541][Python]Add metadata parameter to pyspark.s...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16094
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69602/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16094: [SPARK-18541][Python]Add metadata parameter to pyspark.s...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16094
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16127: [SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1....

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/16127
  
The solution of the bug fix was the feature of predicate pushdown to 
PostgreSQL in latest branch.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16094: [SPARK-18541][Python]Add metadata parameter to pyspark.s...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16094
  
**[Test build #69602 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69602/consoleFull)**
 for PR 16094 at commit 
[`3711ca7`](https://github.com/apache/spark/commit/3711ca72a14e019200358f1d5fc5149010e4ceb2).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16127: [SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1....

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/16127
  
Ur, @rxin . We talked on #16021 about these 6 commits. 
You delegated this review work to @gatorsmile , didn't you?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16098: [SPARK-18672][CORE] Close recordwriter in SparkHadoopMap...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16098
  
**[Test build #69606 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69606/consoleFull)**
 for PR 16098 at commit 
[`4804862`](https://github.com/apache/spark/commit/48048622067f092ed247bc555e5461c073894a9c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16098: [SPARK-18672][CORE] Close recordwriter in SparkHadoopMap...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/16098
  
It seems the failed test is not related and passed in my local.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16098: [SPARK-18672][CORE] Close recordwriter in SparkHadoopMap...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/16098
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16109: [SPARK-18588][Tests]Fix flaky test: KafkaSourceStressFor...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16109
  
**[Test build #3461 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3461/consoleFull)**
 for PR 16109 at commit 
[`8c5ae2d`](https://github.com/apache/spark/commit/8c5ae2d4b123f7872365cc2e27f2c93cdd74af4e).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16127: [SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1....

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/16127
  
Why are we adding features? I thought they were bug fixes.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16128: [SPARK-18671][SS] Added tests to ensure stability of tha...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16128
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69605/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16128: [SPARK-18671][SS] Added tests to ensure stability of tha...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16128
  
**[Test build #69605 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69605/consoleFull)**
 for PR 16128 at commit 
[`49e940b`](https://github.com/apache/spark/commit/49e940b3bd566d0502d52c0a974ddd75650d9f7a).
 * This patch **fails RAT tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16128: [SPARK-18671][SS] Added tests to ensure stability of tha...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16128
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16128: [SPARK-18671][SS] Added tests to ensure stability of tha...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16128
  
**[Test build #69605 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69605/consoleFull)**
 for PR 16128 at commit 
[`49e940b`](https://github.com/apache/spark/commit/49e940b3bd566d0502d52c0a974ddd75650d9f7a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16128: [SPARK-18671][SS] Added tests to ensure stability...

[tdas]

GitHub user tdas opened a pull request:

https://github.com/apache/spark/pull/16128

[SPARK-18671][SS] Added tests to ensure stability of that all Structured 
Streaming log formats

## What changes were proposed in this pull request?

To be able to restart StreamingQueries across Spark version, we have 
already made the logs (offset log, file source log, file sink log) use json. We 
should added tests with actual json files in the Spark such that any 
incompatible changes in reading the logs is immediately caught. This PR add 
tests for FileStreamSourceLog, FileStreamSinkLog, and OffsetSeqLog.

## How was this patch tested?
new unit tests

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tdas/spark SPARK-18671

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16128.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16128


commit 84efce97313c5244dbeb30db6ff70f70b48f1555
Author: Tathagata Das 
Date:   2016-12-03T01:48:22Z

Added test for FileStremaSinkLog

commit 3d5449418da6453b608e39ab19d32caa3ffe9d7b
Author: Tathagata Das 
Date:   2016-12-03T02:23:28Z

Added test for FileStreamSourceLog

commit 49e940b3bd566d0502d52c0a974ddd75650d9f7a
Author: Tathagata Das 
Date:   2016-12-03T02:53:13Z

Added test for OffsetSeqLog




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16127: [SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1....

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16127
  
**[Test build #69604 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69604/consoleFull)**
 for PR 16127 at commit 
[`21fbe74`](https://github.com/apache/spark/commit/21fbe7456667541638b64e123b235acc03d06a4f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16127: [SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACK...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/16127#discussion_r90749070
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala 
---
@@ -180,12 +181,32 @@ class JDBCSuite extends SparkFunSuite
   }
 
   test("SELECT * WHERE (simple predicates)") {
--- End diff --

We should update the whole content of this testcase.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16127: [SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1....

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/16127
  
This is the second commit for #16021.
cc @gatorsmile .


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16127: [SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACK...

[dongjoon-hyun]

GitHub user dongjoon-hyun opened a pull request:

https://github.com/apache/spark/pull/16127

[SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1.6] Support 
AND/OR/IN/LIKE push-down filters for JDBC

This is rework from #10386 and add more tests and LIKE push-down support.

Author: Takeshi YAMAMURO 

Closes #10468 from maropu/SupportMorePushdownInJdbc.

(cherry picked from commit 5c2682b0c8fd2aeae2af1adb716ee0d5f8b85135)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dongjoon-hyun/spark SPARK-12409-BACKPORT-1.6

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16127.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16127


commit 21fbe7456667541638b64e123b235acc03d06a4f
Author: Takeshi YAMAMURO 
Date:   2015-12-30T21:34:37Z

[SPARK-12409][SPARK-12387][SPARK-12391][SQL][BACKPORT-1.6] Support 
AND/OR/IN/LIKE push-down filters for JDBC

This is rework from #10386 and add more tests and LIKE push-down support.

Author: Takeshi YAMAMURO 

Closes #10468 from maropu/SupportMorePushdownInJdbc.

(cherry picked from commit 5c2682b0c8fd2aeae2af1adb716ee0d5f8b85135)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16053: [SPARK-17931] Eliminate unncessary task (de) serializati...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16053
  
Build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16053: [SPARK-17931] Eliminate unncessary task (de) serializati...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16053
  
**[Test build #69603 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69603/consoleFull)**
 for PR 16053 at commit 
[`063ff52`](https://github.com/apache/spark/commit/063ff52140962c0fcae3a2806959318aa3684d6c).
 * This patch **fails to build**.
 * This patch **does not merge cleanly**.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16053: [SPARK-17931] Eliminate unncessary task (de) serializati...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16053
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69603/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16090: [SPARK-18661] [SQL] Creating a partitioned datasource ta...

[ericl]

Github user ericl commented on the issue:

https://github.com/apache/spark/pull/16090
  
cc @rxin please merge unless wenchen gets to it first


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16124: [SPARK-12446][SQL][BACKPORT-1.6] Add unit tests f...

[dongjoon-hyun]

Github user dongjoon-hyun closed the pull request at:

https://github.com/apache/spark/pull/16124


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16124: [SPARK-12446][SQL][BACKPORT-1.6] Add unit tests for JDBC...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/16124
  
Thank you! I'll close this and proceed to next.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16053: [SPARK-17931] Eliminate unncessary task (de) serializati...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16053
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69601/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16053: [SPARK-17931] Eliminate unncessary task (de) serializati...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16053
  
**[Test build #69601 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69601/consoleFull)**
 for PR 16053 at commit 
[`8e2de02`](https://github.com/apache/spark/commit/8e2de02f7e365403e2c551fa035df766a8f12d39).
 * This patch **fails to build**.
 * This patch **does not merge cleanly**.
 * This patch adds the following public classes _(experimental)_:
  * `  case class LaunchTask(`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16053: [SPARK-17931] Eliminate unncessary task (de) serializati...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16053
  
Build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16125: [SPARK-18694][SS]Add StreamingQuery.explain and e...

[tdas]

Github user tdas commented on a diff in the pull request:

https://github.com/apache/spark/pull/16125#discussion_r90748623
  
--- Diff: 
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala
 ---
@@ -103,10 +103,12 @@ class StreamingQuerySuite extends StreamTest with 
BeforeAndAfter with Logging {
   TestAwaitTermination(ExpectException[SparkException]),
   TestAwaitTermination(ExpectException[SparkException], timeoutMs = 
2000),
   TestAwaitTermination(ExpectException[SparkException], timeoutMs = 
10),
-  AssertOnQuery(
-q => q.exception.get.startOffset.get.offsets ===
-  q.committedOffsets.toOffsetSeq(Seq(inputData), "{}").offsets,
-"incorrect start offset on exception")
+  AssertOnQuery(q => {
+q.exception.get.startOffset ===
+  q.committedOffsets.toOffsetSeq(Seq(inputData), "{}").toString &&
--- End diff --

This line would conflict with my PR #16113 , you may have to fix issues 
after my PR gets merged.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16094: [SPARK-18541][Python]Add metadata parameter to py...

[shea-parkes]

Github user shea-parkes commented on a diff in the pull request:

https://github.com/apache/spark/pull/16094#discussion_r90748611
  
--- Diff: python/pyspark/sql/column.py ---
@@ -298,19 +299,34 @@ def isin(self, *cols):
 isNotNull = _unary_op("isNotNull", "True if the current expression is 
not null.")
 
 @since(1.3)
-def alias(self, *alias):
+def alias(self, *alias, **kwargs):
 """
 Returns this column aliased with a new name or names (in the case 
of expressions that
 return more than one column, such as explode).
 
+Optional ``metadata`` keyword argument can be passed when aliasing 
a single column.
--- End diff --

Oh, and I inferred that this would be a v2.2 feature since v2.1 is already 
in pre-release.  If I got the version wrong, just let me know.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org