[GitHub] spark issue #15958: [SPARK-17932][SQL] Support SHOW TABLES EXTENDED LIKE 'id...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15958
  
**[Test build #69393 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69393/consoleFull)**
 for PR 15958 at commit 
[`958fe8b`](https://github.com/apache/spark/commit/958fe8b083feb6a312b02abe8325b973bc91500f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15975: [SPARK-18538] [SQL] Fix Concurrent Table Fetching...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/15975#discussion_r90183049
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala 
---
@@ -209,6 +209,16 @@ class JDBCSuite extends SparkFunSuite
 conn.close()
   }
 
+  // Check whether the tables are fetched in the expected degree of 
parallelism
+  def checkNumPartitions(df: DataFrame, expectedNumPartitions: Int): Unit 
= {
+df.queryExecution.analyzed.collectFirst {
--- End diff --

nit:
```
val jdbcRelations = df.queryExecutioncollect {
  case LogicalRelation(r: JDBCRelation, _, _) => r
}
assert(jdbcRelations.length == 1)
assert(jdbcRelations.head.parts.length == expected, "error message")
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15975: [SPARK-18538] [SQL] Fix Concurrent Table Fetching Using ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15975
  
**[Test build #69392 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69392/consoleFull)**
 for PR 15975 at commit 
[`404aa22`](https://github.com/apache/spark/commit/404aa223dc66419319de15798bf43abe00fd6e64).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15975: [SPARK-18538] [SQL] Fix Concurrent Table Fetching Using ...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/15975
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15995: [SPARK-18566][SQL] remove OverwriteOptions

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/15995#discussion_r90182739
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
 ---
@@ -129,65 +129,67 @@ case class DataSourceAnalysis(conf: CatalystConf) 
extends Rule[LogicalPlan] {
   }
 
   override def apply(plan: LogicalPlan): LogicalPlan = plan transform {
-// If the InsertIntoTable command is for a partitioned 
HadoopFsRelation and
-// the user has specified static partitions, we add a Project operator 
on top of the query
-// to include those constant column values in the query result.
-//
-// Example:
-// Let's say that we have a table "t", which is created by
-// CREATE TABLE t (a INT, b INT, c INT) USING parquet PARTITIONED BY 
(b, c)
-// The statement of "INSERT INTO TABLE t PARTITION (b=2, c) SELECT 1, 
3"
-// will be converted to "INSERT INTO TABLE t PARTITION (b, c) SELECT 
1, 2, 3".
-//
-// Basically, we will put those partition columns having a assigned 
value back
-// to the SELECT clause. The output of the SELECT clause is organized 
as
-// normal_columns static_partitioning_columns 
dynamic_partitioning_columns.
-// static_partitioning_columns are partitioning columns having assigned
-// values in the PARTITION clause (e.g. b in the above example).
-// dynamic_partitioning_columns are partitioning columns that do not 
assigned
-// values in the PARTITION clause (e.g. c in the above example).
 case insert @ logical.InsertIntoTable(
-  relation @ LogicalRelation(t: HadoopFsRelation, _, _), parts, query, 
overwrite, false)
-  if query.resolved && parts.exists(_._2.isDefined) =>
-
-  val projectList = convertStaticPartitions(
-sourceAttributes = query.output,
-providedPartitions = parts,
-targetAttributes = relation.output,
-targetPartitionSchema = t.partitionSchema)
-
-  // We will remove all assigned values to static partitions because 
they have been
-  // moved to the projectList.
-  insert.copy(partition = parts.map(p => (p._1, None)), child = 
Project(projectList, query))
-
+  l @ LogicalRelation(t: HadoopFsRelation, _, table), parts, query, 
overwrite, false)
+  if query.resolved =>
+
+  // If the InsertIntoTable command is for a partitioned 
HadoopFsRelation and
+  // the user has specified static partitions, we add a Project 
operator on top of the query
+  // to include those constant column values in the query result.
+  //
+  // Example:
+  // Let's say that we have a table "t", which is created by
+  // CREATE TABLE t (a INT, b INT, c INT) USING parquet PARTITIONED BY 
(b, c)
+  // The statement of "INSERT INTO TABLE t PARTITION (b=2, c) SELECT 
1, 3"
+  // will be converted to "INSERT INTO TABLE t PARTITION (b, c) SELECT 
1, 2, 3".
+  //
+  // Basically, we will put those partition columns having a assigned 
value back
+  // to the SELECT clause. The output of the SELECT clause is 
organized as
+  // normal_columns static_partitioning_columns 
dynamic_partitioning_columns.
+  // static_partitioning_columns are partitioning columns having 
assigned
+  // values in the PARTITION clause (e.g. b in the above example).
+  // dynamic_partitioning_columns are partitioning columns that do not 
assigned
+  // values in the PARTITION clause (e.g. c in the above example).
+  val actualQuery = if (parts.exists(_._2.isDefined)) {
+val projectList = convertStaticPartitions(
+  sourceAttributes = query.output,
+  providedPartitions = parts,
+  targetAttributes = l.output,
+  targetPartitionSchema = t.partitionSchema)
+Project(projectList, query)
+  } else {
+query
+  }
 
-case i @ logical.InsertIntoTable(
-   l @ LogicalRelation(t: HadoopFsRelation, _, table), part, 
query, overwrite, false)
-if query.resolved && t.schema.asNullable == 
query.schema.asNullable =>
+  // Sanity check, this should be guaranteed by 
`PreprocessTableInsertion`
+  if (t.schema.asNullable != actualQuery.schema.asNullable) {
+throw new AnalysisException(
+  s"Can not insert into ${l.simpleString} because of schema 
mismatch.")
--- End diff --

I'm going to submit another PR to tweak the extended analyzer rules 
execution order, currently these rules like `PreProcessCreateTable`, 
`PreWriteCheck`, `DataSourceAnalysis`, etc. may have some dependencies and may 
worth to put them in different batches.

BTW I have retargeted this JIRA ticket 

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15915
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15915
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69382/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16052: [SPARK-18617][CORE][STREAMING] Close "kryo auto p...

[zsxwing]

Github user zsxwing commented on a diff in the pull request:

https://github.com/apache/spark/pull/16052#discussion_r90182581
  
--- Diff: 
streaming/src/test/scala/org/apache/spark/streaming/StreamingContextSuite.scala 
---
@@ -869,6 +891,31 @@ object TestReceiver {
   val counter = new AtomicInteger(1)
 }
 
+class FakeByteArrayReceiver extends 
Receiver[Array[Byte]](StorageLevel.MEMORY_ONLY) with Logging {
--- End diff --

nit: Why create a new class? Is there any concern to just use TestReceiver?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16048
  
**[Test build #69391 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69391/consoleFull)**
 for PR 16048 at commit 
[`eaa4a73`](https://github.com/apache/spark/commit/eaa4a73b4c30a446e8144339de3eca71d0b5dfdf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15915
  
**[Test build #69382 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69382/consoleFull)**
 for PR 15915 at commit 
[`ad85da0`](https://github.com/apache/spark/commit/ad85da0121d56487bb2f87b203d81a2c451035c8).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16052: [SPARK-18617][CORE][STREAMING] Close "kryo auto p...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/16052


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15998: [SPARK-18572][SQL] Add a method `listPartitionNames` to ...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/15998
  
I think this PR is for performance improvement, why we update the 
`HiveCommandSuite` here? Otherwise LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16052: [SPARK-18617][CORE][STREAMING] Close "kryo auto pick" fe...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/16052
  
This doesn't merge cleanly into branch-2.0. @uncleGen can you submit a pull 
request for branch-2.0?



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/16048
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16052: [SPARK-18617][CORE][STREAMING] Close "kryo auto pick" fe...

[rxin]

Github user rxin commented on the issue:

https://github.com/apache/spark/pull/16052
  
Merging in master/branch-2.1.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15998: [SPARK-18572][SQL] Add a method `listPartitionNam...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/15998#discussion_r90181898
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala ---
@@ -155,6 +155,25 @@ private[hive] trait HiveClient {
 }
   }
 
+  /**
+   * Returns the partition names for the given table that match the 
supplied partition spec.
+   * If no partition spec is specified, all partitions are returned.
+   */
+  final def getPartitionNames(
+  db: String,
+  table: String,
+  partialSpec: Option[TablePartitionSpec]): Seq[String] = {
+getPartitionNames(getTable(db, table), partialSpec)
+  }
+
+  /**
+   * Returns the partition names for the given table that match the 
supplied partition spec.
+   * If no partition spec is specified, all partitions are returned.
+   */
+  def getPartitionNames(
--- End diff --

why we create 2 APIs? who will call this method directly?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15998: [SPARK-18572][SQL] Add a method `listPartitionNam...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/15998#discussion_r90181774
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -922,6 +924,25 @@ private[spark] class HiveExternalCatalog(conf: 
SparkConf, hadoopConf: Configurat
   /**
* Returns the partition names from hive metastore for a given table in 
a database.
*/
+  override def listPartitionNames(
+  db: String,
+  table: String,
+  partialSpec: Option[TablePartitionSpec] = None): Seq[String] = 
withClient {
+val actualPartColNames = getTable(db, table).partitionColumnNames
+val clientPartitionNames =
+  client.getPartitionNames(db, table, 
partialSpec.map(lowerCasePartitionSpec))
+clientPartitionNames.map { partName =>
+  val partSpec = PartitioningUtils.parsePathFragmentAsSeq(partName)
+  partSpec.map { case (partName, partValue) =>
+
escapePathName(actualPartColNames.find(_.equalsIgnoreCase(partName)).get) + "=" 
+
--- End diff --

can we just build a map with lower cased partition column name as key, and 
path-escaped actual partition column name as value? then we can do a simple map 
lookup here, i.e. `map.get(partName.toLowerCase)`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16052: [SPARK-18617][CORE][STREAMING] Close "kryo auto pick" fe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16052
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69380/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16052: [SPARK-18617][CORE][STREAMING] Close "kryo auto pick" fe...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16052
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16048
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16048
  
**[Test build #69390 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69390/consoleFull)**
 for PR 16048 at commit 
[`eaa4a73`](https://github.com/apache/spark/commit/eaa4a73b4c30a446e8144339de3eca71d0b5dfdf).
 * This patch **fails executing the `dev/run-tests` script**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16048
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69390/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16052: [SPARK-18617][CORE][STREAMING] Close "kryo auto pick" fe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16052
  
**[Test build #69380 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69380/consoleFull)**
 for PR 16052 at commit 
[`39b4867`](https://github.com/apache/spark/commit/39b486769bfdab8515a4f188bea8a37c23546ad7).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class FakeByteArrayReceiver extends 
Receiver[Array[Byte]](StorageLevel.MEMORY_ONLY) with Logging `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16063: [SPARK-18622][SQL] Fix the datatype of the Sum ag...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/16063


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16063: [SPARK-18622][SQL] Fix the datatype of the Sum aggregate...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/16063
  
LGTM, merging to master/2.1!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15255: [SPARK-17680] [SQL] [TEST] Added a Testcase for V...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/15255


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15255: [SPARK-17680] [SQL] [TEST] Added a Testcase for Verifyin...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/15255
  
thanks, merging to master/2.1!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15780: [SPARK-18284][SQL] Make ExpressionEncoder.seriali...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/15780#discussion_r90180025
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala 
---
@@ -590,7 +591,11 @@ object ScalaReflection extends ScalaReflection {
   "cannot be used as field name\n" + 
walkedTypePath.mkString("\n"))
   }
 
-  val fieldValue = Invoke(inputObject, fieldName, 
dataTypeFor(fieldType))
+  // primitive take only non-null or struct takes non-null object 
guarded by isNull
--- End diff --

ExpressionEncoder can't encode `None` at the root level, as Spark SQL 
doesn't allow top-level row to be null.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16075: [SPARK-18516][Structured Streaming] Follow up PR ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/16075


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16075: [SPARK-18516][Structured Streaming] Follow up PR to add ...

[tdas]

Github user tdas commented on the issue:

https://github.com/apache/spark/pull/16075
  
Merging to master and 2.1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16048
  
**[Test build #69390 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69390/consoleFull)**
 for PR 16048 at commit 
[`eaa4a73`](https://github.com/apache/spark/commit/eaa4a73b4c30a446e8144339de3eca71d0b5dfdf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16067: [SPARK-17897] [SQL] Fixed IsNotNull Constraint Inference...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16067
  
**[Test build #69388 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69388/consoleFull)**
 for PR 16067 at commit 
[`54c0dd1`](https://github.com/apache/spark/commit/54c0dd10d4aabc4700d4a33206c481703c16fb83).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16060: [SPARK-18220][SQL] read Hive orc table with varchar colu...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16060
  
**[Test build #69389 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69389/consoleFull)**
 for PR 16060 at commit 
[`8b697be`](https://github.com/apache/spark/commit/8b697be520bb9c070462bebc8c72796eca8c8517).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16048: [DO_NOT_MERGE]Test kafka deletion

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/16048
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16077: [SPARK-18643][SPARKR] SparkR hangs at session start when...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16077
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69386/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16077: [SPARK-18643][SPARKR] SparkR hangs at session start when...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16077
  
**[Test build #69386 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69386/consoleFull)**
 for PR 16077 at commit 
[`866727d`](https://github.com/apache/spark/commit/866727d775c45bc8f2f6891ab685f3b6e20109b3).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16077: [SPARK-18643][SPARKR] SparkR hangs at session start when...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16077
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15979: [SPARK-18251][SQL] the type of Dataset can't be Option o...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15979
  
**[Test build #69387 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69387/consoleFull)**
 for PR 15979 at commit 
[`70dd650`](https://github.com/apache/spark/commit/70dd650a7e43a44a056c4aa95dbbd88d23cbfbee).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16067: [SPARK-17897] [SQL] Fixed IsNotNull Constraint In...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16067#discussion_r90177599
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala 
---
@@ -1697,6 +1697,12 @@ class DataFrameSuite extends QueryTest with 
SharedSQLContext {
   expr = "cast((_1 + _2) as boolean)", expectedNonNullableColumns = 
Seq("_1", "_2"))
   }
 
+  test("SPARK-17897: Fixed IsNotNull Constraint Inference Rule") {
+val data = Seq[java.lang.Integer](1, null).toDF("key")
+checkAnswer(data.filter("not key is not null"), Row(null))
--- End diff --

sure. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15987: [SPARK-17732][SPARK-18515][SQL] ALTER TABLE DROP PARTITI...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15987
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69383/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15987: [SPARK-17732][SPARK-18515][SQL] ALTER TABLE DROP PARTITI...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15987
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16067: [SPARK-17897] [SQL] Fixed IsNotNull Constraint In...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16067#discussion_r90177515
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala 
---
@@ -58,13 +57,28 @@ abstract class QueryPlan[PlanType <: 
QueryPlan[PlanType]] extends TreeNode[PlanT
   }
 
   /**
+   * Infer the Attribute-specific IsNotNull constraints from the null 
intolerant child expressions
+   * of constraints.
+   */
+  private def inferIsNotNullConstraints(constraint: Expression): 
Seq[Expression] =
+constraint match {
+  case IsNotNull(_: Attribute) => constraint :: Nil
--- End diff --

Yeah, my original idea is to do a fast stop. After rethinking it, it might 
be fine. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16067: [SPARK-17897] [SQL] Fixed IsNotNull Constraint In...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16067#discussion_r90177567
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala 
---
@@ -58,13 +57,28 @@ abstract class QueryPlan[PlanType <: 
QueryPlan[PlanType]] extends TreeNode[PlanT
   }
 
   /**
+   * Infer the Attribute-specific IsNotNull constraints from the null 
intolerant child expressions
+   * of constraints.
+   */
+  private def inferIsNotNullConstraints(constraint: Expression): 
Seq[Expression] =
+constraint match {
+  case IsNotNull(_: Attribute) => constraint :: Nil
+  // When the root is IsNotNull, we can push IsNotNull through the 
child null intolerant
+  // expressions
+  case IsNotNull(expr) => 
scanNullIntolerantExpr(expr).map(IsNotNull(_))
+  // Constraints always return true for all the inputs. That means, 
null will never be returned.
+  // Thus, we can infer `IsNotNull(constraint)`, and also push 
IsNotNull through the child
+  // null intolerant expressions.
+  case _ => scanNullIntolerantExpr(constraint).map(IsNotNull(_))
+}
+
+  /**
* Recursively explores the expressions which are null intolerant and 
returns all attributes
* in these expressions.
*/
   private def scanNullIntolerantExpr(expr: Expression): Seq[Attribute] = 
expr match {
--- End diff --

Sure


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15987: [SPARK-17732][SPARK-18515][SQL] ALTER TABLE DROP PARTITI...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15987
  
**[Test build #69383 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69383/consoleFull)**
 for PR 15987 at commit 
[`9db07af`](https://github.com/apache/spark/commit/9db07afafd010d969ca18f8365673ddd828736d2).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `trait CommandWithExpression extends LeafNode `


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16063: [SPARK-18622][SQL] Fix the datatype of the Sum aggregate...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/16063
  
LGTM cc @cloud-fan for double check


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16067: [SPARK-17897] [SQL] Fixed IsNotNull Constraint Inference...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/16067
  
LGTM except some minor comments


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16067: [SPARK-17897] [SQL] Fixed IsNotNull Constraint In...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/16067#discussion_r90177275
  
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala 
---
@@ -1697,6 +1697,12 @@ class DataFrameSuite extends QueryTest with 
SharedSQLContext {
   expr = "cast((_1 + _2) as boolean)", expectedNonNullableColumns = 
Seq("_1", "_2"))
   }
 
+  test("SPARK-17897: Fixed IsNotNull Constraint Inference Rule") {
+val data = Seq[java.lang.Integer](1, null).toDF("key")
+checkAnswer(data.filter("not key is not null"), Row(null))
--- End diff --

shall we use DataFrame API? i.e. `data.filter(!$"key".isNotNull)`. The 
string version looks weird...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16067: [SPARK-17897] [SQL] Fixed IsNotNull Constraint In...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/16067#discussion_r90176972
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala 
---
@@ -58,13 +57,28 @@ abstract class QueryPlan[PlanType <: 
QueryPlan[PlanType]] extends TreeNode[PlanT
   }
 
   /**
+   * Infer the Attribute-specific IsNotNull constraints from the null 
intolerant child expressions
+   * of constraints.
+   */
+  private def inferIsNotNullConstraints(constraint: Expression): 
Seq[Expression] =
+constraint match {
+  case IsNotNull(_: Attribute) => constraint :: Nil
+  // When the root is IsNotNull, we can push IsNotNull through the 
child null intolerant
+  // expressions
+  case IsNotNull(expr) => 
scanNullIntolerantExpr(expr).map(IsNotNull(_))
+  // Constraints always return true for all the inputs. That means, 
null will never be returned.
+  // Thus, we can infer `IsNotNull(constraint)`, and also push 
IsNotNull through the child
+  // null intolerant expressions.
+  case _ => scanNullIntolerantExpr(constraint).map(IsNotNull(_))
+}
+
+  /**
* Recursively explores the expressions which are null intolerant and 
returns all attributes
* in these expressions.
*/
   private def scanNullIntolerantExpr(expr: Expression): Seq[Attribute] = 
expr match {
--- End diff --

shall we rename it to `scanNullIntolerantAttribute`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16067: [SPARK-17897] [SQL] Fixed IsNotNull Constraint In...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/16067#discussion_r90176867
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala 
---
@@ -58,13 +57,28 @@ abstract class QueryPlan[PlanType <: 
QueryPlan[PlanType]] extends TreeNode[PlanT
   }
 
   /**
+   * Infer the Attribute-specific IsNotNull constraints from the null 
intolerant child expressions
+   * of constraints.
+   */
+  private def inferIsNotNullConstraints(constraint: Expression): 
Seq[Expression] =
+constraint match {
+  case IsNotNull(_: Attribute) => constraint :: Nil
--- End diff --

we don't this case, I think it can be covered by the next case?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16076: [SPARK-18324][ML][DOC] Update ML programming and migrati...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16076
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69385/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16076: [SPARK-18324][ML][DOC] Update ML programming and migrati...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16076
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16076: [SPARK-18324][ML][DOC] Update ML programming and migrati...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16076
  
**[Test build #69385 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69385/consoleFull)**
 for PR 16076 at commit 
[`f3f1fc3`](https://github.com/apache/spark/commit/f3f1fc36391a180e0e554f84e8030cc66853982a).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16077: [SPARK-18643][SPARKR] SparkR hangs at session start when...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16077
  
**[Test build #69386 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69386/consoleFull)**
 for PR 16077 at commit 
[`866727d`](https://github.com/apache/spark/commit/866727d775c45bc8f2f6891ab685f3b6e20109b3).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16077: [SPARK-18643][SPARKR] SparkR hangs at session start when...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/16077
  
@yanboliang @shivaram



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16077: [SPARK-18643][SPARKR] SparkR hangs at session sta...

[felixcheung]

GitHub user felixcheung opened a pull request:

https://github.com/apache/spark/pull/16077

[SPARK-18643][SPARKR] SparkR hangs at session start when installed as a 
package without Spark

## What changes were proposed in this pull request?

If SparkR is running as a package and it has previously downloaded Spark 
Jar it should be able to run as before without having to set SPARK_HOME. 
Basically with this bug the auto install Spark will only work in the first 
session.

This seems to be a regression on the earlier behavior.
Fix is to always try to install or check for the cached Spark if running in 
an interactive session.
As discussed before, we should probably only install Spark iff running in 
an interactive session (R shell, RStudio etc)

## How was this patch tested?

Manually


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/felixcheung/spark rsessioninteractive

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16077.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16077


commit 866727d775c45bc8f2f6891ab685f3b6e20109b3
Author: Felix Cheung 
Date:   2016-11-30T06:16:54Z

install or check for cached installation if interactive




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15998: [SPARK-18572][SQL] Add a method `listPartitionNames` to ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15998
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69378/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15998: [SPARK-18572][SQL] Add a method `listPartitionNames` to ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15998
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16009: [SPARK-18318][ML] ML, Graph 2.1 QA: API: New Scala APIs,...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16009
  
**[Test build #69381 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69381/consoleFull)**
 for PR 16009 at commit 
[`27b07ef`](https://github.com/apache/spark/commit/27b07ef168fcaeb26b45968d4a82f7df349a2f65).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16009: [SPARK-18318][ML] ML, Graph 2.1 QA: API: New Scala APIs,...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16009
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16009: [SPARK-18318][ML] ML, Graph 2.1 QA: API: New Scala APIs,...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16009
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69381/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15998: [SPARK-18572][SQL] Add a method `listPartitionNames` to ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15998
  
**[Test build #69378 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69378/consoleFull)**
 for PR 15998 at commit 
[`48ae2a3`](https://github.com/apache/spark/commit/48ae2a39f8c5365c9f8e087333fc719b901517c2).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16067: [SPARK-17897] [SQL] Fixed IsNotNull Constraint Inference...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16067
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16067: [SPARK-17897] [SQL] Fixed IsNotNull Constraint Inference...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16067
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69377/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16067: [SPARK-17897] [SQL] Fixed IsNotNull Constraint Inference...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16067
  
**[Test build #69377 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69377/consoleFull)**
 for PR 16067 at commit 
[`a835f80`](https://github.com/apache/spark/commit/a835f804d593151578c7227cc4c240b0c5c27754).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16076: [SPARK-18324][ML][DOC] Update ML programming and migrati...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16076
  
**[Test build #69385 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69385/consoleFull)**
 for PR 16076 at commit 
[`f3f1fc3`](https://github.com/apache/spark/commit/f3f1fc36391a180e0e554f84e8030cc66853982a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16076: [SPARK-18324][ML][DOC] ML programming guide updat...

[yanboliang]

GitHub user yanboliang opened a pull request:

https://github.com/apache/spark/pull/16076

[SPARK-18324][ML][DOC] ML programming guide update and migration guide for 
2.1 release

## What changes were proposed in this pull request?
Update ML programming guide and migration guide for 2.1 release.

## How was this patch tested?
Doc change, no test.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yanboliang/spark spark-18324

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/16076.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #16076


commit f3f1fc36391a180e0e554f84e8030cc66853982a
Author: Yanbo Liang 
Date:   2016-11-30T05:59:19Z

Programming guide update and migration guide




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15255: [SPARK-17680] [SQL] [TEST] Added a Testcase for Verifyin...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15255
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15255: [SPARK-17680] [SQL] [TEST] Added a Testcase for Verifyin...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15255
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69379/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15255: [SPARK-17680] [SQL] [TEST] Added a Testcase for Verifyin...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15255
  
**[Test build #69379 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69379/consoleFull)**
 for PR 15255 at commit 
[`57817a1`](https://github.com/apache/spark/commit/57817a1c96c9577725ee8766834b20b06adfe521).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16075: [SPARK-18516][Structured Streaming] Follow up PR to add ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16075
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69376/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16075: [SPARK-18516][Structured Streaming] Follow up PR to add ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16075
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16075: [SPARK-18516][Structured Streaming] Follow up PR to add ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16075
  
**[Test build #69376 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69376/consoleFull)**
 for PR 16075 at commit 
[`f936595`](https://github.com/apache/spark/commit/f936595019ef5fbe41df8cea5ef08d6a9c132700).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16072: [SPARK-18639] Build only a single pip package

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16072
  
**[Test build #69384 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69384/consoleFull)**
 for PR 16072 at commit 
[`88b53c3`](https://github.com/apache/spark/commit/88b53c3b542d1423c169af7b4e52ecd6da067ced).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16072: [SPARK-18639] Build only a single pip package

[dongjoon-hyun]

Github user dongjoon-hyun commented on the issue:

https://github.com/apache/spark/pull/16072
  
Retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15780: [SPARK-18284][SQL] Make ExpressionEncoder.seriali...

[ueshin]

Github user ueshin commented on a diff in the pull request:

https://github.com/apache/spark/pull/15780#discussion_r90172243
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala 
---
@@ -590,7 +591,11 @@ object ScalaReflection extends ScalaReflection {
   "cannot be used as field name\n" + 
walkedTypePath.mkString("\n"))
   }
 
-  val fieldValue = Invoke(inputObject, fieldName, 
dataTypeFor(fieldType))
+  // primitive take only non-null or struct takes non-null object 
guarded by isNull
--- End diff --

Hmm, I found something wrong with the combination of `Option` and something 
the serializer of which is made by `t if definedByConstructorParams(t)` case 
like `Option[Tuple2[Int, String]]` (even at master).

If we add a following test to `ExpressionEncoderSuite` (at master):

```scala
encodeDecodeTest(Option.empty[Tuple2[Int, Sring]], "empty option of tuple2 
of int and string")
```

it fails as:

```
Encoded/Decoded data does not match input data

in:  None
out: Some((0,null))
types: scala.Tuple2

Encoded Data: [null,null]
Schema: _1#381,_2#382
root
-- _1: integer (nullable = false)
-- _2: string (nullable = true)
```

(BTW, there are no `Option` related tests in `ExpressionEncoderSuite`..)

The failure of `roundtrip in to_json and from_json` would be related to 
this.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15987: [SPARK-17732][SPARK-18515][SQL] ALTER TABLE DROP PARTITI...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15987
  
**[Test build #69383 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69383/consoleFull)**
 for PR 15987 at commit 
[`9db07af`](https://github.com/apache/spark/commit/9db07afafd010d969ca18f8365673ddd828736d2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15915: [SPARK-18485][CORE] Underlying integer overflow when cre...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15915
  
**[Test build #69382 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69382/consoleFull)**
 for PR 15915 at commit 
[`ad85da0`](https://github.com/apache/spark/commit/ad85da0121d56487bb2f87b203d81a2c451035c8).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15982: [SPARK-18546][core] Fix merging shuffle spills when usin...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15982
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69375/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15982: [SPARK-18546][core] Fix merging shuffle spills when usin...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15982
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15982: [SPARK-18546][core] Fix merging shuffle spills when usin...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15982
  
**[Test build #69375 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69375/consoleFull)**
 for PR 15982 at commit 
[`1025c6b`](https://github.com/apache/spark/commit/1025c6bb384968a7fc474d35a1bb18d82eb21938).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16009: [SPARK-18318][ML] ML, Graph 2.1 QA: API: New Scal...

[yanboliang]

Github user yanboliang commented on a diff in the pull request:

https://github.com/apache/spark/pull/16009#discussion_r90169879
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/ml/feature/ChiSqSelector.scala ---
@@ -49,15 +49,13 @@ private[feature] trait ChiSqSelectorParams extends 
Params
*
* @group param
*/
-  @Since("1.6.0")
--- End diff --

Yeah, it's safe for this case. However, I found lots of other traits which 
is also safe enough to add since tag but did not add. I reverted this part of 
change in this PR to make it catch another RC of 2.1, I think we should unify 
them in a separate work. Thanks. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...

[shivaram]

Github user shivaram commented on the issue:

https://github.com/apache/spark/pull/16014
  
Sorry I was caught up with some other stuff today. Will take a final look 
tomm morning. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16014: [SPARK-18590][SPARKR] build R source package when making...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/16014
  
Any more thought on this? Without this we don't really have a signed 
tarball in the official release to release to CRAN...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16067: [SPARK-17897] [SQL] Fixed IsNotNull Constraint In...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/16067#discussion_r90169798
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala 
---
@@ -58,13 +57,28 @@ abstract class QueryPlan[PlanType <: 
QueryPlan[PlanType]] extends TreeNode[PlanT
   }
 
   /**
+   * Infer the Attribute-specific IsNotNull constraints from the null 
intolerant child expressions
+   * of constraints.
+   */
+  private def inferIsNotNullConstraints(constraint: Expression): 
Seq[Expression] =
--- End diff --

Yeah.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16009: [SPARK-18318][ML] ML, Graph 2.1 QA: API: New Scala APIs,...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16009
  
**[Test build #69381 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69381/consoleFull)**
 for PR 16009 at commit 
[`27b07ef`](https://github.com/apache/spark/commit/27b07ef168fcaeb26b45968d4a82f7df349a2f65).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16073: [SPARK-18640] Add synchronization to TaskScheduler.runni...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16073
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16073: [SPARK-18640] Add synchronization to TaskScheduler.runni...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16073
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69372/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16073: [SPARK-18640] Add synchronization to TaskScheduler.runni...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16073
  
**[Test build #69372 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69372/consoleFull)**
 for PR 16073 at commit 
[`c866a1a`](https://github.com/apache/spark/commit/c866a1ab6fbc2b1440e594edaecb4e88f59d6a1d).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16052: [SPARK-18617][CORE][STREAMING] Close "kryo auto pick" fe...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16052
  
**[Test build #69380 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69380/consoleFull)**
 for PR 16052 at commit 
[`39b4867`](https://github.com/apache/spark/commit/39b486769bfdab8515a4f188bea8a37c23546ad7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15780: [SPARK-18284][SQL] Make ExpressionEncoder.serializer.nul...

[kiszk]

Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/15780
  
ping @cloud-fan 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16067: [SPARK-17897] [SQL] Fixed IsNotNull Constraint In...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/16067#discussion_r90168753
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala 
---
@@ -58,13 +57,28 @@ abstract class QueryPlan[PlanType <: 
QueryPlan[PlanType]] extends TreeNode[PlanT
   }
 
   /**
+   * Infer the Attribute-specific IsNotNull constraints from the null 
intolerant child expressions
+   * of constraints.
+   */
+  private def inferIsNotNullConstraints(constraint: Expression): 
Seq[Expression] =
--- End diff --

Yes. After this PR, we do not support it. This is a pretty rare case, 
right? 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #13557: [SPARK-15819][PYSPARK][ML] Add KMeanSummary in KM...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/13557


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16061: [SPARK-18278] [Scheduler] Support native submissi...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/16061#discussion_r90168600
  
--- Diff: kubernetes/README.md ---
@@ -0,0 +1,21 @@
+# Pre-requisites
+* maven, JDK and all other pre-requisites for building Spark.
+
+# Steps to compile
+
+* Clone the fork of spark: https://github.com/foxish/spark/ and switch to 
the k8s-support branch.
--- End diff --

I think this is not correct now?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16061: [SPARK-18278] [Scheduler] Support native submissi...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/16061#discussion_r90168566
  
--- Diff: kubernetes/README.md ---
@@ -0,0 +1,21 @@
+# Pre-requisites
+* maven, JDK and all other pre-requisites for building Spark.
+
+# Steps to compile
+
+* Clone the fork of spark: https://github.com/foxish/spark/ and switch to 
the k8s-support branch.
+* Build the project
+* ./build/mvn -Pkubernetes -Phadoop-2.4 -Dhadoop.version=2.4.0 
-DskipTests package
+* Ensure that you are pointing to a k8s cluster (kubectl config 
current-context), which you want to use with spark.
+* Launch a spark-submit job:
+   * `./bin/spark-submit --deploy-mode cluster --class 
org.apache.spark.examples.SparkPi --master k8s://default --conf 
spark.executor.instances=5 --conf 
spark.kubernetes.sparkImage=manyangled/kube-spark:dynamic  
http://storage.googleapis.com/foxish-spark-distro/original-spark-examples_2.11-2.1.0-SNAPSHOT.jar
 1`
--- End diff --

Do we need to prepare an official image for this?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #13557: [SPARK-15819][PYSPARK][ML] Add KMeanSummary in KMeans of...

[yanboliang]

Github user yanboliang commented on the issue:

https://github.com/apache/spark/pull/13557
  
LGTM, merged into master and branch-2.1. Thank you all.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #15994: [SPARK-18555][SQL]DataFrameNaFunctions.fill miss ...

[windpiger]

Github user windpiger commented on a diff in the pull request:

https://github.com/apache/spark/pull/15994#discussion_r90167716
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/DataFrameNaFunctions.scala ---
@@ -437,4 +438,38 @@ final class DataFrameNaFunctions private[sql](df: 
DataFrame) {
 case v => throw new IllegalArgumentException(
   s"Unsupported value type ${v.getClass.getName} ($v).")
   }
+
+  /**
+   * Returns a new `DataFrame` that replaces null or NaN values in 
specified
+   * numeric, string columns. If a specified column is not a numeric, 
string column,
+   * it is ignored.
+   */
+  private def fill1[T](value: T, cols: Seq[String]): DataFrame = {
--- End diff --

@srowen 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16072: [SPARK-18639] Build only a single pip package

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16072
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16072: [SPARK-18639] Build only a single pip package

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/16072
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69371/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16072: [SPARK-18639] Build only a single pip package

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/16072
  
**[Test build #69371 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69371/consoleFull)**
 for PR 16072 at commit 
[`88b53c3`](https://github.com/apache/spark/commit/88b53c3b542d1423c169af7b4e52ecd6da067ced).
 * This patch **fails PySpark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #16067: [SPARK-17897] [SQL] Fixed IsNotNull Constraint In...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/16067#discussion_r90167277
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala 
---
@@ -58,13 +57,28 @@ abstract class QueryPlan[PlanType <: 
QueryPlan[PlanType]] extends TreeNode[PlanT
   }
 
   /**
+   * Infer the Attribute-specific IsNotNull constraints from the null 
intolerant child expressions
+   * of constraints.
+   */
+  private def inferIsNotNullConstraints(constraint: Expression): 
Seq[Expression] =
--- End diff --

This change simply ignores all `IsNotNull`s which are not the top 
expression. The above case works because `Filter` splits it. But if the 
constraint looks like `Cast(IsNotNull(a), Integer) == 1`, we won't infer 
`IsNotNull(a)` from it, right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org