[GitHub] spark issue #18846: [SPARK-21642][CORE] Use FQDN for DRIVER_HOST_ADDRESS ins...

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/18846
  
@thide I think DRIVER_HOST_ADDRESS will be used to generate the driver 
url. Could you check if this line still works after your change? 
https://github.com/apache/spark/blob/b35660dd0e930f4b484a079d9e2516b0a7dacf1d/core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala#L131


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18890: [SPARK-21596][SS] Ensure places calling HDFSMetadataLog....

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/18890
  
Merged. Could you close the PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18867: [SPARK-21663][TESTS]test("remote fetch below max RPC mes...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18867
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18867: [SPARK-21663][TESTS]test("remote fetch below max RPC mes...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18867
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80437/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18867: [SPARK-21663][TESTS]test("remote fetch below max RPC mes...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18867
  
**[Test build #80437 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80437/testReport)**
 for PR 18867 at commit 
[`5fb7403`](https://github.com/apache/spark/commit/5fb7403b4b207e5239b04231938b5c9b20479506).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18890: [SPARK-21596][SS] Ensure places calling HDFSMetadataLog....

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/18890
  
Thanks! Merging to branch-2.2.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18797: [SPARK-21523][ML] update breeze to 0.13.2 for an ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18797


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18797: [SPARK-21523][ML] update breeze to 0.13.2 for an emergen...

[yanboliang]

Github user yanboliang commented on the issue:

https://github.com/apache/spark/pull/18797
  
Merged into master and branch-2.2. Thanks for all.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18789: SPARK-20433 Bump jackson from 2.6.5 to 2.6.7.1

[ash211]

Github user ash211 commented on the issue:

https://github.com/apache/spark/pull/18789
  
@srowen sorry for not picking up on this -- thanks for pushing it over the 
finish line in your PR!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18499: [SPARK-21176][WEB UI] Use a single ProxyServlet to proxy...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18499
  
@aosagie  do we still need the previous fix for SPARK-21176?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18499: [SPARK-21176][WEB UI] Use a single ProxyServlet t...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18499


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18421: [SPARK-21213][SQL] Support collecting partition-level st...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18421
  
@mbasmanova Really thank you for your contribution! LGTM except the above 
four comments.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18499: [SPARK-21176][WEB UI] Use a single ProxyServlet to proxy...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18499
  
thanks, merging to master!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18421: [SPARK-21213][SQL] Support collecting partition-l...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18421#discussion_r132099901
  
--- Diff: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/StatisticsSuite.scala ---
@@ -256,6 +257,222 @@ class StatisticsSuite extends 
StatisticsCollectionTestBase with TestHiveSingleto
 }
   }
 
+  test("analyze single partition") {
+val tableName = "analyzeTable_part"
+
+def queryStats(ds: String): CatalogStatistics = {
+  val partition =
+
spark.sessionState.catalog.getPartition(TableIdentifier(tableName), Map("ds" -> 
ds))
+  partition.stats.get
+}
+
+def createPartition(ds: String, query: String): Unit = {
+  sql(s"INSERT INTO TABLE $tableName PARTITION (ds='$ds') $query")
+}
+
+withTable(tableName) {
+  sql(s"CREATE TABLE $tableName (key STRING, value STRING) PARTITIONED 
BY (ds STRING)")
+
+  createPartition("2010-01-01", "SELECT '1', 'A' from src")
+  createPartition("2010-01-02", "SELECT '1', 'A' from src UNION ALL 
SELECT '1', 'A' from src")
+  createPartition("2010-01-03", "SELECT '1', 'A' from src")
+
+  sql(s"ANALYZE TABLE $tableName PARTITION (ds='2010-01-01') COMPUTE 
STATISTICS NOSCAN")
+
+  sql(s"ANALYZE TABLE $tableName PARTITION (ds='2010-01-02') COMPUTE 
STATISTICS NOSCAN")
+
+  assert(queryStats("2010-01-01").rowCount === None)
+  assert(queryStats("2010-01-01").sizeInBytes === 2000)
+
+  assert(queryStats("2010-01-02").rowCount === None)
+  assert(queryStats("2010-01-02").sizeInBytes === 2*2000)
+
+  sql(s"ANALYZE TABLE $tableName PARTITION (ds='2010-01-01') COMPUTE 
STATISTICS")
+
+  sql(s"ANALYZE TABLE $tableName PARTITION (ds='2010-01-02') COMPUTE 
STATISTICS")
+
+  assert(queryStats("2010-01-01").rowCount.get === 500)
+  assert(queryStats("2010-01-01").sizeInBytes === 2000)
+
+  assert(queryStats("2010-01-02").rowCount.get === 2*500)
+  assert(queryStats("2010-01-02").sizeInBytes === 2*2000)
+}
+  }
+
+  test("analyze a set of partitions") {
+val tableName = "analyzeTable_part"
+
+def queryStats(ds: String, hr: String): Option[CatalogStatistics] = {
+  val tableId = TableIdentifier(tableName)
+  val partition =
+spark.sessionState.catalog.getPartition(tableId, Map("ds" -> ds, 
"hr" -> hr))
+  partition.stats
+}
+
+def assertPartitionStats(
+ds: String,
+hr: String,
+rowCount: Option[BigInt],
+sizeInBytes: BigInt): Unit = {
+  val stats = queryStats(ds, hr).get
+  assert(stats.rowCount === rowCount)
+  assert(stats.sizeInBytes === sizeInBytes)
+}
+
+def createPartition(ds: String, hr: Int, query: String): Unit = {
+  sql(s"INSERT INTO TABLE $tableName PARTITION (ds='$ds', hr=$hr) 
$query")
+}
+
+withTable(tableName) {
+  sql(s"CREATE TABLE $tableName (key STRING, value STRING) PARTITIONED 
BY (ds STRING, hr INT)")
+
+  createPartition("2010-01-01", 10, "SELECT '1', 'A' from src")
+  createPartition("2010-01-01", 11, "SELECT '1', 'A' from src")
+  createPartition("2010-01-02", 10, "SELECT '1', 'A' from src")
+  createPartition("2010-01-02", 11,
+"SELECT '1', 'A' from src UNION ALL SELECT '1', 'A' from src")
+
+  sql(s"ANALYZE TABLE $tableName PARTITION (ds='2010-01-01') COMPUTE 
STATISTICS NOSCAN")
+
+  assertPartitionStats("2010-01-01", "10", rowCount = None, 
sizeInBytes = 2000)
+  assertPartitionStats("2010-01-01", "11", rowCount = None, 
sizeInBytes = 2000)
+  assert(queryStats("2010-01-02", "10") === None)
+  assert(queryStats("2010-01-02", "11") === None)
+
+  sql(s"ANALYZE TABLE $tableName PARTITION (ds='2010-01-02') COMPUTE 
STATISTICS NOSCAN")
+
+  assertPartitionStats("2010-01-01", "10", rowCount = None, 
sizeInBytes = 2000)
+  assertPartitionStats("2010-01-01", "11", rowCount = None, 
sizeInBytes = 2000)
+  assertPartitionStats("2010-01-02", "10", rowCount = None, 
sizeInBytes = 2000)
+  assertPartitionStats("2010-01-02", "11", rowCount = None, 
sizeInBytes = 2*2000)
+
+  sql(s"ANALYZE TABLE $tableName PARTITION (ds='2010-01-01') COMPUTE 
STATISTICS")
+
+  assertPartitionStats("2010-01-01", "10", rowCount = Some(500), 
sizeInBytes = 2000)
+  assertPartitionStats("2010-01-01", "11", rowCount = Some(500), 
sizeInBytes = 2000)
+  assertPartitionStats("2010-01-02", "10", rowCount = None, 
sizeInBytes = 2000)
+  assertPartitionStats("2010-01-02", "11", rowCount = None, 
sizeInBytes = 2*2000)
+
+  sql(s"ANALYZE TABLE $tableName PARTITIO

[GitHub] spark pull request #18421: [SPARK-21213][SQL] Support collecting partition-l...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18421#discussion_r132099286
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -1033,25 +998,126 @@ private[spark] class HiveExternalCatalog(conf: 
SparkConf, hadoopConf: Configurat
 currentFullPath
   }
 
+  private def statsToProperties(
+  stats: CatalogStatistics,
+  schema: StructType): Map[String, String] = {
+
+var statsProperties: Map[String, String] =
+  Map(STATISTICS_TOTAL_SIZE -> stats.sizeInBytes.toString())
+if (stats.rowCount.isDefined) {
+  statsProperties += STATISTICS_NUM_ROWS -> 
stats.rowCount.get.toString()
+}
+
+val colNameTypeMap: Map[String, DataType] =
+  schema.fields.map(f => (f.name, f.dataType)).toMap
+stats.colStats.foreach { case (colName, colStat) =>
+  colStat.toMap(colName, colNameTypeMap(colName)).foreach { case (k, 
v) =>
+statsProperties += (columnStatKeyPropName(colName, k) -> v)
+  }
+}
+
+statsProperties
+  }
+
+  private def statsFromProperties(
+  properties: Map[String, String],
+  table: String,
+  schema: StructType): Option[CatalogStatistics] = {
+
+val statsProps = properties.filterKeys(_.startsWith(STATISTICS_PREFIX))
+if (statsProps.isEmpty) {
+  None
+} else {
+
+  val colStats = new mutable.HashMap[String, ColumnStat]
+
+  // For each column, recover its column stats. Note that this is 
currently a O(n^2) operation,
+  // but given the number of columns it usually not enormous, this is 
probably OK as a start.
+  // If we want to map this a linear operation, we'd need a stronger 
contract between the
+  // naming convention used for serialization.
+  schema.foreach { field =>
+if (statsProps.contains(columnStatKeyPropName(field.name, 
ColumnStat.KEY_VERSION))) {
+  // If "version" field is defined, then the column stat is 
defined.
+  val keyPrefix = columnStatKeyPropName(field.name, "")
+  val colStatMap = 
statsProps.filterKeys(_.startsWith(keyPrefix)).map { case (k, v) =>
+(k.drop(keyPrefix.length), v)
+  }
+
+  ColumnStat.fromMap(table, field, colStatMap).foreach {
+colStat => colStats += field.name -> colStat
+  }
+}
+  }
+
+  Some(CatalogStatistics(
+sizeInBytes = BigInt(statsProps(STATISTICS_TOTAL_SIZE)),
+rowCount = statsProps.get(STATISTICS_NUM_ROWS).map(BigInt(_)),
+colStats = colStats.toMap))
+}
+  }
+
   override def alterPartitions(
   db: String,
   table: String,
   newParts: Seq[CatalogTablePartition]): Unit = withClient {
 val lowerCasedParts = newParts.map(p => p.copy(spec = 
lowerCasePartitionSpec(p.spec)))
+
+val rawTable = getRawTable(db, table)
+
+// For datasource tables and hive serde tables created by spark 2.1 or 
higher,
+// the data schema is stored in the table properties.
+val schema = restoreTableMetadata(rawTable).schema
+
+// convert partition statistics to properties so that we can persist 
them through hive api
+val withStatsProps = lowerCasedParts.map(p => {
+  if (p.stats.isDefined) {
+val statsProperties = statsToProperties(p.stats.get, schema)
+p.copy(parameters = p.parameters ++ statsProperties)
+  } else {
+p
+  }
+})
+
 // Note: Before altering table partitions in Hive, you *must* set the 
current database
 // to the one that contains the table of interest. Otherwise you will 
end up with the
 // most helpful error message ever: "Unable to alter partition. alter 
is not possible."
 // See HIVE-2742 for more detail.
 client.setCurrentDatabase(db)
-client.alterPartitions(db, table, lowerCasedParts)
+client.alterPartitions(db, table, withStatsProps)
   }
 
   override def getPartition(
   db: String,
   table: String,
   spec: TablePartitionSpec): CatalogTablePartition = withClient {
 val part = client.getPartition(db, table, lowerCasePartitionSpec(spec))
-part.copy(spec = restorePartitionSpec(part.spec, getTable(db, 
table).partitionColumnNames))
+restorePartitionMetadata(part, getTable(db, table))
+  }
+
+  /**
+   * Restores partition metadata from the partition properties.
+   *
+   * Reads partition-level statistics from partition properties, puts these
+   * into [[CatalogTablePartition#stats]] and removes these special entries
+   * from the partition properties.
  

[GitHub] spark issue #18787: [SPARK-21583][SQL] Create a ColumnarBatch from ArrowColu...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18787
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80438/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18787: [SPARK-21583][SQL] Create a ColumnarBatch from ArrowColu...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18787
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18787: [SPARK-21583][SQL] Create a ColumnarBatch from ArrowColu...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18787
  
**[Test build #80438 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80438/testReport)**
 for PR 18787 at commit 
[`23d19df`](https://github.com/apache/spark/commit/23d19dfde53d02c37a2c20f67c9816a73bd57cd2).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18421: [SPARK-21213][SQL] Support collecting partition-level st...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18421
  
There are still two remaining comments. Could you also address them?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18869: [SPARK-21654][SQL] Complement SQL predicates expression ...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/18869
  
@gatorsmile Sure. Let's wait after #18818. I'll look at it too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18869: [SPARK-21654][SQL] Complement SQL predicates expression ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18869
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18818: [SPARK-21110][SQL] Structs, arrays, and other orderable ...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18818
  
Also add `NULL` in the test case?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18869: [SPARK-21654][SQL] Complement SQL predicates expression ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18869
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80435/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18869: [SPARK-21654][SQL] Complement SQL predicates expression ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18869
  
**[Test build #80435 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80435/testReport)**
 for PR 18869 at commit 
[`1369fd5`](https://github.com/apache/spark/commit/1369fd5d9e6e78db90fa4a1af15ceeff9be38478).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18869: [SPARK-21654][SQL] Complement SQL predicates expression ...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18869
  
@viirya Could you hold this PR at first? There is a related PR 
https://github.com/apache/spark/pull/18818 that is changing the underlying 
semantics. We also need to update the description after this. Also, welcome to 
review that PR too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18709: [SPARK-21504] [SQL] Add spark version info into table me...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18709
  
**[Test build #80445 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80445/testReport)**
 for PR 18709 at commit 
[`f5a26cc`](https://github.com/apache/spark/commit/f5a26cccdbc2c670db60610edc756e4f758bd30e).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18880: [SPARK-21665][Core]Need to close resources after use

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18880
  
**[Test build #80444 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80444/testReport)**
 for PR 18880 at commit 
[`7bd2904`](https://github.com/apache/spark/commit/7bd29049ff6544013f80a2b55d229203bee28b57).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18890: [SPARK-21596][SS] Ensure places calling HDFSMetadataLog....

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18890
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18890: [SPARK-21596][SS] Ensure places calling HDFSMetadataLog....

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18890
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80436/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18790: [SPARK-21587][SS] Added pushdown through watermarks.

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18790
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80434/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18790: [SPARK-21587][SS] Added pushdown through watermarks.

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18790
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18890: [SPARK-21596][SS] Ensure places calling HDFSMetadataLog....

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18890
  
**[Test build #80436 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80436/testReport)**
 for PR 18890 at commit 
[`532529a`](https://github.com/apache/spark/commit/532529a485fb884579c13271a43d39167b7fbf16).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18790: [SPARK-21587][SS] Added pushdown through watermarks.

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18790
  
**[Test build #80434 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80434/testReport)**
 for PR 18790 at commit 
[`4cae897`](https://github.com/apache/spark/commit/4cae8973f52078afae2a9d92d59c91edaab0ba88).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18866: [SPARK-21649][SQL] Support writing data into hive bucket...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18866
  
Hash function is not the only issue, one important difference is: hive will 
shuffle before write, and make sure one bucket has only one file. Spark doesn't 
shuffle and each write task may write a file for a bucket. More details please 
refer to 
https://docs.google.com/document/d/1a8IDh23RAkrkg9YYAeO51F4aGO8-xAlupKwdshve2fc/edit#heading=h.ualze2k709kj

also cc @tejasapatil 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #16774: [SPARK-19357][ML] Adding parallel model evaluation in ML...

[WeichenXu123]

Github user WeichenXu123 commented on the issue:

https://github.com/apache/spark/pull/16774
  
@BryanCutler You are right. Once `Future` complete the model can be cleaned 
by GC. So the memory cost of the code has been optimized already. I didn't look 
at the code carefully a few days ago.
Thanks for your careful work for this. I will take a deeper look later. ^_^


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18790: [SPARK-21587][SS] Added pushdown through watermarks.

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18790
  
It sounds like we need to revisit all the optimizer rules for 
`EventTimeWatermark`? We can do it in the separate PRs.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18846: [SPARK-21642][CORE] Use FQDN for DRIVER_HOST_ADDRESS ins...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18846
  
**[Test build #80443 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80443/testReport)**
 for PR 18846 at commit 
[`afc07ee`](https://github.com/apache/spark/commit/afc07ee14974a38c3b6912dfd2943084d25eeccf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18846: [SPARK-21642][CORE] Use FQDN for DRIVER_HOST_ADDRESS ins...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18846
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18707: [SPARK-21503][UI]: Spark UI shows incorrect task ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18707


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18790: [SPARK-21587][SS] Added pushdown through watermar...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18790#discussion_r132095065
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -410,6 +413,14 @@ object PushProjectionThroughUnion extends 
Rule[LogicalPlan] with PredicateHelper
   } else {
 p
   }
+
+case p @ Project(projectList, watermark: EventTimeWatermark) =>
+  // Push as long as the project doesn't eliminates the attribute.
--- End diff --

Do we still need `Project` when `projectList` are identical to the output 
of `EventTimeWatermark`, except the metadata?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18707: [SPARK-21503][UI]: Spark UI shows incorrect task status ...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18707
  
I was just asking about different fixes, I'm ok with the current fix.

thanks, merging to master/2.2!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18756: [SPARK-21548][SQL] "Support insert into serial columns o...

[lvdongr]

Github user lvdongr commented on the issue:

https://github.com/apache/spark/pull/18756
  
The target of this pr is support  to insert into specified  columns， all  
columns is  no need ， like   insert into t(a, c) values (1, 0.8) .  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18790: [SPARK-21587][SS] Added pushdown through watermar...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18790#discussion_r132094610
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -410,6 +413,14 @@ object PushProjectionThroughUnion extends 
Rule[LogicalPlan] with PredicateHelper
   } else {
 p
   }
+
+case p @ Project(projectList, watermark: EventTimeWatermark) =>
+  // Push as long as the project doesn't eliminates the attribute.
--- End diff --

Do we need to check `deterministic` here, like what we do for the above 
case?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18709: [SPARK-21504] [SQL] Add spark version info into table me...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18709
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18709: [SPARK-21504] [SQL] Add spark version info into t...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/18709#discussion_r132094303
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
 ---
@@ -205,6 +205,9 @@ case class BucketSpec(
  *   configured.
  * @param ignoredProperties is a list of table properties that are used by 
the underlying table
  *  but ignored by Spark SQL yet.
+ * @param createVersion records the version of Spark that created this 
table metadata. The default
+ *  is '2.2 or prior'. We expect it will be read from 
the catalog or filled by
--- End diff --

the default is empty string.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18756: [SPARK-21548][SQL] "Support insert into serial columns o...

[maropu]

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/18756
  
I think this pr should be fixed in that way, maybe.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18756: [SPARK-21548][SQL] "Support insert into serial columns o...

[maropu]

Github user maropu commented on the issue:

https://github.com/apache/spark/pull/18756
  
I'm still not looking into this pr though, IMHO it would be good for users 
to select parts of columns like other databases;
```
postgres=# create table t(a int, b text, c float8, d int);

postgres=# insert into t(a, c) values (1, 0.8);
INSERT 0 1

postgres=# select * from t;
 a | b |  c  | d 
---+---+-+---
 1 |   | 0.8 |  
(1 row)
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18648: [SPARK-21428] Turn IsolatedClientLoader off while using ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18648
  
**[Test build #80442 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80442/testReport)**
 for PR 18648 at commit 
[`6c0bf70`](https://github.com/apache/spark/commit/6c0bf709f95dec3ee3bce3e905e51f31fc5b0e64).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18648: [SPARK-21428] Turn IsolatedClientLoader off while...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/18648#discussion_r132093341
  
--- Diff: 
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveCliSessionStateSuite.scala
 ---
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.thriftserver
+
+import org.apache.hadoop.hive.cli.CliSessionState
+import org.apache.hadoop.hive.conf.HiveConf
+import org.apache.hadoop.hive.ql.session.SessionState
+
+import org.apache.spark.{SparkConf, SparkFunSuite}
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.sql.hive.HiveUtils
+
+class HiveCliSessionStateSuite extends SparkFunSuite {
+
+  test("CliSessionState will be reused") {
+val hiveConf = new HiveConf(classOf[SessionState])
+HiveUtils.newTemporaryConfiguration(useInMemoryDerby = false).foreach {
+  case (key, value) => hiveConf.set(key, value)
+}
+val sessionState: SessionState = new CliSessionState(hiveConf)
+SessionState.start(sessionState)
+val s1 = SessionState.get
+val sparkConf = new SparkConf()
+val hadoopConf = SparkHadoopUtil.get.newConfiguration(sparkConf)
+val s2 = HiveUtils.newClientForMetadata(sparkConf, hadoopConf).getState
--- End diff --

weird, `HiveClientImpl` is the only implementation of the `HiveClient` 
interface.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18790: [SPARK-21587][SS] Added pushdown through watermar...

[gatorsmile]

Github user gatorsmile commented on a diff in the pull request:

https://github.com/apache/spark/pull/18790#discussion_r132093307
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 ---
@@ -872,6 +886,25 @@ object PushDownPredicate extends Rule[LogicalPlan] 
with PredicateHelper {
   pushDownPredicate(filter, u.child) { predicate =>
 u.withNewChildren(Seq(Filter(predicate, u.child)))
   }
+
+case filter @ Filter(condition, watermark: EventTimeWatermark) =>
--- End diff --

I see. How about moving this case above `UnaryNode`? The following two 
functions `canPushThrough ` and `pushDownPredicate ` are just for UnaryNode.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18648: [SPARK-21428] Turn IsolatedClientLoader off while using ...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18648
  
ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #17894: [WIP][SPARK-17134][ML] Use level 2 BLAS operations in Lo...

[WeichenXu123]

Github user WeichenXu123 commented on the issue:

https://github.com/apache/spark/pull/17894
  
I am also interested in implementation by level-3 BLAS. Can you post a 
design doc first?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18855
  
since this PR only focus on `DiskStore`, shall we remove the new tests in 
`BlockManagerSuite`? Seems the OOM only happens in `BlockManagerSuite`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18814: [SPARK-21608][SPARK-9221][SQL] Window rangeBetwee...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18814


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18814: [SPARK-21608][SPARK-9221][SQL] Window rangeBetween() API...

[cloud-fan]

Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/18814
  
thanks, merging to master!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18756: [SPARK-21548][SQL] "Support insert into serial columns o...

[gatorsmile]

Github user gatorsmile commented on the issue:

https://github.com/apache/spark/pull/18756
  
I see. This PR still requires users to provide all the columns, right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15435
  
**[Test build #80441 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80441/testReport)**
 for PR 15435 at commit 
[`403680c`](https://github.com/apache/spark/commit/403680ca8aaf2e14993993d6ecbabe7b58881ed9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18869: [SPARK-21654][SQL] Complement SQL predicates expression ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18869
  
**[Test build #80440 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80440/testReport)**
 for PR 18869 at commit 
[`b64c9e6`](https://github.com/apache/spark/commit/b64c9e67dfb34668ba93089d84f932778c005d7b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18869: [SPARK-21654][SQL] Complement SQL predicates expression ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18869
  
**[Test build #80439 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80439/testReport)**
 for PR 18869 at commit 
[`bca2b0b`](https://github.com/apache/spark/commit/bca2b0bc71f5a7ea2d909dfad5e6e094e587d288).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18869: [SPARK-21654][SQL] Complement SQL predicates expr...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/18869#discussion_r132088990
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
 ---
@@ -564,8 +611,27 @@ case class EqualNullSafe(left: Expression, right: 
Expression) extends BinaryComp
   }
 }
 
+// scalastyle:off line.size.limit
 @ExpressionDescription(
-  usage = "expr1 _FUNC_ expr2 - Returns true if `expr1` is less than 
`expr2`.")
+  usage = "expr1 _FUNC_ expr2 - Returns true if `expr1` is less than 
`expr2`.",
+  arguments = """
+Arguments:
+  * expr1, expr2 - the two expressions must be same type or can be 
casted to a common type, and must be a type that can be ordered/compared.
--- End diff --

Thanks. Fixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18849: [SPARK-21617][SQL] Store correct table metadata w...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/18849#discussion_r132088495
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala ---
@@ -1175,6 +1205,27 @@ private[spark] class HiveExternalCatalog(conf: 
SparkConf, hadoopConf: Configurat
 client.listFunctions(db, pattern)
   }
 
+  /** Detect whether a table is stored with Hive-compatible metadata. */
+  private def isHiveCompatible(table: CatalogTable): Boolean = {
+val provider = 
table.provider.orElse(table.properties.get(DATASOURCE_PROVIDER))
+if (provider.isDefined && provider != Some(DDLUtils.HIVE_PROVIDER)) {
+  table.properties.get(DATASOURCE_HIVE_COMPATIBLE) match {
+case Some(value) =>
+  value.toBoolean
+case _ =>
+  // If the property is not set, the table may have been created 
by an old version
+  // of Spark. Detect Hive compatibility by comparing the table's 
serde with the
+  // serde for the table's data source. If they match, the table 
is Hive-compatible.
+  // If they don't, they're not, because of some other table 
property that made it
+  // not initially Hive-compatible.
+  HiveSerDe.sourceToSerDe(provider.get) == table.storage.serde
--- End diff --

There is a change regarding treating case-sensitive DS tables as 
Hive-incompatible. Once the given table is this kind of table without the new 
`DATASOURCE_HIVE_COMPATIBLE` property, we should treat it as Hive compatible or 
incompatible? Looks like for now we treat it as compatible?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18869: [SPARK-21654][SQL] Complement SQL predicates expr...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/18869#discussion_r132088507
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
 ---
@@ -564,8 +611,27 @@ case class EqualNullSafe(left: Expression, right: 
Expression) extends BinaryComp
   }
 }
 
+// scalastyle:off line.size.limit
 @ExpressionDescription(
-  usage = "expr1 _FUNC_ expr2 - Returns true if `expr1` is less than 
`expr2`.")
+  usage = "expr1 _FUNC_ expr2 - Returns true if `expr1` is less than 
`expr2`.",
+  arguments = """
+Arguments:
+  * expr1, expr2 - the two expressions must be same type or can be 
casted to a common type, and must be a type that can be ordered/compared.
--- End diff --

@viirya, BTW, I think we could also do this as below:

```diff
-  * expr1, expr2 - the two expressions must be same type or can be 
casted to a common type, and must be a type that can be ordered/compared.
+  * expr1, expr2 - the two expressions must be same type or can be 
casted to a common type,
+  and must be a type that can be ordered/compared.
```

Just double checked it renders fine in the doc as well.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18630
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18630
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80433/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18630: [SPARK-12559][SPARK SUBMIT] fix --packages for stand-alo...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18630
  
**[Test build #80433 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80433/testReport)**
 for PR 18630 at commit 
[`a177992`](https://github.com/apache/spark/commit/a177992ec0a5a6797422c6b445c95b5b50d436b5).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18875: [SPARK-21513][SQL] Allow UDF to_json support conv...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

https://github.com/apache/spark/pull/18875#discussion_r132088249
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala
 ---
@@ -659,13 +660,19 @@ case class StructsToJson(
 (arr: Any) =>
   gen.write(arr.asInstanceOf[ArrayData])
   getAndReset()
+  case MapType(_: DataType, _: StructType, _: Boolean) =>
+(map: Any) =>
+  val mapType = child.dataType.asInstanceOf[MapType]
+  gen.write(map.asInstanceOf[MapData], mapType)
+  getAndReset()
 }
   }
 
   override def dataType: DataType = StringType
 
   override def checkInputDataTypes(): TypeCheckResult = child.dataType 
match {
-case _: StructType | ArrayType(_: StructType, _) =>
+case _: StructType | ArrayType(_: StructType, _) |
+ MapType(_: DataType, _: StructType, _: Boolean) =>
--- End diff --

I see. Looks now we have assumption that `JacksonGenerator` only takes 
`StructType` now. I think expression name should be fine and I guess we are 
okay to rename it if it looks required (actually I did it to `StructsToJson` :) 
).

Would it be difficult to support `MapType` like `StructType`? I took a 
quick look and it looks quite a lot of places to fix though. Up to my 
understanding, we could load JSON both `MapType` and `StructType` too.

```
scala> spark.read.schema("a map").json(Seq("""{"a": {"a": 
2}}""").toDS).show()
+---+
|  a|
+---+
|Map(a -> 2)|
+---+


scala> spark.read.schema("a struct").json(Seq("""{"a": {"a": 
2}}""").toDS).show()
+---+
|  a|
+---+
|[2]|
+---+
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18849: [SPARK-21617][SQL] Store correct table metadata w...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/18849#discussion_r132087575
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala 
---
@@ -908,7 +909,13 @@ private[hive] object HiveClientImpl {
 }
 // after SPARK-19279, it is not allowed to create a hive table with an 
empty schema,
 // so here we should not add a default col schema
--- End diff --

This comment looks like to move accordingly?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18887: [SPARK-20642][core] Store FsHistoryProvider listing data...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18887
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80431/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18887: [SPARK-20642][core] Store FsHistoryProvider listing data...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18887
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18887: [SPARK-20642][core] Store FsHistoryProvider listing data...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18887
  
**[Test build #80431 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80431/testReport)**
 for PR 18887 at commit 
[`1ec1a67`](https://github.com/apache/spark/commit/1ec1a672673eafa9f04ad81c1fff3feadfcddd2c).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18474: [SPARK-21235][TESTS] UTest should clear temp results whe...

[HyukjinKwon]

Github user HyukjinKwon commented on the issue:

https://github.com/apache/spark/pull/18474
  
Thanks for reproduction @kiszk. @wangjiaochun, I think the cause should be 
explained here. I remember I reviewed similar PRs before where the cause was 
`Utils.getOrCreateLocalRootDirs` is being cached.

However, what I still don't get is, if my understanding is correct, we are 
moving all temp directories in shutdown hooks at least. Did I maybe miss 
something - @jiangxb1987?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18787: [SPARK-21583][SQL] Create a ColumnarBatch from ArrowColu...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18787
  
**[Test build #80438 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80438/testReport)**
 for PR 18787 at commit 
[`23d19df`](https://github.com/apache/spark/commit/23d19dfde53d02c37a2c20f67c9816a73bd57cd2).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18890: [SPARK-21596][SS] Ensure places calling HDFSMetadataLog....

[zsxwing]

Github user zsxwing commented on the issue:

https://github.com/apache/spark/pull/18890
  
LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18787: [SPARK-21583][SQL] Create a ColumnarBatch from ArrowColu...

[BryanCutler]

Github user BryanCutler commented on the issue:

https://github.com/apache/spark/pull/18787
  
jenkins retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18884: [SPARK-21669] Internal API for collecting metrics...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/18884#discussion_r132086218
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
 ---
@@ -269,50 +278,57 @@ object FileFormatWriter extends Logging {
   }
 
   /**
+   * For every registered [[WriteJobStatsTracker]], call `processStats()` 
on it, passing it
+   * the corresponding [[WriteTaskStats]] from all executors.
+   */
+  private def processStats(
+  statsTrackers: Seq[WriteJobStatsTracker],
+  statsPerTask: Seq[Seq[WriteTaskStats]])
--- End diff --

The current framework looks like the trackers can't share the collection of 
states or some same metrics. Isn't a likely happened use case? When two 
trackers needs the same metrics, we will need to collect it in two copies of 
stats.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18849
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80432/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18849
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18884: [SPARK-21669] Internal API for collecting metrics...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/18884#discussion_r132085558
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
 ---
@@ -269,50 +278,57 @@ object FileFormatWriter extends Logging {
   }
 
   /**
+   * For every registered [[WriteJobStatsTracker]], call `processStats()` 
on it, passing it
+   * the corresponding [[WriteTaskStats]] from all executors.
+   */
+  private def processStats(
+  statsTrackers: Seq[WriteJobStatsTracker],
+  statsPerTask: Seq[Seq[WriteTaskStats]])
+: Unit = {
+
+val statsPerTracker = if (statsPerTask.nonEmpty) {
+  statsPerTask.transpose
+} else {
+  statsTrackers.map(_ => Seq.empty)
+}
+assert(statsTrackers.length == statsPerTracker.length,
+  s"""Every WriteTask should have produced one `WriteTaskStats` object 
for every tracker.
+ |statsTrackers = ${statsTrackers}
+ |statsPerTracker = ${statsPerTracker}
+ """.stripMargin)
--- End diff --

In case of many states, this might result big error message. Just printing 
length is enough?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18867: [SPARK-21663][TESTS]test("remote fetch below max RPC mes...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18867
  
**[Test build #80437 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80437/testReport)**
 for PR 18867 at commit 
[`5fb7403`](https://github.com/apache/spark/commit/5fb7403b4b207e5239b04231938b5c9b20479506).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18849
  
**[Test build #80432 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80432/testReport)**
 for PR 18849 at commit 
[`7b777ed`](https://github.com/apache/spark/commit/7b777edccddf91f2dfb99b30855265284188e00b).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18799: [SPARK-21596][SS]Ensure places calling HDFSMetadataLog.g...

[tdas]

Github user tdas commented on the issue:

https://github.com/apache/spark/pull/18799
  
Never mind @zsxwing I already opened a PR #18890


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18890: [SPARK-21596][SS] Ensure places calling HDFSMetadataLog....

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18890
  
**[Test build #80436 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80436/testReport)**
 for PR 18890 at commit 
[`532529a`](https://github.com/apache/spark/commit/532529a485fb884579c13271a43d39167b7fbf16).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18890: [SPARK-21596][SS] Ensure places calling HDFSMetad...

[tdas]

GitHub user tdas opened a pull request:

https://github.com/apache/spark/pull/18890

[SPARK-21596][SS] Ensure places calling HDFSMetadataLog.get check the 
return value

When I was investigating a flaky test, I realized that many places don't 
check the return value of `HDFSMetadataLog.get(batchId: Long): Option[T]`. When 
a batch is supposed to be there, the caller just ignores None rather than 
throwing an error. If some bug causes a query doesn't generate a batch metadata 
file, this behavior will hide it and allow the query continuing to run and 
finally delete metadata logs and make it hard to debug.

This PR ensures that places calling HDFSMetadataLog.get always check the 
return value.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tdas/spark SPARK-21596-2.2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/18890.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #18890


commit 532529a485fb884579c13271a43d39167b7fbf16
Author: Shixiong Zhu 
Date:   2017-08-09T03:20:26Z

[SPARK-21596][SS] Ensure places calling HDFSMetadataLog.get check the 
return value

When I was investigating a flaky test, I realized that many places don't 
check the return value of `HDFSMetadataLog.get(batchId: Long): Option[T]`. When 
a batch is supposed to be there, the caller just ignores None rather than 
throwing an error. If some bug causes a query doesn't generate a batch metadata 
file, this behavior will hide it and allow the query continuing to run and 
finally delete metadata logs and make it hard to debug.

This PR ensures that places calling HDFSMetadataLog.get always check the 
return value.

Jenkins

Author: Shixiong Zhu 

Closes #18799 from zsxwing/SPARK-21596.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18869: [SPARK-21654][SQL] Complement SQL predicates expression ...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18869
  
**[Test build #80435 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80435/testReport)**
 for PR 18869 at commit 
[`1369fd5`](https://github.com/apache/spark/commit/1369fd5d9e6e78db90fa4a1af15ceeff9be38478).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18869: [SPARK-21654][SQL] Complement SQL predicates expression ...

[viirya]

Github user viirya commented on the issue:

https://github.com/apache/spark/pull/18869
  
@gatorsmile I've checked the binary predicates in this change. Related test 
cases are added.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18869: [SPARK-21654][SQL] Complement SQL predicates expr...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/18869#discussion_r132084124
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
 ---
@@ -484,7 +501,20 @@ object Equality {
 }
 
 @ExpressionDescription(
-  usage = "expr1 _FUNC_ expr2 - Returns true if `expr1` equals `expr2`, or 
false otherwise.")
+  usage = "expr1 _FUNC_ expr2 - Returns true if `expr1` equals `expr2`, or 
false otherwise.",
+  arguments = """
+Arguments:
+  * expr1, expr2 - the two expressions must be same type.
--- End diff --

I've modified the statement.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18799: [SPARK-21596][SS]Ensure places calling HDFSMetadataLog.g...

[tdas]

Github user tdas commented on the issue:

https://github.com/apache/spark/pull/18799
  
Merged to master. But there were conflicts with 2.2. Can you make another 
PR for 2.2.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15435
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18799: [SPARK-21596][SS]Ensure places calling HDFSMetada...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/18799


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/15435
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80428/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18790: [SPARK-21587][SS] Added pushdown through watermarks.

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18790
  
**[Test build #80434 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80434/testReport)**
 for PR 18790 at commit 
[`4cae897`](https://github.com/apache/spark/commit/4cae8973f52078afae2a9d92d59c91edaab0ba88).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #15435: [SPARK-17139][ML] Add model summary for MultinomialLogis...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/15435
  
**[Test build #80428 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80428/testReport)**
 for PR 15435 at commit 
[`cb0a818`](https://github.com/apache/spark/commit/cb0a818ff6bfede1d8e0551e81c955692bb4817a).
 * This patch **fails SparkR unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18790: [SPARK-21587][SS] Added pushdown through watermarks.

[tdas]

Github user tdas commented on the issue:

https://github.com/apache/spark/pull/18790
  
retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18869: [SPARK-21654][SQL] Complement SQL predicates expr...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/18869#discussion_r132081911
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
 ---
@@ -484,7 +501,20 @@ object Equality {
 }
 
 @ExpressionDescription(
-  usage = "expr1 _FUNC_ expr2 - Returns true if `expr1` equals `expr2`, or 
false otherwise.")
+  usage = "expr1 _FUNC_ expr2 - Returns true if `expr1` equals `expr2`, or 
false otherwise.",
+  arguments = """
+Arguments:
+  * expr1, expr2 - the two expressions must be same type.
--- End diff --

Right. The rule `ImplicitTypeCasts` will specially do type casting for 
`BinaryOperator`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18799: [SPARK-21596][SS]Ensure places calling HDFSMetadataLog.g...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18799
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80429/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18799: [SPARK-21596][SS]Ensure places calling HDFSMetadataLog.g...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/18799
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18799: [SPARK-21596][SS]Ensure places calling HDFSMetadataLog.g...

[SparkQA]

Github user SparkQA commented on the issue:

https://github.com/apache/spark/pull/18799
  
**[Test build #80429 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80429/testReport)**
 for PR 18799 at commit 
[`16b02da`](https://github.com/apache/spark/commit/16b02da4f75f7838006ce8a03d63e04d8bc21455).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #18867: [SPARK-21663][TESTS]test("remote fetch below max ...

[jiangxb1987]

Github user jiangxb1987 commented on a diff in the pull request:

https://github.com/apache/spark/pull/18867#discussion_r132081417
  
--- Diff: core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala 
---
@@ -176,7 +176,7 @@ class MapOutputTrackerSuite extends SparkFunSuite {
 val masterTracker = newTrackerMaster(newConf)
 val rpcEnv = createRpcEnv("spark")
 val masterEndpoint = new MapOutputTrackerMasterEndpoint(rpcEnv, 
masterTracker, newConf)
-rpcEnv.setupEndpoint(MapOutputTracker.ENDPOINT_NAME, masterEndpoint)
+masterTracker.trackerEndpoint = 
rpcEnv.setupEndpoint(MapOutputTracker.ENDPOINT_NAME, masterEndpoint)
--- End diff --

nit:
```
masterTracker.trackerEndpoint =
  rpcEnv.setupEndpoint(MapOutputTracker.ENDPOINT_NAME, masterEndpoint)
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18790: [SPARK-21587][SS] Added pushdown through watermarks.

[tdas]

Github user tdas commented on the issue:

https://github.com/apache/spark/pull/18790
  
Ok to test

On Aug 8, 2017 7:48 PM, "UCB AMPLab"  wrote:

> Can one of the admins verify this patch?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18790: [SPARK-21587][SS] Added pushdown through watermarks.

[tdas]

Github user tdas commented on the issue:

https://github.com/apache/spark/pull/18790
  
Ok to test


On Aug 8, 2017 7:48 PM, "UCB AMPLab"  wrote:

> Can one of the admins verify this patch?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org