date:20190731

[GitHub] [spark] HyukjinKwon edited a comment on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

2019-07-31 Thread GitBox

HyukjinKwon edited a comment on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517131664
 
 
   Let me cc Spark committers to inform although it might be a bit of noise .. 
because I know it's annoying to use this script with Python 2 in Mac due to 
some pre-installed lib issues. After this PR, we can now use Python 3 for this 
script. cc @apache/spark-committers.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517131144
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13597/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517131141
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

2019-07-31 Thread GitBox

HyukjinKwon commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr 
script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517131664
 
 
   Let me cc Spark committers to inform although it might be a bit of noise .. 
because I know it's annoying to use Python 2 in Mac due to some pre-installed 
lib issues. After this PR, we can now use Python 3 for this script. cc 
@apache/spark-committers.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

2019-07-31 Thread GitBox

SparkQA commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr 
script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517131560
 
 
   **[Test build #108503 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108503/testReport)**
 for PR 25322 at commit 
[`0d0fdc9`](https://github.com/apache/spark/commit/0d0fdc99b82e650a4fbc30dd5edbca5e48e6621a).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517131141
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

2019-07-31 Thread GitBox

HyukjinKwon commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr 
script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517131263
 
 
   Yea, it's deprecated but not removed out yet.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517131144
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13597/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

2019-07-31 Thread GitBox

maropu commented on issue #25322: [SPARK-28586][INFRA] Make merge-spark-pr 
script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322#issuecomment-517130856
 
 
   We still need to keep the Python2 support?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon opened a new pull request #25322: [SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

2019-07-31 Thread GitBox

HyukjinKwon opened a new pull request #25322: [SPARK-28586][INFRA] Make 
merge-spark-pr script compatible with Python 3
URL: https://github.com/apache/spark/pull/25322
 
 
   ## What changes were proposed in this pull request?
   
   This PR proposes to make `merge_spark_pr.py` script Python 3 compatible.
   
   ## How was this patch tested?
   
   Manually tested against my forked remote with the PR and JIRA below:
   
   https://github.com/apache/spark/pull/25321
   https://github.com/apache/spark/pull/25286
   https://issues.apache.org/jira/browse/SPARK-28153


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #24938: [SPARK-27946][SQL] Hive DDL to Spark DDL conversion USING "show create table"

2019-07-31 Thread GitBox

maropu commented on a change in pull request #24938: [SPARK-27946][SQL] Hive 
DDL to Spark DDL conversion USING "show create table"
URL: https://github.com/apache/spark/pull/24938#discussion_r309530837
 
 

 ##
 File path: 
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveShowCreateTableSuite.scala
 ##
 @@ -195,4 +203,214 @@ class HiveShowCreateTableSuite extends 
ShowCreateTableSuite with TestHiveSinglet
 
hiveContext.sharedState.externalCatalog.unwrapped.asInstanceOf[HiveExternalCatalog]
   .client.runSqlHive(ddl)
   }
+
+  private def checkCreateSparkTable(tableName: String): Unit = {
+val table = TableIdentifier(tableName, Some("default"))
+val db = table.database.get
+val hiveTable = spark.sharedState.externalCatalog.getTable(db, table.table)
+val shownSparkDDL = sql(s"SHOW CREATE TABLE ${table.quotedString} AS 
SPARK").head().getString(0)
+// Drops original Hive table.
+sql(s"DROP TABLE ${table.quotedString}")
+
+try {
+  sql(shownSparkDDL)
+  val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+  val shownDDL = sql(s"SHOW CREATE TABLE 
${table.quotedString}").head().getString(0)
+
+  // Drops created Spark table using `SHOW CREATE TABLE AS SPARK`.
+  sql(s"DROP TABLE ${table.quotedString}")
+
+  sql(shownDDL)
+  val expected = spark.sharedState.externalCatalog.getTable(db, 
table.table)
+
+  checkCatalogTables(expected, actual)
+  checkHiveCatalogTables(hiveTable, actual)
+} finally {
+  sql(s"DROP TABLE IF EXISTS ${table.table}")
+}
+  }
+
+  private def checkHiveCatalogTables(expected: CatalogTable, actual: 
CatalogTable): Unit = {
+def normalize(table: CatalogTable): CatalogTable = {
+  val nondeterministicProps = Set(
+"CreateTime",
+"transient_lastDdlTime",
+"grantTime",
+"lastUpdateTime",
+"last_modified_by",
+"last_modified_time",
+"Owner:",
+// The following are hive specific schema parameters which we do not 
need to match exactly.
+"totalNumberFiles",
+"maxFileSize",
+"minFileSize"
+  )
+
+  table.copy(
+createTime = 0L,
+lastAccessTime = 0L,
+properties = 
table.properties.filterKeys(!nondeterministicProps.contains(_)),
+stats = None,
+ignoredProperties = Map.empty,
+storage = CatalogStorageFormat.empty,
+provider = None,
+tracksPartitionsInCatalog = false
+  )
+}
+assert(normalize(actual) == normalize(expected))
+  }
+
+  test("simple hive table as spark") {
+withTable("t1") {
+  sql(
+s"""CREATE TABLE t1 (
 
 Review comment:
   nit: format issue? 
https://github.com/apache/spark/pull/25204#discussion_r307234253


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] gatorsmile commented on a change in pull request #25107: [SPARK-28344][SQL] detect ambiguous self-join and fail the query

2019-07-31 Thread GitBox

gatorsmile commented on a change in pull request #25107: [SPARK-28344][SQL] 
detect ambiguous self-join and fail the query
URL: https://github.com/apache/spark/pull/25107#discussion_r309530391
 
 

 ##
 File path: sql/core/src/main/scala/org/apache/spark/sql/Column.scala
 ##
 @@ -144,11 +153,16 @@ class Column(val expr: Expression) extends Logging {
   override def toString: String = toPrettySQL(expr)
 
   override def equals(that: Any): Boolean = that match {
-case that: Column => that.expr.equals(this.expr)
+case that: Column => that.normalizedExpr().equals(this.normalizedExpr())
 case _ => false
   }
 
-  override def hashCode: Int = this.expr.hashCode()
+  override def hashCode: Int = this.normalizedExpr().hashCode()
+
+  private def normalizedExpr(): Expression = expr match {
 
 Review comment:
   When we add the metadata in addDataFrameIdToCol, we use transform. Do we 
need to use it here when removing it? 
   
   Our test case can cover it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] gatorsmile commented on issue #24938: [SPARK-27946][SQL] Hive DDL to Spark DDL conversion USING "show create table"

2019-07-31 Thread GitBox

gatorsmile commented on issue #24938: [SPARK-27946][SQL] Hive DDL to Spark DDL 
conversion USING "show create table"
URL: https://github.com/apache/spark/pull/24938#issuecomment-517125964
 
 
   cc @cloud-fan @gengliangwang
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] gatorsmile commented on a change in pull request #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

2019-07-31 Thread GitBox

gatorsmile commented on a change in pull request #25294: [SPARK-28476][SQL] 
Support ALTER DATABASE SET LOCATION
URL: https://github.com/apache/spark/pull/25294#discussion_r309527041
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala
 ##
 @@ -184,6 +184,15 @@ class DDLParserSuite extends AnalysisTest with 
SharedSQLContext {
   containsThesePhrases = Seq("key_without_value"))
   }
 
+  test("alter database set location") {
+// ALTER (DATABASE|SCHEMA) database_name SET LOCATION
+val sql1 = "ALTER DATABASE database_name SET LOCATION '/home/user/db'"
 
 Review comment:
   1. A database that does not exist before issuing this command?
   2. Setting an illegal path that could trigger an exception inside the 
function `stringToURI`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #25286: [SPARK-25584][ML][DOC] datasource for libsvm user guide

2019-07-31 Thread GitBox

HyukjinKwon commented on issue #25286: [SPARK-25584][ML][DOC] datasource for 
libsvm user guide
URL: https://github.com/apache/spark/pull/25286#issuecomment-517121115
 
 
   (ignore the logs above. I was testing merge script Python 3 compatibility)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] rdblue commented on a change in pull request #24382: [SPARK-27330][SS] support task abort in foreach writer

2019-07-31 Thread GitBox

rdblue commented on a change in pull request #24382: [SPARK-27330][SS] support 
task abort in foreach writer
URL: https://github.com/apache/spark/pull/24382#discussion_r309521630
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ForeachWriterTable.scala
 ##
 @@ -141,17 +143,33 @@ class ForeachDataWriter[T](
   writer.process(rowConverter(record))
 } catch {
   case t: Throwable =>
-writer.close(t)
+closeWriter(t)
 throw t
 }
   }
 
   override def commit(): WriterCommitMessage = {
-writer.close(null)
+closeWriter(null)
 ForeachWriterCommitMessage
   }
 
-  override def abort(): Unit = {}
+  override def abort(): Unit = {
 
 Review comment:
   I can see that the exception is passed to close. My question is: what does 
the writer do differently based on the exception? If this is to satisfy an API 
and you can pass any exception, then I don't think it matters. There's effort 
required to get the exception to pass through and without a reasonable use case 
I'm wondering why it is necessary to do it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

2019-07-31 Thread GitBox

HyukjinKwon commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use 
AtomicReference at InputFileBlockHolder (to support input_file_name with Python 
UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517117629
 
 
   I am testing mergine script Python 3 compatibility. please ignore the noise 
above.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

2019-07-31 Thread GitBox

SparkQA commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use 
AtomicReference at InputFileBlockHolder (to support input_file_name with Python 
UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517113023
 
 
   **[Test build #108502 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108502/testReport)**
 for PR 25321 at commit 
[`4275f82`](https://github.com/apache/spark/commit/4275f8284d13c49a46a15e48aed08a4114201e7e).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25262: [SPARK-28486][CORE][PYTHON] 
Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517112556
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25321: 
[SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder 
(to support input_file_name with Python UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517112639
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13596/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use 
AtomicReference at InputFileBlockHolder (to support input_file_name with Python 
UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517112639
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13596/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25321: 
[SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder 
(to support input_file_name with Python UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517112634
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use 
AtomicReference at InputFileBlockHolder (to support input_file_name with Python 
UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517112634
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25262: [SPARK-28486][CORE][PYTHON] 
Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517112559
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108499/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map 
PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517112556
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map 
PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517112559
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108499/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

2019-07-31 Thread GitBox

SparkQA removed a comment on issue #25262: [SPARK-28486][CORE][PYTHON] Map 
PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517082869
 
 
   **[Test build #108499 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108499/testReport)**
 for PR 25262 at commit 
[`5cdbab3`](https://github.com/apache/spark/commit/5cdbab3659edd219bff462ce4d004724cff68e10).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

2019-07-31 Thread GitBox

HyukjinKwon commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use 
AtomicReference at InputFileBlockHolder (to support input_file_name with Python 
UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517112094
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

2019-07-31 Thread GitBox

SparkQA commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map 
PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517112137
 
 
   **[Test build #108499 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108499/testReport)**
 for PR 25262 at commit 
[`5cdbab3`](https://github.com/apache/spark/commit/5cdbab3659edd219bff462ce4d004724cff68e10).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] beliefer commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array

2019-07-31 Thread GitBox

beliefer commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI 
SQL: OVERLAY function support byte array
URL: https://github.com/apache/spark/pull/25172#discussion_r309516995
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
 ##
 @@ -141,6 +141,19 @@ class StringFunctionsSuite extends QueryTest with 
SharedSQLContext {
 // scalastyle:on
   }
 
+  test("binary overlay function") {
+// non ascii characters are not allowed in the code, so we disable the 
scalastyle here.
+val df = Seq((Array[Byte](1, 2, 3, 4, 5, 6, 7, 8, 9))).toDF("a")
+checkAnswer(df.select(overlay(
+  $"a", Array[Byte](-1), 6)), Row(Array[Byte](1, 2, 3, 4, 5, -1, 7, 8, 9)))
+checkAnswer(df.select(overlay(
+  $"a", Array[Byte](-1, -1, -1, -1), 7)), Row(Array[Byte](1, 2, 3, 4, 5, 
6, -1, -1, -1, -1)))
+checkAnswer(df.select(overlay(
+  $"a", Array[Byte](-1, -1), 7, 0)), Row(Array[Byte](1, 2, 3, 4, 5, 6, -1, 
-1, 7, 8, 9)))
+checkAnswer(df.select(overlay($"a", Array[Byte](-1, -1, -1, -1, -1), 2, 
4)),
+  Row(Array[Byte](1, -1, -1, -1, -1, -1, 6, 7, 8, 9)))
+  }
 
 Review comment:
   OK. Thanks for the decision. @maropu 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25321: 
[SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder 
(to support input_file_name with Python UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517110949
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108497/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

2019-07-31 Thread GitBox

SparkQA removed a comment on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] 
Use AtomicReference at InputFileBlockHolder (to support input_file_name with 
Python UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517070221
 
 
   **[Test build #108497 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108497/testReport)**
 for PR 25321 at commit 
[`4275f82`](https://github.com/apache/spark/commit/4275f8284d13c49a46a15e48aed08a4114201e7e).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25321: 
[SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder 
(to support input_file_name with Python UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517110943
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use 
AtomicReference at InputFileBlockHolder (to support input_file_name with Python 
UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517110949
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108497/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use 
AtomicReference at InputFileBlockHolder (to support input_file_name with Python 
UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517110943
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

2019-07-31 Thread GitBox

SparkQA commented on issue #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use 
AtomicReference at InputFileBlockHolder (to support input_file_name with Python 
UDF)
URL: https://github.com/apache/spark/pull/25321#issuecomment-517110757
 
 
   **[Test build #108497 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108497/testReport)**
 for PR 25321 at commit 
[`4275f82`](https://github.com/apache/spark/commit/4275f8284d13c49a46a15e48aed08a4114201e7e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] beliefer commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array

2019-07-31 Thread GitBox

beliefer commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI 
SQL: OVERLAY function support byte array
URL: https://github.com/apache/spark/pull/25172#discussion_r309515031
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
 ##
 @@ -496,19 +509,39 @@ case class Overlay(input: Expression, replace: 
Expression, pos: Expression, len:
 this(str, replace, pos, Literal.create(-1, IntegerType))
   }
 
-  override def dataType: DataType = StringType
+  override def dataType: DataType = input.dataType
 
-  override def inputTypes: Seq[AbstractDataType] =
-Seq(StringType, StringType, IntegerType, IntegerType)
+  override def inputTypes: Seq[AbstractDataType] = 
Seq(TypeCollection(StringType, BinaryType),
+TypeCollection(StringType, BinaryType), IntegerType, IntegerType)
 
   override def children: Seq[Expression] = input :: replace :: pos :: len :: 
Nil
 
+  override def checkInputDataTypes(): TypeCheckResult = {
+val (inputType, replaceType, posType, lenType) =
+  (input.dataType, replace.dataType, pos.dataType, len.dataType)
+(inputType, replaceType) match {
+  case (StringType, StringType) | (BinaryType, BinaryType) =>
+  case (inputType, replaceType) =>
+return TypeCheckResult.TypeCheckFailure(s"Invalid data type 
${inputType.simpleString}" +
+  s" and ${replaceType.simpleString}. The result type of the input 
expression and the" +
+  "replace expression are either String or Byte Array.")
+}
+if (IntegerType.acceptsType(posType) && IntegerType.acceptsType(lenType)) {
+  TypeCheckResult.TypeCheckSuccess
+} else {
+  TypeCheckResult.TypeCheckFailure("Pos expression and len expression must 
be integer type.")
+}
+  }
+
   override def nullSafeEval(inputEval: Any, replaceEval: Any, posEval: Any, 
lenEval: Any): Any = {
-val inputStr = inputEval.asInstanceOf[UTF8String]
-val replaceStr = replaceEval.asInstanceOf[UTF8String]
 val position = posEval.asInstanceOf[Int]
 val length = lenEval.asInstanceOf[Int]
-Overlay.calculate(inputStr, replaceStr, position, length)
+input.dataType match {
+  case StringType => Overlay.calculate(inputEval.asInstanceOf[UTF8String],
+replaceEval.asInstanceOf[UTF8String], position, length)
+  case BinaryType => Overlay.calculate(inputEval.asInstanceOf[Array[Byte]],
+replaceEval.asInstanceOf[Array[Byte]], position, length)
+}
 
 Review comment:
   IMHO, it seems no difference and more code. Why we need this change?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] viirya commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template

2019-07-31 Thread GitBox

viirya commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull 
request template
URL: https://github.com/apache/spark/pull/25310#issuecomment-517108202
 
 
   Thanks.
   
   Don't have stable Internet connection right now.
   
   May look into it and provide feedback tomorrow or later.
   
   On Wed, Jul 31, 2019, 20:50 Hyukjin Kwon  wrote:
   
   > Let me cc some more active contributors who didn't put input here for
   > better visibility and to collect more feedback. cc @viirya
   >  @mgaido91 
   > @MaxGekk  @HeartSaVioR
   >  @gaborgsomogyi
   >  @dilipbiswal
   > 
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template

2019-07-31 Thread GitBox

SparkQA commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull 
request template
URL: https://github.com/apache/spark/pull/25310#issuecomment-517107671
 
 
   **[Test build #108501 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108501/testReport)**
 for PR 25310 at commit 
[`eb76d8d`](https://github.com/apache/spark/commit/eb76d8d947147b9c89f53ab609c49b06b9a66b9d).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve 
Github pull request template
URL: https://github.com/apache/spark/pull/25310#issuecomment-517107269
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13595/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25310: [SPARK-28578][INFRA] Improve 
Github pull request template
URL: https://github.com/apache/spark/pull/25310#issuecomment-517107264
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template

2019-07-31 Thread GitBox

HyukjinKwon commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull 
request template
URL: https://github.com/apache/spark/pull/25310#issuecomment-517107290
 
 
   Let me cc some more active contributors who didn't put input here for better 
visibility and to collect more feedback. cc @viirya @mgaido91 @MaxGekk 
@HeartSaVioR @gaborgsomogyi @dilipbiswal 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github 
pull request template
URL: https://github.com/apache/spark/pull/25310#issuecomment-517107269
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13595/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25310: [SPARK-28578][INFRA] Improve Github 
pull request template
URL: https://github.com/apache/spark/pull/25310#issuecomment-517107264
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array

2019-07-31 Thread GitBox

maropu commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI 
SQL: OVERLAY function support byte array
URL: https://github.com/apache/spark/pull/25172#discussion_r309512454
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala
 ##
 @@ -496,19 +509,39 @@ case class Overlay(input: Expression, replace: 
Expression, pos: Expression, len:
 this(str, replace, pos, Literal.create(-1, IntegerType))
   }
 
-  override def dataType: DataType = StringType
+  override def dataType: DataType = input.dataType
 
-  override def inputTypes: Seq[AbstractDataType] =
-Seq(StringType, StringType, IntegerType, IntegerType)
+  override def inputTypes: Seq[AbstractDataType] = 
Seq(TypeCollection(StringType, BinaryType),
+TypeCollection(StringType, BinaryType), IntegerType, IntegerType)
 
   override def children: Seq[Expression] = input :: replace :: pos :: len :: 
Nil
 
+  override def checkInputDataTypes(): TypeCheckResult = {
+val (inputType, replaceType, posType, lenType) =
+  (input.dataType, replace.dataType, pos.dataType, len.dataType)
+(inputType, replaceType) match {
+  case (StringType, StringType) | (BinaryType, BinaryType) =>
+  case (inputType, replaceType) =>
+return TypeCheckResult.TypeCheckFailure(s"Invalid data type 
${inputType.simpleString}" +
+  s" and ${replaceType.simpleString}. The result type of the input 
expression and the" +
+  "replace expression are either String or Byte Array.")
+}
+if (IntegerType.acceptsType(posType) && IntegerType.acceptsType(lenType)) {
+  TypeCheckResult.TypeCheckSuccess
+} else {
+  TypeCheckResult.TypeCheckFailure("Pos expression and len expression must 
be integer type.")
+}
+  }
+
   override def nullSafeEval(inputEval: Any, replaceEval: Any, posEval: Any, 
lenEval: Any): Any = {
-val inputStr = inputEval.asInstanceOf[UTF8String]
-val replaceStr = replaceEval.asInstanceOf[UTF8String]
 val position = posEval.asInstanceOf[Int]
 val length = lenEval.asInstanceOf[Int]
-Overlay.calculate(inputStr, replaceStr, position, length)
+input.dataType match {
+  case StringType => Overlay.calculate(inputEval.asInstanceOf[UTF8String],
+replaceEval.asInstanceOf[UTF8String], position, length)
+  case BinaryType => Overlay.calculate(inputEval.asInstanceOf[Array[Byte]],
+replaceEval.asInstanceOf[Array[Byte]], position, length)
+}
 
 Review comment:
   ```
 private lazy val replaceFunc = input.dataType match {
   case StringType =>
 (inputEval: Any, replaceEval: Any, posEval: Int, lenEval: Int) => {
   Overlay.calculate(
 inputEval.asInstanceOf[UTF8String],
 replaceEval.asInstanceOf[UTF8String],
 posEval, lenEval)
 }
   case BinaryType =>
 (inputEval: Any, replaceEval: Any, posEval: Int, lenEval: Int) => {
   Overlay.calculate(
 inputEval.asInstanceOf[Array[Byte]],
 replaceEval.asInstanceOf[Array[Byte]],
 posEval, lenEval)
 }
 }
   
 override def nullSafeEval(inputEval: Any, replaceEval: Any, posEval: Any, 
lenEval: Any): Any = {
   replaceFunc(inputEval, replaceEval, posEval.asInstanceOf[Int], 
lenEval.asInstanceOf[Int])
 }
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull request template

2019-07-31 Thread GitBox

HyukjinKwon commented on issue #25310: [SPARK-28578][INFRA] Improve Github pull 
request template
URL: https://github.com/apache/spark/pull/25310#issuecomment-517105598
 
 
   I think basically JIRA describes what issue and PR describes how it is fixed.
   
   Each focuses on each's purpose but other information can be duplicated for 
better and faster reviews. I think it's case-by-case.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array

2019-07-31 Thread GitBox

maropu commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI 
SQL: OVERLAY function support byte array
URL: https://github.com/apache/spark/pull/25172#discussion_r309511156
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
 ##
 @@ -141,6 +141,19 @@ class StringFunctionsSuite extends QueryTest with 
SharedSQLContext {
 // scalastyle:on
   }
 
+  test("binary overlay function") {
+// non ascii characters are not allowed in the code, so we disable the 
scalastyle here.
+val df = Seq((Array[Byte](1, 2, 3, 4, 5, 6, 7, 8, 9))).toDF("a")
+checkAnswer(df.select(overlay(
+  $"a", Array[Byte](-1), 6)), Row(Array[Byte](1, 2, 3, 4, 5, -1, 7, 8, 9)))
+checkAnswer(df.select(overlay(
+  $"a", Array[Byte](-1, -1, -1, -1), 7)), Row(Array[Byte](1, 2, 3, 4, 5, 
6, -1, -1, -1, -1)))
+checkAnswer(df.select(overlay(
+  $"a", Array[Byte](-1, -1), 7, 0)), Row(Array[Byte](1, 2, 3, 4, 5, 6, -1, 
-1, 7, 8, 9)))
+checkAnswer(df.select(overlay($"a", Array[Byte](-1, -1, -1, -1, -1), 2, 
4)),
+  Row(Array[Byte](1, -1, -1, -1, -1, -1, 6, 7, 8, 9)))
+  }
 
 Review comment:
   Ur, I forgot the discussion. ok, its ok to keep as it is. If necessary to 
support that, we can make a follow-up pr for that.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #25310: [SPARK-28578][INFRA] Improve Github pull request template

2019-07-31 Thread GitBox

HyukjinKwon commented on a change in pull request #25310: [SPARK-28578][INFRA] 
Improve Github pull request template
URL: https://github.com/apache/spark/pull/25310#discussion_r309511122
 
 

 ##
 File path: .github/PULL_REQUEST_TEMPLATE
 ##
 @@ -1,10 +1,31 @@
-## What changes were proposed in this pull request?

[GitHub] [spark] HyukjinKwon commented on a change in pull request #25310: [SPARK-28578][INFRA] Improve Github pull request template

2019-07-31 Thread GitBox

HyukjinKwon commented on a change in pull request #25310: [SPARK-28578][INFRA] 
Improve Github pull request template
URL: https://github.com/apache/spark/pull/25310#discussion_r309511122
 
 

 ##
 File path: .github/PULL_REQUEST_TEMPLATE
 ##
 @@ -1,10 +1,31 @@
-## What changes were proposed in this pull request?

[GitHub] [spark] WeichenXu123 commented on a change in pull request #25315: [SPARK-28582][PYSPARK] Fix pyspark daemon exit failed when receive SIGTERM on Python 3.7

2019-07-31 Thread GitBox

WeichenXu123 commented on a change in pull request #25315: 
[SPARK-28582][PYSPARK] Fix pyspark daemon exit failed when receive SIGTERM on 
Python 3.7
URL: https://github.com/apache/spark/pull/25315#discussion_r309503553
 
 

 ##
 File path: python/pyspark/daemon.py
 ##
 @@ -102,7 +102,7 @@ def shutdown(code):
 signal.signal(SIGTERM, SIG_DFL)
 # Send SIGHUP to notify workers of shutdown
 os.kill(0, SIGHUP)
-sys.exit(code)
+os._exit(code)
 
 Review comment:
   This line `os.kill(0, SIGHUP)` kill child processes by sending SIGHUP to 
them.
   so looks like calling sys.exit is the last thing to do in daemon. We have 
already done necessary clean before calling exit.
   
   So I prefer `os._exit` rather than `sys.exit` here, because `sys.exit` will 
throw `SystemExit` exception in main thread. If this `SystemExit` exception is 
swallowed by user code somewhere or in finally block run into some blocking 
code, then the kill will fail.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] xuanyuanking commented on a change in pull request #24382: [SPARK-27330][SS] support task abort in foreach writer

2019-07-31 Thread GitBox

xuanyuanking commented on a change in pull request #24382: [SPARK-27330][SS] 
support task abort in foreach writer
URL: https://github.com/apache/spark/pull/24382#discussion_r309508694
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ForeachWriterTable.scala
 ##
 @@ -141,17 +143,33 @@ class ForeachDataWriter[T](
   writer.process(rowConverter(record))
 } catch {
   case t: Throwable =>
-writer.close(t)
+closeWriter(t)
 throw t
 }
   }
 
   override def commit(): WriterCommitMessage = {
-writer.close(null)
+closeWriter(null)
 ForeachWriterCommitMessage
   }
 
-  override def abort(): Unit = {}
+  override def abort(): Unit = {
 
 Review comment:
   In this case, we need to call `writer.close` in `abort`, the exception is 
necessary for the close API. 
https://github.com/apache/spark/blob/b3ffd8be14779cbb824d14b409f0a6eab93444ba/sql/core/src/main/scala/org/apache/spark/sql/ForeachWriter.scala#L129-L141


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25294: [SPARK-28476][SQL] Support 
ALTER DATABASE SET LOCATION
URL: https://github.com/apache/spark/pull/25294#issuecomment-517100795
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108498/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25294: [SPARK-28476][SQL] Support 
ALTER DATABASE SET LOCATION
URL: https://github.com/apache/spark/pull/25294#issuecomment-517100787
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

2019-07-31 Thread GitBox

SparkQA commented on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE 
SET LOCATION
URL: https://github.com/apache/spark/pull/25294#issuecomment-517100666
 
 
   **[Test build #108498 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108498/testReport)**
 for PR 25294 at commit 
[`93d76d6`](https://github.com/apache/spark/commit/93d76d68c493074e568cae1661d0b314a1948f4a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

2019-07-31 Thread GitBox

SparkQA removed a comment on issue #25294: [SPARK-28476][SQL] Support ALTER 
DATABASE SET LOCATION
URL: https://github.com/apache/spark/pull/25294#issuecomment-517082849
 
 
   **[Test build #108498 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108498/testReport)**
 for PR 25294 at commit 
[`93d76d6`](https://github.com/apache/spark/commit/93d76d68c493074e568cae1661d0b314a1948f4a).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25294: [SPARK-28476][SQL] Support ALTER 
DATABASE SET LOCATION
URL: https://github.com/apache/spark/pull/25294#issuecomment-517100795
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108498/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25294: [SPARK-28476][SQL] Support ALTER 
DATABASE SET LOCATION
URL: https://github.com/apache/spark/pull/25294#issuecomment-517100787
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] WeichenXu123 commented on a change in pull request #25315: [SPARK-28582][PYSPARK] Fix pyspark daemon exit failed when receive SIGTERM on Python 3.7

2019-07-31 Thread GitBox

WeichenXu123 commented on a change in pull request #25315: 
[SPARK-28582][PYSPARK] Fix pyspark daemon exit failed when receive SIGTERM on 
Python 3.7
URL: https://github.com/apache/spark/pull/25315#discussion_r309503553
 
 

 ##
 File path: python/pyspark/daemon.py
 ##
 @@ -102,7 +102,7 @@ def shutdown(code):
 signal.signal(SIGTERM, SIG_DFL)
 # Send SIGHUP to notify workers of shutdown
 os.kill(0, SIGHUP)
-sys.exit(code)
+os._exit(code)
 
 Review comment:
   This line `os.kill(0, SIGHUP)` kill child processes by sending SIGHUP to 
them.
   And I prefer `os._exit` rather than `sys.exit` here, because `sys.exit` will 
throw `SystemExit` exception in main thread. If this `SystemExit` exception is 
caught by user code then the kill will fail.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] WeichenXu123 commented on a change in pull request #25315: [SPARK-28582][PYSPARK] Fix pyspark daemon exit failed when receive SIGTERM on Python 3.7

2019-07-31 Thread GitBox

WeichenXu123 commented on a change in pull request #25315: 
[SPARK-28582][PYSPARK] Fix pyspark daemon exit failed when receive SIGTERM on 
Python 3.7
URL: https://github.com/apache/spark/pull/25315#discussion_r309503553
 
 

 ##
 File path: python/pyspark/daemon.py
 ##
 @@ -102,7 +102,7 @@ def shutdown(code):
 signal.signal(SIGTERM, SIG_DFL)
 # Send SIGHUP to notify workers of shutdown
 os.kill(0, SIGHUP)
-sys.exit(code)
+os._exit(code)
 
 Review comment:
   This line `os.kill(0, SIGHUP)` kill child processes by sending SIGHUP to 
them.
   And I prefer `os._exit` rather than `sys.exit` here, because `sys.exit` will 
throw `SystemExit` exception in main thread. If this `SystemExit` exception is 
caught by user code or there're a dead loop in user code then kill will fail.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25261: [SPARK-28521][SQL] Fix error message for built-in functions

2019-07-31 Thread GitBox

SparkQA commented on issue #25261: [SPARK-28521][SQL] Fix error message for 
built-in functions
URL: https://github.com/apache/spark/pull/25261#issuecomment-517092086
 
 
   **[Test build #108500 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108500/testReport)**
 for PR 25261 at commit 
[`3c46a84`](https://github.com/apache/spark/commit/3c46a84a4a847f0eb2e68b3f978dfeb5ec61f4df).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25261: [SPARK-28521][SQL] Fix error message for built-in functions

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25261: [SPARK-28521][SQL] Fix error 
message for built-in functions
URL: https://github.com/apache/spark/pull/25261#issuecomment-517091636
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25261: [SPARK-28521][SQL] Fix error message for built-in functions

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25261: [SPARK-28521][SQL] Fix error 
message for built-in functions
URL: https://github.com/apache/spark/pull/25261#issuecomment-517091643
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13594/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25261: [SPARK-28521][SQL] Fix error message for built-in functions

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25261: [SPARK-28521][SQL] Fix error message 
for built-in functions
URL: https://github.com/apache/spark/pull/25261#issuecomment-517091636
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25261: [SPARK-28521][SQL] Fix error message for built-in functions

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25261: [SPARK-28521][SQL] Fix error message 
for built-in functions
URL: https://github.com/apache/spark/pull/25261#issuecomment-517091643
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13594/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] advancedxy commented on issue #25306: [SPARK-28573][SQL] Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table

2019-07-31 Thread GitBox

advancedxy commented on issue #25306: [SPARK-28573][SQL] Convert 
InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table
URL: https://github.com/apache/spark/pull/25306#issuecomment-517091163
 
 
   > I think we should have a config for it.
   
   I could add a config for this. But is that really necessary?  Giving users 
the option to fallback to the old behaviour? 
   We don't have a config for the read path, the write path should be the same 
eventually. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] wangyum commented on a change in pull request #25261: [SPARK-28521][SQL] Fix error message for built-in functions

2019-07-31 Thread GitBox

wangyum commented on a change in pull request #25261: [SPARK-28521][SQL] Fix 
error message for built-in functions
URL: https://github.com/apache/spark/pull/25261#discussion_r309500306
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
 ##
 @@ -588,14 +588,19 @@ object FunctionRegistry {
   val validParametersCount = constructors
 .filter(_.getParameterTypes.forall(_ == classOf[Expression]))
 .map(_.getParameterCount).distinct.sorted
-  val expectedNumberOfParameters = if (validParametersCount.length == 
1) {
-validParametersCount.head.toString
-  } else {
-validParametersCount.init.mkString("one of ", ", ", " and ") +
-  validParametersCount.last
+  val expectedErrorMsg = validParametersCount.length match {
+case 0 =>
+  ""
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25296: [SPARK-28561][WEBUI] DAG viz for barrier-execution mode

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25296: [SPARK-28561][WEBUI] DAG viz 
for barrier-execution mode
URL: https://github.com/apache/spark/pull/25296#issuecomment-517089466
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25296: [SPARK-28561][WEBUI] DAG viz for barrier-execution mode

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25296: [SPARK-28561][WEBUI] DAG viz 
for barrier-execution mode
URL: https://github.com/apache/spark/pull/25296#issuecomment-517089471
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108496/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25320: [SPARK-28532][SPARK-28530][SQL][FOLLOWUP] Inline doc for FixedPoint(1) batches "Subquery" and "Join Reorder"

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25320: 
[SPARK-28532][SPARK-28530][SQL][FOLLOWUP] Inline doc for FixedPoint(1) batches 
"Subquery" and "Join Reorder"
URL: https://github.com/apache/spark/pull/25320#issuecomment-517089323
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108494/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25320: [SPARK-28532][SPARK-28530][SQL][FOLLOWUP] Inline doc for FixedPoint(1) batches "Subquery" and "Join Reorder"

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25320: 
[SPARK-28532][SPARK-28530][SQL][FOLLOWUP] Inline doc for FixedPoint(1) batches 
"Subquery" and "Join Reorder"
URL: https://github.com/apache/spark/pull/25320#issuecomment-517089320
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25296: [SPARK-28561][WEBUI] DAG viz for barrier-execution mode

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25296: [SPARK-28561][WEBUI] DAG viz for 
barrier-execution mode
URL: https://github.com/apache/spark/pull/25296#issuecomment-517089471
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108496/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25296: [SPARK-28561][WEBUI] DAG viz for barrier-execution mode

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25296: [SPARK-28561][WEBUI] DAG viz for 
barrier-execution mode
URL: https://github.com/apache/spark/pull/25296#issuecomment-517089466
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25320: [SPARK-28532][SPARK-28530][SQL][FOLLOWUP] Inline doc for FixedPoint(1) batches "Subquery" and "Join Reorder"

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25320: 
[SPARK-28532][SPARK-28530][SQL][FOLLOWUP] Inline doc for FixedPoint(1) batches 
"Subquery" and "Join Reorder"
URL: https://github.com/apache/spark/pull/25320#issuecomment-517089323
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108494/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25320: [SPARK-28532][SPARK-28530][SQL][FOLLOWUP] Inline doc for FixedPoint(1) batches "Subquery" and "Join Reorder"

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25320: 
[SPARK-28532][SPARK-28530][SQL][FOLLOWUP] Inline doc for FixedPoint(1) batches 
"Subquery" and "Join Reorder"
URL: https://github.com/apache/spark/pull/25320#issuecomment-517089320
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #25296: [SPARK-28561][WEBUI] DAG viz for barrier-execution mode

2019-07-31 Thread GitBox

SparkQA removed a comment on issue #25296: [SPARK-28561][WEBUI] DAG viz for 
barrier-execution mode
URL: https://github.com/apache/spark/pull/25296#issuecomment-517057193
 
 
   **[Test build #108496 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108496/testReport)**
 for PR 25296 at commit 
[`1d71e4e`](https://github.com/apache/spark/commit/1d71e4e762e92863017c2ed8c8c7ca93fdcc2c34).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25296: [SPARK-28561][WEBUI] DAG viz for barrier-execution mode

2019-07-31 Thread GitBox

SparkQA commented on issue #25296: [SPARK-28561][WEBUI] DAG viz for 
barrier-execution mode
URL: https://github.com/apache/spark/pull/25296#issuecomment-517089081
 
 
   **[Test build #108496 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108496/testReport)**
 for PR 25296 at commit 
[`1d71e4e`](https://github.com/apache/spark/commit/1d71e4e762e92863017c2ed8c8c7ca93fdcc2c34).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25320: [SPARK-28532][SPARK-28530][SQL][FOLLOWUP] Inline doc for FixedPoint(1) batches "Subquery" and "Join Reorder"

2019-07-31 Thread GitBox

SparkQA commented on issue #25320: [SPARK-28532][SPARK-28530][SQL][FOLLOWUP] 
Inline doc for FixedPoint(1) batches "Subquery" and "Join Reorder"
URL: https://github.com/apache/spark/pull/25320#issuecomment-517088875
 
 
   **[Test build #108494 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108494/testReport)**
 for PR 25320 at commit 
[`b02e7b6`](https://github.com/apache/spark/commit/b02e7b6956e63b17ad6ce88576a8cb3babe28243).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25320: [SPARK-28532][SPARK-28530][SQL][FOLLOWUP] Inline doc for FixedPoint(1) batches "Subquery" and "Join Reorder"

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25320: 
[SPARK-28532][SPARK-28530][SQL][FOLLOWUP] Inline doc for FixedPoint(1) batches 
"Subquery" and "Join Reorder"
URL: https://github.com/apache/spark/pull/25320#issuecomment-517039638
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #25320: [SPARK-28532][SPARK-28530][SQL][FOLLOWUP] Inline doc for FixedPoint(1) batches "Subquery" and "Join Reorder"

2019-07-31 Thread GitBox

SparkQA removed a comment on issue #25320: 
[SPARK-28532][SPARK-28530][SQL][FOLLOWUP] Inline doc for FixedPoint(1) batches 
"Subquery" and "Join Reorder"
URL: https://github.com/apache/spark/pull/25320#issuecomment-517038352
 
 
   **[Test build #108494 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108494/testReport)**
 for PR 25320 at commit 
[`b02e7b6`](https://github.com/apache/spark/commit/b02e7b6956e63b17ad6ce88576a8cb3babe28243).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #24881: [SPARK-23160][SQL][TEST] Port window.sql

2019-07-31 Thread GitBox

maropu commented on a change in pull request #24881: [SPARK-23160][SQL][TEST] 
Port window.sql
URL: https://github.com/apache/spark/pull/24881#discussion_r309497538
 
 

 ##
 File path: sql/core/src/test/resources/sql-tests/results/pgSQL/window.sql.out
 ##
 @@ -0,0 +1,2554 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 161
+
+
+-- !query 0
+CREATE TEMPORARY VIEW tenk2 AS SELECT * FROM tenk1
+-- !query 0 schema
+struct<>
+-- !query 0 output
+
+
+
+-- !query 1
+CREATE TABLE empsalary (
+depname string,
+empno integer,
+salary int,
+enroll_date date
+) USING parquet
+-- !query 1 schema
+struct<>
+-- !query 1 output
+
+
+
+-- !query 2
+INSERT INTO empsalary VALUES
+('develop', 10, 5200, '2007-08-01'),
+('sales', 1, 5000, '2006-10-01'),
+('personnel', 5, 3500, '2007-12-10'),
+('sales', 4, 4800, '2007-08-08'),
+('personnel', 2, 3900, '2006-12-23'),
+('develop', 7, 4200, '2008-01-01'),
+('develop', 9, 4500, '2008-01-01'),
+('sales', 3, 4800, '2007-08-01'),
+('develop', 8, 6000, '2006-10-01'),
+('develop', 11, 5200, '2007-08-15')
+-- !query 2 schema
+struct<>
+-- !query 2 output
+
+
+
+-- !query 3
+SELECT depname, empno, salary, sum(salary) OVER (PARTITION BY depname) FROM 
empsalary ORDER BY depname, salary
+-- !query 3 schema
+struct
+-- !query 3 output
+develop7   420025100
+develop9   450025100
+develop10  520025100
+develop11  520025100
+develop8   600025100
+personnel  5   35007400
+personnel  2   39007400
+sales  4   480014600
+sales  3   480014600
+sales  1   500014600
+
+
+-- !query 4
+SELECT depname, empno, salary, rank() OVER (PARTITION BY depname ORDER BY 
salary) FROM empsalary
+-- !query 4 schema
+struct
+-- !query 4 output
+develop10  52003
+develop11  52003
+develop7   42001
+develop8   60005
+develop9   45002
+personnel  2   39002
+personnel  5   35001
+sales  1   50003
+sales  3   48001
+sales  4   48001
+
+
+-- !query 5
+SELECT four, ten, SUM(SUM(four)) OVER (PARTITION BY four), AVG(ten) FROM tenk1
+GROUP BY four, ten ORDER BY four, ten
+-- !query 5 schema
+struct
+-- !query 5 output
+0  0   0   0.0
+0  2   0   2.0
+0  4   0   4.0
+0  6   0   6.0
+0  8   0   8.0
+1  1   25001.0
+1  3   25003.0
+1  5   25005.0
+1  7   25007.0
+1  9   25009.0
+2  0   50000.0
+2  2   50002.0
+2  4   50004.0
+2  6   50006.0
+2  8   50008.0
+3  1   75001.0
+3  3   75003.0
+3  5   75005.0
+3  7   75007.0
+3  9   75009.0
+
+
+-- !query 6
+SELECT depname, empno, salary, sum(salary) OVER w FROM empsalary WINDOW w AS 
(PARTITION BY depname)
+-- !query 6 schema
+struct
+-- !query 6 output
+develop10  520025100
+develop11  520025100
+develop7   420025100
+develop8   600025100
+develop9   450025100
+personnel  2   39007400
+personnel  5   35007400
+sales  1   500014600
+sales  3   480014600
+sales  4   480014600
+
+
+-- !query 7
+SELECT COUNT(*) OVER () FROM tenk1 WHERE unique2 < 10
+-- !query 7 schema
+struct
+-- !query 7 output
+10
+10
+10
+10
+10
+10
+10
+10
+10
+10
+
+
+-- !query 8
+SELECT COUNT(*) OVER w FROM tenk1 WHERE unique2 < 10 WINDOW w AS ()
+-- !query 8 schema
+struct
+-- !query 8 output
+10
+10
+10
+10
+10
+10
+10
+10
+10
+10
+
+
+-- !query 9
+SELECT four FROM tenk1 WHERE FALSE WINDOW w AS (PARTITION BY ten)
+-- !query 9 schema
+struct
+-- !query 9 output
+
+
+
+-- !query 10
+SELECT sum(four) OVER (PARTITION BY ten ORDER BY unique2) AS sum_1, ten, four 
FROM tenk1 WHERE unique2 < 10
+-- !query 10 schema
+struct
+-- !query 10 output
+0  0   0
+0  0   0
+0  4   0
+1  7   1
+1  9   1
+2  0   2
+3  1   3
+3  3   3
+4  1   1
+5  1   1
+
+
+-- !query 11
+SELECT row_number() OVER (ORDER BY unique2) FROM tenk1 WHERE unique2 < 10
+-- !query 11 schema
+struct
+-- !query 11 output
+1
+10
+2
+3
+4
+5
+6
+7
+8
+9
+
+
+-- !query 12
+SELECT rank() OVER (PARTITION BY four ORDER BY ten) AS rank_1, ten, four FROM 
tenk1 WHERE unique2 < 10
+-- !query 12 schema
+struct
+-- !query 12 output
+1  0   0
+1  0   0
+1  0   2
+1  1   1
+1  1   1
+1  1   3
+2  3   3
+3  4   0
+3  7   1
+4  9   1
+
+
+-- !query 13
+SELECT dense_rank() OVER (PARTITION BY four ORDER BY ten), ten, four FROM 
tenk1 WHERE unique2 < 10
+-- !query 13 schema
+struct
+-- !query 13 output
+1  0   0
+1  0   0
+1  0

[GitHub] [spark] maropu commented on a change in pull request #24881: [SPARK-23160][SQL][TEST] Port window.sql

2019-07-31 Thread GitBox

maropu commented on a change in pull request #24881: [SPARK-23160][SQL][TEST] 
Port window.sql
URL: https://github.com/apache/spark/pull/24881#discussion_r309497069
 
 

 ##
 File path: sql/core/src/test/resources/sql-tests/results/pgSQL/window.sql.out
 ##
 @@ -0,0 +1,2554 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 161
+
+
+-- !query 0
+CREATE TEMPORARY VIEW tenk2 AS SELECT * FROM tenk1
+-- !query 0 schema
+struct<>
+-- !query 0 output
+
+
+
+-- !query 1
+CREATE TABLE empsalary (
+depname string,
+empno integer,
+salary int,
+enroll_date date
+) USING parquet
+-- !query 1 schema
+struct<>
+-- !query 1 output
+
+
+
+-- !query 2
+INSERT INTO empsalary VALUES
+('develop', 10, 5200, '2007-08-01'),
+('sales', 1, 5000, '2006-10-01'),
+('personnel', 5, 3500, '2007-12-10'),
+('sales', 4, 4800, '2007-08-08'),
+('personnel', 2, 3900, '2006-12-23'),
+('develop', 7, 4200, '2008-01-01'),
+('develop', 9, 4500, '2008-01-01'),
+('sales', 3, 4800, '2007-08-01'),
+('develop', 8, 6000, '2006-10-01'),
+('develop', 11, 5200, '2007-08-15')
+-- !query 2 schema
+struct<>
+-- !query 2 output
+
+
+
+-- !query 3
+SELECT depname, empno, salary, sum(salary) OVER (PARTITION BY depname) FROM 
empsalary ORDER BY depname, salary
+-- !query 3 schema
+struct
+-- !query 3 output
+develop7   420025100
+develop9   450025100
+develop10  520025100
+develop11  520025100
+develop8   600025100
+personnel  5   35007400
+personnel  2   39007400
+sales  4   480014600
+sales  3   480014600
+sales  1   500014600
+
+
+-- !query 4
+SELECT depname, empno, salary, rank() OVER (PARTITION BY depname ORDER BY 
salary) FROM empsalary
+-- !query 4 schema
+struct
+-- !query 4 output
+develop10  52003
+develop11  52003
+develop7   42001
+develop8   60005
+develop9   45002
+personnel  2   39002
+personnel  5   35001
+sales  1   50003
+sales  3   48001
+sales  4   48001
+
+
+-- !query 5
+SELECT four, ten, SUM(SUM(four)) OVER (PARTITION BY four), AVG(ten) FROM tenk1
+GROUP BY four, ten ORDER BY four, ten
+-- !query 5 schema
+struct
+-- !query 5 output
+0  0   0   0.0
+0  2   0   2.0
+0  4   0   4.0
+0  6   0   6.0
+0  8   0   8.0
+1  1   25001.0
+1  3   25003.0
+1  5   25005.0
+1  7   25007.0
+1  9   25009.0
+2  0   50000.0
+2  2   50002.0
+2  4   50004.0
+2  6   50006.0
+2  8   50008.0
+3  1   75001.0
+3  3   75003.0
+3  5   75005.0
+3  7   75007.0
+3  9   75009.0
+
+
+-- !query 6
+SELECT depname, empno, salary, sum(salary) OVER w FROM empsalary WINDOW w AS 
(PARTITION BY depname)
+-- !query 6 schema
+struct
+-- !query 6 output
+develop10  520025100
+develop11  520025100
+develop7   420025100
+develop8   600025100
+develop9   450025100
+personnel  2   39007400
+personnel  5   35007400
+sales  1   500014600
+sales  3   480014600
+sales  4   480014600
+
+
+-- !query 7
+SELECT COUNT(*) OVER () FROM tenk1 WHERE unique2 < 10
+-- !query 7 schema
+struct
+-- !query 7 output
+10
+10
+10
+10
+10
+10
+10
+10
+10
+10
+
+
+-- !query 8
+SELECT COUNT(*) OVER w FROM tenk1 WHERE unique2 < 10 WINDOW w AS ()
+-- !query 8 schema
+struct
+-- !query 8 output
+10
+10
+10
+10
+10
+10
+10
+10
+10
+10
+
+
+-- !query 9
+SELECT four FROM tenk1 WHERE FALSE WINDOW w AS (PARTITION BY ten)
+-- !query 9 schema
+struct
+-- !query 9 output
+
+
+
+-- !query 10
+SELECT sum(four) OVER (PARTITION BY ten ORDER BY unique2) AS sum_1, ten, four 
FROM tenk1 WHERE unique2 < 10
+-- !query 10 schema
+struct
+-- !query 10 output
+0  0   0
+0  0   0
+0  4   0
+1  7   1
+1  9   1
+2  0   2
+3  1   3
+3  3   3
+4  1   1
+5  1   1
+
+
+-- !query 11
+SELECT row_number() OVER (ORDER BY unique2) FROM tenk1 WHERE unique2 < 10
+-- !query 11 schema
+struct
+-- !query 11 output
+1
+10
+2
+3
+4
+5
+6
+7
+8
+9
+
+
+-- !query 12
+SELECT rank() OVER (PARTITION BY four ORDER BY ten) AS rank_1, ten, four FROM 
tenk1 WHERE unique2 < 10
+-- !query 12 schema
+struct
+-- !query 12 output
+1  0   0
+1  0   0
+1  0   2
+1  1   1
+1  1   1
+1  1   3
+2  3   3
+3  4   0
+3  7   1
+4  9   1
+
+
+-- !query 13
+SELECT dense_rank() OVER (PARTITION BY four ORDER BY ten), ten, four FROM 
tenk1 WHERE unique2 < 10
+-- !query 13 schema
+struct
+-- !query 13 output
+1  0   0
+1  0   0
+1  0

[GitHub] [spark] maropu commented on a change in pull request #24881: [SPARK-23160][SQL][TEST] Port window.sql

2019-07-31 Thread GitBox

maropu commented on a change in pull request #24881: [SPARK-23160][SQL][TEST] 
Port window.sql
URL: https://github.com/apache/spark/pull/24881#discussion_r309496851
 
 

 ##
 File path: sql/core/src/test/resources/sql-tests/results/pgSQL/window.sql.out
 ##
 @@ -0,0 +1,2554 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 161
+
+
+-- !query 0
+CREATE TEMPORARY VIEW tenk2 AS SELECT * FROM tenk1
+-- !query 0 schema
+struct<>
+-- !query 0 output
+
+
+
+-- !query 1
+CREATE TABLE empsalary (
+depname string,
+empno integer,
+salary int,
+enroll_date date
+) USING parquet
+-- !query 1 schema
+struct<>
+-- !query 1 output
+
+
+
+-- !query 2
+INSERT INTO empsalary VALUES
+('develop', 10, 5200, '2007-08-01'),
+('sales', 1, 5000, '2006-10-01'),
+('personnel', 5, 3500, '2007-12-10'),
+('sales', 4, 4800, '2007-08-08'),
+('personnel', 2, 3900, '2006-12-23'),
+('develop', 7, 4200, '2008-01-01'),
+('develop', 9, 4500, '2008-01-01'),
+('sales', 3, 4800, '2007-08-01'),
+('develop', 8, 6000, '2006-10-01'),
+('develop', 11, 5200, '2007-08-15')
+-- !query 2 schema
+struct<>
+-- !query 2 output
+
+
+
+-- !query 3
+SELECT depname, empno, salary, sum(salary) OVER (PARTITION BY depname) FROM 
empsalary ORDER BY depname, salary
+-- !query 3 schema
+struct
+-- !query 3 output
+develop7   420025100
+develop9   450025100
+develop10  520025100
+develop11  520025100
+develop8   600025100
+personnel  5   35007400
+personnel  2   39007400
+sales  4   480014600
+sales  3   480014600
+sales  1   500014600
+
+
+-- !query 4
+SELECT depname, empno, salary, rank() OVER (PARTITION BY depname ORDER BY 
salary) FROM empsalary
+-- !query 4 schema
+struct
+-- !query 4 output
+develop10  52003
+develop11  52003
+develop7   42001
+develop8   60005
+develop9   45002
+personnel  2   39002
+personnel  5   35001
+sales  1   50003
+sales  3   48001
+sales  4   48001
+
+
+-- !query 5
+SELECT four, ten, SUM(SUM(four)) OVER (PARTITION BY four), AVG(ten) FROM tenk1
+GROUP BY four, ten ORDER BY four, ten
+-- !query 5 schema
+struct
+-- !query 5 output
+0  0   0   0.0
+0  2   0   2.0
+0  4   0   4.0
+0  6   0   6.0
+0  8   0   8.0
+1  1   25001.0
+1  3   25003.0
+1  5   25005.0
+1  7   25007.0
+1  9   25009.0
+2  0   50000.0
+2  2   50002.0
+2  4   50004.0
+2  6   50006.0
+2  8   50008.0
+3  1   75001.0
+3  3   75003.0
+3  5   75005.0
+3  7   75007.0
+3  9   75009.0
+
+
+-- !query 6
+SELECT depname, empno, salary, sum(salary) OVER w FROM empsalary WINDOW w AS 
(PARTITION BY depname)
+-- !query 6 schema
+struct
+-- !query 6 output
+develop10  520025100
+develop11  520025100
+develop7   420025100
+develop8   600025100
+develop9   450025100
+personnel  2   39007400
+personnel  5   35007400
+sales  1   500014600
+sales  3   480014600
+sales  4   480014600
+
+
+-- !query 7
+SELECT COUNT(*) OVER () FROM tenk1 WHERE unique2 < 10
+-- !query 7 schema
+struct
+-- !query 7 output
+10
+10
+10
+10
+10
+10
+10
+10
+10
+10
+
+
+-- !query 8
+SELECT COUNT(*) OVER w FROM tenk1 WHERE unique2 < 10 WINDOW w AS ()
+-- !query 8 schema
+struct
+-- !query 8 output
+10
+10
+10
+10
+10
+10
+10
+10
+10
+10
+
+
+-- !query 9
+SELECT four FROM tenk1 WHERE FALSE WINDOW w AS (PARTITION BY ten)
+-- !query 9 schema
+struct
+-- !query 9 output
+
+
+
+-- !query 10
+SELECT sum(four) OVER (PARTITION BY ten ORDER BY unique2) AS sum_1, ten, four 
FROM tenk1 WHERE unique2 < 10
+-- !query 10 schema
+struct
+-- !query 10 output
+0  0   0
+0  0   0
+0  4   0
+1  7   1
+1  9   1
+2  0   2
+3  1   3
+3  3   3
+4  1   1
+5  1   1
+
+
+-- !query 11
+SELECT row_number() OVER (ORDER BY unique2) FROM tenk1 WHERE unique2 < 10
+-- !query 11 schema
+struct
+-- !query 11 output
+1
+10
+2
+3
+4
+5
+6
+7
+8
+9
+
+
+-- !query 12
+SELECT rank() OVER (PARTITION BY four ORDER BY ten) AS rank_1, ten, four FROM 
tenk1 WHERE unique2 < 10
+-- !query 12 schema
+struct
+-- !query 12 output
+1  0   0
+1  0   0
+1  0   2
+1  1   1
+1  1   1
+1  1   3
+2  3   3
+3  4   0
+3  7   1
+4  9   1
+
+
+-- !query 13
+SELECT dense_rank() OVER (PARTITION BY four ORDER BY ten), ten, four FROM 
tenk1 WHERE unique2 < 10
+-- !query 13 schema
+struct
+-- !query 13 output
+1  0   0
+1  0   0
+1  0

[GitHub] [spark] beliefer commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI SQL: OVERLAY function support byte array

2019-07-31 Thread GitBox

beliefer commented on a change in pull request #25172: [SPARK-28412][SQL] ANSI 
SQL: OVERLAY function support byte array
URL: https://github.com/apache/spark/pull/25172#discussion_r309496562
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
 ##
 @@ -141,6 +141,19 @@ class StringFunctionsSuite extends QueryTest with 
SharedSQLContext {
 // scalastyle:on
   }
 
+  test("binary overlay function") {
+// non ascii characters are not allowed in the code, so we disable the 
scalastyle here.
+val df = Seq((Array[Byte](1, 2, 3, 4, 5, 6, 7, 8, 9))).toDF("a")
+checkAnswer(df.select(overlay(
+  $"a", Array[Byte](-1), 6)), Row(Array[Byte](1, 2, 3, 4, 5, -1, 7, 8, 9)))
+checkAnswer(df.select(overlay(
+  $"a", Array[Byte](-1, -1, -1, -1), 7)), Row(Array[Byte](1, 2, 3, 4, 5, 
6, -1, -1, -1, -1)))
+checkAnswer(df.select(overlay(
+  $"a", Array[Byte](-1, -1), 7, 0)), Row(Array[Byte](1, 2, 3, 4, 5, 6, -1, 
-1, 7, 8, 9)))
+checkAnswer(df.select(overlay($"a", Array[Byte](-1, -1, -1, -1, -1), 2, 
4)),
+  Row(Array[Byte](1, -1, -1, -1, -1, -1, 6, 7, 8, 9)))
+  }
 
 Review comment:
   Are we determined to support `overlay(string, binary)` and `overlay(binary, 
string)`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #24881: [SPARK-23160][SQL][TEST] Port window.sql

2019-07-31 Thread GitBox

maropu commented on a change in pull request #24881: [SPARK-23160][SQL][TEST] 
Port window.sql
URL: https://github.com/apache/spark/pull/24881#discussion_r309495722
 
 

 ##
 File path: sql/core/src/test/resources/sql-tests/inputs/pgSQL/window.sql
 ##
 @@ -0,0 +1,1169 @@
+-- Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+--
+-- Window Functions Testing
+-- 
https://github.com/postgres/postgres/blob/REL_12_BETA2/src/test/regress/sql/window.sql
+
+CREATE TEMPORARY VIEW tenk2 AS SELECT * FROM tenk1;
+
+CREATE TABLE empsalary (
+depname string,
+empno integer,
+salary int,
+enroll_date date
+) USING parquet;
+
+INSERT INTO empsalary VALUES
+('develop', 10, 5200, '2007-08-01'),
+('sales', 1, 5000, '2006-10-01'),
+('personnel', 5, 3500, '2007-12-10'),
+('sales', 4, 4800, '2007-08-08'),
+('personnel', 2, 3900, '2006-12-23'),
+('develop', 7, 4200, '2008-01-01'),
+('develop', 9, 4500, '2008-01-01'),
+('sales', 3, 4800, '2007-08-01'),
+('develop', 8, 6000, '2006-10-01'),
+('develop', 11, 5200, '2007-08-15');
+
+SELECT depname, empno, salary, sum(salary) OVER (PARTITION BY depname) FROM 
empsalary ORDER BY depname, salary;
+
+SELECT depname, empno, salary, rank() OVER (PARTITION BY depname ORDER BY 
salary) FROM empsalary;
+
+-- with GROUP BY
+SELECT four, ten, SUM(SUM(four)) OVER (PARTITION BY four), AVG(ten) FROM tenk1
+GROUP BY four, ten ORDER BY four, ten;
+
+SELECT depname, empno, salary, sum(salary) OVER w FROM empsalary WINDOW w AS 
(PARTITION BY depname);
+
+-- [SPARK-28064] Order by does not accept a call to rank()
+-- SELECT depname, empno, salary, rank() OVER w FROM empsalary WINDOW w AS 
(PARTITION BY depname ORDER BY salary) ORDER BY rank() OVER w;
+
+-- empty window specification
+SELECT COUNT(*) OVER () FROM tenk1 WHERE unique2 < 10;
+
+SELECT COUNT(*) OVER w FROM tenk1 WHERE unique2 < 10 WINDOW w AS ();
+
+-- no window operation
+SELECT four FROM tenk1 WHERE FALSE WINDOW w AS (PARTITION BY ten);
+
+-- cumulative aggregate
+SELECT sum(four) OVER (PARTITION BY ten ORDER BY unique2) AS sum_1, ten, four 
FROM tenk1 WHERE unique2 < 10;
+
+SELECT row_number() OVER (ORDER BY unique2) FROM tenk1 WHERE unique2 < 10;
+
+SELECT rank() OVER (PARTITION BY four ORDER BY ten) AS rank_1, ten, four FROM 
tenk1 WHERE unique2 < 10;
+
+SELECT dense_rank() OVER (PARTITION BY four ORDER BY ten), ten, four FROM 
tenk1 WHERE unique2 < 10;
+
+SELECT percent_rank() OVER (PARTITION BY four ORDER BY ten), ten, four FROM 
tenk1 WHERE unique2 < 10;
+
+SELECT cume_dist() OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 
WHERE unique2 < 10;
+
+SELECT ntile(3) OVER (ORDER BY ten, four), ten, four FROM tenk1 WHERE unique2 
< 10;
+
+-- [SPARK-28065] ntile does not accept NULL as input
+-- SELECT ntile(NULL) OVER (ORDER BY ten, four), ten, four FROM tenk1 LIMIT 2;
+
+SELECT lag(ten) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 
WHERE unique2 < 10;
+
+-- [SPARK-28068] `lag` second argument must be a literal in Spark
+-- SELECT lag(ten, four) OVER (PARTITION BY four ORDER BY ten), ten, four FROM 
tenk1 WHERE unique2 < 10;
+-- SELECT lag(ten, four, 0) OVER (PARTITION BY four ORDER BY ten), ten, four 
FROM tenk1 WHERE unique2 < 10;
+
+SELECT lead(ten) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 
WHERE unique2 < 10;
+
+SELECT lead(ten * 2, 1) OVER (PARTITION BY four ORDER BY ten), ten, four FROM 
tenk1 WHERE unique2 < 10;
+
+SELECT lead(ten * 2, 1, -1) OVER (PARTITION BY four ORDER BY ten), ten, four 
FROM tenk1 WHERE unique2 < 10;
+
+SELECT first(ten) OVER (PARTITION BY four ORDER BY ten), ten, four FROM tenk1 
WHERE unique2 < 10;
+
+-- last returns the last row of the frame, which is CURRENT ROW in ORDER BY 
window.
+SELECT last(four) OVER (ORDER BY ten), ten, four FROM tenk1 WHERE unique2 < 10;
+
+SELECT last(ten) OVER (PARTITION BY four), ten, four FROM
+(SELECT * FROM tenk1 WHERE unique2 < 10 ORDER BY four, ten)s
+ORDER BY four, ten;
+
+-- [SPARK-27951] ANSI SQL: NTH_VALUE function
+-- SELECT nth_value(ten, four + 1) OVER (PARTITION BY four), ten, four
+-- FROM (SELECT * FROM tenk1 WHERE unique2 < 10 ORDER BY four, ten)s;
+
+SELECT ten, two, sum(hundred) AS gsum, sum(sum(hundred)) OVER (PARTITION BY 
two ORDER BY ten) AS wsum
+FROM tenk1 GROUP BY ten, two;
+
+SELECT count(*) OVER (PARTITION BY four), four FROM (SELECT * FROM tenk1 WHERE 
two = 1)s WHERE unique2 < 10;
+
+SELECT (count(*) OVER (PARTITION BY four ORDER BY ten) +
+  sum(hundred) OVER (PARTITION BY four ORDER BY ten)) AS cntsum
+  FROM tenk1 WHERE unique2 < 10;
+
+-- opexpr with different windows evaluation.
+SELECT * FROM(
+  SELECT count(*) OVER (PARTITION BY four ORDER BY ten) +
+sum(hundred) OVER (PARTITION BY two ORDER BY ten) AS total,
+count(*) OVER (PARTITION BY four ORDER BY ten) AS fourcount,
+sum(hundred) OVER (PARTITION BY two ORDER BY ten) AS twosum
+FROM tenk1
+)sub WHERE total <> fourcount + twosum;
+
+SELECT avg(four) OVER (PARTITION BY four

[GitHub] [spark] maropu commented on issue #25287: [SPARK-28552][SQL]Identification of different dialects insensitive to case by JDBC URL prefix

2019-07-31 Thread GitBox

maropu commented on issue #25287: [SPARK-28552][SQL]Identification of different 
dialects insensitive to case by JDBC URL prefix
URL: https://github.com/apache/spark/pull/25287#issuecomment-517084252
 
 
   For example,
   ```
   @DeveloperApi
   @Evolving
   abstract class JdbcDialect extends Serializable {
   
 def urlName: String
   
 /**
  * Check if this dialect instance can handle a certain jdbc url.
  * @param url the jdbc url.
  * @return True if the dialect can be applied on the given jdbc url.
  * @throws NullPointerException if the url is null.
  */
 def canHandle(url : String): Boolean = {
   url.toLowerCase(Locale.ROOT).startsWith(s"jdbc:$urlName")
 }
   ...
   }
   
   private object PostgresDialect extends JdbcDialect {
 override val urlName: String = "postgresql"
   ...
   }
   ```
   ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

2019-07-31 Thread GitBox

SparkQA commented on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE 
SET LOCATION
URL: https://github.com/apache/spark/pull/25294#issuecomment-517082849
 
 
   **[Test build #108498 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108498/testReport)**
 for PR 25294 at commit 
[`93d76d6`](https://github.com/apache/spark/commit/93d76d68c493074e568cae1661d0b314a1948f4a).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

2019-07-31 Thread GitBox

SparkQA commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map 
PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517082869
 
 
   **[Test build #108499 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/108499/testReport)**
 for PR 25262 at commit 
[`5cdbab3`](https://github.com/apache/spark/commit/5cdbab3659edd219bff462ce4d004724cff68e10).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25294: [SPARK-28476][SQL] Support 
ALTER DATABASE SET LOCATION
URL: https://github.com/apache/spark/pull/25294#issuecomment-517082398
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13592/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25262: [SPARK-28486][CORE][PYTHON] 
Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517082383
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13593/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25262: [SPARK-28486][CORE][PYTHON] 
Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517082380
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

2019-07-31 Thread GitBox

AmplabJenkins removed a comment on issue #25294: [SPARK-28476][SQL] Support 
ALTER DATABASE SET LOCATION
URL: https://github.com/apache/spark/pull/25294#issuecomment-517082393
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25294: [SPARK-28476][SQL] Support ALTER 
DATABASE SET LOCATION
URL: https://github.com/apache/spark/pull/25294#issuecomment-517082398
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13592/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map 
PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517082383
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/13593/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25294: [SPARK-28476][SQL] Support ALTER DATABASE SET LOCATION

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25294: [SPARK-28476][SQL] Support ALTER 
DATABASE SET LOCATION
URL: https://github.com/apache/spark/pull/25294#issuecomment-517082393
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

2019-07-31 Thread GitBox

AmplabJenkins commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map 
PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517082380
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] huangtianhua commented on issue #25279: [WIP][SPARK-28519][SQL] Use StrictMath log, pow functions for platform independence

2019-07-31 Thread GitBox

huangtianhua commented on issue #25279: [WIP][SPARK-28519][SQL] Use StrictMath 
log, pow functions for platform independence
URL: https://github.com/apache/spark/pull/25279#issuecomment-517081503
 
 
   @srowen Thanks a lot. And a good news:) all tests passed on x86_64 and 
aarch64 based on your pr(only two tests now we increase the timeout, later we 
will tests on larger server), see: 
https://logs.openlabtesting.org/logs/6/6/947ddad683ad7a2e0a0cc4c2310e352ace21a86f/check/spark-build-arm64/8e39061/
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC

2019-07-31 Thread GitBox

HyukjinKwon commented on issue #25262: [SPARK-28486][CORE][PYTHON] Map 
PythonBroadcast's data file to a BroadcastBlock to avoid delete by GC
URL: https://github.com/apache/spark/pull/25262#issuecomment-517081265
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] jzhuge commented on issue #24623: [SPARK-27739][SQL] df.persist should save stats from optimized plan

2019-07-31 Thread GitBox

jzhuge commented on issue #24623: [SPARK-27739][SQL] df.persist should save 
stats from optimized plan
URL: https://github.com/apache/spark/pull/24623#issuecomment-517078978
 
 
   Like the idea @maropu 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25321: [SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

2019-07-31 Thread GitBox

dongjoon-hyun commented on a change in pull request #25321: 
[SPARK-28153][PYTHON][BRANCH-2.4] Use AtomicReference at InputFileBlockHolder 
(to support input_file_name with Python UDF)
URL: https://github.com/apache/spark/pull/25321#discussion_r309488096
 
 

 ##
 File path: python/pyspark/sql/tests.py
 ##
 @@ -847,6 +847,14 @@ def test_input_file_name_reset_for_rdd(self):
 for result in results:
 self.assertEqual(result[0], '')
 
+def test_input_file_name_udf(self):
 
 Review comment:
   Got it~


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #25315: [SPARK-28582][PYSPARK] Fix pyspark daemon exit failed when receive SIGTERM on Python 3.7

2019-07-31 Thread GitBox

HyukjinKwon commented on a change in pull request #25315: 
[SPARK-28582][PYSPARK] Fix pyspark daemon exit failed when receive SIGTERM on 
Python 3.7
URL: https://github.com/apache/spark/pull/25315#discussion_r309484449
 
 

 ##
 File path: python/pyspark/daemon.py
 ##
 @@ -102,7 +102,7 @@ def shutdown(code):
 signal.signal(SIGTERM, SIG_DFL)
 # Send SIGHUP to notify workers of shutdown
 os.kill(0, SIGHUP)
-sys.exit(code)
+os._exit(code)
 
 Review comment:
   If this is really a bug in Python 3.7, let's file a bug in 
https://bugs.python.org/ , confirm it was a bug , and use this workaround only 
for Python 3.7 conditionally. Ideally, we should prepare a minimal reproducer 
and share there.
   
   Lastly, is this test-only issue or actual problem with Python 3.7 (daemon 
and/or workers are not being dead)? If so, this issue is a critical issue that 
leaks resources.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

1 2 3 4 5 6 7 8 9 10 >

1 - 100 of 1003 matches

Mail list logo