[GitHub] [spark] wangyum commented on a change in pull request #22502: [SPARK-25474][SQL] Support `spark.sql.statistics.fallBackToHdfs` in data source tables

2019-08-14 Thread GitBox
wangyum commented on a change in pull request #22502: [SPARK-25474][SQL] 
Support `spark.sql.statistics.fallBackToHdfs` in data source tables
URL: https://github.com/apache/spark/pull/22502#discussion_r314180567
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFsRelation.scala
 ##
 @@ -71,7 +70,13 @@ case class HadoopFsRelation(
 
   override def sizeInBytes: Long = {
 val compressionFactor = sqlContext.conf.fileCompressionFactor
-(location.sizeInBytes * compressionFactor).toLong
+val defaultSize = (location.sizeInBytes * compressionFactor).toLong
+location match {
+  case cfi: CatalogFileIndex if 
sparkSession.sessionState.conf.fallBackToHdfsForStatsEnabled =>
 
 Review comment:
   https://github.com/apache/spark/pull/24715


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] cloud-fan commented on a change in pull request #22502: [SPARK-25474][SQL] Support `spark.sql.statistics.fallBackToHdfs` in data source tables

2019-08-14 Thread GitBox
cloud-fan commented on a change in pull request #22502: [SPARK-25474][SQL] 
Support `spark.sql.statistics.fallBackToHdfs` in data source tables
URL: https://github.com/apache/spark/pull/22502#discussion_r314180114
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFsRelation.scala
 ##
 @@ -71,7 +70,13 @@ case class HadoopFsRelation(
 
   override def sizeInBytes: Long = {
 val compressionFactor = sqlContext.conf.fileCompressionFactor
-(location.sizeInBytes * compressionFactor).toLong
+val defaultSize = (location.sizeInBytes * compressionFactor).toLong
+location match {
+  case cfi: CatalogFileIndex if 
sparkSession.sessionState.conf.fallBackToHdfsForStatsEnabled =>
 
 Review comment:
   @wangyum can you send a PR for your proposal? It's unclear to me what you 
are proposing here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] younggyuchun commented on a change in pull request #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type.

2019-08-14 Thread GitBox
younggyuchun commented on a change in pull request #25458: [SPARK-27931][SQL] 
Accept 'on' and 'off' as input and trim input for the boolean data type.
URL: https://github.com/apache/spark/pull/25458#discussion_r314179583
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala
 ##
 @@ -65,12 +65,15 @@ object StringUtils extends Logging {
 "(?s)" + out.result() // (?s) enables dotall mode, causing "." to match 
new lines
   }
 
-  private[this] val trueStrings = Set("t", "true", "y", "yes", 
"1").map(UTF8String.fromString)
-  private[this] val falseStrings = Set("f", "false", "n", "no", 
"0").map(UTF8String.fromString)
+  private[this] val trueStrings =
+Set("t", "true", "y", "yes", "1", "on").map(UTF8String.fromString)
+
+  private[this] val falseStrings =
+Set("f", "false", "n", "no", "0", "off").map(UTF8String.fromString)
 
 Review comment:
   Ah okay. Let me add that too. Thank you


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] wangyum commented on a change in pull request #22502: [SPARK-25474][SQL] Support `spark.sql.statistics.fallBackToHdfs` in data source tables

2019-08-14 Thread GitBox
wangyum commented on a change in pull request #22502: [SPARK-25474][SQL] 
Support `spark.sql.statistics.fallBackToHdfs` in data source tables
URL: https://github.com/apache/spark/pull/22502#discussion_r314179379
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/HadoopFsRelation.scala
 ##
 @@ -71,7 +70,13 @@ case class HadoopFsRelation(
 
   override def sizeInBytes: Long = {
 val compressionFactor = sqlContext.conf.fileCompressionFactor
-(location.sizeInBytes * compressionFactor).toLong
+val defaultSize = (location.sizeInBytes * compressionFactor).toLong
+location match {
+  case cfi: CatalogFileIndex if 
sparkSession.sessionState.conf.fallBackToHdfsForStatsEnabled =>
 
 Review comment:
   Yes. I have prepared some tests to illustrate this issue. These tests can be 
passed before this commit:
   ```scala
 test("Non-partitioned data source table") {
   withTempDir { dir =>
 withTable("spark_25474") {
   sql(s"CREATE TABLE spark_25474 (c1 BIGINT) USING PARQUET LOCATION 
'${dir.toURI}'")
   
spark.range(5).write.mode(SaveMode.Overwrite).parquet(dir.getCanonicalPath)
   
   assert(getCatalogTable("spark_25474").stats.isEmpty)
   val relation = 
spark.table("spark_25474").queryExecution.analyzed.children.head
   assert(relation.stats.sizeInBytes === 935)
 }
   }
 }
   
 test("Partitioned data source table default") {
   withTempDir { dir =>
 withTable("spark_25474") {
   spark.sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
 s"PARTITIONED BY(a) LOCATION '${dir.toURI}'")
   spark.sql("INSERT INTO TABLE spark_25474 PARTITION(a=1) SELECT 2")
   
   assert(getCatalogTable("spark_25474").stats.isEmpty)
   val relation = 
spark.table("spark_25474").queryExecution.analyzed.children.head
   // scalastyle:off line.size.limit
   // It's 8.0EB in this case. This 8.0EB from:
   // 
https://github.com/apache/spark/blob/c30b5297bc607ae33cc2fcf624b127942154e559/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala#L383-L387
   // scalastyle:on line.size.limit
   assert(relation.stats.sizeInBytes === conf.defaultSizeInBytes)
 }
   }
 }
   
 test("Partitioned data source table and disable 
HIVE_MANAGE_FILESOURCE_PARTITIONS") {
   withSQLConf(SQLConf.HIVE_MANAGE_FILESOURCE_PARTITIONS.key -> "false") {
 withTempDir { dir =>
   withTable("spark_25474") {
 spark.sql("CREATE TABLE spark_25474(a int, b int) USING parquet " +
   s"PARTITIONED BY(a) LOCATION '${dir.toURI}'")
 spark.sql("INSERT INTO TABLE spark_25474 PARTITION(a=1) SELECT 2")
   
 assert(getCatalogTable("spark_25474").stats.isEmpty)
 val relation = 
spark.table("spark_25474").queryExecution.analyzed.children.head
 assert(relation.stats.sizeInBytes === 418)
   }
 }
   }
 }
   ```
   
   
https://github.com/apache/spark/compare/master...wangyum:SPARK-25474-DEV?expand=1
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #24440: [SPARK-27545] [SQL] Uncache table needs to delete the temporary view …

2019-08-14 Thread GitBox
SparkQA commented on issue #24440: [SPARK-27545] [SQL] Uncache table needs to 
delete the temporary view …
URL: https://github.com/apache/spark/pull/24440#issuecomment-521519898
 
 
   **[Test build #109143 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109143/testReport)**
 for PR 24440 at commit 
[`770ee42`](https://github.com/apache/spark/commit/770ee4261335635fafe79afebb1ce7302db96d92).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24440: [SPARK-27545] [SQL] Uncache table needs to delete the temporary view …

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #24440: [SPARK-27545] [SQL] Uncache 
table needs to delete the temporary view …
URL: https://github.com/apache/spark/pull/24440#issuecomment-521519493
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #24440: [SPARK-27545] [SQL] Uncache table needs to delete the temporary view …

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #24440: [SPARK-27545] [SQL] Uncache 
table needs to delete the temporary view …
URL: https://github.com/apache/spark/pull/24440#issuecomment-521519498
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14212/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24440: [SPARK-27545] [SQL] Uncache table needs to delete the temporary view …

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #24440: [SPARK-27545] [SQL] Uncache table 
needs to delete the temporary view …
URL: https://github.com/apache/spark/pull/24440#issuecomment-521519498
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14212/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #24440: [SPARK-27545] [SQL] Uncache table needs to delete the temporary view …

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #24440: [SPARK-27545] [SQL] Uncache table 
needs to delete the temporary view …
URL: https://github.com/apache/spark/pull/24440#issuecomment-521519493
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] httfighter commented on issue #24440: [SPARK-27545] [SQL] Uncache table needs to delete the temporary view …

2019-08-14 Thread GitBox
httfighter commented on issue #24440: [SPARK-27545] [SQL] Uncache table needs 
to delete the temporary view …
URL: https://github.com/apache/spark/pull/24440#issuecomment-521518729
 
 
   @dongjoon-hyun Thank you for reminding.I have added a test case.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25456: [SPARK-28739][SQL] Add a simple cost check for Adaptive Query Execution

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25456: [SPARK-28739][SQL] Add a 
simple cost check for Adaptive Query Execution
URL: https://github.com/apache/spark/pull/25456#issuecomment-521517932
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14211/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25456: [SPARK-28739][SQL] Add a simple cost check for Adaptive Query Execution

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25456: [SPARK-28739][SQL] Add a simple cost 
check for Adaptive Query Execution
URL: https://github.com/apache/spark/pull/25456#issuecomment-521517930
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25456: [SPARK-28739][SQL] Add a simple cost check for Adaptive Query Execution

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25456: [SPARK-28739][SQL] Add a 
simple cost check for Adaptive Query Execution
URL: https://github.com/apache/spark/pull/25456#issuecomment-521517930
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25456: [SPARK-28739][SQL] Add a simple cost check for Adaptive Query Execution

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25456: [SPARK-28739][SQL] Add a simple cost 
check for Adaptive Query Execution
URL: https://github.com/apache/spark/pull/25456#issuecomment-521517932
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14211/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25459: [SPARK-28734[DOC] Initial table of 
content in the left hand side bar for SQL doc
URL: https://github.com/apache/spark/pull/25459#issuecomment-521517572
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109141/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-14 Thread GitBox
SparkQA commented on issue #25459: [SPARK-28734[DOC] Initial table of content 
in the left hand side bar for SQL doc
URL: https://github.com/apache/spark/pull/25459#issuecomment-521517517
 
 
   **[Test build #109141 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109141/testReport)**
 for PR 25459 at commit 
[`791ee67`](https://github.com/apache/spark/commit/791ee67a26230d44b6839e4d414980d9889cea74).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25459: [SPARK-28734[DOC] Initial 
table of content in the left hand side bar for SQL doc
URL: https://github.com/apache/spark/pull/25459#issuecomment-521517570
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-14 Thread GitBox
SparkQA removed a comment on issue #25459: [SPARK-28734[DOC] Initial table of 
content in the left hand side bar for SQL doc
URL: https://github.com/apache/spark/pull/25459#issuecomment-521515799
 
 
   **[Test build #109141 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109141/testReport)**
 for PR 25459 at commit 
[`791ee67`](https://github.com/apache/spark/commit/791ee67a26230d44b6839e4d414980d9889cea74).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25459: [SPARK-28734[DOC] Initial table of 
content in the left hand side bar for SQL doc
URL: https://github.com/apache/spark/pull/25459#issuecomment-521517570
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25459: [SPARK-28734[DOC] Initial 
table of content in the left hand side bar for SQL doc
URL: https://github.com/apache/spark/pull/25459#issuecomment-521517572
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109141/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25456: [SPARK-28739][SQL] Add a simple cost check for Adaptive Query Execution

2019-08-14 Thread GitBox
SparkQA commented on issue #25456: [SPARK-28739][SQL] Add a simple cost check 
for Adaptive Query Execution
URL: https://github.com/apache/spark/pull/25456#issuecomment-521516999
 
 
   **[Test build #109142 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109142/testReport)**
 for PR 25456 at commit 
[`74dd386`](https://github.com/apache/spark/commit/74dd3865e0fe3287d73a7b6aa954cc63bf17e9fd).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25459: [SPARK-28734[DOC] Initial 
table of content in the left hand side bar for SQL doc
URL: https://github.com/apache/spark/pull/25459#issuecomment-521516655
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14210/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25459: [SPARK-28734[DOC] Initial 
table of content in the left hand side bar for SQL doc
URL: https://github.com/apache/spark/pull/25459#issuecomment-521516649
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25459: [SPARK-28734[DOC] Initial table of 
content in the left hand side bar for SQL doc
URL: https://github.com/apache/spark/pull/25459#issuecomment-521516649
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25459: [SPARK-28734[DOC] Initial table of 
content in the left hand side bar for SQL doc
URL: https://github.com/apache/spark/pull/25459#issuecomment-521516655
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14210/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] wangyum commented on a change in pull request #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type.

2019-08-14 Thread GitBox
wangyum commented on a change in pull request #25458: [SPARK-27931][SQL] Accept 
'on' and 'off' as input and trim input for the boolean data type.
URL: https://github.com/apache/spark/pull/25458#discussion_r314176109
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala
 ##
 @@ -65,12 +65,15 @@ object StringUtils extends Logging {
 "(?s)" + out.result() // (?s) enables dotall mode, causing "." to match 
new lines
   }
 
-  private[this] val trueStrings = Set("t", "true", "y", "yes", 
"1").map(UTF8String.fromString)
-  private[this] val falseStrings = Set("f", "false", "n", "no", 
"0").map(UTF8String.fromString)
+  private[this] val trueStrings =
+Set("t", "true", "y", "yes", "1", "on").map(UTF8String.fromString)
+
+  private[this] val falseStrings =
+Set("f", "false", "n", "no", "0", "off").map(UTF8String.fromString)
 
 Review comment:
   But PostgreSQL also accepts`of`, `tru`, `fals`, ...:
   ```sql
   postgres=# select cast('of' as boolean), cast('tru' as boolean), cast('fals' 
as boolean);
bool | bool | bool
   --+--+--
f| t| f
   (1 row)
   ```
   
   
https://github.com/postgres/postgres/commit/9729c9360886bee7feddc6a1124b0742de4b9f3d


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25459: [SPARK-28734[DOC] Initial 
table of content in the left hand side bar for SQL doc
URL: https://github.com/apache/spark/pull/25459#issuecomment-521515466
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25459: [SPARK-28734[DOC] Initial 
table of content in the left hand side bar for SQL doc
URL: https://github.com/apache/spark/pull/25459#issuecomment-521515467
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14209/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-14 Thread GitBox
SparkQA commented on issue #25459: [SPARK-28734[DOC] Initial table of content 
in the left hand side bar for SQL doc
URL: https://github.com/apache/spark/pull/25459#issuecomment-521515799
 
 
   **[Test build #109141 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109141/testReport)**
 for PR 25459 at commit 
[`791ee67`](https://github.com/apache/spark/commit/791ee67a26230d44b6839e4d414980d9889cea74).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25459: [SPARK-28734[DOC] Initial table of 
content in the left hand side bar for SQL doc
URL: https://github.com/apache/spark/pull/25459#issuecomment-521515466
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25459: [SPARK-28734[DOC] Initial table of 
content in the left hand side bar for SQL doc
URL: https://github.com/apache/spark/pull/25459#issuecomment-521515467
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14209/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dilipbiswal opened a new pull request #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-14 Thread GitBox
dilipbiswal opened a new pull request #25459: [SPARK-28734[DOC] Initial table 
of content in the left hand side bar for SQL doc
URL: https://github.com/apache/spark/pull/25459
 
 
   ## What changes were proposed in this pull request?
   This is a initial PR that creates the table of content for SQL reference 
guide. The left side bar will displays additional menu items corresponding to 
supported SQL constructs. One this PR is merged, we will fill in the content 
incrementally.  Additionally this PR contains a minor change to make the left 
sidebar scrollable. Currently it is not possible to scroll in the left hand 
side window.
   
   ## How was this patch tested?
   Used jekyll build and serve to verify.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dilipbiswal commented on issue #25459: [SPARK-28734[DOC] Initial table of content in the left hand side bar for SQL doc

2019-08-14 Thread GitBox
dilipbiswal commented on issue #25459: [SPARK-28734[DOC] Initial table of 
content in the left hand side bar for SQL doc
URL: https://github.com/apache/spark/pull/25459#issuecomment-521514797
 
 
   cc @gatorsmile 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] younggyuchun commented on a change in pull request #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type.

2019-08-14 Thread GitBox
younggyuchun commented on a change in pull request #25458: [SPARK-27931][SQL] 
Accept 'on' and 'off' as input and trim input for the boolean data type.
URL: https://github.com/apache/spark/pull/25458#discussion_r314174879
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala
 ##
 @@ -65,12 +65,15 @@ object StringUtils extends Logging {
 "(?s)" + out.result() // (?s) enables dotall mode, causing "." to match 
new lines
   }
 
-  private[this] val trueStrings = Set("t", "true", "y", "yes", 
"1").map(UTF8String.fromString)
-  private[this] val falseStrings = Set("f", "false", "n", "no", 
"0").map(UTF8String.fromString)
+  private[this] val trueStrings =
+Set("t", "true", "y", "yes", "1", "on").map(UTF8String.fromString)
+
+  private[this] val falseStrings =
+Set("f", "false", "n", "no", "0", "off").map(UTF8String.fromString)
 
 Review comment:
   Yes I guess so. Do you know other common string representattion used in 
other databases?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] wangyum commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-14 Thread GitBox
wangyum commented on a change in pull request #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#discussion_r314174619
 
 

 ##
 File path: pom.xml
 ##
 @@ -115,7 +115,7 @@
   
 UTF-8
 UTF-8
-11
+1.8
 
 Review comment:
   Let's wait for the fix of PySpark and SparkR?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] BestOreo commented on a change in pull request #25342: [SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter

2019-08-14 Thread GitBox
BestOreo commented on a change in pull request #25342: 
[SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the 
SortShuffleWriter
URL: https://github.com/apache/spark/pull/25342#discussion_r314174130
 
 

 ##
 File path: 
core/src/main/scala/org/apache/spark/util/collection/ShufflePartitionPairsWriter.scala
 ##
 @@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.collection
+
+import java.io.{Closeable, FilterOutputStream, OutputStream}
+
+import org.apache.spark.serializer.{SerializationStream, SerializerInstance, 
SerializerManager}
+import org.apache.spark.shuffle.ShuffleWriteMetricsReporter
+import org.apache.spark.shuffle.api.ShufflePartitionWriter
+import org.apache.spark.storage.BlockId
+
+/**
+ * A key-value writer inspired by {@link DiskBlockObjectWriter} that pushes 
the bytes to an
+ * arbitrary partition writer instead of writing to local disk through the 
block manager.
+ */
+private[spark] class ShufflePartitionPairsWriter(
+partitionWriter: ShufflePartitionWriter,
+serializerManager: SerializerManager,
+serializerInstance: SerializerInstance,
+blockId: BlockId,
+writeMetrics: ShuffleWriteMetricsReporter)
+  extends PairsWriter with Closeable {
+
+  private var isOpen = false
+  private var partitionStream: OutputStream = _
+  private var wrappedStream: OutputStream = _
+  private var objOut: SerializationStream = _
+  private var numRecordsWritten = 0
+  private var curNumBytesWritten = 0L
+
+  override def write(key: Any, value: Any): Unit = {
+if (!isOpen) {
+  open()
+  isOpen = true
+}
+objOut.writeKey(key)
+objOut.writeValue(value)
+writeMetrics.incRecordsWritten(1)
+  }
+
+  private def open(): Unit = {
+partitionStream = partitionWriter.openStream
+wrappedStream = serializerManager.wrapStream(blockId, partitionStream)
+objOut = serializerInstance.serializeStream(wrappedStream)
+  }
+
+  override def close(): Unit = {
+if (isOpen) {
 
 Review comment:
   The worry is unnecessary because `wrappedStream` and `objOut` would must be 
initialized successfully if `partitionStream` is opened as OutputStream without 
exception.
   And I think flag `isOpen` makes code easier to understand.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] BestOreo commented on a change in pull request #25342: [SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter

2019-08-14 Thread GitBox
BestOreo commented on a change in pull request #25342: 
[SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the 
SortShuffleWriter
URL: https://github.com/apache/spark/pull/25342#discussion_r314174130
 
 

 ##
 File path: 
core/src/main/scala/org/apache/spark/util/collection/ShufflePartitionPairsWriter.scala
 ##
 @@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.collection
+
+import java.io.{Closeable, FilterOutputStream, OutputStream}
+
+import org.apache.spark.serializer.{SerializationStream, SerializerInstance, 
SerializerManager}
+import org.apache.spark.shuffle.ShuffleWriteMetricsReporter
+import org.apache.spark.shuffle.api.ShufflePartitionWriter
+import org.apache.spark.storage.BlockId
+
+/**
+ * A key-value writer inspired by {@link DiskBlockObjectWriter} that pushes 
the bytes to an
+ * arbitrary partition writer instead of writing to local disk through the 
block manager.
+ */
+private[spark] class ShufflePartitionPairsWriter(
+partitionWriter: ShufflePartitionWriter,
+serializerManager: SerializerManager,
+serializerInstance: SerializerInstance,
+blockId: BlockId,
+writeMetrics: ShuffleWriteMetricsReporter)
+  extends PairsWriter with Closeable {
+
+  private var isOpen = false
+  private var partitionStream: OutputStream = _
+  private var wrappedStream: OutputStream = _
+  private var objOut: SerializationStream = _
+  private var numRecordsWritten = 0
+  private var curNumBytesWritten = 0L
+
+  override def write(key: Any, value: Any): Unit = {
+if (!isOpen) {
+  open()
+  isOpen = true
+}
+objOut.writeKey(key)
+objOut.writeValue(value)
+writeMetrics.incRecordsWritten(1)
+  }
+
+  private def open(): Unit = {
+partitionStream = partitionWriter.openStream
+wrappedStream = serializerManager.wrapStream(blockId, partitionStream)
+objOut = serializerInstance.serializeStream(wrappedStream)
+  }
+
+  override def close(): Unit = {
+if (isOpen) {
 
 Review comment:
   The worry is unnecessary because wrappedStream and objOut would must be 
initialized successfully if partitionStream is opened as OutputStream without 
exception.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] BestOreo commented on a change in pull request #25342: [SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the SortShuffleWriter

2019-08-14 Thread GitBox
BestOreo commented on a change in pull request #25342: 
[SPARK-28571][CORE][SHUFFLE] Use the shuffle writer plugin for the 
SortShuffleWriter
URL: https://github.com/apache/spark/pull/25342#discussion_r314174130
 
 

 ##
 File path: 
core/src/main/scala/org/apache/spark/util/collection/ShufflePartitionPairsWriter.scala
 ##
 @@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util.collection
+
+import java.io.{Closeable, FilterOutputStream, OutputStream}
+
+import org.apache.spark.serializer.{SerializationStream, SerializerInstance, 
SerializerManager}
+import org.apache.spark.shuffle.ShuffleWriteMetricsReporter
+import org.apache.spark.shuffle.api.ShufflePartitionWriter
+import org.apache.spark.storage.BlockId
+
+/**
+ * A key-value writer inspired by {@link DiskBlockObjectWriter} that pushes 
the bytes to an
+ * arbitrary partition writer instead of writing to local disk through the 
block manager.
+ */
+private[spark] class ShufflePartitionPairsWriter(
+partitionWriter: ShufflePartitionWriter,
+serializerManager: SerializerManager,
+serializerInstance: SerializerInstance,
+blockId: BlockId,
+writeMetrics: ShuffleWriteMetricsReporter)
+  extends PairsWriter with Closeable {
+
+  private var isOpen = false
+  private var partitionStream: OutputStream = _
+  private var wrappedStream: OutputStream = _
+  private var objOut: SerializationStream = _
+  private var numRecordsWritten = 0
+  private var curNumBytesWritten = 0L
+
+  override def write(key: Any, value: Any): Unit = {
+if (!isOpen) {
+  open()
+  isOpen = true
+}
+objOut.writeKey(key)
+objOut.writeValue(value)
+writeMetrics.incRecordsWritten(1)
+  }
+
+  private def open(): Unit = {
+partitionStream = partitionWriter.openStream
+wrappedStream = serializerManager.wrapStream(blockId, partitionStream)
+objOut = serializerInstance.serializeStream(wrappedStream)
+  }
+
+  override def close(): Unit = {
+if (isOpen) {
 
 Review comment:
   The worry is unnecessary because wrappedStream and objOut would be 
initialized successfully if partitionStream is opened as OutputStream without 
exception.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on issue #25446: [SPARK-28724] [SQL] Throw error message when cast out range decimal to long

2019-08-14 Thread GitBox
dongjoon-hyun commented on issue #25446: [SPARK-28724] [SQL] Throw error 
message when cast out range decimal to long
URL: https://github.com/apache/spark/pull/25446#issuecomment-521513201
 
 
   Thank you for your understanding, @LiShuMing .
   Thank you, @maropu .


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] cloud-fan commented on a change in pull request #25348: [SPARK-28554][SQL] Adds a v1 fallback writer implementation for v2 data source codepaths

2019-08-14 Thread GitBox
cloud-fan commented on a change in pull request #25348: [SPARK-28554][SQL] Adds 
a v1 fallback writer implementation for v2 data source codepaths
URL: https://github.com/apache/spark/pull/25348#discussion_r314173218
 
 

 ##
 File path: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
 ##
 @@ -200,24 +202,37 @@ object DataSourceV2Strategy extends Strategy with 
PredicateHelper {
 catalog,
 ident,
 parts,
+query,
 planLater(query),
 props,
 writeOptions,
 orCreate = orCreate) :: Nil
   }
 
 case AppendData(r: DataSourceV2Relation, query, _) =>
-  AppendDataExec(r.table.asWritable, r.options, planLater(query)) :: Nil
 
 Review comment:
   If end-users look at the SQL tab and see `AppendDataExecV1`, they would 
expect to see v1 version of CTAS physical plan as well, and may report a bug if 
they don't see it.
   
   BTW I think there are other ways to implement this feature (users know if v1 
fallback is triggered from SQL tab), e.g. we can use SQLMetrics to report it, 
which can be updated at runtime and support CTAS as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25457: [SPARK-27234][SS][PYTHON][BRANCH-2.4] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25457: 
[SPARK-27234][SS][PYTHON][BRANCH-2.4] Use InheritableThreadLocal for current 
epoch in EpochTracker (to support Python UDFs)
URL: https://github.com/apache/spark/pull/25457#issuecomment-521511185
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25457: [SPARK-27234][SS][PYTHON][BRANCH-2.4] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25457: 
[SPARK-27234][SS][PYTHON][BRANCH-2.4] Use InheritableThreadLocal for current 
epoch in EpochTracker (to support Python UDFs)
URL: https://github.com/apache/spark/pull/25457#issuecomment-521511188
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109135/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25457: [SPARK-27234][SS][PYTHON][BRANCH-2.4] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25457: [SPARK-27234][SS][PYTHON][BRANCH-2.4] 
Use InheritableThreadLocal for current epoch in EpochTracker (to support Python 
UDFs)
URL: https://github.com/apache/spark/pull/25457#issuecomment-521511188
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109135/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25457: [SPARK-27234][SS][PYTHON][BRANCH-2.4] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25457: [SPARK-27234][SS][PYTHON][BRANCH-2.4] 
Use InheritableThreadLocal for current epoch in EpochTracker (to support Python 
UDFs)
URL: https://github.com/apache/spark/pull/25457#issuecomment-521511185
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #25457: [SPARK-27234][SS][PYTHON][BRANCH-2.4] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-08-14 Thread GitBox
SparkQA removed a comment on issue #25457: 
[SPARK-27234][SS][PYTHON][BRANCH-2.4] Use InheritableThreadLocal for current 
epoch in EpochTracker (to support Python UDFs)
URL: https://github.com/apache/spark/pull/25457#issuecomment-521473909
 
 
   **[Test build #109135 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109135/testReport)**
 for PR 25457 at commit 
[`4c5fdd6`](https://github.com/apache/spark/commit/4c5fdd668be1f31561849e5fe485e814e318a3ef).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25457: [SPARK-27234][SS][PYTHON][BRANCH-2.4] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-08-14 Thread GitBox
SparkQA commented on issue #25457: [SPARK-27234][SS][PYTHON][BRANCH-2.4] Use 
InheritableThreadLocal for current epoch in EpochTracker (to support Python 
UDFs)
URL: https://github.com/apache/spark/pull/25457#issuecomment-521511008
 
 
   **[Test build #109135 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109135/testReport)**
 for PR 25457 at commit 
[`4c5fdd6`](https://github.com/apache/spark/commit/4c5fdd668be1f31561849e5fe485e814e318a3ef).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on a change in pull request #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-14 Thread GitBox
dongjoon-hyun commented on a change in pull request #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#discussion_r314172342
 
 

 ##
 File path: pom.xml
 ##
 @@ -115,7 +115,7 @@
   
 UTF-8
 UTF-8
-11
+1.8
 
 Review comment:
   Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun commented on issue #25457: [SPARK-27234][SS][PYTHON][BRANCH-2.4] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-08-14 Thread GitBox
dongjoon-hyun commented on issue #25457: [SPARK-27234][SS][PYTHON][BRANCH-2.4] 
Use InheritableThreadLocal for current epoch in EpochTracker (to support Python 
UDFs)
URL: https://github.com/apache/spark/pull/25457#issuecomment-521510518
 
 
   Merged to `branch-2.4`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] dongjoon-hyun closed pull request #25457: [SPARK-27234][SS][PYTHON][BRANCH-2.4] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)

2019-08-14 Thread GitBox
dongjoon-hyun closed pull request #25457: [SPARK-27234][SS][PYTHON][BRANCH-2.4] 
Use InheritableThreadLocal for current epoch in EpochTracker (to support Python 
UDFs)
URL: https://github.com/apache/spark/pull/25457
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] cloud-fan closed pull request #25402: [SPARK-28666] Support saveAsTable for V2 tables through Session Catalog

2019-08-14 Thread GitBox
cloud-fan closed pull request #25402: [SPARK-28666] Support saveAsTable for V2 
tables through Session Catalog
URL: https://github.com/apache/spark/pull/25402
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] wangyum commented on a change in pull request #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type.

2019-08-14 Thread GitBox
wangyum commented on a change in pull request #25458: [SPARK-27931][SQL] Accept 
'on' and 'off' as input and trim input for the boolean data type.
URL: https://github.com/apache/spark/pull/25458#discussion_r314171694
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala
 ##
 @@ -65,12 +65,15 @@ object StringUtils extends Logging {
 "(?s)" + out.result() // (?s) enables dotall mode, causing "." to match 
new lines
   }
 
-  private[this] val trueStrings = Set("t", "true", "y", "yes", 
"1").map(UTF8String.fromString)
-  private[this] val falseStrings = Set("f", "false", "n", "no", 
"0").map(UTF8String.fromString)
+  private[this] val trueStrings =
+Set("t", "true", "y", "yes", "1", "on").map(UTF8String.fromString)
+
+  private[this] val falseStrings =
+Set("f", "false", "n", "no", "0", "off").map(UTF8String.fromString)
 
 Review comment:
   It seems only PostgreSQL accepts `on` and `off`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#issuecomment-521509487
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25455: [WIP][SPARK-28737][CORE] Update Jersey to 2.29

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25455: [WIP][SPARK-28737][CORE] Update Jersey 
to 2.29
URL: https://github.com/apache/spark/pull/25455#issuecomment-521509388
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109137/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] cloud-fan commented on issue #25402: [SPARK-28666] Support saveAsTable for V2 tables through Session Catalog

2019-08-14 Thread GitBox
cloud-fan commented on issue #25402: [SPARK-28666] Support saveAsTable for V2 
tables through Session Catalog
URL: https://github.com/apache/spark/pull/25402#issuecomment-521509547
 
 
   thanks, merging to master! Please address 
https://github.com/apache/spark/pull/25402#discussion_r313962683 in a followup.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25455: [WIP][SPARK-28737][CORE] Update Jersey to 2.29

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25455: [WIP][SPARK-28737][CORE] 
Update Jersey to 2.29
URL: https://github.com/apache/spark/pull/25455#issuecomment-521509386
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25455: [WIP][SPARK-28737][CORE] Update Jersey to 2.29

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25455: [WIP][SPARK-28737][CORE] 
Update Jersey to 2.29
URL: https://github.com/apache/spark/pull/25455#issuecomment-521509388
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109137/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-14 Thread GitBox
SparkQA commented on issue #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#issuecomment-521509799
 
 
   **[Test build #109140 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109140/testReport)**
 for PR 25443 at commit 
[`0ac0b30`](https://github.com/apache/spark/commit/0ac0b30947dc1da33d0ac5ed0c8201a4c3d54c8a).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#issuecomment-521509487
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#issuecomment-521509492
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14208/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#issuecomment-521509492
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14208/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #25455: [WIP][SPARK-28737][CORE] Update Jersey to 2.29

2019-08-14 Thread GitBox
SparkQA removed a comment on issue #25455: [WIP][SPARK-28737][CORE] Update 
Jersey to 2.29
URL: https://github.com/apache/spark/pull/25455#issuecomment-521493509
 
 
   **[Test build #109137 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109137/testReport)**
 for PR 25455 at commit 
[`3e0b534`](https://github.com/apache/spark/commit/3e0b5341dee62dc1d54fc0d6856b7a387ab8d968).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25455: [WIP][SPARK-28737][CORE] Update Jersey to 2.29

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25455: [WIP][SPARK-28737][CORE] Update Jersey 
to 2.29
URL: https://github.com/apache/spark/pull/25455#issuecomment-521509386
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25455: [WIP][SPARK-28737][CORE] Update Jersey to 2.29

2019-08-14 Thread GitBox
SparkQA commented on issue #25455: [WIP][SPARK-28737][CORE] Update Jersey to 
2.29
URL: https://github.com/apache/spark/pull/25455#issuecomment-521509245
 
 
   **[Test build #109137 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109137/testReport)**
 for PR 25455 at commit 
[`3e0b534`](https://github.com/apache/spark/commit/3e0b5341dee62dc1d54fc0d6856b7a387ab8d968).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type.

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept 'on' 
and 'off' as input and trim input for the boolean data type.
URL: https://github.com/apache/spark/pull/25458#issuecomment-521505997
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25456: [SPARK-28739][SQL] Add a simple cost check for Adaptive Query Execution

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25456: [SPARK-28739][SQL] Add a 
simple cost check for Adaptive Query Execution
URL: https://github.com/apache/spark/pull/25456#issuecomment-521506406
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109134/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25456: [SPARK-28739][SQL] Add a simple cost check for Adaptive Query Execution

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25456: [SPARK-28739][SQL] Add a 
simple cost check for Adaptive Query Execution
URL: https://github.com/apache/spark/pull/25456#issuecomment-521506404
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25456: [SPARK-28739][SQL] Add a simple cost check for Adaptive Query Execution

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25456: [SPARK-28739][SQL] Add a simple cost 
check for Adaptive Query Execution
URL: https://github.com/apache/spark/pull/25456#issuecomment-521506406
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109134/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25456: [SPARK-28739][SQL] Add a simple cost check for Adaptive Query Execution

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25456: [SPARK-28739][SQL] Add a simple cost 
check for Adaptive Query Execution
URL: https://github.com/apache/spark/pull/25456#issuecomment-521506404
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type.

2019-08-14 Thread GitBox
SparkQA commented on issue #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as 
input and trim input for the boolean data type.
URL: https://github.com/apache/spark/pull/25458#issuecomment-521506392
 
 
   **[Test build #109139 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109139/testReport)**
 for PR 25458 at commit 
[`7d61642`](https://github.com/apache/spark/commit/7d61642860125ff8049578507b6b1143eacad88b).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type.

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept 'on' 
and 'off' as input and trim input for the boolean data type.
URL: https://github.com/apache/spark/pull/25458#issuecomment-521506160
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14207/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type.

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25458: [SPARK-27931][SQL] Accept 'on' 
and 'off' as input and trim input for the boolean data type.
URL: https://github.com/apache/spark/pull/25458#issuecomment-521506156
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #25456: [SPARK-28739][SQL] Add a simple cost check for Adaptive Query Execution

2019-08-14 Thread GitBox
SparkQA removed a comment on issue #25456: [SPARK-28739][SQL] Add a simple cost 
check for Adaptive Query Execution
URL: https://github.com/apache/spark/pull/25456#issuecomment-521470699
 
 
   **[Test build #109134 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109134/testReport)**
 for PR 25456 at commit 
[`4fa256e`](https://github.com/apache/spark/commit/4fa256e8cca3ae807847f17d0e7097dabd9b76c8).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type.

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept 'on' and 
'off' as input and trim input for the boolean data type.
URL: https://github.com/apache/spark/pull/25458#issuecomment-521506160
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14207/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25456: [SPARK-28739][SQL] Add a simple cost check for Adaptive Query Execution

2019-08-14 Thread GitBox
SparkQA commented on issue #25456: [SPARK-28739][SQL] Add a simple cost check 
for Adaptive Query Execution
URL: https://github.com/apache/spark/pull/25456#issuecomment-521506199
 
 
   **[Test build #109134 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109134/testReport)**
 for PR 25456 at commit 
[`4fa256e`](https://github.com/apache/spark/commit/4fa256e8cca3ae807847f17d0e7097dabd9b76c8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type.

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept 'on' and 
'off' as input and trim input for the boolean data type.
URL: https://github.com/apache/spark/pull/25458#issuecomment-521506156
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type.

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25458: [SPARK-27931][SQL] Accept 'on' and 
'off' as input and trim input for the boolean data type.
URL: https://github.com/apache/spark/pull/25458#issuecomment-521505997
 
 
   Can one of the admins verify this patch?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] younggyuchun opened a new pull request #25458: [SPARK-27931][SQL] Accept 'on' and 'off' as input and trim input for the boolean data type.

2019-08-14 Thread GitBox
younggyuchun opened a new pull request #25458: [SPARK-27931][SQL] Accept 'on' 
and 'off' as input and trim input for the boolean data type.
URL: https://github.com/apache/spark/pull/25458
 
 
   ## What changes were proposed in this pull request?
   This PR aims to add "on" and "off" as input for a boolean data type and 
ignore input whitespace.
   Please see the following what string representations are using for the 
boolean type in other databases. This changes only add "on" and "off" as other 
string representations are already added. 
   
   https://www.postgresql.org/docs/devel/datatype-boolean.html
   https://docs.aws.amazon.com/redshift/latest/dg/r_Boolean_type.html
   
   ## How was this patch tested?
   Added new tests to CastSuite.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] advancedxy commented on issue #25306: [SPARK-28573][SQL] Convert InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table

2019-08-14 Thread GitBox
advancedxy commented on issue #25306: [SPARK-28573][SQL] Convert 
InsertIntoTable(HiveTableRelation) to DataSource inserting for partitioned table
URL: https://github.com/apache/spark/pull/25306#issuecomment-521503417
 
 
   > Can we update `DetermineTableStats` to catch 
`InsertIntoTable(HiveTableRelation)` as well? 
   
   will push a new commit later.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] cloud-fan commented on a change in pull request #25402: [SPARK-28666] Support saveAsTable for V2 tables through Session Catalog

2019-08-14 Thread GitBox
cloud-fan commented on a change in pull request #25402: [SPARK-28666] Support 
saveAsTable for V2 tables through Session Catalog
URL: https://github.com/apache/spark/pull/25402#discussion_r314165849
 
 

 ##
 File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
 ##
 @@ -493,13 +488,20 @@ final class DataFrameWriter[T] private[sql](ds: 
Dataset[T]) {
 import df.sparkSession.sessionState.analyzer.{AsTableIdentifier, 
CatalogObjectIdentifier}
 import org.apache.spark.sql.catalog.v2.CatalogV2Implicits._
 
-import org.apache.spark.sql.catalog.v2.CatalogV2Implicits._
 val session = df.sparkSession
+val provider = DataSource.lookupDataSource(source, 
session.sessionState.conf)
 
 Review comment:
   let's do it in a followup.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples

2019-08-14 Thread GitBox
SparkQA removed a comment on issue #25412: [SPARK-28691][EXAMPLES] Add 
Java/Scala DirectKerberizedKafkaWordCount examples
URL: https://github.com/apache/spark/pull/25412#issuecomment-521499531
 
 
   **[Test build #109138 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109138/testReport)**
 for PR 25412 at commit 
[`f79328b`](https://github.com/apache/spark/commit/f79328b9c889dc455975f525385ae8f19cf70fb2).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25412: [SPARK-28691][EXAMPLES] Add 
Java/Scala DirectKerberizedKafkaWordCount examples
URL: https://github.com/apache/spark/pull/25412#issuecomment-521501434
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109138/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25412: [SPARK-28691][EXAMPLES] Add 
Java/Scala DirectKerberizedKafkaWordCount examples
URL: https://github.com/apache/spark/pull/25412#issuecomment-521501428
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala 
DirectKerberizedKafkaWordCount examples
URL: https://github.com/apache/spark/pull/25412#issuecomment-521501428
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala 
DirectKerberizedKafkaWordCount examples
URL: https://github.com/apache/spark/pull/25412#issuecomment-521501434
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109138/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples

2019-08-14 Thread GitBox
SparkQA commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala 
DirectKerberizedKafkaWordCount examples
URL: https://github.com/apache/spark/pull/25412#issuecomment-521501378
 
 
   **[Test build #109138 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109138/testReport)**
 for PR 25412 at commit 
[`f79328b`](https://github.com/apache/spark/commit/f79328b9c889dc455975f525385ae8f19cf70fb2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] cloud-fan commented on a change in pull request #25402: [SPARK-28666] Support saveAsTable for V2 tables through Session Catalog

2019-08-14 Thread GitBox
cloud-fan commented on a change in pull request #25402: [SPARK-28666] Support 
saveAsTable for V2 tables through Session Catalog
URL: https://github.com/apache/spark/pull/25402#discussion_r314163841
 
 

 ##
 File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
 ##
 @@ -493,13 +488,20 @@ final class DataFrameWriter[T] private[sql](ds: 
Dataset[T]) {
 import df.sparkSession.sessionState.analyzer.{AsTableIdentifier, 
CatalogObjectIdentifier}
 import org.apache.spark.sql.catalog.v2.CatalogV2Implicits._
 
-import org.apache.spark.sql.catalog.v2.CatalogV2Implicits._
 val session = df.sparkSession
+val provider = DataSource.lookupDataSource(source, 
session.sessionState.conf)
 
 Review comment:
   +1 on using statements.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] hddong commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples

2019-08-14 Thread GitBox
hddong commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala 
DirectKerberizedKafkaWordCount examples
URL: https://github.com/apache/spark/pull/25412#issuecomment-521500093
 
 
   Thasks for @HyukjinKwon . Move the comment here.
   `This file uses SASL_PLAINTEXT for simplicity; however, SASL_PLAINTEXT has 
no SSL encryption and likely be less secure. Consider using SASL_SSL.`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] cloud-fan commented on issue #24892: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files

2019-08-14 Thread GitBox
cloud-fan commented on issue #24892: [SPARK-25341][Core] Support rolling back a 
shuffle map stage and re-generate the shuffle files
URL: https://github.com/apache/spark/pull/24892#issuecomment-521499643
 
 
   @vanzin I think your concern is valid. Seems the shuffle writing policy is 
contradictory to itself: if a partition has multiple shuffle write tasks, 1) if 
they are on the same node (e.g. a stage becomes zombie and rerun), first write 
wins. 2) if they are on different nodes (speculative tasks), last write wins.
   
   I think we should stick with "first write wins". As you said we should only 
update the map status if the task is the first one that completes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples

2019-08-14 Thread GitBox
SparkQA commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala 
DirectKerberizedKafkaWordCount examples
URL: https://github.com/apache/spark/pull/25412#issuecomment-521499531
 
 
   **[Test build #109138 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109138/testReport)**
 for PR 25412 at commit 
[`f79328b`](https://github.com/apache/spark/commit/f79328b9c889dc455975f525385ae8f19cf70fb2).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] jzhuge commented on issue #24623: [SPARK-27739][SQL] df.persist should save stats from optimized plan

2019-08-14 Thread GitBox
jzhuge commented on issue #24623: [SPARK-27739][SQL] df.persist should save 
stats from optimized plan
URL: https://github.com/apache/spark/pull/24623#issuecomment-521499553
 
 
   thanks all for the reviews. thanks @cloud-fan for the merge.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25412: [SPARK-28691][EXAMPLES] Add 
Java/Scala DirectKerberizedKafkaWordCount examples
URL: https://github.com/apache/spark/pull/25412#issuecomment-521499273
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25412: [SPARK-28691][EXAMPLES] Add 
Java/Scala DirectKerberizedKafkaWordCount examples
URL: https://github.com/apache/spark/pull/25412#issuecomment-521499278
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14206/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala 
DirectKerberizedKafkaWordCount examples
URL: https://github.com/apache/spark/pull/25412#issuecomment-521499273
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala DirectKerberizedKafkaWordCount examples

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25412: [SPARK-28691][EXAMPLES] Add Java/Scala 
DirectKerberizedKafkaWordCount examples
URL: https://github.com/apache/spark/pull/25412#issuecomment-521499278
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/14206/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#issuecomment-521497106
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109136/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-14 Thread GitBox
SparkQA commented on issue #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#issuecomment-521497095
 
 
   **[Test build #109136 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109136/testReport)**
 for PR 25443 at commit 
[`9254dfb`](https://github.com/apache/spark/commit/9254dfbc60464c7b326da6a1688d328e71dffd69).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-14 Thread GitBox
AmplabJenkins removed a comment on issue #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#issuecomment-521497104
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#issuecomment-521497106
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/109136/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] AmplabJenkins commented on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-14 Thread GitBox
AmplabJenkins commented on issue #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#issuecomment-521497104
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



[GitHub] [spark] SparkQA removed a comment on issue #25443: [WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 2.3.6 on jenkins

2019-08-14 Thread GitBox
SparkQA removed a comment on issue #25443: 
[WIP][SPARK-28723][test-hadoop3.2][test-maven] Test JDK 11 with Hadoop-3.2/Hive 
2.3.6 on jenkins
URL: https://github.com/apache/spark/pull/25443#issuecomment-521490729
 
 
   **[Test build #109136 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/109136/testReport)**
 for PR 25443 at commit 
[`9254dfb`](https://github.com/apache/spark/commit/9254dfbc60464c7b326da6a1688d328e71dffd69).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org



  1   2   3   4   5   6   7   8   9   10   >