[GitHub] [spark] gengliangwang commented on issue #24815: [SPARK-27961][SQL] DataSourceV2Relation should not have refresh method

[GitBox]

gengliangwang commented on issue #24815: [SPARK-27961][SQL] 
DataSourceV2Relation should not have refresh method
URL: https://github.com/apache/spark/pull/24815#issuecomment-500098462
 
 
   @dongjoon-hyun Any other concerns on this PR?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] gengliangwang commented on issue #24327: [SPARK-27418][SQL] Migrate Parquet to File Data Source V2

[GitBox]

gengliangwang commented on issue #24327: [SPARK-27418][SQL] Migrate Parquet to 
File Data Source V2
URL: https://github.com/apache/spark/pull/24327#issuecomment-500098384
 
 
   @rdblue Thanks for the review. I have addressed all your comments. Any other 
concerns?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] ConvertToLocalRelation should tolerate expression reusing output object

[GitBox]

AmplabJenkins removed a comment on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] 
ConvertToLocalRelation should tolerate expression reusing output object
URL: https://github.com/apache/spark/pull/24823#issuecomment-500098108
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106291/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] ConvertToLocalRelation should tolerate expression reusing output object

[GitBox]

AmplabJenkins removed a comment on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] 
ConvertToLocalRelation should tolerate expression reusing output object
URL: https://github.com/apache/spark/pull/24823#issuecomment-500098105
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] ConvertToLocalRelation should tolerate expression reusing output object

[GitBox]

AmplabJenkins commented on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] 
ConvertToLocalRelation should tolerate expression reusing output object
URL: https://github.com/apache/spark/pull/24823#issuecomment-500098108
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106291/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] ConvertToLocalRelation should tolerate expression reusing output object

[GitBox]

SparkQA removed a comment on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] 
ConvertToLocalRelation should tolerate expression reusing output object
URL: https://github.com/apache/spark/pull/24823#issuecomment-500089696
 
 
   **[Test build #106291 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106291/testReport)**
 for PR 24823 at commit 
[`bc8d609`](https://github.com/apache/spark/commit/bc8d60993fe2f2611c07eba190eef96ddb1c1dc8).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] ConvertToLocalRelation should tolerate expression reusing output object

[GitBox]

SparkQA commented on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] 
ConvertToLocalRelation should tolerate expression reusing output object
URL: https://github.com/apache/spark/pull/24823#issuecomment-500098066
 
 
   **[Test build #106291 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106291/testReport)**
 for PR 24823 at commit 
[`bc8d609`](https://github.com/apache/spark/commit/bc8d60993fe2f2611c07eba190eef96ddb1c1dc8).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] ConvertToLocalRelation should tolerate expression reusing output object

[GitBox]

AmplabJenkins commented on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] 
ConvertToLocalRelation should tolerate expression reusing output object
URL: https://github.com/apache/spark/pull/24823#issuecomment-500098105
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on a change in pull request #24800: [SPARK-27947][SQL] ParsedStatement subclass toString may throw ClassCastException

[GitBox]

HyukjinKwon commented on a change in pull request #24800: [SPARK-27947][SQL] 
ParsedStatement subclass toString may throw ClassCastException
URL: https://github.com/apache/spark/pull/24800#discussion_r291795355
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/sql/ParsedStatement.scala
 ##
 @@ -36,8 +38,11 @@ import 
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
 private[sql] abstract class ParsedStatement extends LogicalPlan {
   // Redact properties and options when parsed nodes are used by generic 
methods like toString
   override def productIterator: Iterator[Any] = super.productIterator.map {
 
 Review comment:
   We can just redact `options` and `properties` and remove `productIterator` 
override. Wasn't that the purpose of doing this?
   
   I am currently unable to follow why we need to it for every map argument in 
implemented case classes. How about array?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] jzhuge commented on a change in pull request #24800: [SPARK-27947][SQL] ParsedStatement subclass toString may throw ClassCastException

[GitBox]

jzhuge commented on a change in pull request #24800: [SPARK-27947][SQL] 
ParsedStatement subclass toString may throw ClassCastException
URL: https://github.com/apache/spark/pull/24800#discussion_r291794150
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/sql/ParsedStatement.scala
 ##
 @@ -36,8 +38,11 @@ import 
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
 private[sql] abstract class ParsedStatement extends LogicalPlan {
   // Redact properties and options when parsed nodes are used by generic 
methods like toString
   override def productIterator: Iterator[Any] = super.productIterator.map {
 
 Review comment:
   This is not flexible enough to automatically redact other field names. 
Please check out the solution in my new commit.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] jzhuge commented on a change in pull request #24800: [SPARK-27947][SQL] ParsedStatement subclass toString may throw ClassCastException

[GitBox]

jzhuge commented on a change in pull request #24800: [SPARK-27947][SQL] 
ParsedStatement subclass toString may throw ClassCastException
URL: https://github.com/apache/spark/pull/24800#discussion_r291794124
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/sql/ParsedStatement.scala
 ##
 @@ -36,8 +38,11 @@ import 
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
 private[sql] abstract class ParsedStatement extends LogicalPlan {
   // Redact properties and options when parsed nodes are used by generic 
methods like toString
   override def productIterator: Iterator[Any] = super.productIterator.map {
-case mapArg: Map[_, _] => 
conf.redactOptions(mapArg.asInstanceOf[Map[String, String]])
-case other => other
+case mapArg: Map[_, _] =>
+  // May match any Map type, e.g. Map[String, Int], due to type erasure
+  Try(conf.redactOptions(mapArg.asInstanceOf[Map[String, 
String]])).getOrElse(mapArg)
 
 Review comment:
   @dongjoon-hyun @gatorsmile @srowen Please review the new commit.
   
   Please note the commit does not redact Map[String, non-String] because the 
replacement text is string. For non-String, e.g, Int, if we use -1 as 
replacement, it may be confusing.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24800: [SPARK-27947][SQL] ParsedStatement subclass toString may throw ClassCastException

[GitBox]

AmplabJenkins removed a comment on issue #24800: [SPARK-27947][SQL] 
ParsedStatement subclass toString may throw ClassCastException
URL: https://github.com/apache/spark/pull/24800#issuecomment-500093445
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11539/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24800: [SPARK-27947][SQL] ParsedStatement subclass toString may throw ClassCastException

[GitBox]

AmplabJenkins removed a comment on issue #24800: [SPARK-27947][SQL] 
ParsedStatement subclass toString may throw ClassCastException
URL: https://github.com/apache/spark/pull/24800#issuecomment-500093444
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #24800: [SPARK-27947][SQL] ParsedStatement subclass toString may throw ClassCastException

[GitBox]

SparkQA commented on issue #24800: [SPARK-27947][SQL] ParsedStatement subclass 
toString may throw ClassCastException
URL: https://github.com/apache/spark/pull/24800#issuecomment-500093532
 
 
   **[Test build #106293 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106293/testReport)**
 for PR 24800 at commit 
[`2e00113`](https://github.com/apache/spark/commit/2e001135c971e3abccd45ee096beac18f5cc90f6).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24800: [SPARK-27947][SQL] ParsedStatement subclass toString may throw ClassCastException

[GitBox]

AmplabJenkins commented on issue #24800: [SPARK-27947][SQL] ParsedStatement 
subclass toString may throw ClassCastException
URL: https://github.com/apache/spark/pull/24800#issuecomment-500093445
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11539/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24800: [SPARK-27947][SQL] ParsedStatement subclass toString may throw ClassCastException

[GitBox]

AmplabJenkins commented on issue #24800: [SPARK-27947][SQL] ParsedStatement 
subclass toString may throw ClassCastException
URL: https://github.com/apache/spark/pull/24800#issuecomment-500093444
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on issue #24824: [SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in `build/mvn` and `run-tests.py`

[GitBox]

dongjoon-hyun edited a comment on issue #24824: 
[SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in 
`build/mvn` and `run-tests.py`
URL: https://github.com/apache/spark/pull/24824#issuecomment-500091184
 
 
   Hi, @srowen . Can we remove this at 3.0.0 since we deprecated this at 2.0.0? 
:)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24824: [SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in `build/mvn` and `run-tests.py`

[GitBox]

AmplabJenkins removed a comment on issue #24824: 
[SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in 
`build/mvn` and `run-tests.py`
URL: https://github.com/apache/spark/pull/24824#issuecomment-500091438
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24824: [SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in `build/mvn` and `run-tests.py`

[GitBox]

AmplabJenkins removed a comment on issue #24824: 
[SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in 
`build/mvn` and `run-tests.py`
URL: https://github.com/apache/spark/pull/24824#issuecomment-500091441
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11538/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24824: [SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in `build/mvn` and `run-tests.py`

[GitBox]

AmplabJenkins commented on issue #24824: [SPARK-27979][BUILD][test-maven] 
Remove deprecated `--force` option in `build/mvn` and `run-tests.py`
URL: https://github.com/apache/spark/pull/24824#issuecomment-500091441
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11538/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24824: [SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in `build/mvn` and `run-tests.py`

[GitBox]

AmplabJenkins commented on issue #24824: [SPARK-27979][BUILD][test-maven] 
Remove deprecated `--force` option in `build/mvn` and `run-tests.py`
URL: https://github.com/apache/spark/pull/24824#issuecomment-500091438
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on issue #24824: [SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in `build/mvn` and `run-tests.py`

[GitBox]

dongjoon-hyun commented on issue #24824: [SPARK-27979][BUILD][test-maven] 
Remove deprecated `--force` option in `build/mvn` and `run-tests.py`
URL: https://github.com/apache/spark/pull/24824#issuecomment-500091184
 
 
   @srowen . Can we remove this at 3.0.0 since we deprecated this at 2.0.0? :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #24824: [SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in `build/mvn` and `run-tests.py`

[GitBox]

SparkQA commented on issue #24824: [SPARK-27979][BUILD][test-maven] Remove 
deprecated `--force` option in `build/mvn` and `run-tests.py`
URL: https://github.com/apache/spark/pull/24824#issuecomment-500091189
 
 
   **[Test build #106292 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106292/testReport)**
 for PR 24824 at commit 
[`1aeb767`](https://github.com/apache/spark/commit/1aeb7675422072736df150c7c2591d5038d2a4b3).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24824: [SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in `build/mvn` and `run-tests.py`

[GitBox]

AmplabJenkins removed a comment on issue #24824: 
[SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in 
`build/mvn` and `run-tests.py`
URL: https://github.com/apache/spark/pull/24824#issuecomment-500091096
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on issue #24824: [SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in `build/mvn` and `run-tests.py`

[GitBox]

dongjoon-hyun commented on issue #24824: [SPARK-27979][BUILD][test-maven] 
Remove deprecated `--force` option in `build/mvn` and `run-tests.py`
URL: https://github.com/apache/spark/pull/24824#issuecomment-500091135
 
 
   Retest this please.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24824: [SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in `build/mvn` and `run-tests.py`

[GitBox]

AmplabJenkins removed a comment on issue #24824: 
[SPARK-27979][BUILD][test-maven] Remove deprecated `--force` option in 
`build/mvn` and `run-tests.py`
URL: https://github.com/apache/spark/pull/24824#issuecomment-500091097
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11537/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24824: [SPARK-27979][BUILD] Remove deprecated `--force` option in `build/mvn` and `run-tests.py`

[GitBox]

AmplabJenkins commented on issue #24824: [SPARK-27979][BUILD] Remove deprecated 
`--force` option in `build/mvn` and `run-tests.py`
URL: https://github.com/apache/spark/pull/24824#issuecomment-500091097
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11537/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24824: [SPARK-27979][BUILD] Remove deprecated `--force` option in `build/mvn` and `run-tests.py`

[GitBox]

AmplabJenkins commented on issue #24824: [SPARK-27979][BUILD] Remove deprecated 
`--force` option in `build/mvn` and `run-tests.py`
URL: https://github.com/apache/spark/pull/24824#issuecomment-500091096
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] dongjoon-hyun opened a new pull request #24824: [SPARK-27979][BUILD] Remove deprecated `--force` option in `build/mvn` and `run-tests.py`

[GitBox]

dongjoon-hyun opened a new pull request #24824: [SPARK-27979][BUILD] Remove 
deprecated `--force` option in `build/mvn` and `run-tests.py`
URL: https://github.com/apache/spark/pull/24824
 
 
   ## What changes were proposed in this pull request?
   
   Since Apache Spark 2.0.0, SPARK-14867 deprecated `--force` option and made 
it ignored. This PR cleans up the related code completely at 3.0.0.
   
   **BEFORE (Jenkins)**
   ```
   build/mvn --force -DzincPort=3374 -DskipTests -Phadoop-2.7 -Pyarn -Phive 
-Phive-thriftserver -Pkinesis-asl -Pmesos clean package
   WARNING: '--force' is deprecated and ignored.
   ...
   build/mvn --force -DzincPort=3374 -Phadoop-2.7 -Pyarn -Phive 
-Phive-thriftserver -Pkinesis-asl -Pmesos --fail-at-end test
   WARNING: '--force' is deprecated and ignored.
   ```
   
   **AFTER (Jenkins)**
   ```
   build/mvn -DzincPort=3374 -DskipTests -Phadoop-2.7 -Pyarn -Phive 
-Phive-thriftserver -Pkinesis-asl -Pmesos clean package
   ...
   build/mvn -DzincPort=3374 -Phadoop-2.7 -Pyarn -Phive -Phive-thriftserver 
-Pkinesis-asl -Pmesos --fail-at-end test
   ```
   
   ## How was this patch tested?
   
   Manually check the Jenkins logs.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] ConvertToLocalRelation should tolerate expression reusing output object

[GitBox]

dongjoon-hyun commented on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] 
ConvertToLocalRelation should tolerate expression reusing output object
URL: https://github.com/apache/spark/pull/24823#issuecomment-500089895
 
 
   Thank you a lot for backporting with the proper title, @viirya .


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] ConvertToLocalRelation should tolerate expression reusing output object

[GitBox]

AmplabJenkins removed a comment on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] 
ConvertToLocalRelation should tolerate expression reusing output object
URL: https://github.com/apache/spark/pull/24823#issuecomment-500089593
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11536/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] ConvertToLocalRelation should tolerate expression reusing output object

[GitBox]

SparkQA commented on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] 
ConvertToLocalRelation should tolerate expression reusing output object
URL: https://github.com/apache/spark/pull/24823#issuecomment-500089696
 
 
   **[Test build #106291 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106291/testReport)**
 for PR 24823 at commit 
[`bc8d609`](https://github.com/apache/spark/commit/bc8d60993fe2f2611c07eba190eef96ddb1c1dc8).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] ConvertToLocalRelation should tolerate expression reusing output object

[GitBox]

AmplabJenkins removed a comment on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] 
ConvertToLocalRelation should tolerate expression reusing output object
URL: https://github.com/apache/spark/pull/24823#issuecomment-500089591
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] ConvertToLocalRelation should tolerate expression reusing output object

[GitBox]

AmplabJenkins commented on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] 
ConvertToLocalRelation should tolerate expression reusing output object
URL: https://github.com/apache/spark/pull/24823#issuecomment-500089591
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] ConvertToLocalRelation should tolerate expression reusing output object

[GitBox]

AmplabJenkins commented on issue #24823: [SPARK-27798][SQL][BRANCH-2.3] 
ConvertToLocalRelation should tolerate expression reusing output object
URL: https://github.com/apache/spark/pull/24823#issuecomment-500089593
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11536/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] viirya opened a new pull request #24823: [SPARK-27798][SQL][BRANCH-2.3] ConvertToLocalRelation should tolerate expression reusing output object

[GitBox]

viirya opened a new pull request #24823: [SPARK-27798][SQL][BRANCH-2.3] 
ConvertToLocalRelation should tolerate expression reusing output object
URL: https://github.com/apache/spark/pull/24823
 
 
   ## What changes were proposed in this pull request?
   
   The original issue SPARK-27798 was reported on master branch. When using 
`from_avro` to deserialize avro data to catalyst StructType format, if 
`ConvertToLocalRelation` is applied at the time, `from_avro` produces only the 
last value (overriding previous values).
   
   The cause is `AvroDeserializer` reuses output row for StructType. But 
`ConvertToLocalRelation` doesn't tolerate expression reusing output object.
   
   This is backport the fix of ConvertToLocalRelation to branch 2.3.
   
   ## How was this patch tested?
   
   Added test.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] xuanyuanking commented on issue #24565: [SPARK-27665][Core] Split fetch shuffle blocks protocol from OpenBlocks

[GitBox]

xuanyuanking commented on issue #24565: [SPARK-27665][Core] Split fetch shuffle 
blocks protocol from OpenBlocks
URL: https://github.com/apache/spark/pull/24565#issuecomment-500088110
 
 
   Thanks for all your help.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #24821: [SPARK-27823][CORE] Add an abstraction layer for resource handling

[GitBox]

SparkQA commented on issue #24821: [SPARK-27823][CORE] Add an abstraction layer 
for resource handling
URL: https://github.com/apache/spark/pull/24821#issuecomment-500087718
 
 
   **[Test build #106290 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106290/testReport)**
 for PR 24821 at commit 
[`db953d9`](https://github.com/apache/spark/commit/db953d9b73df320d7527a40ef044f2ef664b505c).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

AmplabJenkins removed a comment on issue #24798: [SPARK-27724][SQL] Implement 
REPLACE TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500087536
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106288/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24821: [SPARK-27823][CORE] Add an abstraction layer for resource handling

[GitBox]

AmplabJenkins commented on issue #24821: [SPARK-27823][CORE] Add an abstraction 
layer for resource handling
URL: https://github.com/apache/spark/pull/24821#issuecomment-500087566
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

AmplabJenkins removed a comment on issue #24798: [SPARK-27724][SQL] Implement 
REPLACE TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500087534
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24821: [SPARK-27823][CORE] Add an abstraction layer for resource handling

[GitBox]

AmplabJenkins removed a comment on issue #24821: [SPARK-27823][CORE] Add an 
abstraction layer for resource handling
URL: https://github.com/apache/spark/pull/24821#issuecomment-500087567
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11535/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24821: [SPARK-27823][CORE] Add an abstraction layer for resource handling

[GitBox]

AmplabJenkins removed a comment on issue #24821: [SPARK-27823][CORE] Add an 
abstraction layer for resource handling
URL: https://github.com/apache/spark/pull/24821#issuecomment-500087566
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24821: [SPARK-27823][CORE] Add an abstraction layer for resource handling

[GitBox]

AmplabJenkins commented on issue #24821: [SPARK-27823][CORE] Add an abstraction 
layer for resource handling
URL: https://github.com/apache/spark/pull/24821#issuecomment-500087567
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11535/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

AmplabJenkins commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE 
TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500087534
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

AmplabJenkins commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE 
TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500087536
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106288/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on issue #24821: [SPARK-27823][CORE] Add an abstraction layer for resource handling

[GitBox]

dongjoon-hyun commented on issue #24821: [SPARK-27823][CORE] Add an abstraction 
layer for resource handling
URL: https://github.com/apache/spark/pull/24821#issuecomment-500087431
 
 
   Retest this please.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

SparkQA commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE 
and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500087411
 
 
   **[Test build #106288 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106288/testReport)**
 for PR 24798 at commit 
[`a9142e9`](https://github.com/apache/spark/commit/a9142e9278e61121f279d1aa7bef69ef863e3b36).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

SparkQA removed a comment on issue #24798: [SPARK-27724][SQL] Implement REPLACE 
TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500079376
 
 
   **[Test build #106288 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106288/testReport)**
 for PR 24798 at commit 
[`a9142e9`](https://github.com/apache/spark/commit/a9142e9278e61121f279d1aa7bef69ef863e3b36).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on issue #24822: [SPARK-27798][SQL][BRANCH-2.4] from_avro shouldn't produces same value when converted to local relation

[GitBox]

dongjoon-hyun commented on issue #24822: [SPARK-27798][SQL][BRANCH-2.4] 
from_avro shouldn't produces same value when converted to local relation
URL: https://github.com/apache/spark/pull/24822#issuecomment-500087299
 
 
   Thank you, @viirya !


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24335: [SPARK-27425][SQL] Add count_if function

[GitBox]

dongjoon-hyun commented on a change in pull request #24335: [SPARK-27425][SQL] 
Add count_if function
URL: https://github.com/apache/spark/pull/24335#discussion_r291791927
 
 

 ##
 File path: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/CountIf.scala
 ##
 @@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions.aggregate
+
+import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
+import org.apache.spark.sql.catalyst.expressions.{Expression, 
ExpressionDescription, ImplicitCastInputTypes, UnevaluableAggregate}
+import org.apache.spark.sql.types.{AbstractDataType, BooleanType, DataType, 
LongType}
+
+@ExpressionDescription(
+  usage = """
+_FUNC_(expr) - Returns the number of `TRUE` values for the expression.
+  This function is equivalent to `count(CASE WHEN expr THEN 1 END)`.
 
 Review comment:
   Initially, I recommended this to give a hint to the users like the other SQL 
engines. The reason why I chose this expression instead of `Count(NullIf(...))` 
which is used in this PR with `RuntimeReplaceable` is that `Count(NullIf(...))` 
doesn't work like new `count_if` due to the type casting.
   
   For the following case, `Count(NullIf(...))` works while `count_if` doesn't.
   ```sql
   spark-sql> select count(nullif(a, false)) from values (1) T(a);
   1
   
   spark-sql> select count_if(a) from values (1) T(a);
   Error in query: cannot resolve 'count_if(T.a)' due to data type mismatch: 
function count_if requires boolean type,
   
   spark-sql> select count(case when a then 1 end) from values (1) T(a);
   Error in query: cannot resolve 'CASE WHEN T.`a` THEN 1 END' due to data type 
mismatch: WHEN expressions in CaseWhen
   ```
   
   In short, new `count_if`'s behavior is the same with `count(CASE WHEN expr 
THEN 1 END)`. However, while reviewing this PR again, I notice that this might 
mislead the **developers** because we are using `count(nullif(...))` 
technically.
   
   To sum up, we cannot give the simple fallback example here. Both ones are 
inadequate. We had better remove this line. So, could you remove this line 
again, @cryeo ? Sorry, it's my bad.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24822: [SPARK-27798][SQL][BRANCH-2.4] from_avro shouldn't produces same value when converted to local relation

[GitBox]

AmplabJenkins removed a comment on issue #24822: [SPARK-27798][SQL][BRANCH-2.4] 
from_avro shouldn't produces same value when converted to local relation
URL: https://github.com/apache/spark/pull/24822#issuecomment-500086568
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24822: [SPARK-27798][SQL][BRANCH-2.4] from_avro shouldn't produces same value when converted to local relation

[GitBox]

AmplabJenkins removed a comment on issue #24822: [SPARK-27798][SQL][BRANCH-2.4] 
from_avro shouldn't produces same value when converted to local relation
URL: https://github.com/apache/spark/pull/24822#issuecomment-500086569
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11534/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #24822: [SPARK-27798][SQL][BRANCH-2.4] from_avro shouldn't produces same value when converted to local relation

[GitBox]

SparkQA commented on issue #24822: [SPARK-27798][SQL][BRANCH-2.4] from_avro 
shouldn't produces same value when converted to local relation
URL: https://github.com/apache/spark/pull/24822#issuecomment-500086702
 
 
   **[Test build #106289 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106289/testReport)**
 for PR 24822 at commit 
[`8a6e520`](https://github.com/apache/spark/commit/8a6e5208b26ab0080804c8172be99916925a6f73).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24821: [SPARK-27823][CORE] Add an abstraction layer for resource handling

[GitBox]

AmplabJenkins removed a comment on issue #24821: [SPARK-27823][CORE] Add an 
abstraction layer for resource handling
URL: https://github.com/apache/spark/pull/24821#issuecomment-500086464
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106283/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24822: [SPARK-27798][SQL][BRANCH-2.4] from_avro shouldn't produces same value when converted to local relation

[GitBox]

AmplabJenkins commented on issue #24822: [SPARK-27798][SQL][BRANCH-2.4] 
from_avro shouldn't produces same value when converted to local relation
URL: https://github.com/apache/spark/pull/24822#issuecomment-500086569
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11534/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24821: [SPARK-27823][CORE] Add an abstraction layer for resource handling

[GitBox]

AmplabJenkins removed a comment on issue #24821: [SPARK-27823][CORE] Add an 
abstraction layer for resource handling
URL: https://github.com/apache/spark/pull/24821#issuecomment-500086461
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #24821: [SPARK-27823][CORE] Add an abstraction layer for resource handling

[GitBox]

SparkQA removed a comment on issue #24821: [SPARK-27823][CORE] Add an 
abstraction layer for resource handling
URL: https://github.com/apache/spark/pull/24821#issuecomment-500018622
 
 
   **[Test build #106283 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106283/testReport)**
 for PR 24821 at commit 
[`db953d9`](https://github.com/apache/spark/commit/db953d9b73df320d7527a40ef044f2ef664b505c).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24822: [SPARK-27798][SQL][BRANCH-2.4] from_avro shouldn't produces same value when converted to local relation

[GitBox]

AmplabJenkins commented on issue #24822: [SPARK-27798][SQL][BRANCH-2.4] 
from_avro shouldn't produces same value when converted to local relation
URL: https://github.com/apache/spark/pull/24822#issuecomment-500086568
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24821: [SPARK-27823][CORE] Add an abstraction layer for resource handling

[GitBox]

AmplabJenkins commented on issue #24821: [SPARK-27823][CORE] Add an abstraction 
layer for resource handling
URL: https://github.com/apache/spark/pull/24821#issuecomment-500086464
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106283/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24821: [SPARK-27823][CORE] Add an abstraction layer for resource handling

[GitBox]

AmplabJenkins commented on issue #24821: [SPARK-27823][CORE] Add an abstraction 
layer for resource handling
URL: https://github.com/apache/spark/pull/24821#issuecomment-500086461
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #24821: [SPARK-27823][CORE] Add an abstraction layer for resource handling

[GitBox]

SparkQA commented on issue #24821: [SPARK-27823][CORE] Add an abstraction layer 
for resource handling
URL: https://github.com/apache/spark/pull/24821#issuecomment-500086410
 
 
   **[Test build #106283 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106283/testReport)**
 for PR 24821 at commit 
[`db953d9`](https://github.com/apache/spark/commit/db953d9b73df320d7527a40ef044f2ef664b505c).
* This patch **fails from timeout after a configured wait of `400m`**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] viirya opened a new pull request #24822: [SPARK-27798][SQL][BRANCH-2.4] from_avro shouldn't produces same value when converted to local relation

[GitBox]

viirya opened a new pull request #24822: [SPARK-27798][SQL][BRANCH-2.4] 
from_avro shouldn't produces same value when converted to local relation
URL: https://github.com/apache/spark/pull/24822
 
 
   ## What changes were proposed in this pull request?
   
   When using `from_avro` to deserialize avro data to catalyst StructType 
format, if `ConvertToLocalRelation` is applied at the time, `from_avro` 
produces only the last value (overriding previous values).
   
   The cause is `AvroDeserializer` reuses output row for StructType. Normally, 
it should be fine in Spark SQL. But `ConvertToLocalRelation` just uses 
`InterpretedProjection` to project local rows. `InterpretedProjection` creates 
new row for each output thro, it includes the same nested row object from 
`AvroDeserializer`. By the end, converted local relation has only last value.
   
   This is to backport the fix to branch 2.4 and uses 
`InterpretedMutableProjection` in `ConvertToLocalRelation` and call `copy()` on 
output rows.
   
   ## How was this patch tested?
   
   Added test.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] viirya commented on issue #24805: [SPARK-27798][SQL] from_avro shouldn't produces same value when converted to local relation

[GitBox]

viirya commented on issue #24805: [SPARK-27798][SQL] from_avro shouldn't 
produces same value when converted to local relation
URL: https://github.com/apache/spark/pull/24805#issuecomment-500080167
 
 
   Ok. Will do. Thanks.
   
   
   On Sat, Jun 8, 2019, 04:55 Dongjoon Hyun  wrote:
   
   > Ur, @viirya  . There was some conflicts, could
   > you make two backporting PR for branch-2.4(with AvroTest) and 
branch-2.3(without
   > Avro test)?
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > 
,
   > or mute the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] mccheah commented on a change in pull request #24807: [SPARK-27958] Stopping a SparkSession should not always stop Spark Context

[GitBox]

mccheah commented on a change in pull request #24807: [SPARK-27958] Stopping a 
SparkSession should not always stop Spark Context
URL: https://github.com/apache/spark/pull/24807#discussion_r291788697
 
 

 ##
 File path: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
 ##
 @@ -711,12 +711,15 @@ class SparkSession private(
   // scalastyle:on
 
   /**
-   * Stop the underlying `SparkContext`.
+   * Stop the underlying `SparkContext` if there are are no active sessions 
remaining.
*
* @since 2.0.0
*/
   def stop(): Unit = {
 
 Review comment:
   The idea is that if one creates a multi-tenant Spark process, and you give 
each user a Spark session, you want to be able to close down the resources for 
one session (e.g. connections to JDBC, perhaps), but not stop the entire Spark 
Context, thus keeping the Spark Context alive for the other users.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

SparkQA commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE 
and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500079376
 
 
   **[Test build #106288 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106288/testReport)**
 for PR 24798 at commit 
[`a9142e9`](https://github.com/apache/spark/commit/a9142e9278e61121f279d1aa7bef69ef863e3b36).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

AmplabJenkins removed a comment on issue #24798: [SPARK-27724][SQL] Implement 
REPLACE TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500079229
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

AmplabJenkins removed a comment on issue #24798: [SPARK-27724][SQL] Implement 
REPLACE TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500079233
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11533/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

AmplabJenkins commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE 
TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500079233
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11533/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

AmplabJenkins commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE 
TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500079229
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

AmplabJenkins removed a comment on issue #24798: [SPARK-27724][SQL] Implement 
REPLACE TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500077136
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106287/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

SparkQA removed a comment on issue #24798: [SPARK-27724][SQL] Implement REPLACE 
TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500075708
 
 
   **[Test build #106287 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106287/testReport)**
 for PR 24798 at commit 
[`89aea5e`](https://github.com/apache/spark/commit/89aea5ea3b2bfc6914ada5adf8a138a66129a824).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

AmplabJenkins removed a comment on issue #24798: [SPARK-27724][SQL] Implement 
REPLACE TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500077129
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

AmplabJenkins commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE 
TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500077136
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106287/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

AmplabJenkins commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE 
TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500077129
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

SparkQA commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE 
and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500077123
 
 
   **[Test build #106287 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106287/testReport)**
 for PR 24798 at commit 
[`89aea5e`](https://github.com/apache/spark/commit/89aea5ea3b2bfc6914ada5adf8a138a66129a824).
* This patch **fails to generate documentation**.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

AmplabJenkins removed a comment on issue #24798: [SPARK-27724][SQL] Implement 
REPLACE TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500076359
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

AmplabJenkins removed a comment on issue #24798: [SPARK-27724][SQL] Implement 
REPLACE TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500076361
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11532/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

AmplabJenkins commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE 
TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500076361
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/11532/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

AmplabJenkins commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE 
TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500076359
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

SparkQA commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE 
and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500075708
 
 
   **[Test build #106287 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106287/testReport)**
 for PR 24798 at commit 
[`89aea5e`](https://github.com/apache/spark/commit/89aea5ea3b2bfc6914ada5adf8a138a66129a824).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] mccheah commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

mccheah commented on issue #24798: [SPARK-27724][SQL] Implement REPLACE TABLE 
and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#issuecomment-500075726
 
 
   Ok I addressed all the comments. Let me know if there's anything else to 
address.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] mccheah commented on a change in pull request #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

mccheah commented on a change in pull request #24798: [SPARK-27724][SQL] 
Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#discussion_r291785893
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2SQLSuite.scala
 ##
 @@ -170,6 +173,85 @@ class DataSourceV2SQLSuite extends QueryTest with 
SharedSQLContext with BeforeAn
 checkAnswer(spark.internalCreateDataFrame(rdd, table.schema), 
spark.table("source"))
   }
 
+  test("ReplaceTableAsSelect: basic v2 implementation using atomic catalog.") {
+spark.sql("CREATE TABLE testcatatomic.table_name USING foo AS SELECT id, 
data FROM source")
+val testCatalog = spark.catalog("testcatatomic").asTableCatalog
+val originalTable = testCatalog.loadTable(Identifier.of(Array(), 
"table_name"))
+spark.sql("REPLACE TABLE testcatatomic.table_name USING foo AS SELECT id 
FROM source")
+val replacedTable = testCatalog.loadTable(Identifier.of(Array(), 
"table_name"))
+assert(replacedTable != originalTable, "Table should have been replaced.")
+assert(replacedTable.name == "testcatatomic.table_name")
+assert(replacedTable.partitioning.isEmpty)
+assert(replacedTable.properties == Map("provider" -> "foo").asJava)
+assert(replacedTable.schema == new StructType()
+  .add("id", LongType, nullable = false))
+val rdd = 
spark.sparkContext.parallelize(replacedTable.asInstanceOf[InMemoryTable].rows)
+checkAnswer(
+spark.internalCreateDataFrame(rdd, replacedTable.schema),
+spark.table("source").select("id"))
+  }
 
 Review comment:
   I modified some of the success cases, don't know there are more that need to 
be adjusted. Think we don't have to be completely exhaustive here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions

[GitBox]

AmplabJenkins removed a comment on issue #24749: [SPARK-27890][SQL] Improve SQL 
parser error message for character-only identifier with hyphens except those in 
expressions
URL: https://github.com/apache/spark/pull/24749#issuecomment-500075533
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions

[GitBox]

AmplabJenkins removed a comment on issue #24749: [SPARK-27890][SQL] Improve SQL 
parser error message for character-only identifier with hyphens except those in 
expressions
URL: https://github.com/apache/spark/pull/24749#issuecomment-500075536
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106286/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] mccheah commented on a change in pull request #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

mccheah commented on a change in pull request #24798: [SPARK-27724][SQL] 
Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#discussion_r291785852
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2SQLSuite.scala
 ##
 @@ -170,6 +173,85 @@ class DataSourceV2SQLSuite extends QueryTest with 
SharedSQLContext with BeforeAn
 checkAnswer(spark.internalCreateDataFrame(rdd, table.schema), 
spark.table("source"))
   }
 
+  test("ReplaceTableAsSelect: basic v2 implementation using atomic catalog.") {
+spark.sql("CREATE TABLE testcatatomic.table_name USING foo AS SELECT id, 
data FROM source")
+val testCatalog = spark.catalog("testcatatomic").asTableCatalog
+val originalTable = testCatalog.loadTable(Identifier.of(Array(), 
"table_name"))
+spark.sql("REPLACE TABLE testcatatomic.table_name USING foo AS SELECT id 
FROM source")
+val replacedTable = testCatalog.loadTable(Identifier.of(Array(), 
"table_name"))
+assert(replacedTable != originalTable, "Table should have been replaced.")
+assert(replacedTable.name == "testcatatomic.table_name")
+assert(replacedTable.partitioning.isEmpty)
+assert(replacedTable.properties == Map("provider" -> "foo").asJava)
+assert(replacedTable.schema == new StructType()
+  .add("id", LongType, nullable = false))
+val rdd = 
spark.sparkContext.parallelize(replacedTable.asInstanceOf[InMemoryTable].rows)
+checkAnswer(
+spark.internalCreateDataFrame(rdd, replacedTable.schema),
+spark.table("source").select("id"))
+  }
+
+  test("ReplaceTableAsSelect: Non-atomic catalog creates the empty table, but 
leaves the" +
+" table empty if the write fails.") {
+spark.sql("CREATE TABLE testcat.table_name USING foo AS SELECT id, data 
FROM source")
+val testCatalog = spark.catalog("testcat").asTableCatalog
+val table = testCatalog.loadTable(Identifier.of(Array(), "table_name"))
+assert(table.asInstanceOf[InMemoryTable].rows.nonEmpty)
+intercept[Exception] {
+  spark.sql("REPLACE TABLE testcat.table_name" +
+s" USING foo OPTIONS 
(`${TestInMemoryTableCatalog.SIMULATE_FAILED_WRITE_OPTION}`=true)" +
+s" AS SELECT id FROM source")
+}
+val replacedTable = testCatalog.loadTable(Identifier.of(Array(), 
"table_name"))
+assert(replacedTable != table, "Table should have been replaced.")
+assert(replacedTable.asInstanceOf[InMemoryTable].rows.isEmpty,
+"Rows should not have been successfully written to the replaced 
table.")
+  }
+
+  test("ReplaceTableAsSelect: Non-atomic catalog drops the table permanently 
if the" +
+" subsequent table creation fails.") {
+spark.sql("CREATE TABLE testcat.table_name USING foo AS SELECT id, data 
FROM source")
+val testCatalog = spark.catalog("testcat").asTableCatalog
+val table = testCatalog.loadTable(Identifier.of(Array(), "table_name"))
+assert(table.asInstanceOf[InMemoryTable].rows.nonEmpty)
+intercept[Exception] {
+  spark.sql("REPLACE TABLE testcat.table_name" +
+s" USING foo" +
+s" TBLPROPERTIES 
(`${TestInMemoryTableCatalog.SIMULATE_FAILED_CREATE_PROPERTY}`=true)" +
+s" AS SELECT id FROM source")
+}
+assert(!testCatalog.tableExists(Identifier.of(Array(), "table_name")),
+  "Table should have been dropped and failed to be created.")
+  }
+
+  test("ReplaceTableAsSelect: Atomic catalog does not drop the table when 
replace fails.") {
+spark.sql("CREATE TABLE testcatatomic.table_name USING foo AS SELECT id, 
data FROM source")
+val testCatalog = spark.catalog("testcatatomic").asTableCatalog
+val table = testCatalog.loadTable(Identifier.of(Array(), "table_name"))
+
+assert(table.name == "testcatatomic.table_name")
+assert(table.partitioning.isEmpty)
+assert(table.properties == Map("provider" -> "foo").asJava)
+assert(table.schema == new StructType()
+  .add("id", LongType, nullable = false)
+  .add("data", StringType))
+intercept[Exception] {
+  spark.sql("REPLACE TABLE testcatatomic.table_name" +
+s" USING foo OPTIONS 
(`${TestInMemoryTableCatalog.SIMULATE_FAILED_WRITE_OPTION}=true)" +
+s" AS SELECT id FROM source")
+}
+var maybeReplacedTable: Table = 
testCatalog.loadTable(Identifier.of(Array(), "table_name"))
+assert(maybeReplacedTable === table, "Table should not have changed.")
+intercept[Exception] {
+  spark.sql("REPLACE TABLE testcatatomic.table_name" +
+s" USING foo" +
+s" TBLPROPERTIES 
(`${TestInMemoryTableCatalog.SIMULATE_FAILED_CREATE_PROPERTY}`=true)" +
+s" AS SELECT id FROM source")
+}
+maybeReplacedTable = testCatalog.loadTable(Identifier.of(Array(), 
"table_name"))
+assert(maybeReplacedTable === table, "Table should not have changed.")
+  }
+
 
 Review comment:
   I added test cases.

[GitHub] [spark] AmplabJenkins commented on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions

[GitBox]

AmplabJenkins commented on issue #24749: [SPARK-27890][SQL] Improve SQL parser 
error message for character-only identifier with hyphens except those in 
expressions
URL: https://github.com/apache/spark/pull/24749#issuecomment-500075536
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106286/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions

[GitBox]

AmplabJenkins commented on issue #24749: [SPARK-27890][SQL] Improve SQL parser 
error message for character-only identifier with hyphens except those in 
expressions
URL: https://github.com/apache/spark/pull/24749#issuecomment-500075533
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions

[GitBox]

SparkQA removed a comment on issue #24749: [SPARK-27890][SQL] Improve SQL 
parser error message for character-only identifier with hyphens except those in 
expressions
URL: https://github.com/apache/spark/pull/24749#issuecomment-500041380
 
 
   **[Test build #106286 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106286/testReport)**
 for PR 24749 at commit 
[`32fe31c`](https://github.com/apache/spark/commit/32fe31ce5640f0078877d55a72e662279d313374).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #24749: [SPARK-27890][SQL] Improve SQL parser error message for character-only identifier with hyphens except those in expressions

[GitBox]

SparkQA commented on issue #24749: [SPARK-27890][SQL] Improve SQL parser error 
message for character-only identifier with hyphens except those in expressions
URL: https://github.com/apache/spark/pull/24749#issuecomment-500075350
 
 
   **[Test build #106286 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106286/testReport)**
 for PR 24749 at commit 
[`32fe31c`](https://github.com/apache/spark/commit/32fe31ce5640f0078877d55a72e662279d313374).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24335: [SPARK-27425][SQL] Add count_if function

[GitBox]

AmplabJenkins removed a comment on issue #24335: [SPARK-27425][SQL] Add 
count_if function
URL: https://github.com/apache/spark/pull/24335#issuecomment-500073630
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106284/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24335: [SPARK-27425][SQL] Add count_if function

[GitBox]

AmplabJenkins removed a comment on issue #24335: [SPARK-27425][SQL] Add 
count_if function
URL: https://github.com/apache/spark/pull/24335#issuecomment-500073627
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24335: [SPARK-27425][SQL] Add count_if function

[GitBox]

AmplabJenkins commented on issue #24335: [SPARK-27425][SQL] Add count_if 
function
URL: https://github.com/apache/spark/pull/24335#issuecomment-500073630
 
 
   Test PASSed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106284/
   Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on issue #24335: [SPARK-27425][SQL] Add count_if function

[GitBox]

AmplabJenkins commented on issue #24335: [SPARK-27425][SQL] Add count_if 
function
URL: https://github.com/apache/spark/pull/24335#issuecomment-500073627
 
 
   Merged build finished. Test PASSed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA commented on issue #24335: [SPARK-27425][SQL] Add count_if function

[GitBox]

SparkQA commented on issue #24335: [SPARK-27425][SQL] Add count_if function
URL: https://github.com/apache/spark/pull/24335#issuecomment-500073401
 
 
   **[Test build #106284 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106284/testReport)**
 for PR 24335 at commit 
[`2f4d64e`](https://github.com/apache/spark/commit/2f4d64e9f96e3f03ff59ab4e1f8c3e53a4124c98).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on issue #24335: [SPARK-27425][SQL] Add count_if function

[GitBox]

SparkQA removed a comment on issue #24335: [SPARK-27425][SQL] Add count_if 
function
URL: https://github.com/apache/spark/pull/24335#issuecomment-500036976
 
 
   **[Test build #106284 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/106284/testReport)**
 for PR 24335 at commit 
[`2f4d64e`](https://github.com/apache/spark/commit/2f4d64e9f96e3f03ff59ab4e1f8c3e53a4124c98).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] mccheah commented on a change in pull request #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

mccheah commented on a change in pull request #24798: [SPARK-27724][SQL] 
Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#discussion_r291781039
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2SQLSuite.scala
 ##
 @@ -170,6 +173,85 @@ class DataSourceV2SQLSuite extends QueryTest with 
SharedSQLContext with BeforeAn
 checkAnswer(spark.internalCreateDataFrame(rdd, table.schema), 
spark.table("source"))
   }
 
+  test("ReplaceTableAsSelect: basic v2 implementation using atomic catalog.") {
+spark.sql("CREATE TABLE testcatatomic.table_name USING foo AS SELECT id, 
data FROM source")
+val testCatalog = spark.catalog("testcatatomic").asTableCatalog
+val originalTable = testCatalog.loadTable(Identifier.of(Array(), 
"table_name"))
+spark.sql("REPLACE TABLE testcatatomic.table_name USING foo AS SELECT id 
FROM source")
+val replacedTable = testCatalog.loadTable(Identifier.of(Array(), 
"table_name"))
+assert(replacedTable != originalTable, "Table should have been replaced.")
+assert(replacedTable.name == "testcatatomic.table_name")
+assert(replacedTable.partitioning.isEmpty)
+assert(replacedTable.properties == Map("provider" -> "foo").asJava)
+assert(replacedTable.schema == new StructType()
+  .add("id", LongType, nullable = false))
+val rdd = 
spark.sparkContext.parallelize(replacedTable.asInstanceOf[InMemoryTable].rows)
+checkAnswer(
+spark.internalCreateDataFrame(rdd, replacedTable.schema),
+spark.table("source").select("id"))
+  }
+
+  test("ReplaceTableAsSelect: Non-atomic catalog creates the empty table, but 
leaves the" +
+" table empty if the write fails.") {
 
 Review comment:
   Hm so I took a closer look at this. It turns out that using 
`Utils.tryWithSafeFinallyAndFailureCallbacks` is risky in all of these code 
paths as it is currently implemented.
   
   That method, when it tries to run the catch block, will first try to set the 
failure reason on the task context via `TaskContext.get().markTaskFailed`. But 
since we're running this try...finally block on the _driver_, there is no such 
task context to get via `TaskContext.get`.
   
   What happens in this case then is that this test passes when it should fail, 
because indeed, the table should be dropped. But the catch block that drops the 
table never gets run, because `TaskContext.get().markTaskFailed` NPEs before 
the catch block can be run.
   
   I think there's a few ways forward:
   1) Don't use the Utils method to do try-catch-finally
   2) Patch the Utils method to check for null on the current task context 
before trying to mark the task failure reason on it.
   
   I'm going with 2) for now, but 1) is very reasonable as well.
   
   Either way, yeah the table should end up being dropped at the end, so this 
test also has to be patched.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] mccheah commented on a change in pull request #24798: [SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

[GitBox]

mccheah commented on a change in pull request #24798: [SPARK-27724][SQL] 
Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2
URL: https://github.com/apache/spark/pull/24798#discussion_r291781039
 
 

 ##
 File path: 
sql/core/src/test/scala/org/apache/spark/sql/sources/v2/DataSourceV2SQLSuite.scala
 ##
 @@ -170,6 +173,85 @@ class DataSourceV2SQLSuite extends QueryTest with 
SharedSQLContext with BeforeAn
 checkAnswer(spark.internalCreateDataFrame(rdd, table.schema), 
spark.table("source"))
   }
 
+  test("ReplaceTableAsSelect: basic v2 implementation using atomic catalog.") {
+spark.sql("CREATE TABLE testcatatomic.table_name USING foo AS SELECT id, 
data FROM source")
+val testCatalog = spark.catalog("testcatatomic").asTableCatalog
+val originalTable = testCatalog.loadTable(Identifier.of(Array(), 
"table_name"))
+spark.sql("REPLACE TABLE testcatatomic.table_name USING foo AS SELECT id 
FROM source")
+val replacedTable = testCatalog.loadTable(Identifier.of(Array(), 
"table_name"))
+assert(replacedTable != originalTable, "Table should have been replaced.")
+assert(replacedTable.name == "testcatatomic.table_name")
+assert(replacedTable.partitioning.isEmpty)
+assert(replacedTable.properties == Map("provider" -> "foo").asJava)
+assert(replacedTable.schema == new StructType()
+  .add("id", LongType, nullable = false))
+val rdd = 
spark.sparkContext.parallelize(replacedTable.asInstanceOf[InMemoryTable].rows)
+checkAnswer(
+spark.internalCreateDataFrame(rdd, replacedTable.schema),
+spark.table("source").select("id"))
+  }
+
+  test("ReplaceTableAsSelect: Non-atomic catalog creates the empty table, but 
leaves the" +
+" table empty if the write fails.") {
 
 Review comment:
   Hm so I took a closer look at this. It turns out that using 
`Utils.tryWithSafeFinallyAndFailureCallbacks` is risky in all of these code 
paths as it is currently implemented.
   
   That method, when it tries to run the catch block, will first try to set the 
failure reason on the task context via TaskContext.get().markTaskFailed`. But 
since we're running this try...finally block on the _driver_, there is no such 
task context to get via `TaskContext.get`.
   
   What happens in this case then is that this test passes when it should fail, 
because indeed, the table should be dropped. But the catch block that drops the 
table never gets run, because `TaskContext.get().markTaskFailed` NPEs before 
the catch block can be run.
   
   I think there's a few ways forward:
   1) Don't use the Utils method to do try-catch-finally
   2) Patch the Utils method to check for null on the current task context 
before trying to mark the task failure reason on it.
   
   I'm going with 2) for now, but 1) is very reasonable as well.
   
   Either way, yeah the table should end up being dropped at the end, so this 
test also has to be patched.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24817: [WIP][SPARK-27963][core] Allow dynamic allocation without a shuffle service.

[GitBox]

AmplabJenkins removed a comment on issue #24817: [WIP][SPARK-27963][core] Allow 
dynamic allocation without a shuffle service.
URL: https://github.com/apache/spark/pull/24817#issuecomment-500061293
 
 
   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/106285/
   Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on issue #24817: [WIP][SPARK-27963][core] Allow dynamic allocation without a shuffle service.

[GitBox]

AmplabJenkins removed a comment on issue #24817: [WIP][SPARK-27963][core] Allow 
dynamic allocation without a shuffle service.
URL: https://github.com/apache/spark/pull/24817#issuecomment-500061289
 
 
   Merged build finished. Test FAILed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org