[GitHub] [spark] AmplabJenkins removed a comment on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec
AmplabJenkins removed a comment on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec URL: https://github.com/apache/spark/pull/27019#issuecomment-586862023 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec
AmplabJenkins removed a comment on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec URL: https://github.com/apache/spark/pull/27019#issuecomment-586862032 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118519/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec
AmplabJenkins commented on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec URL: https://github.com/apache/spark/pull/27019#issuecomment-586862023 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec
AmplabJenkins commented on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec URL: https://github.com/apache/spark/pull/27019#issuecomment-586862032 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118519/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #27519: [SPARK-30770][ML][WIP] avoid vector conversion in GMM.transform
zhengruifeng commented on a change in pull request #27519:
[SPARK-30770][ML][WIP] avoid vector conversion in GMM.transform
URL: https://github.com/apache/spark/pull/27519#discussion_r376937067
##
File path:
mllib-local/src/main/scala/org/apache/spark/ml/stat/distribution/MultivariateGaussian.scala
##
@@ -43,46 +43,40 @@ class MultivariateGaussian @Since("2.0.0") (
require(cov.numCols == cov.numRows, "Covariance matrix must be square")
require(mean.size == cov.numCols, "Mean vector length must match covariance
matrix size")
- /** Private constructor taking Breeze types */
- private[ml] def this(mean: BDV[Double], cov: BDM[Double]) = {
-this(Vectors.fromBreeze(mean), Matrices.fromBreeze(cov))
- }
-
- @transient private lazy val breezeMu = mean.asBreeze.toDenseVector
-
/**
* Compute distribution dependent constants:
*rootSigmaInv = D^(-1/2)^ * U.t, where sigma = U * D * U.t
*u = log((2*pi)^(-k/2)^ * det(sigma)^(-1/2)^)
*/
- @transient private lazy val (rootSigmaInv: BDM[Double], u: Double) =
calculateCovarianceConstants
+ @transient private lazy val tuple3 = {
Review comment:
it is said in
[LeastSquaresAggregator](https://github.com/apache/spark/blob/12e1bbaddbb2ef304b5880a62df6683fcc94ea54/mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/LeastSquaresAggregator.scala#L188)
that
> // do not use tuple assignment above because it will circumvent the
@transient tag
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec
SparkQA removed a comment on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec URL: https://github.com/apache/spark/pull/27019#issuecomment-586795791 **[Test build #118519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118519/testReport)** for PR 27019 at commit [`18b865f`](https://github.com/apache/spark/commit/18b865f79ebefafedd0dea8716a7faa8794116d3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior
cloud-fan commented on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior URL: https://github.com/apache/spark/pull/27478#issuecomment-586861735 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec
SparkQA commented on issue #27019: [SPARK-30027][SQL] Support codegen for aggregate filters in HashAggregateExec URL: https://github.com/apache/spark/pull/27019#issuecomment-586861197 **[Test build #118519 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118519/testReport)** for PR 27019 at commit [`18b865f`](https://github.com/apache/spark/commit/18b865f79ebefafedd0dea8716a7faa8794116d3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586860312 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23303/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586859959 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118543/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586860302 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586860302 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior
AmplabJenkins removed a comment on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior URL: https://github.com/apache/spark/pull/27478#issuecomment-586859661 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118547/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586859947 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586860312 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23303/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
SparkQA removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586846187 **[Test build #118543 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118543/testReport)** for PR 27552 at commit [`e491f96`](https://github.com/apache/spark/commit/e491f966b23a734358b222ed8486a547467a22ed). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586859947 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586859959 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118543/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586859902 **[Test build #118543 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118543/testReport)** for PR 27552 at commit [`e491f96`](https://github.com/apache/spark/commit/e491f966b23a734358b222ed8486a547467a22ed). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior
SparkQA removed a comment on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior URL: https://github.com/apache/spark/pull/27478#issuecomment-586854920 **[Test build #118547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118547/testReport)** for PR 27478 at commit [`feb8c0a`](https://github.com/apache/spark/commit/feb8c0aa28034c166f6413bd1eec58a885848192). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior
AmplabJenkins removed a comment on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior URL: https://github.com/apache/spark/pull/27478#issuecomment-586859653 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586859868 **[Test build #118549 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118549/testReport)** for PR 27552 at commit [`ce77628`](https://github.com/apache/spark/commit/ce77628cad487d7d86cb33577a699ac37bf8f5fd). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #27590: [SPARK-30703][SQL][DOCS][FollowUp] Declare the ANSI SQL compliance options as experimental
gengliangwang commented on a change in pull request #27590: [SPARK-30703][SQL][DOCS][FollowUp] Declare the ANSI SQL compliance options as experimental URL: https://github.com/apache/spark/pull/27590#discussion_r380025466 ## File path: docs/sql-ref-ansi-compliance.md ## @@ -19,19 +19,21 @@ license: | limitations under the License. --- -Spark SQL has two options to comply with the SQL standard: `spark.sql.ansi.enabled` and `spark.sql.storeAssignmentPolicy` (See a table below for details). +Since Spark 3.0, Spark SQL introduces two experimental options to comply with the SQL standard: `spark.sql.ansi.enabled` and `spark.sql.storeAssignmentPolicy` (See a table below for details). + When `spark.sql.ansi.enabled` is set to `true`, Spark SQL follows the standard in basic behaviours (e.g., arithmetic operations, type conversion, and SQL parsing). Moreover, Spark SQL has an independent option to control implicit casting behaviours when inserting rows in a table. The casting behaviours are defined as store assignment rules in the standard. -When `spark.sql.storeAssignmentPolicy` is set to `ANSI`, Spark SQL complies with the ANSI store assignment rules. + +When `spark.sql.storeAssignmentPolicy` is set to `ANSI`, Spark SQL complies with the ANSI store assignment rules. This is a separate configuration because its default value is `ANSI`, while the configuration `spark.sql.ansi.enabled` is disabled by default. Review comment: Separate can be an adjective This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior
SparkQA commented on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior URL: https://github.com/apache/spark/pull/27478#issuecomment-586859630 **[Test build #118547 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118547/testReport)** for PR 27478 at commit [`feb8c0a`](https://github.com/apache/spark/commit/feb8c0aa28034c166f6413bd1eec58a885848192). * This patch **fails to generate documentation**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior
AmplabJenkins commented on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior URL: https://github.com/apache/spark/pull/27478#issuecomment-586859661 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118547/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior
AmplabJenkins commented on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior URL: https://github.com/apache/spark/pull/27478#issuecomment-586859653 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26193: [SPARK-25065][k8s] Allow setting up correct logging configuration on driver and executor.
AmplabJenkins removed a comment on issue #26193: [SPARK-25065][k8s] Allow setting up correct logging configuration on driver and executor. URL: https://github.com/apache/spark/pull/26193#issuecomment-586858842 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23294/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27489: [SPARK-30703][SQL][DOCS] Add a document for the ANSI mode
cloud-fan commented on a change in pull request #27489: [SPARK-30703][SQL][DOCS] Add a document for the ANSI mode URL: https://github.com/apache/spark/pull/27489#discussion_r380022536 ## File path: docs/sql-ref-ansi-compliance.md ## @@ -19,6 +19,127 @@ license: | limitations under the License. --- +Spark SQL has two options to comply with the SQL standard: `spark.sql.ansi.enabled` and `spark.sql.storeAssignmentPolicy` (See a table below for details). +When `spark.sql.ansi.enabled` is set to `true`, Spark SQL follows the standard in basic behaviours (e.g., arithmetic operations, type conversion, and SQL parsing). +Moreover, Spark SQL has an independent option to control implicit casting behaviours when inserting rows in a table. +The casting behaviours are defined as store assignment rules in the standard. +When `spark.sql.storeAssignmentPolicy` is set to `ANSI`, Spark SQL complies with the ANSI store assignment rules. + + +Property NameDefaultMeaning + + spark.sql.ansi.enabled + false + +When true, Spark tries to conform to the ANSI SQL specification: +1. Spark will throw a runtime exception if an overflow occurs in any operation on integral/decimal field. +2. Spark will forbid using the reserved keywords of ANSI SQL as identifiers in the SQL parser. + + + + spark.sql.storeAssignmentPolicy + ANSI + +When inserting a value into a column with different data type, Spark will perform type coercion. +Currently, we support 3 policies for the type coercion rules: ANSI, legacy and strict. With ANSI policy, +Spark performs the type coercion as per ANSI SQL. In practice, the behavior is mostly the same as PostgreSQL. +It disallows certain unreasonable type conversions such as converting string to int or double to boolean. +With legacy policy, Spark allows the type coercion as long as it is a valid Cast, which is very loose. +e.g. converting string to int or double to boolean is allowed. +It is also the only behavior in Spark 2.x and it is compatible with Hive. +With strict policy, Spark doesn't allow any possible precision loss or data truncation in type coercion, +e.g. converting double to int or decimal to double is not allowed. + + + + +The following subsections present behaviour changes in arithmetic operations, type conversions, and SQL parsing when the ANSI mode enabled. + +### Arithmetic Operations + +In Spark SQL, arithmetic operations performed on numeric types (with the exception of decimal) are not checked for overflows by default. +This means that in case an operation causes overflows, the result is the same that the same operation returns in a Java/Scala program (e.g., if the sum of 2 integers is higher than the maximum value representable, the result is a negative number). Review comment: `the result is the same that` -> `the result is the same with` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27590: [SPARK-30703][SQL][DOCS][FollowUp] Declare the ANSI SQL compliance options as experimental
cloud-fan commented on a change in pull request #27590: [SPARK-30703][SQL][DOCS][FollowUp] Declare the ANSI SQL compliance options as experimental URL: https://github.com/apache/spark/pull/27590#discussion_r380025070 ## File path: docs/sql-ref-ansi-compliance.md ## @@ -19,19 +19,21 @@ license: | limitations under the License. --- -Spark SQL has two options to comply with the SQL standard: `spark.sql.ansi.enabled` and `spark.sql.storeAssignmentPolicy` (See a table below for details). +Since Spark 3.0, Spark SQL introduces two experimental options to comply with the SQL standard: `spark.sql.ansi.enabled` and `spark.sql.storeAssignmentPolicy` (See a table below for details). + When `spark.sql.ansi.enabled` is set to `true`, Spark SQL follows the standard in basic behaviours (e.g., arithmetic operations, type conversion, and SQL parsing). Moreover, Spark SQL has an independent option to control implicit casting behaviours when inserting rows in a table. The casting behaviours are defined as store assignment rules in the standard. -When `spark.sql.storeAssignmentPolicy` is set to `ANSI`, Spark SQL complies with the ANSI store assignment rules. + +When `spark.sql.storeAssignmentPolicy` is set to `ANSI`, Spark SQL complies with the ANSI store assignment rules. This is a separate configuration because its default value is `ANSI`, while the configuration `spark.sql.ansi.enabled` is disabled by default. Review comment: `separate` -> `separated`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26193: [SPARK-25065][k8s] Allow setting up correct logging configuration on driver and executor.
AmplabJenkins removed a comment on issue #26193: [SPARK-25065][k8s] Allow setting up correct logging configuration on driver and executor. URL: https://github.com/apache/spark/pull/26193#issuecomment-586858834 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27489: [SPARK-30703][SQL][DOCS] Add a document for the ANSI mode
cloud-fan commented on a change in pull request #27489: [SPARK-30703][SQL][DOCS] Add a document for the ANSI mode URL: https://github.com/apache/spark/pull/27489#discussion_r380023202 ## File path: docs/sql-ref-ansi-compliance.md ## @@ -19,6 +19,127 @@ license: | limitations under the License. --- +Spark SQL has two options to comply with the SQL standard: `spark.sql.ansi.enabled` and `spark.sql.storeAssignmentPolicy` (See a table below for details). +When `spark.sql.ansi.enabled` is set to `true`, Spark SQL follows the standard in basic behaviours (e.g., arithmetic operations, type conversion, and SQL parsing). +Moreover, Spark SQL has an independent option to control implicit casting behaviours when inserting rows in a table. +The casting behaviours are defined as store assignment rules in the standard. +When `spark.sql.storeAssignmentPolicy` is set to `ANSI`, Spark SQL complies with the ANSI store assignment rules. + + +Property NameDefaultMeaning + + spark.sql.ansi.enabled + false + +When true, Spark tries to conform to the ANSI SQL specification: +1. Spark will throw a runtime exception if an overflow occurs in any operation on integral/decimal field. +2. Spark will forbid using the reserved keywords of ANSI SQL as identifiers in the SQL parser. + + + + spark.sql.storeAssignmentPolicy + ANSI + +When inserting a value into a column with different data type, Spark will perform type coercion. +Currently, we support 3 policies for the type coercion rules: ANSI, legacy and strict. With ANSI policy, +Spark performs the type coercion as per ANSI SQL. In practice, the behavior is mostly the same as PostgreSQL. +It disallows certain unreasonable type conversions such as converting string to int or double to boolean. +With legacy policy, Spark allows the type coercion as long as it is a valid Cast, which is very loose. +e.g. converting string to int or double to boolean is allowed. +It is also the only behavior in Spark 2.x and it is compatible with Hive. +With strict policy, Spark doesn't allow any possible precision loss or data truncation in type coercion, Review comment: I'm wondering if we should remove the STRICT mode. It's not ANSI compliant and no other SQL system has this behavior. cc @rdblue @brkyvz @rxin This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26193: [SPARK-25065][k8s] Allow setting up correct logging configuration on driver and executor.
AmplabJenkins commented on issue #26193: [SPARK-25065][k8s] Allow setting up correct logging configuration on driver and executor. URL: https://github.com/apache/spark/pull/26193#issuecomment-586858834 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26193: [SPARK-25065][k8s] Allow setting up correct logging configuration on driver and executor.
AmplabJenkins commented on issue #26193: [SPARK-25065][k8s] Allow setting up correct logging configuration on driver and executor. URL: https://github.com/apache/spark/pull/26193#issuecomment-586858842 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23294/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
AmplabJenkins removed a comment on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-586858438 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
AmplabJenkins removed a comment on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-586858445 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118520/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
AmplabJenkins commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-586858445 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118520/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
AmplabJenkins commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-586858438 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26193: [SPARK-25065][k8s] Allow setting up correct logging configuration on driver and executor.
SparkQA commented on issue #26193: [SPARK-25065][k8s] Allow setting up correct logging configuration on driver and executor. URL: https://github.com/apache/spark/pull/26193#issuecomment-586858061 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23294/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #27519: [SPARK-30770][ML][WIP] avoid vector conversion in GMM.transform
zhengruifeng commented on issue #27519: [SPARK-30770][ML][WIP] avoid vector conversion in GMM.transform URL: https://github.com/apache/spark/pull/27519#issuecomment-586858044 The pytest issue happened in https://github.com/apache/spark/pull/26735 recur, GMM is too sensitive to the dataset used in pytest. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
SparkQA removed a comment on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-586795803 **[Test build #118520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118520/testReport)** for PR 26918 at commit [`8cded00`](https://github.com/apache/spark/commit/8cded00b31ed5522aa5ed81ee36381e7b9593584). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27600: [MINOR][ML] ML cleanup
AmplabJenkins commented on issue #27600: [MINOR][ML] ML cleanup URL: https://github.com/apache/spark/pull/27600#issuecomment-586857725 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27600: [MINOR][ML] ML cleanup
AmplabJenkins removed a comment on issue #27600: [MINOR][ML] ML cleanup URL: https://github.com/apache/spark/pull/27600#issuecomment-586857734 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118535/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27600: [MINOR][ML] ML cleanup
AmplabJenkins removed a comment on issue #27600: [MINOR][ML] ML cleanup URL: https://github.com/apache/spark/pull/27600#issuecomment-586857725 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID
SparkQA commented on issue #26918: [SPARK-30279][SQL] Support 32 or more grouping attributes for GROUPING_ID URL: https://github.com/apache/spark/pull/26918#issuecomment-586857623 **[Test build #118520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118520/testReport)** for PR 26918 at commit [`8cded00`](https://github.com/apache/spark/commit/8cded00b31ed5522aa5ed81ee36381e7b9593584). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27600: [MINOR][ML] ML cleanup
SparkQA removed a comment on issue #27600: [MINOR][ML] ML cleanup URL: https://github.com/apache/spark/pull/27600#issuecomment-586833775 **[Test build #118535 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118535/testReport)** for PR 27600 at commit [`54d6fc3`](https://github.com/apache/spark/commit/54d6fc34af7e5f43465c3a2ed77a136dead3c824). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27600: [MINOR][ML] ML cleanup
AmplabJenkins commented on issue #27600: [MINOR][ML] ML cleanup URL: https://github.com/apache/spark/pull/27600#issuecomment-586857734 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118535/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27600: [MINOR][ML] ML cleanup
SparkQA commented on issue #27600: [MINOR][ML] ML cleanup URL: https://github.com/apache/spark/pull/27600#issuecomment-586857344 **[Test build #118535 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118535/testReport)** for PR 27600 at commit [`54d6fc3`](https://github.com/apache/spark/commit/54d6fc34af7e5f43465c3a2ed77a136dead3c824). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest
AmplabJenkins removed a comment on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest URL: https://github.com/apache/spark/pull/27461#issuecomment-586855366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23302/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest
AmplabJenkins removed a comment on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest URL: https://github.com/apache/spark/pull/27461#issuecomment-586855358 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest
AmplabJenkins commented on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest URL: https://github.com/apache/spark/pull/27461#issuecomment-586855358 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest
AmplabJenkins commented on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest URL: https://github.com/apache/spark/pull/27461#issuecomment-586855366 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23302/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27539: [SPARK-30786] [CORE] Fix Block replication failure propogation issue in BlockManager
AmplabJenkins removed a comment on issue #27539: [SPARK-30786] [CORE] Fix Block replication failure propogation issue in BlockManager URL: https://github.com/apache/spark/pull/27539#issuecomment-586854938 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118530/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27539: [SPARK-30786] [CORE] Fix Block replication failure propogation issue in BlockManager
AmplabJenkins removed a comment on issue #27539: [SPARK-30786] [CORE] Fix Block replication failure propogation issue in BlockManager URL: https://github.com/apache/spark/pull/27539#issuecomment-586854935 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27539: [SPARK-30786] [CORE] Fix Block replication failure propogation issue in BlockManager
SparkQA removed a comment on issue #27539: [SPARK-30786] [CORE] Fix Block replication failure propogation issue in BlockManager URL: https://github.com/apache/spark/pull/27539#issuecomment-586817813 **[Test build #118530 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118530/testReport)** for PR 27539 at commit [`4857401`](https://github.com/apache/spark/commit/48574017b43073c58d718d8f7f2c3d8e05d2fa24). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27584: [SPARK-30814][SQL] ALTER TABLE ... ADD COLUMN position should be able to reference columns being added
cloud-fan commented on a change in pull request #27584: [SPARK-30814][SQL]
ALTER TABLE ... ADD COLUMN position should be able to reference columns being
added
URL: https://github.com/apache/spark/pull/27584#discussion_r380020580
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##
@@ -483,8 +484,16 @@ trait CheckAnalysis extends PredicateHelper {
checkColumnNotExists("add", add.fieldNames(), table.schema)
}
val parent = findParentStruct("add", add.fieldNames())
-positionArgumentExists(add.position(), parent)
+add.position match {
+ case after: After =>
+// Handle the case where column position is referencing
new columns being added.
+if (!colsToAdd.contains(add.fieldNames().init :+
after.column)) {
+ positionArgumentExists(add.position(), parent)
Review comment:
just want to make sure all the caller side of `positionArgumentExists` take
care of newly added columns.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27584: [SPARK-30814][SQL] ALTER TABLE ... ADD COLUMN position should be able to reference columns being added
cloud-fan commented on a change in pull request #27584: [SPARK-30814][SQL]
ALTER TABLE ... ADD COLUMN position should be able to reference columns being
added
URL: https://github.com/apache/spark/pull/27584#discussion_r380020469
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
##
@@ -483,8 +484,16 @@ trait CheckAnalysis extends PredicateHelper {
checkColumnNotExists("add", add.fieldNames(), table.schema)
}
val parent = findParentStruct("add", add.fieldNames())
-positionArgumentExists(add.position(), parent)
+add.position match {
+ case after: After =>
+// Handle the case where column position is referencing
new columns being added.
+if (!colsToAdd.contains(add.fieldNames().init :+
after.column)) {
+ positionArgumentExists(add.position(), parent)
Review comment:
shall we just add one more parameter to `positionArgumentExists` for the
newly added columns?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27539: [SPARK-30786] [CORE] Fix Block replication failure propogation issue in BlockManager
AmplabJenkins commented on issue #27539: [SPARK-30786] [CORE] Fix Block replication failure propogation issue in BlockManager URL: https://github.com/apache/spark/pull/27539#issuecomment-586854935 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] liangxs commented on issue #27050: [SPARK-30388][Core] Mark running map stages of finished job as finished, and cancel running tasks
liangxs commented on issue #27050: [SPARK-30388][Core] Mark running map stages of finished job as finished, and cancel running tasks URL: https://github.com/apache/spark/pull/27050#issuecomment-586854840 test this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] liangxs removed a comment on issue #27050: [SPARK-30388][Core] Mark running map stages of finished job as finished, and cancel running tasks
liangxs removed a comment on issue #27050: [SPARK-30388][Core] Mark running map stages of finished job as finished, and cancel running tasks URL: https://github.com/apache/spark/pull/27050#issuecomment-584944080 Jenkins retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior
SparkQA commented on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior URL: https://github.com/apache/spark/pull/27478#issuecomment-586854920 **[Test build #118547 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118547/testReport)** for PR 27478 at commit [`feb8c0a`](https://github.com/apache/spark/commit/feb8c0aa28034c166f6413bd1eec58a885848192). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27539: [SPARK-30786] [CORE] Fix Block replication failure propogation issue in BlockManager
AmplabJenkins commented on issue #27539: [SPARK-30786] [CORE] Fix Block replication failure propogation issue in BlockManager URL: https://github.com/apache/spark/pull/27539#issuecomment-586854938 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118530/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest
SparkQA commented on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest URL: https://github.com/apache/spark/pull/27461#issuecomment-586854930 **[Test build #118548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118548/testReport)** for PR 27461 at commit [`a59724e`](https://github.com/apache/spark/commit/a59724e87e84ca09e6d59a6b8a10f0e035ee4d61). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586853998 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118545/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27539: [SPARK-30786] [CORE] Fix Block replication failure propogation issue in BlockManager
SparkQA commented on issue #27539: [SPARK-30786] [CORE] Fix Block replication failure propogation issue in BlockManager URL: https://github.com/apache/spark/pull/27539#issuecomment-586854621 **[Test build #118530 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118530/testReport)** for PR 27539 at commit [`4857401`](https://github.com/apache/spark/commit/48574017b43073c58d718d8f7f2c3d8e05d2fa24). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
AmplabJenkins removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586853991 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
SparkQA removed a comment on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586848299 **[Test build #118545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118545/testReport)** for PR 27552 at commit [`95597a7`](https://github.com/apache/spark/commit/95597a7244e2c99a9ae267e5705748ad8403a169). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586853991 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586853928 **[Test build #118545 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118545/testReport)** for PR 27552 at commit [`95597a7`](https://github.com/apache/spark/commit/95597a7244e2c99a9ae267e5705748ad8403a169). * This patch **fails to generate documentation**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
AmplabJenkins commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586853998 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118545/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior
AmplabJenkins removed a comment on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior URL: https://github.com/apache/spark/pull/27478#issuecomment-586853038 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23301/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior
AmplabJenkins removed a comment on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior URL: https://github.com/apache/spark/pull/27478#issuecomment-586853035 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior
AmplabJenkins commented on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior URL: https://github.com/apache/spark/pull/27478#issuecomment-586853035 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior
AmplabJenkins commented on issue #27478: [SPARK-25829][SQL] Add config `spark.sql.legacy.allowDuplicatedMapKeys` and change the default behavior URL: https://github.com/apache/spark/pull/27478#issuecomment-586853038 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23301/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27579: [SPARK-30829][SQL] Define LegacyBehaviorPolicy enumeration as the common value for result change configs
cloud-fan commented on a change in pull request #27579: [SPARK-30829][SQL]
Define LegacyBehaviorPolicy enumeration as the common value for result change
configs
URL: https://github.com/apache/spark/pull/27579#discussion_r380016560
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -2106,13 +2106,19 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
- val LEGACY_CTE_PRECEDENCE_ENABLED =
buildConf("spark.sql.legacy.ctePrecedence.enabled")
+ object LegacyBehaviorPolicy extends Enumeration {
+val EXCEPTION, LEGACY, CORRECTED = Value
+ }
+
+ val LEGACY_CTE_PRECEDENCE_POLICY =
buildConf("spark.sql.legacy.ctePrecedencePolicy")
.internal()
-.doc("When true, outer CTE definitions takes precedence over inner
definitions. If set to " +
- "false, inner CTE definitions take precedence. The default value is
empty, " +
+.doc("When LEGACY, outer CTE definitions takes precedence over inner
definitions. If set to " +
+ "CORRECTED, inner CTE definitions take precedence. The default value is
EXCEPTION, " +
"AnalysisException is thrown while name conflict is detected in nested
CTE.")
Review comment:
let's add one more sentence: This config will be removed in future versions
and CORRECTED will be the only behavior.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest
AmplabJenkins removed a comment on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest URL: https://github.com/apache/spark/pull/27461#issuecomment-586850786 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27579: [SPARK-30829][SQL] Define LegacyBehaviorPolicy enumeration as the common value for result change configs
cloud-fan commented on a change in pull request #27579: [SPARK-30829][SQL] Define LegacyBehaviorPolicy enumeration as the common value for result change configs URL: https://github.com/apache/spark/pull/27579#discussion_r380016686 ## File path: sql/core/src/test/resources/sql-tests/inputs/cte-nonlegacy.sql ## @@ -1,2 +1,2 @@ ---SET spark.sql.legacy.ctePrecedence.enabled = false +--SET spark.sql.legacy.ctePrecedencePolicy = new_behavior Review comment: corrected This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest
AmplabJenkins removed a comment on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest URL: https://github.com/apache/spark/pull/27461#issuecomment-586850794 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23300/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest
AmplabJenkins commented on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest URL: https://github.com/apache/spark/pull/27461#issuecomment-586850794 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23300/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest
AmplabJenkins commented on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest URL: https://github.com/apache/spark/pull/27461#issuecomment-586850786 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#discussion_r380015431 ## File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/test/TestHive.scala ## @@ -605,6 +605,8 @@ private[hive] class TestHiveQueryExecution( // Proceed with analysis. sparkSession.sessionState.analyzer.executeAndCheck(logical, tracker) } + + lazy val dataset = Dataset.ofRows(sparkSession, logical) Review comment: If it's only used once in https://github.com/apache/spark/pull/27552/files#diff-6b519cd0a9dc5b0ee9caab7712d252e9R349 , let's just inline it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest
SparkQA commented on issue #27461: [SPARK-30736][ML] One-Pass ChiSquareTest URL: https://github.com/apache/spark/pull/27461#issuecomment-586850395 **[Test build #118546 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118546/testReport)** for PR 27461 at commit [`404f018`](https://github.com/apache/spark/commit/404f018ba048562c60c0eed4ed434492826af18a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r380014884
##
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLDriver.scala
##
@@ -60,9 +60,10 @@ private[hive] class SparkSQLDriver(val context: SQLContext
= SparkSQLEnv.sqlCont
// TODO unify the error code
try {
context.sparkContext.setJobDescription(command)
+ val df = context.sql(command)
val execution =
context.sessionState.executePlan(context.sql(command).logicalPlan)
hiveResponse = SQLExecution.withNewExecutionId(execution) {
Review comment:
`SQLExecution.withNewExecutionId(df.queryExecution)`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r380014991
##
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLEnv.scala
##
@@ -24,6 +24,7 @@ import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.internal.Logging
import org.apache.spark.sql.{SparkSession, SQLContext}
import org.apache.spark.sql.hive.{HiveExternalCatalog, HiveUtils}
+import org.apache.spark.sql.internal.SQLConf
Review comment:
unnecessary change?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r380015103
##
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLDriver.scala
##
@@ -60,9 +60,10 @@ private[hive] class SparkSQLDriver(val context: SQLContext
= SparkSQLEnv.sqlCont
// TODO unify the error code
try {
context.sparkContext.setJobDescription(command)
+ val df = context.sql(command)
val execution =
context.sessionState.executePlan(context.sql(command).logicalPlan)
Review comment:
`val execution = df.queryExecution`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r380014884
##
File path:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLDriver.scala
##
@@ -60,9 +60,10 @@ private[hive] class SparkSQLDriver(val context: SQLContext
= SparkSQLEnv.sqlCont
// TODO unify the error code
try {
context.sparkContext.setJobDescription(command)
+ val df = context.sql(command)
val execution =
context.sessionState.executePlan(context.sql(command).logicalPlan)
hiveResponse = SQLExecution.withNewExecutionId(execution) {
Review comment:
`SQLExecution.withNewExecutionId(df.queryExecution)`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27600: [MINOR][ML] ML cleanup
AmplabJenkins removed a comment on issue #27600: [MINOR][ML] ML cleanup URL: https://github.com/apache/spark/pull/27600#issuecomment-586848923 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118534/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27600: [MINOR][ML] ML cleanup
AmplabJenkins removed a comment on issue #27600: [MINOR][ML] ML cleanup URL: https://github.com/apache/spark/pull/27600#issuecomment-586848917 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27600: [MINOR][ML] ML cleanup
AmplabJenkins commented on issue #27600: [MINOR][ML] ML cleanup URL: https://github.com/apache/spark/pull/27600#issuecomment-586848923 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118534/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27600: [MINOR][ML] ML cleanup
AmplabJenkins commented on issue #27600: [MINOR][ML] ML cleanup URL: https://github.com/apache/spark/pull/27600#issuecomment-586848917 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
cloud-fan commented on a change in pull request #27552: [SPARK-30808][SQL]
Enable Java 8 time API in Thrift server
URL: https://github.com/apache/spark/pull/27552#discussion_r380014294
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
##
@@ -36,27 +37,41 @@ object HiveResult {
* Returns the result as a hive compatible sequence of strings. This is used
in tests and
* `SparkSQLDriver` for CLI applications.
*/
- def hiveResultString(executedPlan: SparkPlan): Seq[String] = executedPlan
match {
-case ExecutedCommandExec(_: DescribeCommandBase) =>
- // If it is a describe command for a Hive table, we want to have the
output format
- // be similar with Hive.
- executedPlan.executeCollectPublic().map {
-case Row(name: String, dataType: String, comment) =>
- Seq(name, dataType,
-Option(comment.asInstanceOf[String]).getOrElse(""))
-.map(s => String.format(s"%-20s", s))
-.mkString("\t")
- }
-// SHOW TABLES in Hive only output table names, while ours output
database, table name, isTemp.
-case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended
=>
- command.executeCollect().map(_.getString(1))
-case other =>
- val result: Seq[Seq[Any]] =
other.executeCollectPublic().map(_.toSeq).toSeq
- // We need the types so we can output struct field names
- val types = executedPlan.output.map(_.dataType)
- // Reformat to match hive tab delimited output.
- result.map(_.zip(types).map(e => toHiveString(e)))
-.map(_.mkString("\t"))
+ def hiveResultString(ds: Dataset[_]): Seq[String] = {
+val executedPlan = ds.queryExecution.executedPlan
+executedPlan match {
+ case ExecutedCommandExec(_: DescribeCommandBase) =>
+// If it is a describe command for a Hive table, we want to have the
output format
+// be similar with Hive.
+executedPlan.executeCollectPublic().map {
+ case Row(name: String, dataType: String, comment) =>
+Seq(name, dataType,
+ Option(comment.asInstanceOf[String]).getOrElse(""))
+ .map(s => String.format(s"%-20s", s))
+ .mkString("\t")
+}
+ // SHOW TABLES in Hive only output table names,
+ // while ours output database, table name, isTemp.
+ case command @ ExecutedCommandExec(s: ShowTablesCommand) if
!s.isExtended =>
+command.executeCollect().map(_.getString(1))
+ case _ =>
+val sessionWithJava8DatetimeEnabled = {
+ val cloned = ds.sparkSession.cloneSession()
+ cloned.conf.set(SQLConf.DATETIME_JAVA8API_ENABLED.key, true)
+ cloned
+}
+sessionWithJava8DatetimeEnabled.withActive {
+ val result: Seq[Seq[Any]] = Dataset.ofRows(ds.sparkSession,
ds.queryExecution.logical)
Review comment:
nit: let's add a code comment to explain why we can't collect the input
`ds`. Its encoder was already created with java 8 datetime disabled.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27584: [SPARK-30814][SQL] ALTER TABLE ... ADD COLUMN position should be able to reference columns being added
AmplabJenkins removed a comment on issue #27584: [SPARK-30814][SQL] ALTER TABLE ... ADD COLUMN position should be able to reference columns being added URL: https://github.com/apache/spark/pull/27584#issuecomment-586848595 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23299/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27584: [SPARK-30814][SQL] ALTER TABLE ... ADD COLUMN position should be able to reference columns being added
AmplabJenkins removed a comment on issue #27584: [SPARK-30814][SQL] ALTER TABLE ... ADD COLUMN position should be able to reference columns being added URL: https://github.com/apache/spark/pull/27584#issuecomment-586848586 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26193: [SPARK-25065][k8s] Allow setting up correct logging configuration on driver and executor.
SparkQA commented on issue #26193: [SPARK-25065][k8s] Allow setting up correct logging configuration on driver and executor. URL: https://github.com/apache/spark/pull/26193#issuecomment-586848842 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/23294/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27584: [SPARK-30814][SQL] ALTER TABLE ... ADD COLUMN position should be able to reference columns being added
AmplabJenkins commented on issue #27584: [SPARK-30814][SQL] ALTER TABLE ... ADD COLUMN position should be able to reference columns being added URL: https://github.com/apache/spark/pull/27584#issuecomment-586848586 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27584: [SPARK-30814][SQL] ALTER TABLE ... ADD COLUMN position should be able to reference columns being added
AmplabJenkins commented on issue #27584: [SPARK-30814][SQL] ALTER TABLE ... ADD COLUMN position should be able to reference columns being added URL: https://github.com/apache/spark/pull/27584#issuecomment-586848595 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23299/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27584: [SPARK-30814][SQL] ALTER TABLE ... ADD COLUMN position should be able to reference columns being added
SparkQA commented on issue #27584: [SPARK-30814][SQL] ALTER TABLE ... ADD COLUMN position should be able to reference columns being added URL: https://github.com/apache/spark/pull/27584#issuecomment-586848328 **[Test build #118544 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118544/testReport)** for PR 27584 at commit [`34b1363`](https://github.com/apache/spark/commit/34b1363d9fe98c6a43d7f3b579b7024113f9d062). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server
SparkQA commented on issue #27552: [SPARK-30808][SQL] Enable Java 8 time API in Thrift server URL: https://github.com/apache/spark/pull/27552#issuecomment-586848299 **[Test build #118545 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118545/testReport)** for PR 27552 at commit [`95597a7`](https://github.com/apache/spark/commit/95597a7244e2c99a9ae267e5705748ad8403a169). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27558: [SPARK-30806][SQL]Evaluate once per group in UnboundedWindowFunctionFrame
cloud-fan commented on issue #27558: [SPARK-30806][SQL]Evaluate once per group in UnboundedWindowFunctionFrame URL: https://github.com/apache/spark/pull/27558#issuecomment-586848148 can you address https://github.com/apache/spark/pull/27558#discussion_r378872179 ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27600: [MINOR][ML] ML cleanup
SparkQA commented on issue #27600: [MINOR][ML] ML cleanup URL: https://github.com/apache/spark/pull/27600#issuecomment-586848335 **[Test build #118534 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118534/testReport)** for PR 27600 at commit [`b261e23`](https://github.com/apache/spark/commit/b261e23c9abbfd4d3603d998cfeeb2ad8241a212). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27600: [MINOR][ML] ML cleanup
SparkQA removed a comment on issue #27600: [MINOR][ML] ML cleanup URL: https://github.com/apache/spark/pull/27600#issuecomment-586829697 **[Test build #118534 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118534/testReport)** for PR 27600 at commit [`b261e23`](https://github.com/apache/spark/commit/b261e23c9abbfd4d3603d998cfeeb2ad8241a212). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
