[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24601: [SPARK-27702][K8S] Allow using some alternatives for service accounts
dongjoon-hyun commented on a change in pull request #24601: [SPARK-27702][K8S] Allow using some alternatives for service accounts URL: https://github.com/apache/spark/pull/24601#discussion_r370505542 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala ## @@ -50,10 +50,14 @@ private[spark] class KubernetesClusterManager extends ExternalClusterManager wit require(sc.conf.get(KUBERNETES_DRIVER_POD_NAME).isDefined, "If the application is deployed using spark-submit in cluster mode, the driver pod name " + "must be provided.") + val serviceAccountToken = +Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_TOKEN_PATH)).filter(_.exists) + val serviceAccountCaCrt = +Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_CA_CRT_PATH)).filter(_.exists) Review comment: The existing code already tries to use the given file first. Where did you check the following? > If it does not exist, other configurations are allowed to use, such as spark.kubernetes.authenticate.driver.mounted.oauthTokenFile. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24601: [SPARK-27702][K8S] Allow using some alternatives for service accounts
dongjoon-hyun commented on a change in pull request #24601: [SPARK-27702][K8S] Allow using some alternatives for service accounts URL: https://github.com/apache/spark/pull/24601#discussion_r370505542 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala ## @@ -50,10 +50,14 @@ private[spark] class KubernetesClusterManager extends ExternalClusterManager wit require(sc.conf.get(KUBERNETES_DRIVER_POD_NAME).isDefined, "If the application is deployed using spark-submit in cluster mode, the driver pod name " + "must be provided.") + val serviceAccountToken = +Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_TOKEN_PATH)).filter(_.exists) + val serviceAccountCaCrt = +Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_CA_CRT_PATH)).filter(_.exists) Review comment: The existing code already try to use the given file first. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24601: [SPARK-27702][K8S] Allow using some alternatives for service accounts
dongjoon-hyun commented on a change in pull request #24601: [SPARK-27702][K8S] Allow using some alternatives for service accounts URL: https://github.com/apache/spark/pull/24601#discussion_r370505542 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala ## @@ -50,10 +50,14 @@ private[spark] class KubernetesClusterManager extends ExternalClusterManager wit require(sc.conf.get(KUBERNETES_DRIVER_POD_NAME).isDefined, "If the application is deployed using spark-submit in cluster mode, the driver pod name " + "must be provided.") + val serviceAccountToken = +Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_TOKEN_PATH)).filter(_.exists) + val serviceAccountCaCrt = +Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_CA_CRT_PATH)).filter(_.exists) Review comment: The existing code already tries to use the given file first. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #24601: [SPARK-27702][K8S] Allow using some alternatives for service accounts
dongjoon-hyun commented on a change in pull request #24601: [SPARK-27702][K8S] Allow using some alternatives for service accounts URL: https://github.com/apache/spark/pull/24601#discussion_r370503859 ## File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala ## @@ -50,10 +50,14 @@ private[spark] class KubernetesClusterManager extends ExternalClusterManager wit require(sc.conf.get(KUBERNETES_DRIVER_POD_NAME).isDefined, "If the application is deployed using spark-submit in cluster mode, the driver pod name " + "must be provided.") + val serviceAccountToken = +Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_TOKEN_PATH)).filter(_.exists) + val serviceAccountCaCrt = +Some(new File(Config.KUBERNETES_SERVICE_ACCOUNT_CA_CRT_PATH)).filter(_.exists) Review comment: The current logic looks like this. 1. The difference of this logic is returning `None` when the file doesn't exist. 2. However, `None` is not used when the given `oauthTokenFileConf` points a valid file because it's called as `orElse`. ```scala val oauthTokenFile = sparkConf.getOption(oauthTokenFileConf) .map(new File(_)) .orElse(defaultServiceAccountToken) ``` Is the above correct? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering
AmplabJenkins removed a comment on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering URL: https://github.com/apache/spark/pull/27128#issuecomment-578022881 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117335/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering
AmplabJenkins removed a comment on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering URL: https://github.com/apache/spark/pull/27128#issuecomment-578022877 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering
SparkQA removed a comment on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering URL: https://github.com/apache/spark/pull/27128#issuecomment-577989466 **[Test build #117335 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117335/testReport)** for PR 27128 at commit [`c297e6a`](https://github.com/apache/spark/commit/c297e6a8b4a27b31b783daf5a1264eb8b4d37447). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering
AmplabJenkins commented on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering URL: https://github.com/apache/spark/pull/27128#issuecomment-578022877 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering
AmplabJenkins commented on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering URL: https://github.com/apache/spark/pull/27128#issuecomment-578022881 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117335/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering
SparkQA commented on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering URL: https://github.com/apache/spark/pull/27128#issuecomment-578022707 **[Test build #117335 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117335/testReport)** for PR 27128 at commit [`c297e6a`](https://github.com/apache/spark/commit/c297e6a8b4a27b31b783daf5a1264eb8b4d37447). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp"
gatorsmile commented on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp" URL: https://github.com/apache/spark/pull/27351#issuecomment-578022258 Thanks! @dongjoon-hyun Both Hive and Impala have such functions. We should simplify the migration by introducing the needed functions if we decide to drop/deprecate the existing functions. Adding a conf is not enough in such cases. I think we should introduce more time/timestamp related built-in functions in the future. Processing date and timestamp is a common task in the data processing/cleaning/analysis. We have to be more careful when introducing a behavior breaking change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27352: [SPARK-30630][ML][2.4] Deprecate numTrees in GBT in 2.4.5
AmplabJenkins removed a comment on issue #27352: [SPARK-30630][ML][2.4] Deprecate numTrees in GBT in 2.4.5 URL: https://github.com/apache/spark/pull/27352#issuecomment-578021608 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27352: [SPARK-30630][ML][2.4] Deprecate numTrees in GBT in 2.4.5
AmplabJenkins removed a comment on issue #27352: [SPARK-30630][ML][2.4] Deprecate numTrees in GBT in 2.4.5 URL: https://github.com/apache/spark/pull/27352#issuecomment-578021617 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22101/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp"
AmplabJenkins removed a comment on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp" URL: https://github.com/apache/spark/pull/27351#issuecomment-578021607 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22102/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp"
AmplabJenkins removed a comment on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp" URL: https://github.com/apache/spark/pull/27351#issuecomment-578021603 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27352: [SPARK-30630][ML][2.4] Deprecate numTrees in GBT in 2.4.5
AmplabJenkins commented on issue #27352: [SPARK-30630][ML][2.4] Deprecate numTrees in GBT in 2.4.5 URL: https://github.com/apache/spark/pull/27352#issuecomment-578021608 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp"
AmplabJenkins commented on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp" URL: https://github.com/apache/spark/pull/27351#issuecomment-578021603 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp"
AmplabJenkins commented on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp" URL: https://github.com/apache/spark/pull/27351#issuecomment-578021607 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22102/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27352: [SPARK-30630][ML][2.4] Deprecate numTrees in GBT in 2.4.5
AmplabJenkins commented on issue #27352: [SPARK-30630][ML][2.4] Deprecate numTrees in GBT in 2.4.5 URL: https://github.com/apache/spark/pull/27352#issuecomment-578021617 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22101/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp"
SparkQA commented on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp" URL: https://github.com/apache/spark/pull/27351#issuecomment-578021286 **[Test build #117342 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117342/testReport)** for PR 27351 at commit [`af5a793`](https://github.com/apache/spark/commit/af5a79375e219bc3acdbbdc1bca3ff9bda277f26). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27352: [SPARK-30630][ML][2.4] Deprecate numTrees in GBT in 2.4.5
SparkQA commented on issue #27352: [SPARK-30630][ML][2.4] Deprecate numTrees in GBT in 2.4.5 URL: https://github.com/apache/spark/pull/27352#issuecomment-578021282 **[Test build #117341 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117341/testReport)** for PR 27352 at commit [`92e12a7`](https://github.com/apache/spark/commit/92e12a72dc145e04dece2742d64292d19878efe0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
AmplabJenkins removed a comment on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578019692 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22100/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
AmplabJenkins removed a comment on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578019684 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
AmplabJenkins commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578019692 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22100/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
AmplabJenkins commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578019684 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
SparkQA commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578019308 **[Test build #117340 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117340/testReport)** for PR 27330 at commit [`0b81823`](https://github.com/apache/spark/commit/0b818238bb576c1d8d2ea012f26281ddb2c9b5f5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27352: [SPARK-30630][ML] Deprecate numTrees in GBT in 2.4.5
dongjoon-hyun commented on a change in pull request #27352: [SPARK-30630][ML] Deprecate numTrees in GBT in 2.4.5 URL: https://github.com/apache/spark/pull/27352#discussion_r370498776 ## File path: mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala ## @@ -319,7 +319,12 @@ class GBTClassificationModel private[ml]( } } - /** Number of trees in ensemble */ + /** + * Number of trees in ensemble + * + * @deprecated Use [[getNumTrees]] instead. This method will be removed in 3.0.0 Review comment: nit. `3.0.0` -> `3.0.0.` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27352: [SPARK-30630][ML] Deprecate numTrees in GBT in 2.4.5
dongjoon-hyun commented on a change in pull request #27352: [SPARK-30630][ML] Deprecate numTrees in GBT in 2.4.5 URL: https://github.com/apache/spark/pull/27352#discussion_r370498824 ## File path: mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala ## @@ -255,10 +255,15 @@ class GBTRegressionModel private[ml]( // TODO: When we add a generic Boosting class, handle transform there? SPARK-7129 // Classifies by thresholding sum of weighted tree predictions val treePredictions = _trees.map(_.rootNode.predictImpl(features).prediction) -blas.ddot(numTrees, treePredictions, 1, _treeWeights, 1) +blas.ddot(getNumTrees, treePredictions, 1, _treeWeights, 1) } - /** Number of trees in ensemble */ + /** + * Number of trees in ensemble + * + * @deprecated Use [[getNumTrees]] instead. This method will be removed in 3.0.0 Review comment: ditto. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27352: [SPARK-30630][ML] Deprecate numTrees in GBT in 2.4.5
AmplabJenkins removed a comment on issue #27352: [SPARK-30630][ML] Deprecate numTrees in GBT in 2.4.5 URL: https://github.com/apache/spark/pull/27352#issuecomment-578015911 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27352: [SPARK-30630][ML] Deprecate numTrees in GBT in 2.4.5
AmplabJenkins commented on issue #27352: [SPARK-30630][ML] Deprecate numTrees in GBT in 2.4.5 URL: https://github.com/apache/spark/pull/27352#issuecomment-578015917 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22099/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27352: [SPARK-30630][ML] Deprecate numTrees in GBT in 2.4.5
AmplabJenkins commented on issue #27352: [SPARK-30630][ML] Deprecate numTrees in GBT in 2.4.5 URL: https://github.com/apache/spark/pull/27352#issuecomment-578015911 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27352: [SPARK-30630][ML] Deprecate numTrees in GBT in 2.4.5
AmplabJenkins removed a comment on issue #27352: [SPARK-30630][ML] Deprecate numTrees in GBT in 2.4.5 URL: https://github.com/apache/spark/pull/27352#issuecomment-578015917 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22099/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #24195: [SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp
MaxGekk commented on issue #24195: [SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp URL: https://github.com/apache/spark/pull/24195#issuecomment-578015476 > From the JIRA REST I get a string that is a timestamp encoded in PST. I guess if timestamp strings in PST, the strings contain timezone info. Looking at https://developer.atlassian.com/cloud/jira/platform/jira-expressions/#date-and-time, they are. - If you parse the strings via JSON datasource, you should set JSON option `timestampFormat` with time zone sub-pattern correctly. - If you want to parse it just as string, use `def to_timestamp(s: Column, fmt: String)`. Spark's TimestampFormatter takes into account time zone info https://github.com/apache/spark/blob/c2734ab1fcc523507174c4f9e7368c1e7a2368e8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala#L59-L60 and shifts local timestamp respectively while converting to UTC timestamps: https://github.com/apache/spark/blob/c2734ab1fcc523507174c4f9e7368c1e7a2368e8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/TimestampFormatter.scala#L61-L62 If input strings don't contain time zone info, you can either: - append time zone id string manually, or - temporary set SQL config `spark.sql.session.timeZone` while parsing timestamp strings. For me, still not clear why did you decide to even look at the functions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp"
dongjoon-hyun commented on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp" URL: https://github.com/apache/spark/pull/27351#issuecomment-578015405 cc @MaxGekk and @cloud-fan This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27352: [SPARK-30630][ML] Deprecate numTrees in GBT in 2.4.5
SparkQA commented on issue #27352: [SPARK-30630][ML] Deprecate numTrees in GBT in 2.4.5 URL: https://github.com/apache/spark/pull/27352#issuecomment-578015565 **[Test build #117339 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117339/testReport)** for PR 27352 at commit [`7ff296b`](https://github.com/apache/spark/commit/7ff296b2db16b8805c374300fc6d660db244ef24). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #27352: [SPARK-30630][ML] Deprecate numTrees in GBT in 2.4.5
huaxingao commented on issue #27352: [SPARK-30630][ML] Deprecate numTrees in GBT in 2.4.5 URL: https://github.com/apache/spark/pull/27352#issuecomment-578015259 cc @dongjoon-hyun @srowen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao opened a new pull request #27352: [SPARK-30630][ML] Deprecate numTrees in GBT in 2.4.5
huaxingao opened a new pull request #27352: [SPARK-30630][ML] Deprecate
numTrees in GBT in 2.4.5
URL: https://github.com/apache/spark/pull/27352
### What changes were proposed in this pull request?
Deprecate numTrees in GBT in 2.4.5 so it can be removed in 3.0.0
### Why are the changes needed?
Currently, GBT has
```
/**
* Number of trees in ensemble
*/
@Since("2.0.0")
val getNumTrees: Int = trees.length
```
and
```
/** Number of trees in ensemble */
val numTrees: Int = trees.length
```
I think we should remove one of them. I will deprecate it in 2.4.5 and
remove it in 3.0.0
### Does this PR introduce any user-facing change?
Deprecate numTrees in 2.4.5
### How was this patch tested?
Existing tests
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #26957: [SPARK-30314] Add identifier and catalog information to DataSourceV2Relation
SparkQA commented on issue #26957: [SPARK-30314] Add identifier and catalog information to DataSourceV2Relation URL: https://github.com/apache/spark/pull/26957#issuecomment-578013796 **[Test build #117338 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117338/testReport)** for PR 26957 at commit [`600d289`](https://github.com/apache/spark/commit/600d289dbf159c3a81d170016bf6f7831f9a3165). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26957: [SPARK-30314] Add identifier and catalog information to DataSourceV2Relation
AmplabJenkins removed a comment on issue #26957: [SPARK-30314] Add identifier and catalog information to DataSourceV2Relation URL: https://github.com/apache/spark/pull/26957#issuecomment-578012372 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #26957: [SPARK-30314] Add identifier and catalog information to DataSourceV2Relation
AmplabJenkins removed a comment on issue #26957: [SPARK-30314] Add identifier and catalog information to DataSourceV2Relation URL: https://github.com/apache/spark/pull/26957#issuecomment-578012377 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22098/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26957: [SPARK-30314] Add identifier and catalog information to DataSourceV2Relation
AmplabJenkins commented on issue #26957: [SPARK-30314] Add identifier and catalog information to DataSourceV2Relation URL: https://github.com/apache/spark/pull/26957#issuecomment-578012377 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22098/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #26957: [SPARK-30314] Add identifier and catalog information to DataSourceV2Relation
AmplabJenkins commented on issue #26957: [SPARK-30314] Add identifier and catalog information to DataSourceV2Relation URL: https://github.com/apache/spark/pull/26957#issuecomment-578012372 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
AmplabJenkins removed a comment on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578007367 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117337/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
huaxingao commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578007676 @dongjoon-hyun Yes. I am trying to make a PR for 2.4. I will also fix the MiMa problem. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
AmplabJenkins removed a comment on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578007363 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
SparkQA removed a comment on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578005580 **[Test build #117337 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117337/testReport)** for PR 27330 at commit [`8744e17`](https://github.com/apache/spark/commit/8744e176e3cc523b8d1cb92cea82c2fb61a66044). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
AmplabJenkins commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578007363 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
SparkQA commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578007343 **[Test build #117337 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117337/testReport)** for PR 27330 at commit [`8744e17`](https://github.com/apache/spark/commit/8744e176e3cc523b8d1cb92cea82c2fb61a66044). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
AmplabJenkins commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578007367 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117337/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] bmarcott commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling.
bmarcott commented on issue #27207: [WIP][SPARK-18886][CORE] Make Locality wait time measure resource under utilization due to delay scheduling. URL: https://github.com/apache/spark/pull/27207#issuecomment-578006160 @cloud-fan thanks for the input @tgravescs yep, that sequence and explanation matches with my understanding > I was also wondering about leaving the old logic in there but configured off by default. While I don't like it, it would be safer if we haven't thought of some corner case and user could turn it back on if necessary, thoughts? I'm not opposed to adding a new config that disables new way and enables old way, but is this standard practice in spark for the more risky changes? I would feel more comfortable knowing there was a fallback too ;) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp"
gatorsmile commented on a change in pull request #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp" URL: https://github.com/apache/spark/pull/27351#discussion_r370487183 ## File path: sql/core/src/test/scala/org/apache/spark/sql/DeprecatedDateFunctionsSuite.scala ## @@ -1,113 +0,0 @@ -/* Review comment: The tests are moved by https://github.com/apache/spark/pull/26885 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp"
AmplabJenkins removed a comment on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp" URL: https://github.com/apache/spark/pull/27351#issuecomment-578005873 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #24195: [SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp
gatorsmile commented on issue #24195: [SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp URL: https://github.com/apache/spark/pull/24195#issuecomment-578006107 The code freeze of 3.0 is close. So far, we do not have a user-friendly way to replace these deprecated functions. Let us first revert it and then discuss what is the next step https://github.com/apache/spark/pull/27351 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
AmplabJenkins removed a comment on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578005904 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp"
AmplabJenkins removed a comment on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp" URL: https://github.com/apache/spark/pull/27351#issuecomment-578005881 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22096/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
AmplabJenkins removed a comment on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578005909 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22097/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
AmplabJenkins commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578005904 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
AmplabJenkins commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578005909 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22097/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp"
AmplabJenkins commented on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp" URL: https://github.com/apache/spark/pull/27351#issuecomment-578005881 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22096/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp"
AmplabJenkins commented on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp" URL: https://github.com/apache/spark/pull/27351#issuecomment-578005873 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp"
SparkQA commented on issue #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp" URL: https://github.com/apache/spark/pull/27351#issuecomment-578005583 **[Test build #117336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117336/testReport)** for PR 27351 at commit [`4a273a1`](https://github.com/apache/spark/commit/4a273a186a87f816ba9655c973381b58111da8b0). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
dongjoon-hyun edited a comment on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578005404 @huaxingao . We need to deprecate this in `branch-2.4` first. We can not proceed in the reverse order. Could you make a PR for `branch-2.4` too? We will review that first. > I will deprecate it in 2.4.5 and remove it in 3.0.0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
SparkQA commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578005580 **[Test build #117337 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117337/testReport)** for PR 27330 at commit [`8744e17`](https://github.com/apache/spark/commit/8744e176e3cc523b8d1cb92cea82c2fb61a66044). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0
dongjoon-hyun commented on issue #27330: [SPARK-30630][ML] Remove numTrees in GBT in 3.0.0 URL: https://github.com/apache/spark/pull/27330#issuecomment-578005404 @huaxingao . We need to deprecate this in `branch-2.4` first. We can not proceed in the reverse order. Could you make a PR for `branch-2.4`? > I will deprecate it in 2.4.5 and remove it in 3.0.0 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile opened a new pull request #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp"
gatorsmile opened a new pull request #27351: Revert "[SPARK-25496][SQL] Deprecate from_utc_timestamp and to_utc_timestamp" URL: https://github.com/apache/spark/pull/27351 This reverts commit 1d20d13149140f53df307f47420740f45b4fa5f6. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #27330: [SPARK-30630][ML] Deprecate numTrees in GBT
huaxingao commented on a change in pull request #27330: [SPARK-30630][ML]
Deprecate numTrees in GBT
URL: https://github.com/apache/spark/pull/27330#discussion_r370483158
##
File path:
mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala
##
@@ -342,7 +342,12 @@ class GBTClassificationModel private[ml](
}
}
- /** Number of trees in ensemble */
+ /**
+ * Number of trees in ensemble
+ *
+ * @deprecated Use [[getNumTrees]] instead. This method will be removed in
3.1.0
+ */
+ @deprecated("Use getNumTrees instead. This method will be removed in
3.1.0.", "3.0.0")
Review comment:
Got it. Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27330: [SPARK-30630][ML] Deprecate numTrees in GBT
dongjoon-hyun commented on a change in pull request #27330: [SPARK-30630][ML]
Deprecate numTrees in GBT
URL: https://github.com/apache/spark/pull/27330#discussion_r370482861
##
File path:
mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala
##
@@ -342,7 +342,12 @@ class GBTClassificationModel private[ml](
}
}
- /** Number of trees in ensemble */
+ /**
+ * Number of trees in ensemble
+ *
+ * @deprecated Use [[getNumTrees]] instead. This method will be removed in
3.1.0
+ */
+ @deprecated("Use getNumTrees instead. This method will be removed in
3.1.0.", "3.0.0")
Review comment:
It's `2.4.5` because RC is not an official release.
RC1 vote fails and we are preparing 2.4.5 RC2.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27330: [SPARK-30630][ML] Deprecate numTrees in GBT
dongjoon-hyun commented on a change in pull request #27330: [SPARK-30630][ML]
Deprecate numTrees in GBT
URL: https://github.com/apache/spark/pull/27330#discussion_r370482861
##
File path:
mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala
##
@@ -342,7 +342,12 @@ class GBTClassificationModel private[ml](
}
}
- /** Number of trees in ensemble */
+ /**
+ * Number of trees in ensemble
+ *
+ * @deprecated Use [[getNumTrees]] instead. This method will be removed in
3.1.0
+ */
+ @deprecated("Use getNumTrees instead. This method will be removed in
3.1.0.", "3.0.0")
Review comment:
It's `2.4.5` because RC is not an official release.
RC1 vote failed and we are preparing 2.4.5 RC2.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on issue #27330: [MINOR][ML] Deprecate numTrees in GBT
huaxingao commented on issue #27330: [MINOR][ML] Deprecate numTrees in GBT URL: https://github.com/apache/spark/pull/27330#issuecomment-578001055 @dongjoon-hyun > Could you file a JIRA for this? Yes, I will open a JIRA. Thank you very much for reviewing this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on a change in pull request #27330: [MINOR][ML] Deprecate numTrees in GBT
huaxingao commented on a change in pull request #27330: [MINOR][ML] Deprecate
numTrees in GBT
URL: https://github.com/apache/spark/pull/27330#discussion_r370482028
##
File path:
mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala
##
@@ -342,7 +342,12 @@ class GBTClassificationModel private[ml](
}
}
- /** Number of trees in ensemble */
+ /**
+ * Number of trees in ensemble
+ *
+ * @deprecated Use [[getNumTrees]] instead. This method will be removed in
3.1.0
+ */
+ @deprecated("Use getNumTrees instead. This method will be removed in
3.1.0.", "3.0.0")
Review comment:
so deprecate this in v2.4.5-rc1, right?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default
dongjoon-hyun closed pull request #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default URL: https://github.com/apache/spark/pull/27348 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default
AmplabJenkins removed a comment on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default URL: https://github.com/apache/spark/pull/27348#issuecomment-577998980 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default
AmplabJenkins removed a comment on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default URL: https://github.com/apache/spark/pull/27348#issuecomment-577998991 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117329/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default
AmplabJenkins commented on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default URL: https://github.com/apache/spark/pull/27348#issuecomment-577998991 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117329/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default
AmplabJenkins commented on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default URL: https://github.com/apache/spark/pull/27348#issuecomment-577998980 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default
SparkQA removed a comment on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default URL: https://github.com/apache/spark/pull/27348#issuecomment-577956884 **[Test build #117329 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117329/testReport)** for PR 27348 at commit [`7afda97`](https://github.com/apache/spark/commit/7afda97dcfa24cdb7315bc8cae39cec7bcdfd9b1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default
SparkQA commented on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default URL: https://github.com/apache/spark/pull/27348#issuecomment-577998642 **[Test build #117329 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117329/testReport)** for PR 27348 at commit [`7afda97`](https://github.com/apache/spark/commit/7afda97dcfa24cdb7315bc8cae39cec7bcdfd9b1). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27329: [SPARK-27324][DOC][CORE] Document configurations related to executor metrics.
AmplabJenkins commented on issue #27329: [SPARK-27324][DOC][CORE] Document configurations related to executor metrics. URL: https://github.com/apache/spark/pull/27329#issuecomment-577996785 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117331/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27329: [SPARK-27324][DOC][CORE] Document configurations related to executor metrics.
AmplabJenkins removed a comment on issue #27329: [SPARK-27324][DOC][CORE] Document configurations related to executor metrics. URL: https://github.com/apache/spark/pull/27329#issuecomment-577996785 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117331/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27329: [SPARK-27324][DOC][CORE] Document configurations related to executor metrics.
AmplabJenkins removed a comment on issue #27329: [SPARK-27324][DOC][CORE] Document configurations related to executor metrics. URL: https://github.com/apache/spark/pull/27329#issuecomment-577996782 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27329: [SPARK-27324][DOC][CORE] Document configurations related to executor metrics.
AmplabJenkins commented on issue #27329: [SPARK-27324][DOC][CORE] Document configurations related to executor metrics. URL: https://github.com/apache/spark/pull/27329#issuecomment-577996782 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27329: [SPARK-27324][DOC][CORE] Document configurations related to executor metrics.
SparkQA removed a comment on issue #27329: [SPARK-27324][DOC][CORE] Document configurations related to executor metrics. URL: https://github.com/apache/spark/pull/27329#issuecomment-577971512 **[Test build #117331 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117331/testReport)** for PR 27329 at commit [`745d2d0`](https://github.com/apache/spark/commit/745d2d0925e0bc905734e94624b3fb206e398f9a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27329: [SPARK-27324][DOC][CORE] Document configurations related to executor metrics.
SparkQA commented on issue #27329: [SPARK-27324][DOC][CORE] Document configurations related to executor metrics. URL: https://github.com/apache/spark/pull/27329#issuecomment-577996386 **[Test build #117331 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117331/testReport)** for PR 27329 at commit [`745d2d0`](https://github.com/apache/spark/commit/745d2d0925e0bc905734e94624b3fb206e398f9a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
gatorsmile commented on issue #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations URL: https://github.com/apache/spark/pull/26995#issuecomment-577991412 Since the mode ANSI is introduced in Spark 3.0 and off by default, this is not a behavior change This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations
gatorsmile commented on a change in pull request #26995: [SPARK-30341][SQL] Overflow check for interval arithmetic operations URL: https://github.com/apache/spark/pull/26995#discussion_r370473845 ## File path: sql/core/src/test/resources/sql-tests/results/interval.sql.out ## @@ -1101,3 +1102,45 @@ select interval '1 ' day struct -- !query 113 output 1 days + + +-- !query 114 +select -(a) from values (interval '-2147483648 months', interval '2147483647 months') t(a, b) +-- !query 114 schema +struct<(- a):interval> +-- !query 114 output +-178956970 years -8 months + + +-- !query 115 +select a - b from values (interval '-2147483648 months', interval '2147483647 months') t(a, b) +-- !query 115 schema +struct<(a - b):interval> +-- !query 115 output +1 months + + +-- !query 116 +select b + interval '1 month' from values (interval '-2147483648 months', interval '2147483647 months') t(a, b) +-- !query 116 schema +struct<(b + INTERVAL '1 months'):interval> +-- !query 116 output +-178956970 years -8 months + + +-- !query 117 +select a * 1.1 from values (interval '-2147483648 months', interval '2147483647 months') t(a, b) +-- !query 117 schema +struct<> +-- !query 117 output +java.lang.ArithmeticException +integer overflow Review comment: Please create a 3.0 blocker JIRA. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default
AmplabJenkins removed a comment on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default URL: https://github.com/apache/spark/pull/27348#issuecomment-577989663 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22094/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default
AmplabJenkins removed a comment on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default URL: https://github.com/apache/spark/pull/27348#issuecomment-577989657 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering
AmplabJenkins removed a comment on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering URL: https://github.com/apache/spark/pull/27128#issuecomment-577989682 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22095/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering
AmplabJenkins removed a comment on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering URL: https://github.com/apache/spark/pull/27128#issuecomment-577989680 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering
AmplabJenkins commented on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering URL: https://github.com/apache/spark/pull/27128#issuecomment-577989682 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22095/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering
AmplabJenkins commented on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering URL: https://github.com/apache/spark/pull/27128#issuecomment-577989680 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default
AmplabJenkins commented on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default URL: https://github.com/apache/spark/pull/27348#issuecomment-577989663 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22094/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default
AmplabJenkins commented on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default URL: https://github.com/apache/spark/pull/27348#issuecomment-577989657 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default
SparkQA commented on issue #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default URL: https://github.com/apache/spark/pull/27348#issuecomment-577989470 **[Test build #117334 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117334/testReport)** for PR 27348 at commit [`058`](https://github.com/apache/spark/commit/05815ba05b00cd13e9b14f0f839c97f62738). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering
SparkQA commented on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering URL: https://github.com/apache/spark/pull/27128#issuecomment-577989466 **[Test build #117335 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117335/testReport)** for PR 27128 at commit [`c297e6a`](https://github.com/apache/spark/commit/c297e6a8b4a27b31b783daf5a1264eb8b4d37447). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #27348: [SPARK-30627][SQL] Disable all the V2 file sources by default
gengliangwang commented on a change in pull request #27348: [SPARK-30627][SQL]
Disable all the V2 file sources by default
URL: https://github.com/apache/spark/pull/27348#discussion_r370472102
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/connector/FileDataSourceV2FallBackSuite.scala
##
@@ -86,6 +87,11 @@ class FileDataSourceV2FallBackSuite extends QueryTest with
SharedSparkSession {
private val dummyReadOnlyFileSourceV2 =
classOf[DummyReadOnlyFileDataSourceV2].getName
private val dummyWriteOnlyFileSourceV2 =
classOf[DummyWriteOnlyFileDataSourceV2].getName
+override protected def sparkConf: SparkConf =
Review comment:
Updated, thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering
AmplabJenkins removed a comment on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering URL: https://github.com/apache/spark/pull/27128#issuecomment-571896544 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering
dongjoon-hyun commented on issue #27128: [SPARK-30421][SQL] Dropped columns still available for filtering URL: https://github.com/apache/spark/pull/27128#issuecomment-577988826 ok to test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
dongjoon-hyun commented on issue #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener URL: https://github.com/apache/spark/pull/27030#issuecomment-577987120 Also, I'm worrying the situation where this implementation causes a sabotage on the event message pipeline. Sometimes, the changed number of partition can be huge in the production. Hi, @hvanhovell . How do you think about this PR? Could you give us some advice? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL] Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
dongjoon-hyun commented on a change in pull request #27030: [SPARK-30244][SQL]
Emit pre/post events for "Partition" methods in ExternalCatalogWithListener
URL: https://github.com/apache/spark/pull/27030#discussion_r370469963
##
File path:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogEventSuite.scala
##
@@ -209,4 +209,67 @@ class ExternalCatalogEventSuite extends SparkFunSuite {
catalog.dropFunction("db5", "fn4")
checkEvents(DropFunctionPreEvent("db5", "fn4") :: DropFunctionEvent("db5",
"fn4") :: Nil)
}
+
+ testWithCatalog("partition") { (catalog, checkEvents) =>
+val path1 = Files.createTempDirectory("db_")
+val path2 = Files.createTempDirectory(path1, "tbl_")
+val uri1 = preparePath(path1)
+val uri2 = preparePath(path2)
+
+// CREATE
+val dbDefinition = createDbDefinition(uri1)
+
+val storage = CatalogStorageFormat.empty.copy(
+ locationUri = Option(uri2))
+
+val tableDefinition = CatalogTable(
+ identifier = TableIdentifier("tbl1", Some("db5")),
+ tableType = CatalogTableType.MANAGED,
+ storage = storage,
+ schema = new StructType().add("id", "long"))
+
+val partition = CatalogTablePartition(spec = Map("key1" -> "1", "key2" ->
"2"),
+ storage = CatalogStorageFormat.empty)
+
+catalog.createDatabase(dbDefinition, ignoreIfExists = false)
+checkEvents(CreateDatabasePreEvent("db5") :: CreateDatabaseEvent("db5") ::
Nil)
+
+catalog.createTable(tableDefinition, ignoreIfExists = false)
+checkEvents(CreateTablePreEvent("db5", "tbl1") :: CreateTableEvent("db5",
"tbl1") :: Nil)
+
+catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists =
false)
+checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+catalog.createPartitions("db5", "tbl1", Seq(partition), ignoreIfExists =
true)
+checkEvents(CreatePartitionPreEvent("db5", "tbl1", Seq(partition)) ::
+ CreatePartitionEvent("db5", "tbl1", Seq(partition)) :: Nil)
+
+// RENAME
+val newPartition = CatalogTablePartition(spec = Map("key1" -> "3", "key2"
-> "4"),
+ storage = CatalogStorageFormat.empty)
+
+catalog.renamePartitions("db5", "tbl1", Seq(partition.spec),
Seq(newPartition.spec))
+checkEvents(
+ RenamePartitionPreEvent("db5", "tbl1", Seq(partition.spec),
Seq(newPartition.spec)) ::
+RenamePartitionEvent("db5", "tbl1", Seq(partition.spec),
Seq(newPartition.spec)) ::
+Nil)
+
+// ALTER
+catalog.alterPartitions("db5", "tbl1", Seq.empty)
+checkEvents(AlterPartitionPreEvent("db5", "tbl1", Seq.empty) ::
+ AlterPartitionEvent("db5", "tbl1", Seq.empty) :: Nil)
+
+// DROP
+intercept[AnalysisException] {
+ catalog.dropPartitions("db5", "tbl2", Seq(newPartition.spec),
+ignoreIfNotExists = false, purge = true, retainData = false)
+}
Review comment:
It's always a good practice to check the error message. Sometime,
`AnalysisException` is changed unexpectedly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
