[GitHub] [spark] AmplabJenkins removed a comment on pull request #28860: [SPARK-32002][SQL]Support ExtractValue from nested ArrayStruct
AmplabJenkins removed a comment on pull request #28860: URL: https://github.com/apache/spark/pull/28860#issuecomment-646952997 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28860: [SPARK-32002][SQL]Support ExtractValue from nested ArrayStruct
AmplabJenkins commented on pull request #28860: URL: https://github.com/apache/spark/pull/28860#issuecomment-646952997 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28860: [SPARK-32002][SQL]Support ExtractValue from nested ArrayStruct
SparkQA commented on pull request #28860: URL: https://github.com/apache/spark/pull/28860#issuecomment-646952841 **[Test build #124314 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124314/testReport)** for PR 28860 at commit [`b6e92c0`](https://github.com/apache/spark/commit/b6e92c025a4f3746165a3ba660fe80cca63bb91f). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on pull request #28860: [SPARK-32002][SQL]Support ExtractValue from nested ArrayStruct
AngersZh commented on pull request #28860: URL: https://github.com/apache/spark/pull/28860#issuecomment-646952561 Seems this way will change the schema, maybe we should add a new class to get multidimensional array form GetArrayStruct This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28875: [SPARK-32030][SQL] Support unlimited MATCHED and NOT MATCHED clauses in MERGE INTO
AmplabJenkins removed a comment on pull request #28875: URL: https://github.com/apache/spark/pull/28875#issuecomment-646951851 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28875: [SPARK-32030][SQL] Support unlimited MATCHED and NOT MATCHED clauses in MERGE INTO
AmplabJenkins commented on pull request #28875: URL: https://github.com/apache/spark/pull/28875#issuecomment-646951851 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28875: [SPARK-32030][SQL] Support unlimited MATCHED and NOT MATCHED clauses in MERGE INTO
SparkQA commented on pull request #28875: URL: https://github.com/apache/spark/pull/28875#issuecomment-646951464 **[Test build #124306 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124306/testReport)** for PR 28875 at commit [`e18a7a5`](https://github.com/apache/spark/commit/e18a7a52ccf3da3689b6bfc3a623c8d608814ab4). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class DeleteAction(override val condition: Option[Expression]) extends MergeAction(condition)` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28875: [SPARK-32030][SQL] Support unlimited MATCHED and NOT MATCHED clauses in MERGE INTO
SparkQA removed a comment on pull request #28875: URL: https://github.com/apache/spark/pull/28875#issuecomment-646914074 **[Test build #124306 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124306/testReport)** for PR 28875 at commit [`e18a7a5`](https://github.com/apache/spark/commit/e18a7a52ccf3da3689b6bfc3a623c8d608814ab4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28863: [SPARK-31336][SQL] Support Oracle Kerberos login in JDBC connector
maropu commented on a change in pull request #28863:
URL: https://github.com/apache/spark/pull/28863#discussion_r443106703
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/jdbc/connection/OracleConnectionProviderSuite.scala
##
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.jdbc.connection
+
+class OracleConnectionProviderSuite extends ConnectionProviderSuiteBase {
+ test("setAuthenticationConfigIfNeeded must set authentication if not set") {
Review comment:
All the `XXXConnectionProviderSuite` has the almost same test, so could
you move it into `ConnectionProviderSuiteBase`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28863: [SPARK-31336][SQL] Support Oracle Kerberos login in JDBC connector
maropu commented on a change in pull request #28863: URL: https://github.com/apache/spark/pull/28863#discussion_r443106215 ## File path: external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala ## @@ -32,35 +32,29 @@ import org.apache.spark.tags.DockerTest /** * This patch was tested using the Oracle docker. Created this integration suite for the same. - * The ojdbc6-11.2.0.2.0.jar was to be downloaded from the maven repository. Since there was - * no jdbc jar available in the maven repository, the jar was downloaded from oracle site - * manually and installed in the local; thus tested. So, for SparkQA test case run, the - * ojdbc jar might be manually placed in the local maven repository(com/oracle/ojdbc6/11.2.0.2.0) - * while Spark QA test run. * * The following would be the steps to test this * 1. Build Oracle database in Docker, please refer below link about how to. * https://github.com/oracle/docker-images/blob/master/OracleDatabase/SingleInstance/README.md * 2. export ORACLE_DOCKER_IMAGE_NAME=$ORACLE_DOCKER_IMAGE_NAME *Pull oracle $ORACLE_DOCKER_IMAGE_NAME image - docker pull $ORACLE_DOCKER_IMAGE_NAME * 3. Start docker - sudo service docker start - * 4. Download oracle 11g driver jar and put it in maven local repo: - *(com/oracle/ojdbc6/11.2.0.2.0/ojdbc6-11.2.0.2.0.jar) - * 5. The timeout and interval parameter to be increased from 60,1 to a high value for oracle test - *in DockerJDBCIntegrationSuite.scala (Locally tested with 200,200 and executed successfully). - * 6. Run spark test - ./build/sbt "test-only org.apache.spark.sql.jdbc.OracleIntegrationSuite" + * 4. The timeout and interval parameter to be increased to a high value for oracle test in + * DockerJDBCIntegrationSuite.scala (Locally tested with timeout(20.minutes), interval(1.second) + * and executed successfully). Review comment: How about rephrasing it a bit like this? ``` * 4. The timeout and interval parameter to be increased to a high value for oracle test in *DockerJDBCIntegrationSuite.scala (Locally tested with 20 min timeout and 1 sec interval *then executed successfully). ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28863: [SPARK-31336][SQL] Support Oracle Kerberos login in JDBC connector
maropu commented on a change in pull request #28863: URL: https://github.com/apache/spark/pull/28863#discussion_r443105992 ## File path: external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala ## @@ -32,35 +32,29 @@ import org.apache.spark.tags.DockerTest /** * This patch was tested using the Oracle docker. Created this integration suite for the same. - * The ojdbc6-11.2.0.2.0.jar was to be downloaded from the maven repository. Since there was - * no jdbc jar available in the maven repository, the jar was downloaded from oracle site - * manually and installed in the local; thus tested. So, for SparkQA test case run, the - * ojdbc jar might be manually placed in the local maven repository(com/oracle/ojdbc6/11.2.0.2.0) - * while Spark QA test run. * * The following would be the steps to test this * 1. Build Oracle database in Docker, please refer below link about how to. * https://github.com/oracle/docker-images/blob/master/OracleDatabase/SingleInstance/README.md * 2. export ORACLE_DOCKER_IMAGE_NAME=$ORACLE_DOCKER_IMAGE_NAME *Pull oracle $ORACLE_DOCKER_IMAGE_NAME image - docker pull $ORACLE_DOCKER_IMAGE_NAME * 3. Start docker - sudo service docker start - * 4. Download oracle 11g driver jar and put it in maven local repo: - *(com/oracle/ojdbc6/11.2.0.2.0/ojdbc6-11.2.0.2.0.jar) - * 5. The timeout and interval parameter to be increased from 60,1 to a high value for oracle test - *in DockerJDBCIntegrationSuite.scala (Locally tested with 200,200 and executed successfully). - * 6. Run spark test - ./build/sbt "test-only org.apache.spark.sql.jdbc.OracleIntegrationSuite" + * 4. The timeout and interval parameter to be increased to a high value for oracle test in + * DockerJDBCIntegrationSuite.scala (Locally tested with timeout(20.minutes), interval(1.second) + * and executed successfully). + * 5. Run spark test - ./build/sbt "test-only org.apache.spark.sql.jdbc.OracleIntegrationSuite" * - * All tests in this suite are ignored because of the dependency with the oracle jar from maven - * repository. + * It has been validated with 18.4.0 Express Edition. Review comment: This info looks useful, so how about moving it into the comment side like this? ``` * ... * 5. Run spark test - ./build/sbt "test-only org.apache.spark.sql.jdbc.OracleIntegrationSuite" * * An actual sequence of commands to run the test is as follows * * $ git clone https://github.com/oracle/docker-images.git * // Head SHA: 3e352a22618070595f823977a0fd1a3a8071a83c * $ cd docker-images/OracleDatabase/SingleInstance/dockerfiles * $ ./buildDockerImage.sh -v 18.4.0 -x * $ export ORACLE_DOCKER_IMAGE_NAME=oracle/database:18.4.0-xe * $ cd $SPARK_HOME * $ ./build/sbt "test-only org.apache.spark.sql.jdbc.OracleIntegrationSuite" * * It has been validated with 18.4.0 Express Edition. */ ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28868: [SPARK-32029][SQL] Make active session null when application end
AmplabJenkins removed a comment on pull request #28868: URL: https://github.com/apache/spark/pull/28868#issuecomment-646947829 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124312/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28868: [SPARK-32029][SQL] Make active session null when application end
AmplabJenkins commented on pull request #28868: URL: https://github.com/apache/spark/pull/28868#issuecomment-646947826 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28868: [SPARK-32029][SQL] Make active session null when application end
AmplabJenkins removed a comment on pull request #28868: URL: https://github.com/apache/spark/pull/28868#issuecomment-646947826 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28868: [SPARK-32029][SQL] Make active session null when application end
SparkQA removed a comment on pull request #28868: URL: https://github.com/apache/spark/pull/28868#issuecomment-646937285 **[Test build #124312 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124312/testReport)** for PR 28868 at commit [`a59119f`](https://github.com/apache/spark/commit/a59119f7369a7e8560c179698c85b9a7437899d5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28868: [SPARK-32029][SQL] Make active session null when application end
SparkQA commented on pull request #28868: URL: https://github.com/apache/spark/pull/28868#issuecomment-646947759 **[Test build #124312 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124312/testReport)** for PR 28868 at commit [`a59119f`](https://github.com/apache/spark/commit/a59119f7369a7e8560c179698c85b9a7437899d5). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28863: [SPARK-31336][SQL] Support Oracle Kerberos login in JDBC connector
maropu commented on a change in pull request #28863:
URL: https://github.com/apache/spark/pull/28863#discussion_r443105104
##
File path:
external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala
##
@@ -69,6 +63,7 @@ class OracleIntegrationSuite extends
DockerJDBCIntegrationSuite with SharedSpark
}
override def dataPreparation(conn: Connection): Unit = {
+conn.setAutoCommit(false)
Review comment:
Could you leave the comment in the code side?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28863: [SPARK-31336][SQL] Support Oracle Kerberos login in JDBC connector
maropu commented on a change in pull request #28863: URL: https://github.com/apache/spark/pull/28863#discussion_r443105054 ## File path: external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala ## @@ -32,35 +32,29 @@ import org.apache.spark.tags.DockerTest /** * This patch was tested using the Oracle docker. Created this integration suite for the same. - * The ojdbc6-11.2.0.2.0.jar was to be downloaded from the maven repository. Since there was - * no jdbc jar available in the maven repository, the jar was downloaded from oracle site - * manually and installed in the local; thus tested. So, for SparkQA test case run, the - * ojdbc jar might be manually placed in the local maven repository(com/oracle/ojdbc6/11.2.0.2.0) - * while Spark QA test run. * * The following would be the steps to test this * 1. Build Oracle database in Docker, please refer below link about how to. * https://github.com/oracle/docker-images/blob/master/OracleDatabase/SingleInstance/README.md * 2. export ORACLE_DOCKER_IMAGE_NAME=$ORACLE_DOCKER_IMAGE_NAME *Pull oracle $ORACLE_DOCKER_IMAGE_NAME image - docker pull $ORACLE_DOCKER_IMAGE_NAME * 3. Start docker - sudo service docker start - * 4. Download oracle 11g driver jar and put it in maven local repo: - *(com/oracle/ojdbc6/11.2.0.2.0/ojdbc6-11.2.0.2.0.jar) - * 5. The timeout and interval parameter to be increased from 60,1 to a high value for oracle test - *in DockerJDBCIntegrationSuite.scala (Locally tested with 200,200 and executed successfully). - * 6. Run spark test - ./build/sbt "test-only org.apache.spark.sql.jdbc.OracleIntegrationSuite" + * 4. The timeout and interval parameter to be increased to a high value for oracle test in + * DockerJDBCIntegrationSuite.scala (Locally tested with timeout(20.minutes), interval(1.second) + * and executed successfully). Review comment: nit: looks wrong indents. ## File path: external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala ## @@ -32,35 +32,29 @@ import org.apache.spark.tags.DockerTest /** * This patch was tested using the Oracle docker. Created this integration suite for the same. - * The ojdbc6-11.2.0.2.0.jar was to be downloaded from the maven repository. Since there was - * no jdbc jar available in the maven repository, the jar was downloaded from oracle site - * manually and installed in the local; thus tested. So, for SparkQA test case run, the - * ojdbc jar might be manually placed in the local maven repository(com/oracle/ojdbc6/11.2.0.2.0) - * while Spark QA test run. * * The following would be the steps to test this * 1. Build Oracle database in Docker, please refer below link about how to. * https://github.com/oracle/docker-images/blob/master/OracleDatabase/SingleInstance/README.md * 2. export ORACLE_DOCKER_IMAGE_NAME=$ORACLE_DOCKER_IMAGE_NAME *Pull oracle $ORACLE_DOCKER_IMAGE_NAME image - docker pull $ORACLE_DOCKER_IMAGE_NAME * 3. Start docker - sudo service docker start - * 4. Download oracle 11g driver jar and put it in maven local repo: - *(com/oracle/ojdbc6/11.2.0.2.0/ojdbc6-11.2.0.2.0.jar) - * 5. The timeout and interval parameter to be increased from 60,1 to a high value for oracle test - *in DockerJDBCIntegrationSuite.scala (Locally tested with 200,200 and executed successfully). - * 6. Run spark test - ./build/sbt "test-only org.apache.spark.sql.jdbc.OracleIntegrationSuite" + * 4. The timeout and interval parameter to be increased to a high value for oracle test in + * DockerJDBCIntegrationSuite.scala (Locally tested with timeout(20.minutes), interval(1.second) + * and executed successfully). + * 5. Run spark test - ./build/sbt "test-only org.apache.spark.sql.jdbc.OracleIntegrationSuite" * - * All tests in this suite are ignored because of the dependency with the oracle jar from maven - * repository. + * It has been validated with 18.4.0 Express Edition. */ + Review comment: nit: plz remove the blank. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28863: [SPARK-31336][SQL] Support Oracle Kerberos login in JDBC connector
maropu commented on a change in pull request #28863: URL: https://github.com/apache/spark/pull/28863#discussion_r443104744 ## File path: external/docker-integration-tests/pom.xml ## @@ -130,15 +130,9 @@ postgresql test - - - com.oracle - ojdbc6 - 11.2.0.1.0 + + com.oracle.database.jdbc + ojdbc8 Review comment: +1; this fix looks separate from this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata
AmplabJenkins removed a comment on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-646945193 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124308/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata
AmplabJenkins removed a comment on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-646945191 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata
AmplabJenkins commented on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-646945191 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata
SparkQA commented on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-646945067 **[Test build #124308 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124308/testReport)** for PR 28616 at commit [`4fd056d`](https://github.com/apache/spark/commit/4fd056dffc46e3db9547133c97589e0e2aba7f77). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata
SparkQA removed a comment on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-646927958 **[Test build #124308 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124308/testReport)** for PR 28616 at commit [`4fd056d`](https://github.com/apache/spark/commit/4fd056dffc46e3db9547133c97589e0e2aba7f77). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on pull request #28850: [SPARK-32015][Core]Remote inheritable thread local variables after spark context is stopped
gatorsmile commented on pull request #28850: URL: https://github.com/apache/spark/pull/28850#issuecomment-646944392 cc @Ngone51 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on pull request #28863: [SPARK-31336][SQL] Support Oracle Kerberos login in JDBC connector
gatorsmile commented on pull request #28863: URL: https://github.com/apache/spark/pull/28863#issuecomment-646940577 cc @maropu @MaxGekk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on pull request #28860: [SPARK-32002][SQL]Support ExtractValue from nested ArrayStruct
gatorsmile commented on pull request #28860: URL: https://github.com/apache/spark/pull/28860#issuecomment-646940477 cc @MaxGekk @HyukjinKwon This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on pull request #28859: [SPARK-32024][WEBUI] Update ApplicationStoreInfo.size during HistoryServerDiskManager initializing
gatorsmile commented on pull request #28859: URL: https://github.com/apache/spark/pull/28859#issuecomment-646937829 cc @gengliangwang This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28840: [SPARK-31999][SQL] Add REFRESH FUNCTION command
AmplabJenkins removed a comment on pull request #28840: URL: https://github.com/apache/spark/pull/28840#issuecomment-646937453 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28840: [SPARK-31999][SQL] Add REFRESH FUNCTION command
AmplabJenkins commented on pull request #28840: URL: https://github.com/apache/spark/pull/28840#issuecomment-646937453 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28840: [SPARK-31999][SQL] Add REFRESH FUNCTION command
SparkQA commented on pull request #28840: URL: https://github.com/apache/spark/pull/28840#issuecomment-646937314 **[Test build #124313 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124313/testReport)** for PR 28840 at commit [`6cb2edd`](https://github.com/apache/spark/commit/6cb2edd60e1a74ee7f0464d816d97b72d4a20ef3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28868: [SPARK-32029][SQL] Make active session null when application end
SparkQA commented on pull request #28868: URL: https://github.com/apache/spark/pull/28868#issuecomment-646937285 **[Test build #124312 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124312/testReport)** for PR 28868 at commit [`a59119f`](https://github.com/apache/spark/commit/a59119f7369a7e8560c179698c85b9a7437899d5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28868: [SPARK-32029][SQL] Make active session null when application end
AmplabJenkins removed a comment on pull request #28868: URL: https://github.com/apache/spark/pull/28868#issuecomment-646936353 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28853: [SPARK-32019][SQL] Add spark.sql.files.minPartitionNum config
AmplabJenkins removed a comment on pull request #28853: URL: https://github.com/apache/spark/pull/28853#issuecomment-646936393 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28868: [SPARK-32029][SQL] Make active session null when application end
AmplabJenkins commented on pull request #28868: URL: https://github.com/apache/spark/pull/28868#issuecomment-646936353 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28853: [SPARK-32019][SQL] Add spark.sql.files.minPartitionNum config
AmplabJenkins commented on pull request #28853: URL: https://github.com/apache/spark/pull/28853#issuecomment-646936393 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28853: [SPARK-32019][SQL] Add spark.sql.files.minPartitionNum config
SparkQA commented on pull request #28853: URL: https://github.com/apache/spark/pull/28853#issuecomment-646936144 **[Test build #124311 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124311/testReport)** for PR 28853 at commit [`1fb9dc6`](https://github.com/apache/spark/commit/1fb9dc651d5e1041fef8612bdbd3299dcea494a5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28804: [SPARK-31973][SQL] Add ability to disable Sort,Spill in Partial aggregation
AmplabJenkins removed a comment on pull request #28804: URL: https://github.com/apache/spark/pull/28804#issuecomment-646935915 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28804: [SPARK-31973][SQL] Add ability to disable Sort,Spill in Partial aggregation
AmplabJenkins commented on pull request #28804: URL: https://github.com/apache/spark/pull/28804#issuecomment-646935915 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28804: [SPARK-31973][SQL] Add ability to disable Sort,Spill in Partial aggregation
SparkQA removed a comment on pull request #28804: URL: https://github.com/apache/spark/pull/28804#issuecomment-646893545 **[Test build #124304 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124304/testReport)** for PR 28804 at commit [`56c95e2`](https://github.com/apache/spark/commit/56c95e242126d7aacdb4862adc5e094b4e29561b). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28804: [SPARK-31973][SQL] Add ability to disable Sort,Spill in Partial aggregation
SparkQA commented on pull request #28804: URL: https://github.com/apache/spark/pull/28804#issuecomment-646935525 **[Test build #124304 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124304/testReport)** for PR 28804 at commit [`56c95e2`](https://github.com/apache/spark/commit/56c95e242126d7aacdb4862adc5e094b4e29561b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] siknezevic edited a comment on pull request #27246: [SPARK-30536][CORE][SQL] Sort-merge join operator spilling performance improvements
siknezevic edited a comment on pull request #27246: URL: https://github.com/apache/spark/pull/27246#issuecomment-646931795 > > Could you please let me know would it be OK to hard-code the read buffer size to 1024? > > You think the performance is independent of running platforms, e.g., CPU arch and disk I/O? I'm not 100% sure that the `1024` value is the best on our supported platforms... > > > With 10TB TPCDS data set I tested spilling with query q14a and buffer size of 1024. Execution with hard-coded read buffer size is faster by 37% (27 min vs 37 min) comparing to the execution when buffer size is parameterized and the same size 1024 is used. Query q14a, for 10TB data set, generates around 180 million joins per partition and when buffer size is parameterized, that translates into 10 min longer execution time. > > Why does the parameterized one have so much overhead? Not sure. It looks that call to package.scala to read parameter takes some time. And that time is big enough to cause performance hit because it is executed for each join row. In the case of 10TB data set there is around 180 million rows per partition. The code change that I did was to put 1024 when array is created and comment call to read parameter from package.scala. That is all. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 commented on pull request #28819: [SPARK-31980][SQL]Function sequence() fails if start and end of range are equal dates
TJX2014 commented on pull request #28819: URL: https://github.com/apache/spark/pull/28819#issuecomment-646934894 Thanks, I will make a PR for branch-2.4. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 removed a comment on pull request #28819: [SPARK-31980][SQL]Function sequence() fails if start and end of range are equal dates
TJX2014 removed a comment on pull request #28819: URL: https://github.com/apache/spark/pull/28819#issuecomment-646933713 @dongjoon-hyun Thanks, I am willing to, I still have a question, Could you please help me check.Could I base `master` to use `CalendarInterval.fromString` instead of `stringToInterval` or just base `branch-2.4` to use `CalendarInterval.fromString` but base `master` to use `stringToInterval`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] TJX2014 commented on pull request #28819: [SPARK-31980][SQL]Function sequence() fails if start and end of range are equal dates
TJX2014 commented on pull request #28819: URL: https://github.com/apache/spark/pull/28819#issuecomment-646933713 @dongjoon-hyun Thanks, I am willing to, I still have a question, Could you please help me check.Could I base `master` to use `CalendarInterval.fromString` instead of `stringToInterval` or just base `branch-2.4` to use `CalendarInterval.fromString` but base `master` to use `stringToInterval`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] siknezevic commented on pull request #27246: [SPARK-30536][CORE][SQL] Sort-merge join operator spilling performance improvements
siknezevic commented on pull request #27246: URL: https://github.com/apache/spark/pull/27246#issuecomment-646931795 > > Could you please let me know would it be OK to hard-code the read buffer size to 1024? > > You think the performance is independent of running platforms, e.g., CPU arch and disk I/O? I'm not 100% sure that the `1024` value is the best on our supported platforms... > > > With 10TB TPCDS data set I tested spilling with query q14a and buffer size of 1024. Execution with hard-coded read buffer size is faster by 37% (27 min vs 37 min) comparing to the execution when buffer size is parameterized and the same size 1024 is used. Query q14a, for 10TB data set, generates around 180 million joins per partition and when buffer size is parameterized, that translates into 10 min longer execution time. > > Why does the parameterized one have so much overhead? Not sure. It looks that call to package.scala to read parameter takes some time. And that time is big enough to cause performance hit because it is executed for each join row. In the case of 10TB data set there is around 180 million rows per partition. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28683: [SPARK-31875][SQL] Provide a option to disable user supplied Hints
AmplabJenkins removed a comment on pull request #28683: URL: https://github.com/apache/spark/pull/28683#issuecomment-646931118 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28683: [SPARK-31875][SQL] Provide a option to disable user supplied Hints
AmplabJenkins commented on pull request #28683: URL: https://github.com/apache/spark/pull/28683#issuecomment-646931118 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dilipbiswal commented on a change in pull request #28683: [SPARK-31875][SQL] Provide a option to disable user supplied Hints
dilipbiswal commented on a change in pull request #28683:
URL: https://github.com/apache/spark/pull/28683#discussion_r443096878
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -2065,6 +2065,15 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val OPTIMIZER_HINTS_ENABLED =
+buildConf("spark.sql.optimizer.hints.enabled")
Review comment:
@dongjoon-hyun Thank you. I have made the change.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28683: [SPARK-31875][SQL] Provide a option to disable user supplied Hints
SparkQA commented on pull request #28683: URL: https://github.com/apache/spark/pull/28683#issuecomment-646930991 **[Test build #124310 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124310/testReport)** for PR 28683 at commit [`dd06548`](https://github.com/apache/spark/commit/dd06548de900ec8fc103624d4cd1d5792448c820). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.
AmplabJenkins removed a comment on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-646929406 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124309/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.
SparkQA removed a comment on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-646929024 **[Test build #124309 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124309/testReport)** for PR 28528 at commit [`e2ebe65`](https://github.com/apache/spark/commit/e2ebe658c7352d393ded48a63f37a85b23e199d3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.
AmplabJenkins removed a comment on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-646929403 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.
AmplabJenkins commented on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-646929403 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.
AmplabJenkins removed a comment on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-646929262 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.
SparkQA commented on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-646929396 **[Test build #124309 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124309/testReport)** for PR 28528 at commit [`e2ebe65`](https://github.com/apache/spark/commit/e2ebe658c7352d393ded48a63f37a85b23e199d3). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.
AmplabJenkins commented on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-646929262 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.
SparkQA commented on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-646929024 **[Test build #124309 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124309/testReport)** for PR 28528 at commit [`e2ebe65`](https://github.com/apache/spark/commit/e2ebe658c7352d393ded48a63f37a85b23e199d3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28618: [SPARK-31801][WIP][API][SHUFFLE] Register map output metadata
AmplabJenkins removed a comment on pull request #28618: URL: https://github.com/apache/spark/pull/28618#issuecomment-646928295 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124307/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28528: [SPARK-31711][CORE] Register the executor source with the metrics system when running in local mode.
dongjoon-hyun commented on pull request #28528: URL: https://github.com/apache/spark/pull/28528#issuecomment-646928529 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28618: [SPARK-31801][WIP][API][SHUFFLE] Register map output metadata
AmplabJenkins removed a comment on pull request #28618: URL: https://github.com/apache/spark/pull/28618#issuecomment-646928291 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28618: [SPARK-31801][WIP][API][SHUFFLE] Register map output metadata
SparkQA removed a comment on pull request #28618: URL: https://github.com/apache/spark/pull/28618#issuecomment-646927922 **[Test build #124307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124307/testReport)** for PR 28618 at commit [`e7c9988`](https://github.com/apache/spark/commit/e7c998814df365c726d6615da4b7a14b2ba2167c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28618: [SPARK-31801][WIP][API][SHUFFLE] Register map output metadata
AmplabJenkins commented on pull request #28618: URL: https://github.com/apache/spark/pull/28618#issuecomment-646928291 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata
AmplabJenkins removed a comment on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-646928173 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28618: [SPARK-31801][WIP][API][SHUFFLE] Register map output metadata
SparkQA commented on pull request #28618: URL: https://github.com/apache/spark/pull/28618#issuecomment-646928282 **[Test build #124307 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124307/testReport)** for PR 28618 at commit [`e7c9988`](https://github.com/apache/spark/commit/e7c998814df365c726d6615da4b7a14b2ba2167c). * This patch **fails build dependency tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28618: [SPARK-31801][WIP][API][SHUFFLE] Register map output metadata
AmplabJenkins commented on pull request #28618: URL: https://github.com/apache/spark/pull/28618#issuecomment-646928142 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata
AmplabJenkins commented on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-646928173 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28618: [SPARK-31801][WIP][API][SHUFFLE] Register map output metadata
AmplabJenkins removed a comment on pull request #28618: URL: https://github.com/apache/spark/pull/28618#issuecomment-646928142 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata
SparkQA commented on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-646927958 **[Test build #124308 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124308/testReport)** for PR 28616 at commit [`4fd056d`](https://github.com/apache/spark/commit/4fd056dffc46e3db9547133c97589e0e2aba7f77). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28618: [SPARK-31801][WIP][API][SHUFFLE] Register map output metadata
SparkQA commented on pull request #28618: URL: https://github.com/apache/spark/pull/28618#issuecomment-646927922 **[Test build #124307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124307/testReport)** for PR 28618 at commit [`e7c9988`](https://github.com/apache/spark/commit/e7c998814df365c726d6615da4b7a14b2ba2167c). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28784: [SPARK-31957][SQL] Cleanup hive scratch dir for the developer api startWithContext
yaooqinn commented on pull request #28784: URL: https://github.com/apache/spark/pull/28784#issuecomment-646927538 Thank you all for reviewing and merging This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata
dongjoon-hyun commented on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-646927265 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28618: [SPARK-31801][WIP][API][SHUFFLE] Register map output metadata
dongjoon-hyun commented on pull request #28618: URL: https://github.com/apache/spark/pull/28618#issuecomment-646927049 Retest this please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #28784: [SPARK-31957][SQL] Cleanup hive scratch dir for the developer api startWithContext
dongjoon-hyun closed pull request #28784: URL: https://github.com/apache/spark/pull/28784 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28683: [SPARK-31875][SQL] Provide a option to disable user supplied Hints
dongjoon-hyun commented on pull request #28683: URL: https://github.com/apache/spark/pull/28683#issuecomment-646925511 Thank you, @dilipbiswal . The feature looks useful to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28683: [SPARK-31875][SQL] Provide a option to disable user supplied Hints
dongjoon-hyun commented on a change in pull request #28683:
URL: https://github.com/apache/spark/pull/28683#discussion_r443094576
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##
@@ -2065,6 +2065,15 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val OPTIMIZER_HINTS_ENABLED =
+buildConf("spark.sql.optimizer.hints.enabled")
Review comment:
Can we have more direct names like `OPTIMIZER_IGNORE_HINTS`? Maybe,
`spark.sql.optimizer.ignoreHints.enabled` like `spark.files.ignoreMissingFiles`
or `spark.files.ignoreCorruptFiles`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28683: [SPARK-31875][SQL] Provide a option to disable user supplied Hints
dongjoon-hyun commented on pull request #28683: URL: https://github.com/apache/spark/pull/28683#issuecomment-646924782 Could you resolve a conflict, @dilipbiswal ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28873: [SPARK-32021][SQL] Increase precision of seconds and fractions of `make_interval`
dongjoon-hyun edited a comment on pull request #28873: URL: https://github.com/apache/spark/pull/28873#issuecomment-646923926 Could you make a backporting PR to branch-3.0? We need to regenerate some files. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28873: [SPARK-32021][SQL] Increase precision of seconds and fractions of `make_interval`
dongjoon-hyun commented on pull request #28873: URL: https://github.com/apache/spark/pull/28873#issuecomment-646923926 Could you make a backporting PR to branch-3.0? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #28873: [SPARK-32021][SQL] Increase precision of seconds and fractions of `make_interval`
dongjoon-hyun closed pull request #28873: URL: https://github.com/apache/spark/pull/28873 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28819: [SPARK-31980][SQL]Function sequence() fails if start and end of range are equal dates
dongjoon-hyun commented on pull request #28819: URL: https://github.com/apache/spark/pull/28819#issuecomment-646923541 Hi, @TJX2014 . In branch-2.4, `stringToInterval` doesn't exist. Could you make a PR for Apache Spark 2.4.7 please? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #28874: [SPARK-32036] Replace references to blacklist/whitelist language with more appropriate terminology, excluding the blacklisting feature.
holdenk commented on pull request #28874: URL: https://github.com/apache/spark/pull/28874#issuecomment-646923353 Sure, I'm taking this weekend away from coding so I'll get to this early next week. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #28819: [SPARK-31980][SQL]Function sequence() fails if start and end of range are equal dates
dongjoon-hyun closed pull request #28819: URL: https://github.com/apache/spark/pull/28819 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28873: [SPARK-32021][SQL] Increase precision of seconds and fractions of `make_interval`
AmplabJenkins removed a comment on pull request #28873: URL: https://github.com/apache/spark/pull/28873#issuecomment-646920959 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28873: [SPARK-32021][SQL] Increase precision of seconds and fractions of `make_interval`
AmplabJenkins commented on pull request #28873: URL: https://github.com/apache/spark/pull/28873#issuecomment-646920959 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28873: [SPARK-32021][SQL] Increase precision of seconds and fractions of `make_interval`
SparkQA removed a comment on pull request #28873: URL: https://github.com/apache/spark/pull/28873#issuecomment-646867588 **[Test build #124301 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124301/testReport)** for PR 28873 at commit [`3764477`](https://github.com/apache/spark/commit/376447782460cabd206dee59e0668b1af96c7683). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28873: [SPARK-32021][SQL] Increase precision of seconds and fractions of `make_interval`
SparkQA commented on pull request #28873: URL: https://github.com/apache/spark/pull/28873#issuecomment-646920751 **[Test build #124301 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124301/testReport)** for PR 28873 at commit [`3764477`](https://github.com/apache/spark/commit/376447782460cabd206dee59e0668b1af96c7683). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28868: [SPARK-32029][SQL] Make active session null when application end
AmplabJenkins removed a comment on pull request #28868: URL: https://github.com/apache/spark/pull/28868#issuecomment-646916561 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124302/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28868: [SPARK-32029][SQL] Make active session null when application end
SparkQA removed a comment on pull request #28868: URL: https://github.com/apache/spark/pull/28868#issuecomment-646881818 **[Test build #124302 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124302/testReport)** for PR 28868 at commit [`4d7ff58`](https://github.com/apache/spark/commit/4d7ff588b27ee054e9812e0d667d67002265676a). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28868: [SPARK-32029][SQL] Make active session null when application end
AmplabJenkins removed a comment on pull request #28868: URL: https://github.com/apache/spark/pull/28868#issuecomment-646916553 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28868: [SPARK-32029][SQL] Make active session null when application end
AmplabJenkins commented on pull request #28868: URL: https://github.com/apache/spark/pull/28868#issuecomment-646916553 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28868: [SPARK-32029][SQL] Make active session null when application end
SparkQA commented on pull request #28868: URL: https://github.com/apache/spark/pull/28868#issuecomment-646916428 **[Test build #124302 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124302/testReport)** for PR 28868 at commit [`4d7ff58`](https://github.com/apache/spark/commit/4d7ff588b27ee054e9812e0d667d67002265676a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #28874: [SPARK-32036] Replace references to blacklist/whitelist language with more appropriate terminology, excluding the blacklisting
venkata91 commented on a change in pull request #28874:
URL: https://github.com/apache/spark/pull/28874#discussion_r443089270
##
File path: core/src/test/scala/org/apache/spark/ui/UISeleniumSuite.scala
##
@@ -48,24 +48,24 @@ import org.apache.spark.util.CallSite
private[spark] class SparkUICssErrorHandler extends DefaultCssErrorHandler {
- private val cssWhiteList = List("bootstrap.min.css",
"vis-timeline-graph2d.min.css")
+ private val cssExcludeList = List("bootstrap.min.css",
"vis-timeline-graph2d.min.css")
Review comment:
also it seems in some cases we use `exclude` for both `whitelist` as
well as `blacklist`. Like here `exclude` is used for `whiteList` and
[here](https://github.com/apache/spark/blob/8f414bc6b4eeb59203ecb33c26c762a57bf5429e/resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala#L542)
for `blackList`. In general, I prefer `allowedList` for `whitelist` and
`denyList` or `rejectList` or `stopList` etc for `blacklist` makes it easier to
comprehend quickly. I understand its hard to use the same word everywhere
because of the context.
##
File path: core/src/test/scala/org/apache/spark/ui/UISeleniumSuite.scala
##
@@ -48,24 +48,24 @@ import org.apache.spark.util.CallSite
private[spark] class SparkUICssErrorHandler extends DefaultCssErrorHandler {
- private val cssWhiteList = List("bootstrap.min.css",
"vis-timeline-graph2d.min.css")
+ private val cssExcludeList = List("bootstrap.min.css",
"vis-timeline-graph2d.min.css")
Review comment:
same here instead of `exclude` how about `allowed`?
##
File path: R/pkg/tests/fulltests/test_sparkSQL.R
##
@@ -3921,14 +3921,14 @@ test_that("No extra files are created in SPARK_HOME by
starting session and maki
# before creating a SparkSession with enableHiveSupport = T at the top of
this test file
# (filesBefore). The test here is to compare that (filesBefore) against the
list of files before
# any test is run in run-all.R (sparkRFilesBefore).
- # sparkRWhitelistSQLDirs is also defined in run-all.R, and should contain
only 2 whitelisted dirs,
+ # sparkRIncludedSQLDirs is also defined in run-all.R, and should contain
only 2 included dirs,
Review comment:
does it make sense to have `allowed` here instead of `included`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] xianyinxin commented on pull request #28875: [SPARK-32030][SQL] Support unlimited MATCHED and NOT MATCHED clauses in MERGE INTO
xianyinxin commented on pull request #28875: URL: https://github.com/apache/spark/pull/28875#issuecomment-646915406 @cloud-fan @brkyvz , pls take a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28826: [SPARK-31988][SQL] Schema pruning may discard attribute metadata
dongjoon-hyun commented on pull request #28826:
URL: https://github.com/apache/spark/pull/28826#issuecomment-646914222
BTW, @guykhazma . Is the example in the PR description enough? If I follow
the direction, the metadata is not lost.
```
scala> sc.version
res1: String = 3.0.0
...
{"key":"value"}
{}
{"key":"value"}
{}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28826: [SPARK-31988][SQL] Schema pruning may discard attribute metadata
dongjoon-hyun edited a comment on pull request #28826:
URL: https://github.com/apache/spark/pull/28826#issuecomment-646914222
BTW, @guykhazma . Is the example in the PR description enough? If I follow
the direction, the metadata is not lost. If that is insufficient, please add
more steps for the other people.
```
scala> sc.version
res1: String = 3.0.0
...
{"key":"value"}
{}
{"key":"value"}
{}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28875: [SPARK-32030][SQL] Support unlimited MATCHED and NOT MATCHED clauses in MERGE INTO
AmplabJenkins removed a comment on pull request #28875: URL: https://github.com/apache/spark/pull/28875#issuecomment-646914223 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28875: [SPARK-32030][SQL] Support unlimited MATCHED and NOT MATCHED clauses in MERGE INTO
AmplabJenkins commented on pull request #28875: URL: https://github.com/apache/spark/pull/28875#issuecomment-646914223 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28875: [SPARK-32030][SQL] Support unlimited MATCHED and NOT MATCHED clauses in MERGE INTO
SparkQA commented on pull request #28875: URL: https://github.com/apache/spark/pull/28875#issuecomment-646914074 **[Test build #124306 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124306/testReport)** for PR 28875 at commit [`e18a7a5`](https://github.com/apache/spark/commit/e18a7a52ccf3da3689b6bfc3a623c8d608814ab4). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28826: [SPARK-31988][SQL] Schema pruning may discard attribute metadata
dongjoon-hyun edited a comment on pull request #28826: URL: https://github.com/apache/spark/pull/28826#issuecomment-646913172 +1, for @maropu 's suggestion. You can use your example in the PR description, @guykhazma . Also, I have the same question like @viirya . This seems to affect both v1 and v2. Please add a test case for both V1/V2. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
