[GitHub] [spark] AmplabJenkins commented on pull request #28360: [SPARK-31569][SQL][DOCS] Add links to subsections in SQL Reference main page
AmplabJenkins commented on pull request #28360: URL: https://github.com/apache/spark/pull/28360#issuecomment-619742496 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28360: [SPARK-31569][SQL][DOCS] Add links to subsections in SQL Reference main page
AmplabJenkins removed a comment on pull request #28360: URL: https://github.com/apache/spark/pull/28360#issuecomment-619742496 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28360: [SPARK-31569][SQL][DOCS] Add links to subsections in SQL Reference main page
SparkQA commented on pull request #28360: URL: https://github.com/apache/spark/pull/28360#issuecomment-619741862 **[Test build #121876 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121876/testReport)** for PR 28360 at commit [`ea6f546`](https://github.com/apache/spark/commit/ea6f546b47f702f1369f50226463cb34a483f6e9). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao opened a new pull request #28360: [SPARK-31569][SQL][DOCS] Add links to subsections in SQL Reference main page
huaxingao opened a new pull request #28360: URL: https://github.com/apache/spark/pull/28360 ### What changes were proposed in this pull request? Add links to subsections in SQL Reference main page ### Why are the changes needed? Make SQL Reference complete ### Does this PR introduce any user-facing change? Yes before: https://user-images.githubusercontent.com/13592258/80338238-a9551080-8810-11ea-8ae8-d6707fde2cac.png;> after: https://user-images.githubusercontent.com/13592258/80338241-ac500100-8810-11ea-8518-95c4f8c0a2eb.png;> ### How was this patch tested? Manually build and check. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28055: [SPARK-31289][TEST][test-hive1.2] Eliminate org.apache.spark.sql.hive.thriftserver.CliSuite flakiness
yaooqinn commented on pull request #28055: URL: https://github.com/apache/spark/pull/28055#issuecomment-619740866 I will run some tests blindly to see whether it introduce a significant delay, since I am not sure how to mock an ungrateful shutdown at the end of each test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MichaelChirico edited a comment on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
MichaelChirico edited a comment on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619740538 really opened a can of worms here huh. To me, it makes sense to leave the PR as is, then do the `dapply`/`dapplyCollect` centralization in another PR. Have filed this follow-up Jira in any case: https://issues.apache.org/jira/browse/SPARK-31570 I could also try combining them in this PR, I don't think it would be substantially difficult, but I don't know the rules for closing multiple Jiras in the same PR. up to you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MichaelChirico commented on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
MichaelChirico commented on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619740538 really opened a can of worms here huh. To me, it makes sense to leave the PR as is, then do the `dapply`/`dapplyCollect` centralization in another PR. Have filed this follow-up Jira in any case: https://issues.apache.org/jira/browse/SPARK-31568 I could also try combining them in this PR, I don't think it would be substantially difficult, but I don't know the rules for closing multiple Jiras in the same PR. up to you. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
HyukjinKwon commented on pull request #28327: URL: https://github.com/apache/spark/pull/28327#issuecomment-619738372 Merged to master and branch-3.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped
dongjoon-hyun commented on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-619738381 Thank you, @sarutak . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28215: [SPARK-31272][SQL] Support DB2 Kerberos login in JDBC connector
maropu commented on a change in pull request #28215:
URL: https://github.com/apache/spark/pull/28215#discussion_r415526378
##
File path:
external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DB2KrbIntegrationSuite.scala
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc
+
+import java.security.PrivilegedExceptionAction
+import java.sql.Connection
+import javax.security.auth.login.Configuration
+
+import com.spotify.docker.client.messages.{ContainerConfig, HostConfig}
+import org.apache.hadoop.security.{SecurityUtil, UserGroupInformation}
+import
org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod.KERBEROS
+
+import org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions
+import
org.apache.spark.sql.execution.datasources.jdbc.connection.{DB2ConnectionProvider,
SecureConnectionProvider}
+import org.apache.spark.tags.DockerTest
+
+@DockerTest
+class DB2KrbIntegrationSuite extends DockerKrbJDBCIntegrationSuite {
+ override protected val userName = s"db2/$dockerIp"
+ override protected val keytabFileName = "db2.keytab"
+
+ override val db = new DatabaseOnDocker {
+override val imageName = "ibmcom/db2:11.5.0.0a"
+override val env = Map(
+ "DB2INST1_PASSWORD" -> "rootpass",
+ "LICENSE" -> "accept",
+ "DBNAME" -> "db2"
+)
+override val usesIpc = false
+override val jdbcPort = 5
+override val privileged = true
+override def getJdbcUrl(ip: String, port: Int): String =
s"jdbc:db2://$ip:$port/db2"
+override def getJdbcProperties() = {
+ val options = new JDBCOptions(Map[String, String](
+JDBCOptions.JDBC_URL -> getJdbcUrl(dockerIp, externalPort),
+JDBCOptions.JDBC_TABLE_NAME -> "bar",
+JDBCOptions.JDBC_KEYTAB -> keytabFileName,
+JDBCOptions.JDBC_PRINCIPAL -> principal
+ ))
+ new DB2ConnectionProvider(null, options).getAdditionalProperties()
+}
+
+override def beforeContainerStart(
+hostConfigBuilder: HostConfig.Builder,
+containerConfigBuilder: ContainerConfig.Builder): Unit = {
+ copyExecutableResource("db2_krb_setup.sh", initDbDir, replaceIp)
+
+ hostConfigBuilder.appendBinds(
+HostConfig.Bind.from(initDbDir.getAbsolutePath)
+ .to("/var/custom").readOnly(true).build()
+ )
+}
+ }
+
+ override protected def setAuthentication(keytabFile: String, principal:
String): Unit = {
+val config = new SecureConnectionProvider.JDBCConfiguration(
+ Configuration.getConfiguration, "JaasClient", keytabFile, principal)
+Configuration.setConfiguration(config)
Review comment:
Is this safe when scanning tables in different secure databases ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28215: [SPARK-31272][SQL] Support DB2 Kerberos login in JDBC connector
maropu commented on a change in pull request #28215:
URL: https://github.com/apache/spark/pull/28215#discussion_r415526378
##
File path:
external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DB2KrbIntegrationSuite.scala
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.jdbc
+
+import java.security.PrivilegedExceptionAction
+import java.sql.Connection
+import javax.security.auth.login.Configuration
+
+import com.spotify.docker.client.messages.{ContainerConfig, HostConfig}
+import org.apache.hadoop.security.{SecurityUtil, UserGroupInformation}
+import
org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod.KERBEROS
+
+import org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions
+import
org.apache.spark.sql.execution.datasources.jdbc.connection.{DB2ConnectionProvider,
SecureConnectionProvider}
+import org.apache.spark.tags.DockerTest
+
+@DockerTest
+class DB2KrbIntegrationSuite extends DockerKrbJDBCIntegrationSuite {
+ override protected val userName = s"db2/$dockerIp"
+ override protected val keytabFileName = "db2.keytab"
+
+ override val db = new DatabaseOnDocker {
+override val imageName = "ibmcom/db2:11.5.0.0a"
+override val env = Map(
+ "DB2INST1_PASSWORD" -> "rootpass",
+ "LICENSE" -> "accept",
+ "DBNAME" -> "db2"
+)
+override val usesIpc = false
+override val jdbcPort = 5
+override val privileged = true
+override def getJdbcUrl(ip: String, port: Int): String =
s"jdbc:db2://$ip:$port/db2"
+override def getJdbcProperties() = {
+ val options = new JDBCOptions(Map[String, String](
+JDBCOptions.JDBC_URL -> getJdbcUrl(dockerIp, externalPort),
+JDBCOptions.JDBC_TABLE_NAME -> "bar",
+JDBCOptions.JDBC_KEYTAB -> keytabFileName,
+JDBCOptions.JDBC_PRINCIPAL -> principal
+ ))
+ new DB2ConnectionProvider(null, options).getAdditionalProperties()
+}
+
+override def beforeContainerStart(
+hostConfigBuilder: HostConfig.Builder,
+containerConfigBuilder: ContainerConfig.Builder): Unit = {
+ copyExecutableResource("db2_krb_setup.sh", initDbDir, replaceIp)
+
+ hostConfigBuilder.appendBinds(
+HostConfig.Bind.from(initDbDir.getAbsolutePath)
+ .to("/var/custom").readOnly(true).build()
+ )
+}
+ }
+
+ override protected def setAuthentication(keytabFile: String, principal:
String): Unit = {
+val config = new SecureConnectionProvider.JDBCConfiguration(
+ Configuration.getConfiguration, "JaasClient", keytabFile, principal)
+Configuration.setConfiguration(config)
Review comment:
Is this safe when simultaneously scanning tables in different secure
databases ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28356: [SPARK-31485][CORE][3.0] Avoid application hang if only partial barrier tasks launched
AmplabJenkins removed a comment on pull request #28356: URL: https://github.com/apache/spark/pull/28356#issuecomment-619735958 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
AmplabJenkins commented on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619736195 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #28328: [SPARK-31553][SQL] Fix isInCollection for collection sizes above the optimisation threshold
MaxGekk commented on a change in pull request #28328:
URL: https://github.com/apache/spark/pull/28328#discussion_r415525400
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
##
@@ -426,7 +426,10 @@ case class In(value: Expression, list: Seq[Expression])
extends Predicate {
* Optimized version of In clause, when all filter values of In clause are
* static.
*/
-case class InSet(child: Expression, hset: Set[Any]) extends UnaryExpression
with Predicate {
+case class InSet(
+child: Expression,
+hset: Set[Any],
+hsetElemType: DataType) extends UnaryExpression with Predicate {
Review comment:
And `InSet.sql()` is called from `Dataset.select` _.named:
```scala
Project(untypedCols.map(_.named), logicalPlan)
```
The `named` method calls `toPrettySQL(expr)`:
```scala
case expr: Expression => Alias(expr, toPrettySQL(expr))()
```
The `toPrettySQL` method calls `sql`:
```scala
def toPrettySQL(e: Expression): String = usePrettyExpression(e).sql
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
SparkQA commented on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619736026 **[Test build #121874 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121874/testReport)** for PR 28350 at commit [`f73b8a8`](https://github.com/apache/spark/commit/f73b8a83481aae569b504fcddfff43a13af5b844). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
SparkQA removed a comment on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619717524 **[Test build #121874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121874/testReport)** for PR 28350 at commit [`f73b8a8`](https://github.com/apache/spark/commit/f73b8a83481aae569b504fcddfff43a13af5b844). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
AmplabJenkins removed a comment on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619736195 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28356: [SPARK-31485][CORE][3.0] Avoid application hang if only partial barrier tasks launched
AmplabJenkins commented on pull request #28356: URL: https://github.com/apache/spark/pull/28356#issuecomment-619735958 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28356: [SPARK-31485][CORE][3.0] Avoid application hang if only partial barrier tasks launched
SparkQA commented on pull request #28356: URL: https://github.com/apache/spark/pull/28356#issuecomment-619735451 **[Test build #121875 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121875/testReport)** for PR 28356 at commit [`0f9a186`](https://github.com/apache/spark/commit/0f9a18655b53630ad4690ce03a066f76f23f5812). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost
cloud-fan commented on pull request #28326: URL: https://github.com/apache/spark/pull/28326#issuecomment-619735441 > I'm wondering why Alias has an explicit metadata and hides the actual attribute's metadata Sometimes we have to, for example, `a + b as c`, what should be the metadata of `c`? I think the root cause is we rely on column metadata to store some important information, and we should make column metadata reliable as possible as we can. For example, `a as c`, if users do not specify the metadata of `c` explicitly, it should inherit metadata from `a`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28356: [SPARK-31485][CORE][3.0] Avoid application hang if only partial barrier tasks launched
cloud-fan commented on pull request #28356: URL: https://github.com/apache/spark/pull/28356#issuecomment-619733515 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28055: [SPARK-31289][TEST][test-hive1.2] Eliminate org.apache.spark.sql.hive.thriftserver.CliSuite flakiness
cloud-fan commented on pull request #28055: URL: https://github.com/apache/spark/pull/28055#issuecomment-619733134 I'm OK to have this patch to reduce test time, but we may need to update the PR description and do some experiments to prove this does reduce test time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28055: [SPARK-31289][TEST][test-hive1.2] Eliminate org.apache.spark.sql.hive.thriftserver.CliSuite flakiness
yaooqinn commented on pull request #28055: URL: https://github.com/apache/spark/pull/28055#issuecomment-619731713 In the latest master branch, I notice that we will wait for the suite process to be shutdown gracefully for 1 minute, which may reduce the flakiness of CliSuite but introduce more test time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #28328: [SPARK-31553][SQL] Fix isInCollection for collection sizes above the optimisation threshold
MaxGekk commented on a change in pull request #28328:
URL: https://github.com/apache/spark/pull/28328#discussion_r415521493
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
##
@@ -426,7 +426,10 @@ case class In(value: Expression, list: Seq[Expression])
extends Predicate {
* Optimized version of In clause, when all filter values of In clause are
* static.
*/
-case class InSet(child: Expression, hset: Set[Any]) extends UnaryExpression
with Predicate {
+case class InSet(
+child: Expression,
+hset: Set[Any],
+hsetElemType: DataType) extends UnaryExpression with Predicate {
Review comment:
When InSet is created from isInCollection, in that case `child.dataType`
is `NullType`. For example, it is NullType in the test
https://github.com/apache/spark/pull/28328/files#diff-aa655ba249e00d2591b21cf6a360cf82R886
because child is PrettyAttribute when the `sql` method is called.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28310: [SPARK-31527][SQL] date add/subtract interval only allow those day precision in ansi mode
cloud-fan commented on pull request #28310: URL: https://github.com/apache/spark/pull/28310#issuecomment-619730253 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28109: [SPARK-31524][SQL][followup] Add metric to the split task number for skew optimization
cloud-fan commented on a change in pull request #28109:
URL: https://github.com/apache/spark/pull/28109#discussion_r415519376
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala
##
@@ -87,6 +87,7 @@ object SQLMetrics {
private val TIMING_METRIC = "timing"
private val NS_TIMING_METRIC = "nsTiming"
private val AVERAGE_METRIC = "average"
+ private val NUMBER_METRIC = "num"
Review comment:
This is no longer needed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28109: [SPARK-31524][SQL][followup] Add metric to the split task number for skew optimization
cloud-fan commented on a change in pull request #28109:
URL: https://github.com/apache/spark/pull/28109#discussion_r415519482
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CustomShuffleReaderExec.scala
##
@@ -153,7 +170,9 @@ case class CustomShuffleReaderExec private(
} ++ {
if (hasSkewedPartition) {
Map("numSkewedPartitions" ->
-SQLMetrics.createMetric(sparkContext, "number of skewed
partitions"))
+SQLMetrics.createMetric(sparkContext, "number of skewed
partitions"),
+"numSkewedSplits" ->
+ SQLMetrics.createNumMetric(sparkContext, "number of skewed
partition splits"))
Review comment:
We just set the value once. This can be simply `createMetric`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28109: [SPARK-31524][SQL][followup] Add metric to the split task number for skew optimization
cloud-fan commented on a change in pull request #28109:
URL: https://github.com/apache/spark/pull/28109#discussion_r415519133
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CustomShuffleReaderExec.scala
##
@@ -112,6 +113,16 @@ case class CustomShuffleReaderExec private(
}
}
+ @transient private lazy val skewedSpecs:
Option[Seq[PartialReducerPartitionSpec]] = {
+if (hasSkewedPartition) {
+ Some(partitionSpecs.collect {
+case p: PartialReducerPartitionSpec => p
Review comment:
This is very short now. Can we just inline this lazy val?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn edited a comment on pull request #28055: [SPARK-31289][TEST][test-hive1.2] Eliminate org.apache.spark.sql.hive.thriftserver.CliSuite flakiness
yaooqinn edited a comment on pull request #28055: URL: https://github.com/apache/spark/pull/28055#issuecomment-619726953 According to the error stack trace in the failed test, the test failed to instantiate a hive metastore client because of derby requirements. ``` Caused by: ERROR XBM0A: The database directory '/home/jenkins/workspace/SparkPullRequestBuilder/target/tmp/spark-9249ce52-0a06-42b6-a3df-e6295e880df0' exists. However, it does not contain the expected 'service.properties' file. Perhaps Derby was brought down in the middle of creating this database. You may want to delete this directory and try creating the database again. ``` The derby requires the metastore dir does not exist, but it does exist probably due to the test case before it failed to clear the metastore dir This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
AmplabJenkins removed a comment on pull request #28327: URL: https://github.com/apache/spark/pull/28327#issuecomment-619726343 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28328: [SPARK-31553][SQL] Fix isInCollection for collection sizes above the optimisation threshold
cloud-fan commented on a change in pull request #28328:
URL: https://github.com/apache/spark/pull/28328#discussion_r415517516
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/predicates.scala
##
@@ -426,7 +426,10 @@ case class In(value: Expression, list: Seq[Expression])
extends Predicate {
* Optimized version of In clause, when all filter values of In clause are
* static.
*/
-case class InSet(child: Expression, hset: Set[Any]) extends UnaryExpression
with Predicate {
+case class InSet(
+child: Expression,
+hset: Set[Any],
+hsetElemType: DataType) extends UnaryExpression with Predicate {
Review comment:
when `hsetElemType` can be different from `child.dataType`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28356: [SPARK-31485][CORE][3.0] Avoid application hang if only partial barrier tasks launched
AmplabJenkins removed a comment on pull request #28356: URL: https://github.com/apache/spark/pull/28356#issuecomment-619725756 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121862/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #26624: [SPARK-8981][core] Add MDC support in Executor
cloud-fan commented on pull request #26624: URL: https://github.com/apache/spark/pull/26624#issuecomment-619727014 cc @Ngone51 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn commented on pull request #28055: [SPARK-31289][TEST][test-hive1.2] Eliminate org.apache.spark.sql.hive.thriftserver.CliSuite flakiness
yaooqinn commented on pull request #28055: URL: https://github.com/apache/spark/pull/28055#issuecomment-619726953 According to the error stack trace in the failed test, the test failed to instantiate a hive metastore client because of derby requirements. ``` Caused by: ERROR XBM0A: The database directory '/home/jenkins/workspace/SparkPullRequestBuilder/target/tmp/spark-9249ce52-0a06-42b6-a3df-e6295e880df0' exists. However, it does not contain the expected 'service.properties' file. Perhaps Derby was brought down in the middle of creating this database. You may want to delete this directory and try creating the database again. ``` The derby requires the metastore dir does not exist, but it does exist probably due to the test case before it failed to clear the metastore dir This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
AmplabJenkins commented on pull request #28327: URL: https://github.com/apache/spark/pull/28327#issuecomment-619726343 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28356: [SPARK-31485][CORE][3.0] Avoid application hang if only partial barrier tasks launched
AmplabJenkins removed a comment on pull request #28356: URL: https://github.com/apache/spark/pull/28356#issuecomment-619725747 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
SparkQA removed a comment on pull request #28327: URL: https://github.com/apache/spark/pull/28327#issuecomment-619709672 **[Test build #121872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121872/testReport)** for PR 28327 at commit [`3ec641d`](https://github.com/apache/spark/commit/3ec641dd49def557323160a1c6c31c945aca4842). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28356: [SPARK-31485][CORE][3.0] Avoid application hang if only partial barrier tasks launched
AmplabJenkins commented on pull request #28356: URL: https://github.com/apache/spark/pull/28356#issuecomment-619725747 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
SparkQA commented on pull request #28327: URL: https://github.com/apache/spark/pull/28327#issuecomment-619725648 **[Test build #121872 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121872/testReport)** for PR 28327 at commit [`3ec641d`](https://github.com/apache/spark/commit/3ec641dd49def557323160a1c6c31c945aca4842). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28356: [SPARK-31485][CORE][3.0] Avoid application hang if only partial barrier tasks launched
SparkQA removed a comment on pull request #28356: URL: https://github.com/apache/spark/pull/28356#issuecomment-619678050 **[Test build #121862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121862/testReport)** for PR 28356 at commit [`0f9a186`](https://github.com/apache/spark/commit/0f9a18655b53630ad4690ce03a066f76f23f5812). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28356: [SPARK-31485][CORE][3.0] Avoid application hang if only partial barrier tasks launched
SparkQA commented on pull request #28356: URL: https://github.com/apache/spark/pull/28356#issuecomment-619725253 **[Test build #121862 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121862/testReport)** for PR 28356 at commit [`0f9a186`](https://github.com/apache/spark/commit/0f9a18655b53630ad4690ce03a066f76f23f5812). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #28326: [SPARK-27340][SS] Alias on TimeWindow expression cause watermark metadata lost
HeartSaVioR commented on pull request #28326:
URL: https://github.com/apache/spark/pull/28326#issuecomment-619722391
(Please consider that I'm not an expert of SQL area.)
I've read through the code around Alias and played with the reproducer
(test), and feel #24457 is the fix addressing root cause.
Looking into the reproducer with debugger,
```
val left = df1.select('key, window('leftTime, "10 second") as 'leftWindow,
'leftValue)
```
`'leftTime` in here is **unresolved** when applying `as` (hence `window` is
also **unresolved** as well), and `as` sets the metadata with what original
column has whereas the metadata of `'leftWindow` cannot be determined here,
hence the problem arises.
Even without the analysis, logically thinking, I'm wondering why Alias has
an explicit metadata and hides the actual attribute's metadata, except the case
of optimization which should be done without side-effects.
Sorry for the dumb question, but is there any real case to do it, and even
if it's valid, is it intentional to hide the metadata of actual attribute?
Shouldn't we retain the metadata of actual attribute's metadata as well?
If we concern about the performance about not having shortcut of Alias
metadata, below fix may bring same effect with #24457, whereas it only changes
the behavior when Alias renames Column which has unresolved expression:
```
def name(alias: String): Column = withExpr {
normalizedExpr() match {
case ne: NamedExpression if ne.resolved =>
Alias(expr, alias)(explicitMetadata = Some(ne.metadata))
case other => Alias(other, alias)()
}
}
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28358: [WIP][SPARK-31567][R][TESTS] Update AppVeyor R version to 4.0.0
dongjoon-hyun edited a comment on pull request #28358:
URL: https://github.com/apache/spark/pull/28358#issuecomment-619711936
AppVeyor seems to complain at `SparkR` installation.
However, on Mac, installation is okay and JVM call is okay and some SQL
queries (CREATE/INSERT/SELECT) works.
```
R version 4.0.0 (2020-04-24) -- "Arbor Day"
...
> head(sql("select version()"))
version()
1 3.1.0 1951fe1451ae7e145f83f97a961172590a7f4d77
> sparkR.callJStatic("java.lang.System", "getProperty", "java.home")
[1] "/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre"
```
The real situation seems that Spark is hitting another behavior change of R
4.0. I'll investigate this later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun edited a comment on pull request #28358: [WIP][SPARK-31567][R][TESTS] Update AppVeyor R version to 4.0.0
dongjoon-hyun edited a comment on pull request #28358:
URL: https://github.com/apache/spark/pull/28358#issuecomment-619711936
AppVeyor seems to complain at `SparkR` installation.
However, on Mac, installation is okay and JVM call is okay and some SQL
queries (CREATE/INSERT/SELECT) works.
```
R version 4.0.0 (2020-04-24) -- "Arbor Day"
...
> sparkR.callJStatic("java.lang.System", "getProperty", "java.home")
[1] "/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre"
```
The real situation seems that Spark is hitting another behavior change of R
4.0. I'll investigate this later.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28345: [SPARK-31557][SQL] Legacy time parser should return Gregorian days rather than Julian days
cloud-fan commented on pull request #28345: URL: https://github.com/apache/spark/pull/28345#issuecomment-619721051 @MaxGekk can you help to fix the timestamp part as well? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28345: [SPARK-31557][SQL] Legacy time parser should return Gregorian days rather than Julian days
cloud-fan commented on pull request #28345: URL: https://github.com/apache/spark/pull/28345#issuecomment-619719490 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
AmplabJenkins commented on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619718080 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
AmplabJenkins removed a comment on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619716830 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
SparkQA removed a comment on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619712257 **[Test build #121873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121873/testReport)** for PR 28350 at commit [`6a04693`](https://github.com/apache/spark/commit/6a04693ed33da945a3b7b21e8871e48980ac9b36). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
SparkQA commented on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619717524 **[Test build #121874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121874/testReport)** for PR 28350 at commit [`f73b8a8`](https://github.com/apache/spark/commit/f73b8a83481aae569b504fcddfff43a13af5b844). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
AmplabJenkins commented on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619716830 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
SparkQA commented on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619716814 **[Test build #121873 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121873/testReport)** for PR 28350 at commit [`6a04693`](https://github.com/apache/spark/commit/6a04693ed33da945a3b7b21e8871e48980ac9b36). * This patch **fails R style tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #26339: [SPARK-27194][SPARK-29302][SQL] Fix the issue that for dynamic partition overwrite a task would conflict with its speculative t
AmplabJenkins removed a comment on pull request #26339: URL: https://github.com/apache/spark/pull/26339#issuecomment-619712667 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
AmplabJenkins removed a comment on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619712716 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #26339: [SPARK-27194][SPARK-29302][SQL] Fix the issue that for dynamic partition overwrite a task would conflict with its speculative task
AmplabJenkins commented on pull request #26339: URL: https://github.com/apache/spark/pull/26339#issuecomment-619712667 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
AmplabJenkins commented on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619712716 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
SparkQA commented on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619712257 **[Test build #121873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121873/testReport)** for PR 28350 at commit [`6a04693`](https://github.com/apache/spark/commit/6a04693ed33da945a3b7b21e8871e48980ac9b36). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #26339: [SPARK-27194][SPARK-29302][SQL] Fix the issue that for dynamic partition overwrite a task would conflict with its speculative task
SparkQA removed a comment on pull request #26339: URL: https://github.com/apache/spark/pull/26339#issuecomment-619657326 **[Test build #121859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121859/testReport)** for PR 26339 at commit [`756`](https://github.com/apache/spark/commit/756c6d086542b9de0d991a6de6660fc86fca). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #28358: [WIP][SPARK-31567][R][TESTS] Update AppVeyor R version to 4.0.0
dongjoon-hyun commented on pull request #28358: URL: https://github.com/apache/spark/pull/28358#issuecomment-619711936 AppVeyor seems to complain at `SparkR` installation. However, on Mac, installation is okay and some SQL queries (CREATE/INSERT/SELECT) works. The real situation seems that Spark is hitting another behavior change of R 4.0. I'll investigate this later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #26339: [SPARK-27194][SPARK-29302][SQL] Fix the issue that for dynamic partition overwrite a task would conflict with its speculative task
SparkQA commented on pull request #26339: URL: https://github.com/apache/spark/pull/26339#issuecomment-619711667 **[Test build #121859 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121859/testReport)** for PR 26339 at commit [`756`](https://github.com/apache/spark/commit/756c6d086542b9de0d991a6de6660fc86fca). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MichaelChirico commented on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
MichaelChirico commented on pull request #28350:
URL: https://github.com/apache/spark/pull/28350#issuecomment-619711106
@HyukjinKwon finished merging the Rds. Checked the output of
`roxygen2::roxygenize('.')` & looks OK. Thanks for pushing in a better
direction.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MichaelChirico commented on a change in pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
MichaelChirico commented on a change in pull request #28350:
URL: https://github.com/apache/spark/pull/28350#discussion_r415503595
##
File path: R/pkg/R/functions.R
##
@@ -3951,7 +3951,6 @@ setMethod("map_values",
#' @rdname column_collection_functions
#' @aliases map_zip_with
map_zip_with,characterOrColumn,characterOrColumn,function-method
#'
-#' @examples
Review comment:
this was throwing a warning from `roxygen2` about empty `@examples`.
should add an example, but just deleted for now
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
AmplabJenkins commented on pull request #28327: URL: https://github.com/apache/spark/pull/28327#issuecomment-619710075 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
AmplabJenkins removed a comment on pull request #28327: URL: https://github.com/apache/spark/pull/28327#issuecomment-619710075 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
SparkQA commented on pull request #28327: URL: https://github.com/apache/spark/pull/28327#issuecomment-619709672 **[Test build #121872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121872/testReport)** for PR 28327 at commit [`3ec641d`](https://github.com/apache/spark/commit/3ec641dd49def557323160a1c6c31c945aca4842). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28156: [SPARK-31388][SQL][TESTS] org.apache.spark.sql.hive.thriftserver.CliSuite doesn't match results correctly
cloud-fan commented on pull request #28156: URL: https://github.com/apache/spark/pull/28156#issuecomment-619708264 thanks, merging to master/3.0! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #28055: [SPARK-31289][TEST][test-hive1.2] Eliminate org.apache.spark.sql.hive.thriftserver.CliSuite flakiness
cloud-fan commented on pull request #28055: URL: https://github.com/apache/spark/pull/28055#issuecomment-619707782 > CliSuite seems to be flaky while using metastoreDir per test Do you know the reason? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
HyukjinKwon commented on pull request #28327: URL: https://github.com/apache/spark/pull/28327#issuecomment-619707526 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
AmplabJenkins removed a comment on pull request #28327: URL: https://github.com/apache/spark/pull/28327#issuecomment-619707013 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121868/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
AmplabJenkins commented on pull request #28327: URL: https://github.com/apache/spark/pull/28327#issuecomment-619707004 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
SparkQA removed a comment on pull request #28327: URL: https://github.com/apache/spark/pull/28327#issuecomment-619698750 **[Test build #121868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121868/testReport)** for PR 28327 at commit [`3ec641d`](https://github.com/apache/spark/commit/3ec641dd49def557323160a1c6c31c945aca4842). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
AmplabJenkins removed a comment on pull request #28327: URL: https://github.com/apache/spark/pull/28327#issuecomment-619707004 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28327: [SPARK-29664][PYTHON][SQL][FOLLOW-UP] Add deprecation warnings for getItem instead
SparkQA commented on pull request #28327: URL: https://github.com/apache/spark/pull/28327#issuecomment-619706792 **[Test build #121868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121868/testReport)** for PR 28327 at commit [`3ec641d`](https://github.com/apache/spark/commit/3ec641dd49def557323160a1c6c31c945aca4842). * This patch **fails PySpark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28353: [WIP] Use R 4.0.0 in AppVeyor for Apache Spark 3.1.0
AmplabJenkins removed a comment on pull request #28353: URL: https://github.com/apache/spark/pull/28353#issuecomment-619705683 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28354: [SPARK-31566][SQL][DOCS] Add SQL Rest API Documentation
AmplabJenkins commented on pull request #28354: URL: https://github.com/apache/spark/pull/28354#issuecomment-619705453 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28354: [SPARK-31566][SQL][DOCS] Add SQL Rest API Documentation
SparkQA removed a comment on pull request #28354: URL: https://github.com/apache/spark/pull/28354#issuecomment-619700668 **[Test build #121869 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121869/testReport)** for PR 28354 at commit [`a082de8`](https://github.com/apache/spark/commit/a082de8654fe1dc3182e4147c59a6f7688ed93f5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28353: [WIP] Use R 4.0.0 in AppVeyor for Apache Spark 3.1.0
AmplabJenkins commented on pull request #28353: URL: https://github.com/apache/spark/pull/28353#issuecomment-619705683 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28354: [SPARK-31566][SQL][DOCS] Add SQL Rest API Documentation
AmplabJenkins removed a comment on pull request #28354: URL: https://github.com/apache/spark/pull/28354#issuecomment-619705453 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28354: [SPARK-31566][SQL][DOCS] Add SQL Rest API Documentation
SparkQA commented on pull request #28354: URL: https://github.com/apache/spark/pull/28354#issuecomment-619705351 **[Test build #121869 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121869/testReport)** for PR 28354 at commit [`a082de8`](https://github.com/apache/spark/commit/a082de8654fe1dc3182e4147c59a6f7688ed93f5). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28353: [WIP] Use R 4.0.0 in AppVeyor for Apache Spark 3.1.0
SparkQA removed a comment on pull request #28353: URL: https://github.com/apache/spark/pull/28353#issuecomment-619654357 **[Test build #121856 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121856/testReport)** for PR 28353 at commit [`27642f6`](https://github.com/apache/spark/commit/27642f69b7f39f130da0c29f32f71d0b0677f650). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28353: [WIP] Use R 4.0.0 in AppVeyor for Apache Spark 3.1.0
SparkQA commented on pull request #28353: URL: https://github.com/apache/spark/pull/28353#issuecomment-619704843 **[Test build #121856 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121856/testReport)** for PR 28353 at commit [`27642f6`](https://github.com/apache/spark/commit/27642f69b7f39f130da0c29f32f71d0b0677f650). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
AmplabJenkins removed a comment on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619704025 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
AmplabJenkins commented on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619704025 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
SparkQA removed a comment on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619692020 **[Test build #121866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121866/testReport)** for PR 28350 at commit [`c5725db`](https://github.com/apache/spark/commit/c5725dbaf4b63d9d5348d180eb4e43b88aeacb64). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #28354: [SPARK-31566][SQL][DOCS] Add SQL Rest API Documentation
HyukjinKwon commented on pull request #28354: URL: https://github.com/apache/spark/pull/28354#issuecomment-619704133 cc @gengliangwang FYI This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28350: [SPARK-31568][R][DOCS] Add detail about func/key in gapply to documentation
SparkQA commented on pull request #28350: URL: https://github.com/apache/spark/pull/28350#issuecomment-619703834 **[Test build #121866 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121866/testReport)** for PR 28350 at commit [`c5725db`](https://github.com/apache/spark/commit/c5725dbaf4b63d9d5348d180eb4e43b88aeacb64). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped
AmplabJenkins removed a comment on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-619702942 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped
AmplabJenkins commented on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-619702942 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped
sarutak commented on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-619702971 The error message mentioned [here](https://github.com/apache/spark/pull/28317#issuecomment-619438678) is not shown in this branch. I found it's because the version of `bootstrap.js` is different between `branch-3.0` and `master`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak opened a new pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped
sarutak opened a new pull request #28359: URL: https://github.com/apache/spark/pull/28359 ### What changes were proposed in this pull request? This PR backports #28317 which escapes text for tooltip for DAG Viz and Timeline View. ### Why are the changes needed? This is a bug. Normally, DAG Viz and Timeline View show tooltip like as follows. https://user-images.githubusercontent.com/4736016/80127481-5a6c6880-85cf-11ea-8daf-cfd59aa3ba09.png;> https://user-images.githubusercontent.com/4736016/80127500-60624980-85cf-11ea-9b0f-cce301019e3a.png;> They contain a callsite properly. However, if a callsite contains characters which should be escaped for HTML without escaping , the corresponding tooltips wouldn't show the callsite and its following text properly. https://user-images.githubusercontent.com/4736016/80128480-b1267200-85d0-11ea-8035-ad68ae5fbcab.png;> https://user-images.githubusercontent.com/4736016/80128492-b5eb2600-85d0-11ea-9556-c48490110244.png;> The reason of this issue is that the source texts of the tooltip texts are not escaped. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? I tested manually. First, I ran a job `sc.parallelize(1 to 10).collect` in Spark Shell then, visited AllJobsPage and JobPage and confirmed tooltip texts. https://user-images.githubusercontent.com/4736016/80332616-41411180-8886-11ea-8d93-28e1c5265115.png;> https://user-images.githubusercontent.com/4736016/80332625-44d49880-8886-11ea-8f2e-de8df1369e62.png;> And I confirmed that the appearance of the label of DAG-viz in StagePage is not changed. https://user-images.githubusercontent.com/4736016/80332725-9b41d700-8886-11ea-9adb-40d50ad29f86.png;> I also added a testcase. With this testcase, an error message related to JavaScript is shown. `TypeError: Cannot call method "indexOf" of undefined (http://192.168.1.209:59376/static/spark-dag-viz.js#378)` This is thrown from `interpretLineBreak` in `spark-dag-viz.js`. HtmlUnit seems not to support `innerHTML` for text content (I tried replacing it with `textContent` and the error message is not shown). But DOMs which is needed by the testcase added is already rendered before `interpretLineBreak` is called. So the testcase successfully passes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28359: [SPARK-31534][WEBUI][3.0] Text for tooltip should be escaped
SparkQA commented on pull request #28359: URL: https://github.com/apache/spark/pull/28359#issuecomment-619702598 **[Test build #121871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121871/testReport)** for PR 28359 at commit [`90d3dbf`](https://github.com/apache/spark/commit/90d3dbf3aa671280b494a82ee25c0a13a5f532f1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] erenavsarogullari commented on pull request #28354: [SPARK-31566][SQL][DOCS] Add SQL Rest API Documentation
erenavsarogullari commented on pull request #28354: URL: https://github.com/apache/spark/pull/28354#issuecomment-619701829 Hi @HyukjinKwon, Thanks for the review. I removed the dependency so current documentation just points the existing functionality on `SqlResource`. So, this can go independently now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] huaxingao commented on pull request #28348: [MINOR][SQL][DOCS] Remove two leading spaces from sql tables
huaxingao commented on pull request #28348:
URL: https://github.com/apache/spark/pull/28348#issuecomment-619701454
@gatorsmile
We actually highlight SQL keywords. We have
```
{% highlight sql %}
..
-- Correlated Subquery in `WHERE` clause.
SELECT * FROM person AS parent
WHERE EXISTS (
SELECT 1 FROM person AS child
WHERE parent.id = child.id AND child.age IS NULL
);
..
{% endhighlight %}
```
and it looks like this
https://user-images.githubusercontent.com/13592258/80332710-74d95880-8800-11ea-9c8a-79f377cc4bb3.png;>
The only exception I am aware of is the identifier doc I added recently. The
backtick totally messed up the format.
https://user-images.githubusercontent.com/13592258/80332862-f6c98180-8800-11ea-95f6-09d4606b702d.png;>
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] mengxr commented on pull request #28279: [SPARK-31497][ML][PYSPARK] Fix Pyspark CrossValidator/TrainValidationSplit with pipeline estimator cannot save and load model
mengxr commented on pull request #28279: URL: https://github.com/apache/spark/pull/28279#issuecomment-619701317 LGTM. Merging into master and spark-3.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28354: [SPARK-31566][SQL][DOCS] Add SQL Rest API Documentation
AmplabJenkins removed a comment on pull request #28354: URL: https://github.com/apache/spark/pull/28354#issuecomment-619700962 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28354: [SPARK-31566][SQL][DOCS] Add SQL Rest API Documentation
AmplabJenkins commented on pull request #28354: URL: https://github.com/apache/spark/pull/28354#issuecomment-619700962 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #27006: [SPARK-30352][SQL] DataSourceV2: Add CURRENT_CATALOG function
SparkQA commented on pull request #27006: URL: https://github.com/apache/spark/pull/27006#issuecomment-619700713 **[Test build #121870 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121870/testReport)** for PR 27006 at commit [`180fe0a`](https://github.com/apache/spark/commit/180fe0a14409043e22d2eb51c17b60a27f678eb7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #28354: [SPARK-31566][SQL][DOCS] Add SQL Rest API Documentation
SparkQA commented on pull request #28354: URL: https://github.com/apache/spark/pull/28354#issuecomment-619700668 **[Test build #121869 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121869/testReport)** for PR 28354 at commit [`a082de8`](https://github.com/apache/spark/commit/a082de8654fe1dc3182e4147c59a6f7688ed93f5). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #28353: [WIP] Use R 4.0.0 in AppVeyor for Apache Spark 3.1.0
AmplabJenkins commented on pull request #28353: URL: https://github.com/apache/spark/pull/28353#issuecomment-619699789 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #28353: [WIP] Use R 4.0.0 in AppVeyor for Apache Spark 3.1.0
AmplabJenkins removed a comment on pull request #28353: URL: https://github.com/apache/spark/pull/28353#issuecomment-619699789 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
