[GitHub] [spark] SparkQA removed a comment on pull request #32516: [SPARK-35364][PYTHON] Renaming the existing Koalas related codes
SparkQA removed a comment on pull request #32516: URL: https://github.com/apache/spark/pull/32516#issuecomment-839451472 **[Test build #138427 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138427/testReport)** for PR 32516 at commit [`fe5b3ef`](https://github.com/apache/spark/commit/fe5b3efa9aa41ba21197a625493f4e6c8fb08f14). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32516: [SPARK-35364][PYTHON] Renaming the existing Koalas related codes
SparkQA commented on pull request #32516: URL: https://github.com/apache/spark/pull/32516#issuecomment-839474981 **[Test build #138427 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138427/testReport)** for PR 32516 at commit [`fe5b3ef`](https://github.com/apache/spark/commit/fe5b3efa9aa41ba21197a625493f4e6c8fb08f14). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32518: [SPARK-35383][CORE] Improve s3a magic committer support by inferring missing configs
dongjoon-hyun commented on pull request #32518: URL: https://github.com/apache/spark/pull/32518#issuecomment-839470621 Hi, @steveloughran . Could you review this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #32199: [SPARK-35100][ML] Refactor AFT - support virtual centering
zhengruifeng commented on pull request #32199: URL: https://github.com/apache/spark/pull/32199#issuecomment-839467796 @srowen It should be safe to wait for BLAS related changes. I will keep this PR open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun opened a new pull request #32518: [SPARK-35383][CORE] Improve s3a magic committer support by inferring missing configs
dongjoon-hyun opened a new pull request #32518: URL: https://github.com/apache/spark/pull/32518 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32494: [SPARK-35362][SQL] Update null count in the column stats for UNION operator stats estimation
SparkQA commented on pull request #32494: URL: https://github.com/apache/spark/pull/32494#issuecomment-839459076 **[Test build #138429 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138429/testReport)** for PR 32494 at commit [`d876e7b`](https://github.com/apache/spark/commit/d876e7bff4a50e79230a6101db6b5ed9704d2f46). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #32494: [SPARK-35362][SQL] Update null count in the column stats for UNION operator stats estimation
shahidki31 commented on a change in pull request #32494:
URL: https://github.com/apache/spark/pull/32494#discussion_r630736286
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/UnionEstimation.scala
##
@@ -103,10 +112,11 @@ object UnionEstimation {
}
(min, max)
}
- val newStat = ColumnStat(min = minMaxValue._1, max = minMaxValue._2)
+ val newStat = ColumnStat(min = minMaxValue._1, max = minMaxValue._2,
+nullCount = nullCount)
outputAttrStats += unionOutput(outputIndex) -> newStat
}
- AttributeMap(outputAttrStats.toSeq)
+ AttributeMap(outputAttrStats)
Review comment:
Sure. Undid the change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] shahidki31 commented on a change in pull request #32494: [SPARK-35362][SQL] Update null count in the column stats for UNION operator stats estimation
shahidki31 commented on a change in pull request #32494:
URL: https://github.com/apache/spark/pull/32494#discussion_r630736159
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/UnionEstimation.scala
##
@@ -86,11 +86,20 @@ object UnionEstimation {
val outputAttrStats = new ArrayBuffer[(Attribute, ColumnStat)]()
attrToComputeMinMaxStats.foreach {
case (attrs, outputIndex) =>
+ var nullCount: Option[BigInt] = None
Review comment:
Yes, Removed the var.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32512: [SPARK-35377][INFRA] Add JS linter to GA
AmplabJenkins removed a comment on pull request #32512: URL: https://github.com/apache/spark/pull/32512#issuecomment-839452238 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138417/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32512: [SPARK-35377][INFRA] Add JS linter to GA
AmplabJenkins commented on pull request #32512: URL: https://github.com/apache/spark/pull/32512#issuecomment-839452238 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138417/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32516: [SPARK-35364][PYTHON] Renaming the existing Koalas related codes
SparkQA commented on pull request #32516: URL: https://github.com/apache/spark/pull/32516#issuecomment-839451472 **[Test build #138427 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138427/testReport)** for PR 32516 at commit [`fe5b3ef`](https://github.com/apache/spark/commit/fe5b3efa9aa41ba21197a625493f4e6c8fb08f14). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32515: [WIP][SPARK-35380][SQL] Loading SparkSessionExtensions from ServiceLoader
SparkQA commented on pull request #32515: URL: https://github.com/apache/spark/pull/32515#issuecomment-839451531 **[Test build #138428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138428/testReport)** for PR 32515 at commit [`ec731fc`](https://github.com/apache/spark/commit/ec731fcf4c5e43d4d0b885d0484db8e92d78b517). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32517: [SPARK-35381][R] Fix lambda variable name issues in nested higher order functions at R APIs
SparkQA commented on pull request #32517: URL: https://github.com/apache/spark/pull/32517#issuecomment-839451396 **[Test build #138426 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138426/testReport)** for PR 32517 at commit [`f17b074`](https://github.com/apache/spark/commit/f17b074c8c5d1e22f87e4b92523f1ceff37b7bff). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32512: [SPARK-35377][INFRA] Add JS linter to GA
SparkQA removed a comment on pull request #32512: URL: https://github.com/apache/spark/pull/32512#issuecomment-839403654 **[Test build #138417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138417/testReport)** for PR 32512 at commit [`2df99a4`](https://github.com/apache/spark/commit/2df99a4bcde7d1ac87775498e3a877cbd6ba058a). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32512: [SPARK-35377][INFRA] Add JS linter to GA
SparkQA commented on pull request #32512: URL: https://github.com/apache/spark/pull/32512#issuecomment-839451134 **[Test build #138417 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138417/testReport)** for PR 32512 at commit [`2df99a4`](https://github.com/apache/spark/commit/2df99a4bcde7d1ac87775498e3a877cbd6ba058a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32512: [SPARK-35377][INFRA] Add JS linter to GA
AmplabJenkins removed a comment on pull request #32512: URL: https://github.com/apache/spark/pull/32512#issuecomment-839449457 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42946/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32473: [SPARK-35345][SQL] Add Parquet tests to BloomFilterBenchmark
AmplabJenkins removed a comment on pull request #32473: URL: https://github.com/apache/spark/pull/32473#issuecomment-839449458 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138410/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32457: [SPARK-35329][SQL] Split generated switch code into pieces in ExpandExec
AmplabJenkins removed a comment on pull request #32457: URL: https://github.com/apache/spark/pull/32457#issuecomment-839449400 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42943/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32452: [SPARK-35243][SQL] Support columnar execution on ANSI interval types
AmplabJenkins removed a comment on pull request #32452: URL: https://github.com/apache/spark/pull/32452#issuecomment-839449456 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42944/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32510: [SPARK-35376][CORE] Fallback config should override defaultValue
AmplabJenkins removed a comment on pull request #32510: URL: https://github.com/apache/spark/pull/32510#issuecomment-839449452 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138415/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32516: [SPARK-35364][PYTHON] Renaming the existing Koalas related codes
AmplabJenkins removed a comment on pull request #32516: URL: https://github.com/apache/spark/pull/32516#issuecomment-839449413 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42945/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32473: [SPARK-35345][SQL] Add Parquet tests to BloomFilterBenchmark
AmplabJenkins commented on pull request #32473: URL: https://github.com/apache/spark/pull/32473#issuecomment-839449458 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138410/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32452: [SPARK-35243][SQL] Support columnar execution on ANSI interval types
AmplabJenkins commented on pull request #32452: URL: https://github.com/apache/spark/pull/32452#issuecomment-839449456 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42944/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32510: [SPARK-35376][CORE] Fallback config should override defaultValue
AmplabJenkins commented on pull request #32510: URL: https://github.com/apache/spark/pull/32510#issuecomment-839449452 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138415/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32512: [SPARK-35377][INFRA] Add JS linter to GA
AmplabJenkins commented on pull request #32512: URL: https://github.com/apache/spark/pull/32512#issuecomment-839449457 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42946/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32457: [SPARK-35329][SQL] Split generated switch code into pieces in ExpandExec
AmplabJenkins commented on pull request #32457: URL: https://github.com/apache/spark/pull/32457#issuecomment-839449400 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42943/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32516: [SPARK-35364][PYTHON] Renaming the existing Koalas related codes
SparkQA commented on pull request #32516: URL: https://github.com/apache/spark/pull/32516#issuecomment-839449369 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32516: [SPARK-35364][PYTHON] Renaming the existing Koalas related codes
AmplabJenkins commented on pull request #32516: URL: https://github.com/apache/spark/pull/32516#issuecomment-839449413 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42945/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32457: [SPARK-35329][SQL] Split generated switch code into pieces in ExpandExec
SparkQA commented on pull request #32457: URL: https://github.com/apache/spark/pull/32457#issuecomment-839449229 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32452: [SPARK-35243][SQL] Support columnar execution on ANSI interval types
SparkQA commented on pull request #32452: URL: https://github.com/apache/spark/pull/32452#issuecomment-839448429 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32512: [SPARK-35377][INFRA] Add JS linter to GA
SparkQA commented on pull request #32512: URL: https://github.com/apache/spark/pull/32512#issuecomment-839446249 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42946/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32510: [SPARK-35376][CORE] Fallback config should override defaultValue
SparkQA removed a comment on pull request #32510: URL: https://github.com/apache/spark/pull/32510#issuecomment-839382143 **[Test build #138415 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138415/testReport)** for PR 32510 at commit [`7f819ea`](https://github.com/apache/spark/commit/7f819ea83270bf95eec7aa346fd2dcdd0a54f1a2). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32510: [SPARK-35376][CORE] Fallback config should override defaultValue
SparkQA commented on pull request #32510: URL: https://github.com/apache/spark/pull/32510#issuecomment-839443187 **[Test build #138415 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138415/testReport)** for PR 32510 at commit [`7f819ea`](https://github.com/apache/spark/commit/7f819ea83270bf95eec7aa346fd2dcdd0a54f1a2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #32494: [SPARK-35362][SQL] Update null count in the column stats for UNION operator stats estimation
maropu commented on a change in pull request #32494:
URL: https://github.com/apache/spark/pull/32494#discussion_r630726896
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/UnionEstimation.scala
##
@@ -103,10 +112,11 @@ object UnionEstimation {
}
(min, max)
}
- val newStat = ColumnStat(min = minMaxValue._1, max = minMaxValue._2)
+ val newStat = ColumnStat(min = minMaxValue._1, max = minMaxValue._2,
+nullCount = nullCount)
outputAttrStats += unionOutput(outputIndex) -> newStat
}
- AttributeMap(outputAttrStats.toSeq)
+ AttributeMap(outputAttrStats)
Review comment:
`.toSeq` is needed for `newer` Scala compilers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #32287: [SPARK-27991][CORE] Defer the fetch request on Netty OOM
Ngone51 commented on pull request #32287: URL: https://github.com/apache/spark/pull/32287#issuecomment-839437814 Kindly ping @dongjoon-hyun Do you have any more comments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #32494: [SPARK-35362][SQL] Update null count in the column stats for UNION operator stats estimation
maropu commented on a change in pull request #32494:
URL: https://github.com/apache/spark/pull/32494#discussion_r630726538
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/UnionEstimation.scala
##
@@ -86,11 +86,20 @@ object UnionEstimation {
val outputAttrStats = new ArrayBuffer[(Attribute, ColumnStat)]()
attrToComputeMinMaxStats.foreach {
case (attrs, outputIndex) =>
+ var nullCount: Option[BigInt] = None
Review comment:
We can avoid using `var` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu edited a comment on pull request #32424: [SPARK-34794][SQL] Fix lambda variable name issues in nested DataFrame functions
maropu edited a comment on pull request #32424:
URL: https://github.com/apache/spark/pull/32424#issuecomment-839430081
> Why it's a problem only in scala API? how about SQL API?
In SQL, since user-specified param names are used as they are, the same
issue cannot happen;
```
scala> val df = Seq((Seq(1,2,3), Seq("a", "b", "c"))).toDF("numbers",
"letters")
scala> df.selectExpr("""
| FLATTEN(
| TRANSFORM(
| numbers,
| number -> TRANSFORM(
| letters,
| letter -> (number AS number, letter AS letter)
| )
| )
| ) AS zipped
| """).explain(true)
== Analyzed Logical Plan ==
zipped: array>
Project [flatten(transform(numbers#7, lambdafunction(transform(letters#8,
lambdafunction(named_struct(number, lambda number#14, letter, lambda
letter#15), lambda letter#15, false)), lambda number#14, false))) AS zipped#13]
^^ ^^^
+- Project [_1#2 AS numbers#7, _2#3 AS letters#8]
+- LocalRelation [_1#2, _2#3]
```
On the other hand, In DataFame APIs, the same param names (`x`, `y`, and
`z`) were used in lambda functions, so the name conflict could happen;
```
scala> df.select(
| flatten(
| transform(
| $"numbers",
| (number: Column) => { transform(
| $"letters",
| (letter: Column) => { struct(
| number.as("number"),
| letter.as("letter")
| ) }
| ) }
| )
| ).as("zipped")
| ).explain(true)
== Analyzed Logical Plan ==
zipped: array>
Project [flatten(transform(numbers#7, lambdafunction(transform(letters#8,
lambdafunction(struct(number, lambda x_0#20, letter, lambda x_1#21), lambda
x_1#21, false)), lambda x_0#20, false))) AS zipped#19]
^^ ^^^
+- Project [_1#2 AS numbers#7, _2#3 AS letters#8]
+- LocalRelation [_1#2, _2#3]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on pull request #32424: [SPARK-34794][SQL] Fix lambda variable name issues in nested DataFrame functions
maropu commented on pull request #32424: URL: https://github.com/apache/spark/pull/32424#issuecomment-839430240 > BTW, it has the same problem in Python and R too. I and @ueshin are working on them as followups. Ur, I missed that. Thank you, @HyukjinKwon @ueshin -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] maropu commented on pull request #32424: [SPARK-34794][SQL] Fix lambda variable name issues in nested DataFrame functions
maropu commented on pull request #32424:
URL: https://github.com/apache/spark/pull/32424#issuecomment-839430081
> Why it's a problem only in scala API? how about SQL API?
In SQL, since user-specified param names are used as they are, the same
issue cannot happen;
```
scala> val df = Seq((Seq(1,2,3), Seq("a", "b", "c"))).toDF("numbers",
"letters")
scala> df.selectExpr("""
| FLATTEN(
| TRANSFORM(
| numbers,
| number -> TRANSFORM(
| letters,
| letter -> (number AS number, letter AS letter)
| )
| )
| ) AS zipped
| """).explain(true)
== Analyzed Logical Plan ==
zipped: array>
Project [flatten(transform(numbers#7, lambdafunction(transform(letters#8,
lambdafunction(named_struct(number, lambda number#14, letter, lambda
letter#15), lambda letter#15, false)), lambda number#14, false))) AS zipped#13]
^^ ^^^
+- Project [_1#2 AS numbers#7, _2#3 AS letters#8]
+- LocalRelation [_1#2, _2#3]
```
On the other hand, In DataFame APIs, the same param names were used in
lambda functions, so the name conflict could happen;
```
scala> df.select(
| flatten(
| transform(
| $"numbers",
| (number: Column) => { transform(
| $"letters",
| (letter: Column) => { struct(
| number.as("number"),
| letter.as("letter")
| ) }
| ) }
| )
| ).as("zipped")
| ).explain(true)
== Analyzed Logical Plan ==
zipped: array>
Project [flatten(transform(numbers#7, lambdafunction(transform(letters#8,
lambdafunction(struct(number, lambda x_0#20, letter, lambda x_1#21), lambda
x_1#21, false)), lambda x_0#20, false))) AS zipped#19]
^^ ^^^
+- Project [_1#2 AS numbers#7, _2#3 AS letters#8]
+- LocalRelation [_1#2, _2#3]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32473: [SPARK-35345][SQL] Add Parquet tests to BloomFilterBenchmark
SparkQA removed a comment on pull request #32473: URL: https://github.com/apache/spark/pull/32473#issuecomment-839322042 **[Test build #138410 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138410/testReport)** for PR 32473 at commit [`47f70b7`](https://github.com/apache/spark/commit/47f70b7fa00512e9c971c74ba3714062c7fb216e). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32473: [SPARK-35345][SQL] Add Parquet tests to BloomFilterBenchmark
SparkQA commented on pull request #32473: URL: https://github.com/apache/spark/pull/32473#issuecomment-839428571 **[Test build #138410 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138410/testReport)** for PR 32473 at commit [`47f70b7`](https://github.com/apache/spark/commit/47f70b7fa00512e9c971c74ba3714062c7fb216e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32517: [SPARK-35381][R] Fix lambda variable name issues in nested higher order functions at R APIs
HyukjinKwon commented on pull request #32517: URL: https://github.com/apache/spark/pull/32517#issuecomment-839427517 cc @maropu @zero323 @ueshin FYI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon opened a new pull request #32517: [SPARK-35381][R] Fix lambda variable name issues in nested higher order functions at R APIs
HyukjinKwon opened a new pull request #32517:
URL: https://github.com/apache/spark/pull/32517
### What changes were proposed in this pull request?
This PR fixes the same issue as https://github.com/apache/spark/pull/32424
```r
df <- sql("SELECT array(1, 2, 3) as numbers, array('a', 'b', 'c') as
letters")
collect(select(
df,
array_transform("numbers", function(number) {
array_transform("letters", function(latter) {
struct(alias(number, "n"), alias(latter, "l"))
})
})
))
```
**Before:**
```
transform(numbers, lambdafunction(transform(letters,
lambdafunction(struct(namedlambdavariable() AS n, namedlambdavariable() AS l),
namedlambdavariable())), namedlambdavariable()))
1
a, a, b, b, c, c, a, a, b,
b, c, c, a, a, b, b, c, c
```
**After:**
```
transform(numbers, lambdafunction(transform(letters,
lambdafunction(struct(namedlambdavariable() AS n, namedlambdavariable() AS l),
namedlambdavariable())), namedlambdavariable()))
1
1, a, 1, b, 1, c, 2, a, 2,
b, 2, c, 3, a, 3, b, 3, c
```
### Why are the changes needed?
To produce the correct results.
### Does this PR introduce _any_ user-facing change?
Yes, it fixes the results to be correct as mentioned above.
### How was this patch tested?
Manually tested as above, and unit test was added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32516: [SPARK-35364][PYTHON] Renaming the existing Koalas related codes
AmplabJenkins removed a comment on pull request #32516: URL: https://github.com/apache/spark/pull/32516#issuecomment-839425516 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138424/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32516: [SPARK-35364][PYTHON] Renaming the existing Koalas related codes
SparkQA removed a comment on pull request #32516: URL: https://github.com/apache/spark/pull/32516#issuecomment-839425081 **[Test build #138424 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138424/testReport)** for PR 32516 at commit [`a287ee1`](https://github.com/apache/spark/commit/a287ee11cc089aa7569d993a08241630302abf7b). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32516: [SPARK-35364][PYTHON] Renaming the existing Koalas related codes
AmplabJenkins commented on pull request #32516: URL: https://github.com/apache/spark/pull/32516#issuecomment-839425516 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138424/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32516: [SPARK-35364][PYTHON] Renaming the existing Koalas related codes
SparkQA commented on pull request #32516: URL: https://github.com/apache/spark/pull/32516#issuecomment-839425510 **[Test build #138424 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138424/testReport)** for PR 32516 at commit [`a287ee1`](https://github.com/apache/spark/commit/a287ee11cc089aa7569d993a08241630302abf7b). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32515: [WIP][SPARK-35380][SQL] Loading SparkSessionExtensions from ServiceLoader
SparkQA removed a comment on pull request #32515: URL: https://github.com/apache/spark/pull/32515#issuecomment-839422615 **[Test build #138420 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138420/testReport)** for PR 32515 at commit [`baa61a3`](https://github.com/apache/spark/commit/baa61a3731629cc09f0859f02f18092ade2150c7). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32512: [SPARK-35377][INFRA] Add JS linter to GA
SparkQA commented on pull request #32512: URL: https://github.com/apache/spark/pull/32512#issuecomment-839425100 **[Test build #138425 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138425/testReport)** for PR 32512 at commit [`946dd27`](https://github.com/apache/spark/commit/946dd274680b831ec7e9fcebb80e1d11cfdf42f2). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32515: [WIP][SPARK-35380][SQL] Loading SparkSessionExtensions from ServiceLoader
AmplabJenkins removed a comment on pull request #32515: URL: https://github.com/apache/spark/pull/32515#issuecomment-839424800 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32511: [WIP][SPARK-XXXXX][SQL] Show explain formatted strings for logical plans
AmplabJenkins removed a comment on pull request #32511: URL: https://github.com/apache/spark/pull/32511#issuecomment-839424925 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138414/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32516: [SPARK-35364][PYTHON] Renaming the existing Koalas related codes
SparkQA commented on pull request #32516: URL: https://github.com/apache/spark/pull/32516#issuecomment-839425081 **[Test build #138424 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138424/testReport)** for PR 32516 at commit [`a287ee1`](https://github.com/apache/spark/commit/a287ee11cc089aa7569d993a08241630302abf7b). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32511: [WIP][SPARK-XXXXX][SQL] Show explain formatted strings for logical plans
SparkQA removed a comment on pull request #32511: URL: https://github.com/apache/spark/pull/32511#issuecomment-839382118 **[Test build #138414 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138414/testReport)** for PR 32511 at commit [`416dd90`](https://github.com/apache/spark/commit/416dd90e2a539ffbd36446820f80359200fd8cef). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32511: [WIP][SPARK-XXXXX][SQL] Show explain formatted strings for logical plans
AmplabJenkins commented on pull request #32511: URL: https://github.com/apache/spark/pull/32511#issuecomment-839424925 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138414/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32515: [WIP][SPARK-35380][SQL] Loading SparkSessionExtensions from ServiceLoader
AmplabJenkins commented on pull request #32515: URL: https://github.com/apache/spark/pull/32515#issuecomment-839424800 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32511: [WIP][SPARK-XXXXX][SQL] Show explain formatted strings for logical plans
SparkQA commented on pull request #32511: URL: https://github.com/apache/spark/pull/32511#issuecomment-839424810 **[Test build #138414 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138414/testReport)** for PR 32511 at commit [`416dd90`](https://github.com/apache/spark/commit/416dd90e2a539ffbd36446820f80359200fd8cef). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class BaseExplainUtils[PlanType <: QueryPlan[PlanType]] ` * `trait BinaryNode extends LogicalPlan with BinaryLike[LogicalPlan] ` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32515: [WIP][SPARK-35380][SQL] Loading SparkSessionExtensions from ServiceLoader
SparkQA commented on pull request #32515: URL: https://github.com/apache/spark/pull/32515#issuecomment-839424791 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32516: [SPARK-35364][PYTHON] Renaming the existing Koalas related codes
SparkQA removed a comment on pull request #32516: URL: https://github.com/apache/spark/pull/32516#issuecomment-839423915 **[Test build #138423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138423/testReport)** for PR 32516 at commit [`99f55c6`](https://github.com/apache/spark/commit/99f55c6d106355fac2436c6788c999e46a042e2b). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32516: [SPARK-35364][PYTHON] Renaming the existing Koalas related codes
AmplabJenkins removed a comment on pull request #32516: URL: https://github.com/apache/spark/pull/32516#issuecomment-839424336 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138423/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32516: [SPARK-35364][PYTHON] Renaming the existing Koalas related codes
AmplabJenkins commented on pull request #32516: URL: https://github.com/apache/spark/pull/32516#issuecomment-839424336 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138423/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32516: [SPARK-35364][PYTHON] Renaming the existing Koalas related codes
SparkQA commented on pull request #32516: URL: https://github.com/apache/spark/pull/32516#issuecomment-839424322 **[Test build #138423 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138423/testReport)** for PR 32516 at commit [`99f55c6`](https://github.com/apache/spark/commit/99f55c6d106355fac2436c6788c999e46a042e2b). * This patch **fails Python style tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still co
AmplabJenkins removed a comment on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-839423903 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138413/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue r
AmplabJenkins commented on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-839423903 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138413/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32516: [SPARK-35364][PYTHON] Renaming the existing Koalas related codes
SparkQA commented on pull request #32516: URL: https://github.com/apache/spark/pull/32516#issuecomment-839423915 **[Test build #138423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138423/testReport)** for PR 32516 at commit [`99f55c6`](https://github.com/apache/spark/commit/99f55c6d106355fac2436c6788c999e46a042e2b). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #32512: [SPARK-35377][INFRA] Add JS linter to GA
sarutak commented on a change in pull request #32512:
URL: https://github.com/apache/spark/pull/32512#discussion_r630715420
##
File path: .github/workflows/build_and_test.yml
##
@@ -356,6 +356,10 @@ jobs:
Rscript -e "install.packages(c('devtools'),
repos='https://cloud.r-project.org/')"
Rscript -e "devtools::install_github('jimhester/lintr@v2.0.1')"
./R/install-dev.sh
+- name: Instll JS linter dependencies
Review comment:
O.K.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue
SparkQA removed a comment on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-839362651 **[Test build #138413 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138413/testReport)** for PR 32399 at commit [`dd431b9`](https://github.com/apache/spark/commit/dd431b9a0a1bdd861a18f24219d09fba8a31b46d). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] itholic opened a new pull request #32516: [SPARK-35364][PYTHON] Renaming the existing Koalas related codes
itholic opened a new pull request #32516: URL: https://github.com/apache/spark/pull/32516 ### What changes were proposed in this pull request? There are still naming related to Koalas in test and function name. This PR addressed them to fit pandas-on-spark. - kdf -> psdf - kser -> psser - kidx -> psidx - kmidx -> psmidx - to_koalas() -> to_pandas_on_spark() ### Why are the changes needed? This is because the name Koalas is no longer used in PySpark. ### Does this PR introduce _any_ user-facing change? `to_koalas()` function is renamed to `to_pandas_on_spark()` ### How was this patch tested? Tested in local manually. After changing the related naming, I checked them one by one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32399: [SPARK-35271][ML][PYSPARK] Fix: After CrossValidator/TrainValidationSplit fit raised error, some backgroud threads may still continue run or
SparkQA commented on pull request #32399: URL: https://github.com/apache/spark/pull/32399#issuecomment-839423094 **[Test build #138413 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138413/testReport)** for PR 32399 at commit [`dd431b9`](https://github.com/apache/spark/commit/dd431b9a0a1bdd861a18f24219d09fba8a31b46d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32452: [SPARK-35243][SQL] Support columnar execution on ANSI interval types
SparkQA commented on pull request #32452: URL: https://github.com/apache/spark/pull/32452#issuecomment-839422755 **[Test build #138422 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138422/testReport)** for PR 32452 at commit [`8c53f5a`](https://github.com/apache/spark/commit/8c53f5a01af93269aee46683a59248aa77655de0). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32457: [SPARK-35329][SQL] Split generated switch code into pieces in ExpandExec
SparkQA commented on pull request #32457: URL: https://github.com/apache/spark/pull/32457#issuecomment-839422709 **[Test build #138421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138421/testReport)** for PR 32457 at commit [`cdbbffe`](https://github.com/apache/spark/commit/cdbbffed6f811cfc8e475cbe69a8d3592c7923de). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32512: [SPARK-35377][INFRA] Add JS linter to GA
AmplabJenkins removed a comment on pull request #32512: URL: https://github.com/apache/spark/pull/32512#issuecomment-839422410 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42938/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32515: [WIP][SPARK-35380][SQL] Loading SparkSessionExtensions from ServiceLoader
SparkQA commented on pull request #32515: URL: https://github.com/apache/spark/pull/32515#issuecomment-839422615 **[Test build #138420 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138420/testReport)** for PR 32515 at commit [`baa61a3`](https://github.com/apache/spark/commit/baa61a3731629cc09f0859f02f18092ade2150c7). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32513: [SPARK-35378][SQL] Convert LeafRunnableCommand to LocalRelation when query with CTE
AmplabJenkins removed a comment on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-839422531 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42941/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32513: [SPARK-35378][SQL] Convert LeafRunnableCommand to LocalRelation when query with CTE
AmplabJenkins commented on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-839422531 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42941/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32513: [SPARK-35378][SQL] Convert LeafRunnableCommand to LocalRelation when query with CTE
SparkQA commented on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-839422508 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32512: [SPARK-35377][INFRA] Add JS linter to GA
SparkQA commented on pull request #32512: URL: https://github.com/apache/spark/pull/32512#issuecomment-839422390 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32512: [SPARK-35377][INFRA] Add JS linter to GA
AmplabJenkins commented on pull request #32512: URL: https://github.com/apache/spark/pull/32512#issuecomment-839422410 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42938/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32506: [SPARK-35374][SQL] Add string-to-number conversion support to JacksonParser
AmplabJenkins removed a comment on pull request #32506: URL: https://github.com/apache/spark/pull/32506#issuecomment-839421689 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42939/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32439: [SPARK-35298][SQL] Migrate to transformWithPruning for rules in Optimizer.scala
AmplabJenkins removed a comment on pull request #32439: URL: https://github.com/apache/spark/pull/32439#issuecomment-839421688 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32512: [SPARK-35377][INFRA] Add JS linter to GA
HyukjinKwon commented on a change in pull request #32512:
URL: https://github.com/apache/spark/pull/32512#discussion_r630713623
##
File path: .github/workflows/build_and_test.yml
##
@@ -356,6 +356,10 @@ jobs:
Rscript -e "install.packages(c('devtools'),
repos='https://cloud.r-project.org/')"
Rscript -e "devtools::install_github('jimhester/lintr@v2.0.1')"
./R/install-dev.sh
+- name: Instll JS linter dependencies
Review comment:
Should we rename JS -> JavaScript?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32514: [SPARK-35379][SQL]Improve InferFiltersFromConstraints rule performance when parsing spark sql
AmplabJenkins commented on pull request #32514: URL: https://github.com/apache/spark/pull/32514#issuecomment-839421759 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32506: [SPARK-35374][SQL] Add string-to-number conversion support to JacksonParser
AmplabJenkins commented on pull request #32506: URL: https://github.com/apache/spark/pull/32506#issuecomment-839421689 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42939/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32439: [SPARK-35298][SQL] Migrate to transformWithPruning for rules in Optimizer.scala
AmplabJenkins commented on pull request #32439: URL: https://github.com/apache/spark/pull/32439#issuecomment-839421688 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32506: [SPARK-35374][SQL] Add string-to-number conversion support to JacksonParser
SparkQA commented on pull request #32506: URL: https://github.com/apache/spark/pull/32506#issuecomment-839420084 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #32424: [SPARK-34794][SQL] Fix lambda variable name issues in nested DataFrame functions
HyukjinKwon edited a comment on pull request #32424: URL: https://github.com/apache/spark/pull/32424#issuecomment-839414684 Since R and Python ones are merged into 3.1, I will create separate JIRAs: https://issues.apache.org/jira/browse/SPARK-35381 https://issues.apache.org/jira/browse/SPARK-35382 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] yaooqinn opened a new pull request #32515: [WIP][SPARK-35380][SQL] Loading SparkSessionExtensions from ServiceLoader
yaooqinn opened a new pull request #32515: URL: https://github.com/apache/spark/pull/32515 ### What changes were proposed in this pull request? In https://github.com/yaooqinn/itachi/issues/8, we had a discussion about the current extension injection for the spark session. We've agreed that the current way is not that convenient for both third-party developers and end-users. ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32439: [SPARK-35298][SQL] Migrate to transformWithPruning for rules in Optimizer.scala
SparkQA commented on pull request #32439: URL: https://github.com/apache/spark/pull/32439#issuecomment-839417163 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42940/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32424: [SPARK-34794][SQL] Fix lambda variable name issues in nested DataFrame functions
HyukjinKwon commented on pull request #32424: URL: https://github.com/apache/spark/pull/32424#issuecomment-839414684 Since R and Python ones are merged into 3.1, I will create separate JIRAs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32424: [SPARK-34794][SQL] Fix lambda variable name issues in nested DataFrame functions
HyukjinKwon commented on pull request #32424: URL: https://github.com/apache/spark/pull/32424#issuecomment-839414276 BTW, it has the same problem in Python and R too. I and @ueshin are working on them as followups. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon removed a comment on pull request #32424: [SPARK-34794][SQL] Fix lambda variable name issues in nested DataFrame functions
HyukjinKwon removed a comment on pull request #32424: URL: https://github.com/apache/spark/pull/32424#issuecomment-839414140 Yeah, it has in Python and R too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32439: [SPARK-35298][SQL] Migrate to transformWithPruning for rules in Optimizer.scala
SparkQA removed a comment on pull request #32439: URL: https://github.com/apache/spark/pull/32439#issuecomment-839275149 **[Test build #138408 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138408/testReport)** for PR 32439 at commit [`0df10d1`](https://github.com/apache/spark/commit/0df10d1eff99b1b710684503fd7b0636ffb7de68). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32424: [SPARK-34794][SQL] Fix lambda variable name issues in nested DataFrame functions
HyukjinKwon commented on pull request #32424: URL: https://github.com/apache/spark/pull/32424#issuecomment-839414140 Yeah, it has in Python and R too -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32439: [SPARK-35298][SQL] Migrate to transformWithPruning for rules in Optimizer.scala
SparkQA commented on pull request #32439: URL: https://github.com/apache/spark/pull/32439#issuecomment-839414101 **[Test build #138408 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138408/testReport)** for PR 32439 at commit [`0df10d1`](https://github.com/apache/spark/commit/0df10d1eff99b1b710684503fd7b0636ffb7de68). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] linhongliu-db commented on a change in pull request #32497: [SPARK-35366][SQL] Avoid using deprecated `buildForBatch` and `buildForStreaming`
linhongliu-db commented on a change in pull request #32497:
URL: https://github.com/apache/spark/pull/32497#discussion_r630706338
##
File path:
external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaWrite.scala
##
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.kafka010
+
+import java.{util => ju}
+
+import org.apache.spark.sql.connector.write.{BatchWrite, Write}
+import org.apache.spark.sql.connector.write.streaming.StreamingWrite
+import org.apache.spark.sql.types.StructType
+
+case class KafkaWrite(
Review comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #32424: [SPARK-34794][SQL] Fix lambda variable name issues in nested DataFrame functions
cloud-fan commented on pull request #32424: URL: https://github.com/apache/spark/pull/32424#issuecomment-839410836 Why it's a problem only in scala API? how about SQL API? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] wankunde opened a new pull request #32514: Improve InferFiltersFromConstraints rule performance when parsing spark sql
wankunde opened a new pull request #32514:
URL: https://github.com/apache/spark/pull/32514
### What changes were proposed in this pull request?
This PR try to improve `InferFiltersFromConstraints` performance via avoid
generating too many constraints.
For example:
```java
test("Expression explosion when analyze test") {
RuleExecutor.resetMetrics()
Seq((1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))
.toDF("a", "b", "c", "d", "e", "f", "g", "h", "i", "j",
"k", "l", "m", "n")
.write.saveAsTable("test")
val df = spark.table("test")
val df2 = df.filter("a+b+c+d+e+f+g+h+i+j+k+l+m+n > 100")
val df3 = df2.select('a as 'a1, 'b as 'b1,
'c as 'c1, 'd as 'd1, 'e as 'e1, 'f as 'f1,
'g as 'g1, 'h as 'h1, 'i as 'i1, 'j as 'j1,
'k as 'k1, 'l as 'l1, 'm as 'm1, 'n as 'n1)
val df4 = df3.join(df2, df3("a1") === df2("a"))
df4.explain(true)
logWarning(RuleExecutor.dumpTimeSpent())
}
```
### Why are the changes needed?
Improve `InferFiltersFromConstraints` performance
Before this PR:
```
=== Metrics of Analyzer/Optimizer Rules ===
Total number of runs: 1187
Total time: 5.022786805 seconds
Rule
Effective Time / Total Time Effective Runs / Total Runs
org.apache.spark.sql.catalyst.optimizer.InferFiltersFromConstraints
4528820409 / 4529498144 1 / 2
org.apache.spark.sql.catalyst.analysis.ResolveSessionCatalog
[GitHub] [spark] SparkQA commented on pull request #32439: [SPARK-35298][SQL] Migrate to transformWithPruning for rules in Optimizer.scala
SparkQA commented on pull request #32439: URL: https://github.com/apache/spark/pull/32439#issuecomment-839403755 **[Test build #138419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138419/testReport)** for PR 32439 at commit [`2408a18`](https://github.com/apache/spark/commit/2408a182db3201093b3189f02409dc15e16298fc). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32512: [SPARK-35377][INFRA] Add JS linter to GA
SparkQA commented on pull request #32512: URL: https://github.com/apache/spark/pull/32512#issuecomment-839403654 **[Test build #138417 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138417/testReport)** for PR 32512 at commit [`2df99a4`](https://github.com/apache/spark/commit/2df99a4bcde7d1ac87775498e3a877cbd6ba058a). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32506: [SPARK-35374][SQL] Add string-to-number conversion support to JacksonParser
SparkQA commented on pull request #32506: URL: https://github.com/apache/spark/pull/32506#issuecomment-839403661 **[Test build #138418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138418/testReport)** for PR 32506 at commit [`4cf72e8`](https://github.com/apache/spark/commit/4cf72e83687f9982bdff7e511f37fbe35726e4d2). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32513: [SPARK-35378][SQL] Convert LeafRunnableCommand to LocalRelation when query with CTE
SparkQA commented on pull request #32513: URL: https://github.com/apache/spark/pull/32513#issuecomment-839403607 **[Test build #138416 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138416/testReport)** for PR 32513 at commit [`40cea6b`](https://github.com/apache/spark/commit/40cea6bb85ea5b5020be4dcb07780ab0aa72ad64). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
