[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
SparkQA commented on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876962291 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45352/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33216: [SPARK-36018][CORE][SQL] Some Improvement for Spark Core
SparkQA removed a comment on pull request #33216: URL: https://github.com/apache/spark/pull/33216#issuecomment-876897756 **[Test build #140835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140835/testReport)** for PR 33216 at commit [`19b516f`](https://github.com/apache/spark/commit/19b516f27c9ab8479b6b47764b63d2fd799d5b30). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33216: [SPARK-36018][CORE][SQL] Some Improvement for Spark Core
SparkQA commented on pull request #33216: URL: https://github.com/apache/spark/pull/33216#issuecomment-876958743 **[Test build #140835 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140835/testReport)** for PR 33216 at commit [`19b516f`](https://github.com/apache/spark/commit/19b516f27c9ab8479b6b47764b63d2fd799d5b30). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak edited a comment on pull request #33276: [SPARK-36067][BUILD][TEST][YARN] YarnClusterSuite fails due to NoClassDefFoundError unless hadoop-3.2 profile is activated explicitly
sarutak edited a comment on pull request #33276: URL: https://github.com/apache/spark/pull/33276#issuecomment-876954025 @HyukjinKwon @sunchao Thank you for the review. > I guess this wasn't caught by Jenkins or Github CI because they always add -Phadoop-3.2 or -Phadoop-2.7? Yes, I think so. > I think it's something good to have for hadoop-cloud too Ah, I focused on YARN but `hadoop-cloud` seems to have the similar issue. O.K, I'll fix it too. > ideally we should make hadoop-3.2 as default in root pom but Maven only allows one default profile Yeah it's the pain point... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33276: [SPARK-36067][BUILD][TEST][YARN] YarnClusterSuite fails due to NoClassDefFoundError unless hadoop-3.2 profile is activated explicitly
sarutak commented on pull request #33276: URL: https://github.com/apache/spark/pull/33276#issuecomment-876954025 @HyukjinKwon @sunchao Thank you for the review. > I guess this wasn't caught by Jenkins or Github CI because they always add -Phadoop-3.2 or -Phadoop-2.7? Yes, I think so. > I think it's something good to have for hadoop-cloud too Ah, I focused on YARN but `hadoop-cloud` seems to have the similar issue. O.K, I'll fix it too. > ideally we should make hadoop-3.2 as default in root pom but Maven only allows one default profile Year, it's the pain point... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range
SparkQA commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-876953584 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45350/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cfmcgrady commented on a change in pull request #33212: [SPARK-35912][SQL] Fix nullability of `spark.read.json`
cfmcgrady commented on a change in pull request #33212:
URL: https://github.com/apache/spark/pull/33212#discussion_r666705951
##
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala
##
@@ -404,6 +407,19 @@ class JacksonParser(
}
}
+// When the input schema is setting to `nullable = false`, make sure the
field is not null.
+var index = 0
+while (badRecordException.isEmpty && !skipRow && index < schema.length) {
Review comment:
We may not return the JSON string, the reason show in
[test](https://github.com/apache/spark/pull/33212/files#r666705223)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33206: [SPARK-36002][PYTHON] Consolidate tests for data-type-based operations of decimal Series
SparkQA removed a comment on pull request #33206: URL: https://github.com/apache/spark/pull/33206#issuecomment-876897765 **[Test build #140836 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140836/testReport)** for PR 33206 at commit [`88ccafc`](https://github.com/apache/spark/commit/88ccafc90f0c4d5aa5a8306b5639635588ebcf97). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cfmcgrady commented on a change in pull request #33212: [SPARK-35912][SQL] Fix nullability of `spark.read.json`
cfmcgrady commented on a change in pull request #33212:
URL: https://github.com/apache/spark/pull/33212#discussion_r666705153
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
##
@@ -2924,6 +2924,50 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-35912: nullability with different parse mode -- struct") {
+val input =
+ """
+|{
+| "c1": {
+|"c2": 1
+| }
+|}
+|""".stripMargin
+val json = spark.createDataset(spark.sparkContext.parallelize(input ::
Nil))(Encoders.STRING)
+
+val load = (mode: String, schema: StructType) => {
+ spark.read
+.option("mode", mode)
+.schema(schema)
+.json(json)
+}
+
+Seq(true, false).foreach { nullable =>
Review comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cfmcgrady commented on a change in pull request #33212: [SPARK-35912][SQL] Fix nullability of `spark.read.json`
cfmcgrady commented on a change in pull request #33212:
URL: https://github.com/apache/spark/pull/33212#discussion_r666705223
##
File path:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
##
@@ -2924,6 +2924,53 @@ abstract class JsonSuite
}
}
}
+
+ test("SPARK-35912: nullability with different parse mode -- struct") {
+// JSON field is missing.
+val input = """{"c1": 1}"""
+// JSON filed is null.
+val input2 =
+ """
+|{
+| "c1": 1,
+| "c2": null
+|}
+|""".stripMargin
+
+val load = (mode: String, schema: StructType, inputJson: String) => {
+ val json = spark.createDataset(
+spark.sparkContext.parallelize(inputJson :: Nil))(Encoders.STRING)
+ spark.read
+.option("mode", mode)
+.schema(schema)
+.json(json)
+}
+Seq(true, false).foreach { nullable =>
+ val schema = StructType(Seq(
+ StructField("c1", IntegerType, nullable = false),
+ StructField("c2", IntegerType, nullable = nullable)))
+
+ Seq(input, input2).foreach { jsonString =>
+if (nullable) {
+ checkAnswer(load("DROPMALFORMED", schema, jsonString), Row(1, null)
:: Nil)
+ checkAnswer(load("FAILFAST", schema, jsonString), Row(1, null) ::
Nil)
+ checkAnswer(load("PERMISSIVE", schema, jsonString), Row(1, null) ::
Nil)
+} else {
+ checkAnswer(load("DROPMALFORMED", schema, jsonString), Seq.empty)
+ val exceptionMsg1 = intercept[SparkException] {
+load("FAILFAST", schema, jsonString).collect
+ }.getMessage
+ assert(exceptionMsg1.contains(
+"the null value found when parsing non-nullable field c2."))
+ val exceptionMsg2 = intercept[SparkException] {
+load("PERMISSIVE", schema, jsonString).collect
Review comment:
@cloud-fan Since the field is non-nullable, do we still return Row(1,
null) for PERMISSIVE mode? if yes, this may cause the cast struct problem as we
talked about before, the field is non-nullable but row.isNullAt(index) is true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33206: [SPARK-36002][PYTHON] Consolidate tests for data-type-based operations of decimal Series
SparkQA commented on pull request #33206: URL: https://github.com/apache/spark/pull/33206#issuecomment-876950917 **[Test build #140836 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140836/testReport)** for PR 33206 at commit [`88ccafc`](https://github.com/apache/spark/commit/88ccafc90f0c4d5aa5a8306b5639635588ebcf97). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
sarutak commented on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876950853 @venkata91 I checked out your change and run it on GA on my account but it fails to build. ``` [error] /home/runner/work/spark/spark/core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala: illegal start of simple expression: Token(RPAREN,),10124,)) ``` Could you enable your GA? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33276: [SPARK-36067][BUILD][TEST][YARN] YarnClusterSuite fails due to NoClassDefFoundError unless hadoop-3.2 profile is activated explicitly
HyukjinKwon closed pull request #33276: URL: https://github.com/apache/spark/pull/33276 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33276: [SPARK-36067][BUILD][TEST][YARN] YarnClusterSuite fails due to NoClassDefFoundError unless hadoop-3.2 profile is activated explicitly
HyukjinKwon commented on pull request #33276: URL: https://github.com/apache/spark/pull/33276#issuecomment-876944524 Javascript linter dependency installation failure seems unrelated. I am merging it in. Merged to master and branch-3.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] ulysses-you commented on pull request #33244: [SPARK-36032][SQL] Use inputPlan instead of currentPhysicalPlan to initialize logical link
ulysses-you commented on pull request #33244: URL: https://github.com/apache/spark/pull/33244#issuecomment-876944083 thank you @viirya @cloud-fan ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on pull request #33260: [SPARK-36047][CORE] Replace the handwriting compare methods with static compare methods in Java code
LuciferYang commented on pull request #33260: URL: https://github.com/apache/spark/pull/33260#issuecomment-876941902 > Make sense. Are these only places where we can replace? Yes，that's all -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
SparkQA removed a comment on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876940801 **[Test build #140841 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140841/testReport)** for PR 33253 at commit [`be65ed1`](https://github.com/apache/spark/commit/be65ed153015c4bd3c97ca54cdb567cc787e69fe). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
AmplabJenkins removed a comment on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876941144 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140841/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
AmplabJenkins commented on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876941144 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140841/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
SparkQA commented on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876941119 **[Test build #140841 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140841/testReport)** for PR 33253 at commit [`be65ed1`](https://github.com/apache/spark/commit/be65ed153015c4bd3c97ca54cdb567cc787e69fe). * This patch **fails RAT tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
SparkQA commented on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876940801 **[Test build #140841 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140841/testReport)** for PR 33253 at commit [`be65ed1`](https://github.com/apache/spark/commit/be65ed153015c4bd3c97ca54cdb567cc787e69fe). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33260: [SPARK-36047][CORE] Replace the handwriting compare methods with static compare methods in Java code
SparkQA commented on pull request #33260: URL: https://github.com/apache/spark/pull/33260#issuecomment-876940828 **[Test build #140840 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140840/testReport)** for PR 33260 at commit [`f7c194d`](https://github.com/apache/spark/commit/f7c194dc3c05665cebf5a7b544c37259ca883d89). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33216: [SPARK-36018][CORE][SQL] Some Improvement for Spark Core
AmplabJenkins removed a comment on pull request #33216: URL: https://github.com/apache/spark/pull/33216#issuecomment-876939978 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45346/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33206: [SPARK-36002][PYTHON] Consolidate tests for data-type-based operations of decimal Series
AmplabJenkins removed a comment on pull request #33206: URL: https://github.com/apache/spark/pull/33206#issuecomment-876939980 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45347/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
AmplabJenkins commented on pull request #31517: URL: https://github.com/apache/spark/pull/31517#issuecomment-876939984 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45348/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33244: [SPARK-36032][SQL] Use inputPlan instead of currentPhysicalPlan to initialize logical link
AmplabJenkins commented on pull request #33244: URL: https://github.com/apache/spark/pull/33244#issuecomment-876939985 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140829/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33276: [SPARK-36067][BUILD][TEST][YARN] YarnClusterSuite fails due to NoClassDefFoundError unless hadoop-3.2 profile is activated expl
AmplabJenkins removed a comment on pull request #33276: URL: https://github.com/apache/spark/pull/33276#issuecomment-876939981 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45349/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33244: [SPARK-36032][SQL] Use inputPlan instead of currentPhysicalPlan to initialize logical link
AmplabJenkins removed a comment on pull request #33244: URL: https://github.com/apache/spark/pull/33244#issuecomment-876939985 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140829/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
AmplabJenkins removed a comment on pull request #31517: URL: https://github.com/apache/spark/pull/31517#issuecomment-876939984 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45348/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33216: [SPARK-36018][CORE][SQL] Some Improvement for Spark Core
AmplabJenkins commented on pull request #33216: URL: https://github.com/apache/spark/pull/33216#issuecomment-876939978 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45346/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33276: [SPARK-36067][BUILD][TEST][YARN] YarnClusterSuite fails due to NoClassDefFoundError unless hadoop-3.2 profile is activated explicitly
AmplabJenkins commented on pull request #33276: URL: https://github.com/apache/spark/pull/33276#issuecomment-876939981 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45349/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33206: [SPARK-36002][PYTHON] Consolidate tests for data-type-based operations of decimal Series
AmplabJenkins commented on pull request #33206: URL: https://github.com/apache/spark/pull/33206#issuecomment-876939980 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45347/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range
SparkQA commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-876937911 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45350/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33276: [SPARK-36067][BUILD][TEST][YARN] YarnClusterSuite fails due to NoClassDefFoundError unless hadoop-3.2 profile is activated explicitly
SparkQA commented on pull request #33276: URL: https://github.com/apache/spark/pull/33276#issuecomment-876937716 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45349/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] kiszk commented on pull request #33260: [SPARK-36047][CORE] Replace the handwriting compare methods with static compare methods in Java code
kiszk commented on pull request #33260: URL: https://github.com/apache/spark/pull/33260#issuecomment-876936107 Make sense. Are these only places where we can replace? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on pull request #33239: [SPARK-36030][SQL] Support DS v2 metrics at writing path
viirya commented on pull request #33239: URL: https://github.com/apache/spark/pull/33239#issuecomment-876932276 @cloud-fan Do we consider this in 3.2 to make the API complete in this release? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya closed pull request #33244: [SPARK-36032][SQL] Use inputPlan instead of currentPhysicalPlan to initialize logical link
viirya closed pull request #33244: URL: https://github.com/apache/spark/pull/33244 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] viirya commented on pull request #33244: [SPARK-36032][SQL] Use inputPlan instead of currentPhysicalPlan to initialize logical link
viirya commented on pull request #33244: URL: https://github.com/apache/spark/pull/33244#issuecomment-876928400 Thanks! Merging to master/3.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
SparkQA commented on pull request #31517: URL: https://github.com/apache/spark/pull/31517#issuecomment-876928255 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45348/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33244: [SPARK-36032][SQL] Use inputPlan instead of currentPhysicalPlan to initialize logical link
SparkQA removed a comment on pull request #33244: URL: https://github.com/apache/spark/pull/33244#issuecomment-876845908 **[Test build #140829 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140829/testReport)** for PR 33244 at commit [`ed90d58`](https://github.com/apache/spark/commit/ed90d588467b668a1ef314907c2f5ef777bdb95e). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33244: [SPARK-36032][SQL] Use inputPlan instead of currentPhysicalPlan to initialize logical link
SparkQA commented on pull request #33244: URL: https://github.com/apache/spark/pull/33244#issuecomment-876927517 **[Test build #140829 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140829/testReport)** for PR 33244 at commit [`ed90d58`](https://github.com/apache/spark/commit/ed90d588467b668a1ef314907c2f5ef777bdb95e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33206: [SPARK-36002][PYTHON] Consolidate tests for data-type-based operations of decimal Series
SparkQA commented on pull request #33206: URL: https://github.com/apache/spark/pull/33206#issuecomment-876925741 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45347/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r50648
##
File path:
core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala
##
@@ -62,21 +61,25 @@ private[history] class ApplicationCache(
/**
* Removal event notifies the provider to detach the UI.
- * @param rm removal notification
+ * @param key removal key
+ * @param value removal value
*/
-override def onRemoval(rm: RemovalNotification[CacheKey, CacheEntry]):
Unit = {
+override def onRemoval(key: CacheKey, value: CacheEntry,
+cause: RemovalCause): Unit = {
metrics.evictionCount.inc()
- val key = rm.getKey
- logDebug(s"Evicting entry ${key}")
- operations.detachSparkUI(key.appId, key.attemptId,
rm.getValue().loadedUI.ui)
+ logDebug(s"Evicting entry $key")
+ operations.detachSparkUI(key.appId, key.attemptId, value.loadedUI.ui)
}
}
private val appCache: LoadingCache[CacheKey, CacheEntry] = {
-CacheBuilder.newBuilder()
-.maximumSize(retainedApplications)
-.removalListener(removalListener)
-.build(appLoader)
+val builder = Caffeine.newBuilder()
+ .maximumSize(retainedApplications)
+ .removalListener(removalListener)
+ // SPARK-34309: Use custom Executor to compatible with
+ // the data eviction behavior of Guava cache
+ .executor((command: Runnable) => command.run())
Review comment:
Yes, it's a compromise for compatibility with old behaviour at present.
In the future, I will optimize this behaviour through other pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33216: [SPARK-36018][CORE][SQL] Some Improvement for Spark Core
SparkQA commented on pull request #33216: URL: https://github.com/apache/spark/pull/33216#issuecomment-876924668 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45346/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33258: [WIP][SPARK-36037][SQL] Support new function localtimestamp()
AmplabJenkins removed a comment on pull request #33258: URL: https://github.com/apache/spark/pull/33258#issuecomment-876918238 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45345/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32959: [SPARK-35780][SQL] Support DATE/TIMESTAMP literals across the full range
SparkQA commented on pull request #32959: URL: https://github.com/apache/spark/pull/32959#issuecomment-876918281 **[Test build #140839 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140839/testReport)** for PR 32959 at commit [`b383571`](https://github.com/apache/spark/commit/b38357115c7c890ad0f198cd43b3c66d20abf73a). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33258: [WIP][SPARK-36037][SQL] Support new function localtimestamp()
SparkQA commented on pull request #33258: URL: https://github.com/apache/spark/pull/33258#issuecomment-876918212 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45345/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33258: [WIP][SPARK-36037][SQL] Support new function localtimestamp()
AmplabJenkins commented on pull request #33258: URL: https://github.com/apache/spark/pull/33258#issuecomment-876918238 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45345/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33276: [SPARK-36067][BUILD][TEST][YARN] YarnClusterSuite fails due to NoClassDefFoundError unless hadoop-3.2 profile is activated explicitly
SparkQA commented on pull request #33276: URL: https://github.com/apache/spark/pull/33276#issuecomment-876918087 **[Test build #140838 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140838/testReport)** for PR 33276 at commit [`aafc23b`](https://github.com/apache/spark/commit/aafc23ba3e46a62eed3fbd99987a9db167bed2cf). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33038: [SPARK-35861][SS] Introduce "prefix match scan" feature on state store
AmplabJenkins removed a comment on pull request #33038: URL: https://github.com/apache/spark/pull/33038#issuecomment-876917578 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140826/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
AmplabJenkins removed a comment on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876917577 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45343/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33038: [SPARK-35861][SS] Introduce "prefix match scan" feature on state store
AmplabJenkins commented on pull request #33038: URL: https://github.com/apache/spark/pull/33038#issuecomment-876917578 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140826/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33206: [SPARK-36002][PYTHON] Consolidate tests for data-type-based operations of decimal Series
HyukjinKwon closed pull request #33206: URL: https://github.com/apache/spark/pull/33206 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
AmplabJenkins commented on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876917577 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45343/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r73952
##
File path:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
##
@@ -315,7 +301,7 @@ private ManagedBuffer getSortBasedShuffleBlockData(
"shuffle_" + shuffleId + "_" + mapId + "_0.data"),
shuffleIndexRecord.getOffset(),
shuffleIndexRecord.getLength());
-} catch (ExecutionException e) {
+} catch (CompletionException e) {
Review comment:
@holdenk `testFetchWrongExecutor` and `testFetchNonexistent` in
`ExternalBlockHandlerSuite` already cover this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33206: [SPARK-36002][PYTHON] Consolidate tests for data-type-based operations of decimal Series
HyukjinKwon commented on pull request #33206: URL: https://github.com/apache/spark/pull/33206#issuecomment-876917371 GA passed. Merged to master and branch-3.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak opened a new pull request #33276: [SPARK-36067][BUILD][TEST][YARN] YarnClusterSuite fails due to NoClassDefFoundError unless hadoop-3.2 profile is activated explicitly
sarutak opened a new pull request #33276: URL: https://github.com/apache/spark/pull/33276 ### What changes were proposed in this pull request? This PR fixes an issue that `YarnClusterSuite` fails due to `NoClassDefFoundError unless `hadoop-3.2` profile is activated explicitly regardless of building with SBT or Maven. ``` build/sbt -Pyarn "yarn/testOnly org.apache.spark.deploy.yarn.YarnClusterSuite" ... [info] YarnClusterSuite: [info] org.apache.spark.deploy.yarn.YarnClusterSuite *** ABORTED *** (598 milliseconds) [info] java.lang.NoClassDefFoundError: org/bouncycastle/operator/OperatorCreationException [info] at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:888) [info] at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) [info] at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1410) [info] at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:344) [info] at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) [info] at org.apache.hadoop.yarn.server.MiniYARNCluster.initResourceManager(MiniYARNCluster.java:359) ``` The solution is modifying `yarn/pom.xml` to activate `hadoop-3.2` profiles by default. ### Why are the changes needed? hadoop-3.2 profile should be enabled by default so `YarnClusterSuite` successfully finishes without `-Phadoop-3.2`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Run `YarnClusterSuite` with both SBT and Maven without `-Phadoop-3.2` and it successfully finished. ``` build/sbt -Pyarn "yarn/testOnly org.apache.spark.deploy.yarn.YarnClusterSuite" ... [info] Run completed in 5 minutes, 38 seconds. [info] Total number of tests run: 27 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 27, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. build/mvn -Pyarn -pl resource-managers/yarn test -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnClusterSuite ... Run completed in 5 minutes, 49 seconds. Total number of tests run: 27 Suites: completed 2, aborted 0 Tests: succeeded 27, failed 0, canceled 0, ignored 0, pending 0 All tests passed. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
SparkQA commented on pull request #31517: URL: https://github.com/apache/spark/pull/31517#issuecomment-876914660 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45348/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33206: [SPARK-36002][PYTHON] Consolidate tests for data-type-based operations of decimal Series
SparkQA commented on pull request #33206: URL: https://github.com/apache/spark/pull/33206#issuecomment-876912823 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45347/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33216: [SPARK-36018][CORE][SQL] Some Improvement for Spark Core
SparkQA commented on pull request #33216: URL: https://github.com/apache/spark/pull/33216#issuecomment-876912583 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45346/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
venkata91 commented on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876907600 > @venkata91 > > @sarutak It seems like Github actions is disabled for my fork. I have requested them to enable it. Just curious, why do I need to enable it for my fork (repo)? > > It's to run tests. > > BTW, please show the screenshot in the description rather than conversation. I did link it in the description as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
SparkQA commented on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876904685 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45343/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33038: [SPARK-35861][SS] Introduce "prefix match scan" feature on state store
SparkQA removed a comment on pull request #33038: URL: https://github.com/apache/spark/pull/33038#issuecomment-876820847 **[Test build #140826 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140826/testReport)** for PR 33038 at commit [`a0f34b7`](https://github.com/apache/spark/commit/a0f34b77b25e32bd5002f9df5e80cdc22089a84c). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33038: [SPARK-35861][SS] Introduce "prefix match scan" feature on state store
SparkQA commented on pull request #33038: URL: https://github.com/apache/spark/pull/33038#issuecomment-876904325 **[Test build #140826 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140826/testReport)** for PR 33038 at commit [`a0f34b7`](https://github.com/apache/spark/commit/a0f34b77b25e32bd5002f9df5e80cdc22089a84c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32949: [SPARK-35749][SPARK-35773][SQL] Parse unit list interval literals as tightest year-month/day-time interval types
AmplabJenkins removed a comment on pull request #32949: URL: https://github.com/apache/spark/pull/32949#issuecomment-876903732 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45344/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AngersZhuuuu commented on a change in pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
AngersZh commented on a change in pull request #33253: URL: https://github.com/apache/spark/pull/33253#discussion_r61788 ## File path: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala ## @@ -572,6 +577,7 @@ private[spark] class AppStatusStore( accumulatorUpdates = stage.accumulatorUpdates, tasks = tasks, executorSummary = executorSummaries, +speculationSummary = speculationSummary(stage.stageId, stage.attemptId), Review comment: This should be control by `withSummaries` parameter -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32949: [SPARK-35749][SPARK-35773][SQL] Parse unit list interval literals as tightest year-month/day-time interval types
AmplabJenkins commented on pull request #32949: URL: https://github.com/apache/spark/pull/32949#issuecomment-876903732 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45344/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32949: [SPARK-35749][SPARK-35773][SQL] Parse unit list interval literals as tightest year-month/day-time interval types
SparkQA commented on pull request #32949: URL: https://github.com/apache/spark/pull/32949#issuecomment-876903714 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45344/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
sarutak commented on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876901452 @venkata91 > @sarutak It seems like Github actions is disabled for my fork. I have requested them to enable it. Just curious, why do I need to enable it for my fork (repo)? It's to run tests. BTW, please show the screenshot in the description rather than conversation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r59227
##
File path:
core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala
##
@@ -62,21 +61,25 @@ private[history] class ApplicationCache(
/**
* Removal event notifies the provider to detach the UI.
- * @param rm removal notification
+ * @param key removal key
+ * @param value removal value
*/
-override def onRemoval(rm: RemovalNotification[CacheKey, CacheEntry]):
Unit = {
+override def onRemoval(key: CacheKey, value: CacheEntry,
+cause: RemovalCause): Unit = {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
SparkQA commented on pull request #31517: URL: https://github.com/apache/spark/pull/31517#issuecomment-876898266 **[Test build #140837 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140837/testReport)** for PR 31517 at commit [`68193a9`](https://github.com/apache/spark/commit/68193a9bfd1a1d465e093a8991df9bdc602903c4). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33216: [SPARK-36018][CORE][SQL] Some Improvement for Spark Core
SparkQA commented on pull request #33216: URL: https://github.com/apache/spark/pull/33216#issuecomment-876897756 **[Test build #140835 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140835/testReport)** for PR 33216 at commit [`19b516f`](https://github.com/apache/spark/commit/19b516f27c9ab8479b6b47764b63d2fd799d5b30). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33206: [SPARK-36002][PYTHON] Consolidate tests for data-type-based operations of decimal Series
SparkQA commented on pull request #33206: URL: https://github.com/apache/spark/pull/33206#issuecomment-876897765 **[Test build #140836 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140836/testReport)** for PR 33206 at commit [`88ccafc`](https://github.com/apache/spark/commit/88ccafc90f0c4d5aa5a8306b5639635588ebcf97). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33258: [WIP][SPARK-36037][SQL] Support new function localtimestamp()
SparkQA commented on pull request #33258: URL: https://github.com/apache/spark/pull/33258#issuecomment-876897662 **[Test build #140834 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140834/testReport)** for PR 33258 at commit [`3cba604`](https://github.com/apache/spark/commit/3cba6045862b99340e7f7b3529caf9ffc7833cc0). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33275: [SPARK-36064][PYTHON] Manage InternalField more in DataTypeOps
AmplabJenkins removed a comment on pull request #33275: URL: https://github.com/apache/spark/pull/33275#issuecomment-876897383 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33275: [SPARK-36064][PYTHON] Manage InternalField more in DataTypeOps
AmplabJenkins commented on pull request #33275: URL: https://github.com/apache/spark/pull/33275#issuecomment-876897384 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
SparkQA commented on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876893546 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45343/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33275: [SPARK-36064][PYTHON] Manage InternalField more in DataTypeOps
SparkQA commented on pull request #33275: URL: https://github.com/apache/spark/pull/33275#issuecomment-876893316 Kubernetes integration test unable to build dist. exiting with code: 1 URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45342/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32949: [SPARK-35749][SPARK-35773][SQL] Parse unit list interval literals as tightest year-month/day-time interval types
SparkQA commented on pull request #32949: URL: https://github.com/apache/spark/pull/32949#issuecomment-876892939 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45344/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #33216: [SPARK-36018][CORE][SQL] Some Improvement for Spark Core
beliefer commented on a change in pull request #33216:
URL: https://github.com/apache/spark/pull/33216#discussion_r52043
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/ExperimentalMethods.scala
##
@@ -37,7 +37,7 @@ import org.apache.spark.sql.catalyst.rules.Rule
class ExperimentalMethods private[sql]() {
/**
- * Allows extra strategies to be injected into the query planner at runtime.
Note this API
Review comment:
I have reverted it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #33216: [SPARK-36018][CORE][SQL] Some Improvement for Spark Core
beliefer commented on a change in pull request #33216:
URL: https://github.com/apache/spark/pull/33216#discussion_r52043
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/ExperimentalMethods.scala
##
@@ -37,7 +37,7 @@ import org.apache.spark.sql.catalyst.rules.Rule
class ExperimentalMethods private[sql]() {
/**
- * Allows extra strategies to be injected into the query planner at runtime.
Note this API
Review comment:
I reverted it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] Yikun commented on a change in pull request #33275: [SPARK-36064][PYTHON] Manage InternalField more in DataTypeOps
Yikun commented on a change in pull request #33275: URL: https://github.com/apache/spark/pull/33275#discussion_r47257 ## File path: python/pyspark/pandas/data_type_ops/base.py ## @@ -79,7 +79,7 @@ def is_valid_operand_for_numeric_arithmetic(operand: Any, *, allow_bool: bool = def transform_boolean_operand_to_numeric( -operand: Any, spark_type: Optional[DataType] = None +operand: Any, *, dtype: Optional[Dtype] = None, spark_type: Optional[DataType] = None Review comment: nit: the doc should be updated, we could do it in followup. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r50648
##
File path:
core/src/main/scala/org/apache/spark/deploy/history/ApplicationCache.scala
##
@@ -62,21 +61,25 @@ private[history] class ApplicationCache(
/**
* Removal event notifies the provider to detach the UI.
- * @param rm removal notification
+ * @param key removal key
+ * @param value removal value
*/
-override def onRemoval(rm: RemovalNotification[CacheKey, CacheEntry]):
Unit = {
+override def onRemoval(key: CacheKey, value: CacheEntry,
+cause: RemovalCause): Unit = {
metrics.evictionCount.inc()
- val key = rm.getKey
- logDebug(s"Evicting entry ${key}")
- operations.detachSparkUI(key.appId, key.attemptId,
rm.getValue().loadedUI.ui)
+ logDebug(s"Evicting entry $key")
+ operations.detachSparkUI(key.appId, key.attemptId, value.loadedUI.ui)
}
}
private val appCache: LoadingCache[CacheKey, CacheEntry] = {
-CacheBuilder.newBuilder()
-.maximumSize(retainedApplications)
-.removalListener(removalListener)
-.build(appLoader)
+val builder = Caffeine.newBuilder()
+ .maximumSize(retainedApplications)
+ .removalListener(removalListener)
+ // SPARK-34309: Use custom Executor to compatible with
+ // the data eviction behavior of Guava cache
+ .executor((command: Runnable) => command.run())
Review comment:
Yes, it's a compromise for compatibility with old behaviour at present.
In the future, I will optimize this problem through other pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #33274: [SPARK-36001][PYTHON] Assume result's index to be disordered in tests with operations on different Series
HyukjinKwon closed pull request #33274: URL: https://github.com/apache/spark/pull/33274 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33274: [SPARK-36001][PYTHON] Assume result's index to be disordered in tests with operations on different Series
HyukjinKwon commented on pull request #33274: URL: https://github.com/apache/spark/pull/33274#issuecomment-876889888 Merged to master and branch-3.2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33275: [SPARK-36064][PYTHON] Manage InternalField more in DataTypeOps
SparkQA removed a comment on pull request #33275: URL: https://github.com/apache/spark/pull/33275#issuecomment-876879166 **[Test build #140831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140831/testReport)** for PR 33275 at commit [`43fd5f0`](https://github.com/apache/spark/commit/43fd5f0592fa309a61c54fc9bc73531f25cb8dba). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33275: [SPARK-36064][PYTHON] Manage InternalField more in DataTypeOps
SparkQA commented on pull request #33275: URL: https://github.com/apache/spark/pull/33275#issuecomment-876887986 **[Test build #140831 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140831/testReport)** for PR 33275 at commit [`43fd5f0`](https://github.com/apache/spark/commit/43fd5f0592fa309a61c54fc9bc73531f25cb8dba). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #31517: [SPARK-34309][BUILD][CORE][SQL][K8S]Use Caffeine instead of Guava Cache
LuciferYang commented on a change in pull request #31517:
URL: https://github.com/apache/spark/pull/31517#discussion_r42523
##
File path:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
##
@@ -315,7 +301,7 @@ private ManagedBuffer getSortBasedShuffleBlockData(
"shuffle_" + shuffleId + "_" + mapId + "_0.data"),
shuffleIndexRecord.getOffset(),
shuffleIndexRecord.getLength());
-} catch (ExecutionException e) {
+} catch (CompletionException e) {
Review comment:
> Do we have test coverage for this?
Because the `ExecutionException (now is CompletionException)` is re-throw as
`RuntimeException`, I need to further check whether it is covered by existing
test case
##
File path:
common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java
##
@@ -315,7 +301,7 @@ private ManagedBuffer getSortBasedShuffleBlockData(
"shuffle_" + shuffleId + "_" + mapId + "_0.data"),
shuffleIndexRecord.getOffset(),
shuffleIndexRecord.getLength());
-} catch (ExecutionException e) {
+} catch (CompletionException e) {
Review comment:
> Do we have test coverage for this?
The `ExecutionException (now is CompletionException)` is re-throw as
`RuntimeException`, I need to further check whether it is covered by existing
test case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
AmplabJenkins removed a comment on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876563353 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
SparkQA removed a comment on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876879218 **[Test build #140832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140832/testReport)** for PR 33253 at commit [`0593c30`](https://github.com/apache/spark/commit/0593c309c16a7a2de1422f459d2f4a50194f78c3). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
AmplabJenkins commented on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876879436 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140832/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
SparkQA commented on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876879420 **[Test build #140832 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140832/testReport)** for PR 33253 at commit [`0593c30`](https://github.com/apache/spark/commit/0593c309c16a7a2de1422f459d2f4a50194f78c3). * This patch **fails RAT tests**. * This patch merges cleanly. * This patch adds no public classes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32949: [SPARK-35749][SPARK-35773][SQL] Parse unit list interval literals as tightest year-month/day-time interval types
SparkQA commented on pull request #32949: URL: https://github.com/apache/spark/pull/32949#issuecomment-876879372 **[Test build #140833 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140833/testReport)** for PR 32949 at commit [`03d2522`](https://github.com/apache/spark/commit/03d252266fd780d9e3f3c742a1b263fb2998e1a1). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33244: [SPARK-36032][SQL] Use inputPlan instead of currentPhysicalPlan to initialize logical link
AmplabJenkins removed a comment on pull request #33244: URL: https://github.com/apache/spark/pull/33244#issuecomment-876879026 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45340/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33274: [SPARK-36001][PYTHON] Assume result's index to be disordered in tests with operations on different Series
AmplabJenkins removed a comment on pull request #33274: URL: https://github.com/apache/spark/pull/33274#issuecomment-876879027 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33273: [SPARK-36062][PYTHON] Try to capture faulthanlder when a Python worker crashes.
AmplabJenkins removed a comment on pull request #33273: URL: https://github.com/apache/spark/pull/33273#issuecomment-876879024 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140827/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33253: [SPARK-36038][CORE] Speculation metrics summary at stage level
SparkQA commented on pull request #33253: URL: https://github.com/apache/spark/pull/33253#issuecomment-876879218 **[Test build #140832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140832/testReport)** for PR 33253 at commit [`0593c30`](https://github.com/apache/spark/commit/0593c309c16a7a2de1422f459d2f4a50194f78c3). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33275: [SPARK-36064][PYTHON] Manage InternalField more in DataTypeOps
SparkQA commented on pull request #33275: URL: https://github.com/apache/spark/pull/33275#issuecomment-876879166 **[Test build #140831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140831/testReport)** for PR 33275 at commit [`43fd5f0`](https://github.com/apache/spark/commit/43fd5f0592fa309a61c54fc9bc73531f25cb8dba). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33274: [SPARK-36001][PYTHON] Assume result's index to be disordered in tests with operations on different Series
AmplabJenkins commented on pull request #33274: URL: https://github.com/apache/spark/pull/33274#issuecomment-876879028 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33273: [SPARK-36062][PYTHON] Try to capture faulthanlder when a Python worker crashes.
AmplabJenkins commented on pull request #33273: URL: https://github.com/apache/spark/pull/33273#issuecomment-876879024 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140827/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
