[GitHub] [spark] gengliangwang commented on pull request #34092: [SPARK-36827][CORE] Improve the perf and memory usage of cleaning up stage UI data

gengliangwang commented on pull request #34092: URL: https://github.com/apache/spark/pull/34092#issuecomment-926422744 @mridulm Yes. At first the benchmark of job start: ``` listener.onJobStart(SparkListenerJobStart(4, time, stages, null)) ``` Later on, I change the

[GitHub] [spark] SparkQA commented on pull request #34092: [SPARK-36827][CORE] Improve the perf and memory usage of cleaning up stage UI data

SparkQA commented on pull request #34092: URL: https://github.com/apache/spark/pull/34092#issuecomment-926419740 **[Test build #143587 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143587/testReport)** for PR 34092 at commit

[GitHub] [spark] SparkQA commented on pull request #34089: [SPARK-36837][BUILD] Upgrade Kafka to 3.0.0

SparkQA commented on pull request #34089: URL: https://github.com/apache/spark/pull/34089#issuecomment-926416340 **[Test build #143588 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143588/testReport)** for PR 34089 at commit

[GitHub] [spark] Peng-Lei commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes

Peng-Lei commented on pull request #34093: URL: https://github.com/apache/spark/pull/34093#issuecomment-926416101 @karenfeng @HyukjinKwon Could you take a look ? Thank you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [spark] AmplabJenkins commented on pull request #34091: [SPARK-36839][INFRA] Add daily build with Hadoop 2 profile in GitHub Actions build

AmplabJenkins commented on pull request #34091: URL: https://github.com/apache/spark/pull/34091#issuecomment-926414847 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48102/ --

[GitHub] [spark] SparkQA commented on pull request #34091: [SPARK-36839][INFRA] Add daily build with Hadoop 2 profile in GitHub Actions build

SparkQA commented on pull request #34091: URL: https://github.com/apache/spark/pull/34091#issuecomment-926414804 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48102/ -- This is an automated message from the

[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes

Peng-Lei commented on a change in pull request #34093: URL: https://github.com/apache/spark/pull/34093#discussion_r715386251 ## File path: core/src/main/resources/error/error-classes.json ## @@ -134,13 +181,28 @@ "message" : [ "The second argument of '%s' function needs

[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes

Peng-Lei commented on a change in pull request #34093: URL: https://github.com/apache/spark/pull/34093#discussion_r715386015 ## File path: core/src/main/resources/error/error-classes.json ## @@ -134,13 +181,28 @@ "message" : [ "The second argument of '%s' function needs

[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes

Peng-Lei commented on a change in pull request #34093: URL: https://github.com/apache/spark/pull/34093#discussion_r715385928 ## File path: core/src/main/resources/error/error-classes.json ## @@ -134,13 +181,28 @@ "message" : [ "The second argument of '%s' function needs

[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes

Peng-Lei commented on a change in pull request #34093: URL: https://github.com/apache/spark/pull/34093#discussion_r715385719 ## File path: core/src/main/resources/error/error-classes.json ## @@ -110,6 +153,10 @@ "message" : [ "Unknown static partition column: %s" ],

[GitHub] [spark] Peng-Lei commented on a change in pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes

Peng-Lei commented on a change in pull request #34093: URL: https://github.com/apache/spark/pull/34093#discussion_r715385403 ## File path: core/src/main/resources/error/error-classes.json ## @@ -39,9 +57,31 @@ "message" : [ "Found duplicate keys '%s'" ], "sqlState" :

[GitHub] [spark] mridulm commented on pull request #34092: [SPARK-36827][CORE] Improve the perf and memory usage of cleaning up stage UI data

mridulm commented on pull request #34092: URL: https://github.com/apache/spark/pull/34092#issuecomment-926413763 Btw, can you share the details of the regression ? What I had seen earlier was: ``` Before changes: InMemoryStore: 2.8s LevelDB: 68.9s After changes:

[GitHub] [spark] gengliangwang commented on pull request #34092: [SPARK-36827][CORE] Improve the perf and memory usage of cleaning up stage UI data

gengliangwang commented on pull request #34092: URL: https://github.com/apache/spark/pull/34092#issuecomment-926412080 > One option is to batch all the deletes together, instead of doing it one by one - which should make level db updates also faster ? Then the same optimization can

[GitHub] [spark] mridulm edited a comment on pull request #34092: [SPARK-36827][CORE] Improve the perf and memory usage of cleaning up stage UI data

mridulm edited a comment on pull request #34092: URL: https://github.com/apache/spark/pull/34092#issuecomment-926406914 I am guessing the minor perf difference for level db would be the sort being done after pulling all data vs pushing the sort to the store. One option is to batch all

[GitHub] [spark] SparkQA commented on pull request #34070: [SPARK-36840][SQL] Support DPP if there is no selective predicate on the filtering side

SparkQA commented on pull request #34070: URL: https://github.com/apache/spark/pull/34070#issuecomment-926408733 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48103/ -- This is an automated message from the Apache

[GitHub] [spark] mridulm commented on pull request #34092: [SPARK-36827][CORE] Improve the perf and memory usage of cleaning up stage UI data

mridulm commented on pull request #34092: URL: https://github.com/apache/spark/pull/34092#issuecomment-926406914 I am guessing the minor perf difference for level db would be the sort being done after pulling all data vs pushing the sort to the store. One option is to batch all the

[GitHub] [spark] AmplabJenkins commented on pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes

AmplabJenkins commented on pull request #34093: URL: https://github.com/apache/spark/pull/34093#issuecomment-926403952 Can one of the admins verify this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [spark] Peng-Lei opened a new pull request #34093: [SPARK-36294][SQL] Refactor fifth set of 20 query execution errors to use error classes

Peng-Lei opened a new pull request #34093: URL: https://github.com/apache/spark/pull/34093 ### What changes were proposed in this pull request? Refactor fifth set of 20 query execution errors to use error classes. as follows: ` createStreamingSourceNotSpecifySchemaError

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34009: [SPARK-34378][SQL][AVRO] Enhance AvroSerializer validation to allow extra nullable Avro fields

AmplabJenkins removed a comment on pull request #34009: URL: https://github.com/apache/spark/pull/34009#issuecomment-926403057 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48101/

[GitHub] [spark] AmplabJenkins commented on pull request #34009: [SPARK-34378][SQL][AVRO] Enhance AvroSerializer validation to allow extra nullable Avro fields

AmplabJenkins commented on pull request #34009: URL: https://github.com/apache/spark/pull/34009#issuecomment-926403057 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48101/ --

[GitHub] [spark] SparkQA commented on pull request #34009: [SPARK-34378][SQL][AVRO] Enhance AvroSerializer validation to allow extra nullable Avro fields

SparkQA commented on pull request #34009: URL: https://github.com/apache/spark/pull/34009#issuecomment-926403030 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48101/ -- This is an automated message from the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns

AmplabJenkins removed a comment on pull request #34038: URL: https://github.com/apache/spark/pull/34038#issuecomment-926399922 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48100/

[GitHub] [spark] AmplabJenkins commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns

AmplabJenkins commented on pull request #34038: URL: https://github.com/apache/spark/pull/34038#issuecomment-926399922 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48100/ --

[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns

SparkQA commented on pull request #34038: URL: https://github.com/apache/spark/pull/34038#issuecomment-926399893 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48100/ -- This is an automated message from the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34089: [SPARK-36837][BUILD] Upgrade Kafka to 3.0.0

AmplabJenkins removed a comment on pull request #34089: URL: https://github.com/apache/spark/pull/34089#issuecomment-926399792 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48099/

[GitHub] [spark] AmplabJenkins commented on pull request #34089: [SPARK-36837][BUILD] Upgrade Kafka to 3.0.0

AmplabJenkins commented on pull request #34089: URL: https://github.com/apache/spark/pull/34089#issuecomment-926399792 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48099/ --

[GitHub] [spark] SparkQA commented on pull request #34089: [SPARK-36837][BUILD] Upgrade Kafka to 3.0.0

SparkQA commented on pull request #34089: URL: https://github.com/apache/spark/pull/34089#issuecomment-926399770 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48099/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

SparkQA commented on pull request #34058: URL: https://github.com/apache/spark/pull/34058#issuecomment-926398174 **[Test build #143594 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143594/testReport)** for PR 34058 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

AmplabJenkins removed a comment on pull request #34058: URL: https://github.com/apache/spark/pull/34058#issuecomment-926398197 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143594/

[GitHub] [spark] AmplabJenkins commented on pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

AmplabJenkins commented on pull request #34058: URL: https://github.com/apache/spark/pull/34058#issuecomment-926398197 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143594/ -- This

[GitHub] [spark] SparkQA removed a comment on pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

SparkQA removed a comment on pull request #34058: URL: https://github.com/apache/spark/pull/34058#issuecomment-926397312 **[Test build #143594 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143594/testReport)** for PR 34058 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34091: [SPARK-36839][INFRA] Add daily build with Hadoop 2 profile in GitHub Actions build

AmplabJenkins removed a comment on pull request #34091: URL: https://github.com/apache/spark/pull/34091#issuecomment-926396364 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143584/

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34092: [SPARK-36827][CORE] Improve the perf and memory usage of cleaning up stage UI data

AmplabJenkins removed a comment on pull request #34092: URL: https://github.com/apache/spark/pull/34092#issuecomment-926396888 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48098/

[GitHub] [spark] SparkQA commented on pull request #34070: [SPARK-36840][SQL] Support DPP if there is no selective predicate on the filtering side

SparkQA commented on pull request #34070: URL: https://github.com/apache/spark/pull/34070#issuecomment-926397273 **[Test build #143593 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143593/testReport)** for PR 34070 at commit

[GitHub] [spark] SparkQA commented on pull request #34058: [SPARK-36711][PYTHON] Support multi-index in new syntax

SparkQA commented on pull request #34058: URL: https://github.com/apache/spark/pull/34058#issuecomment-926397312 **[Test build #143594 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143594/testReport)** for PR 34058 at commit

[GitHub] [spark] SparkQA commented on pull request #34092: [SPARK-36827][CORE] Improve the perf and memory usage of cleaning up stage UI data

SparkQA commented on pull request #34092: URL: https://github.com/apache/spark/pull/34092#issuecomment-926397098 **[Test build #143592 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143592/testReport)** for PR 34092 at commit

[GitHub] [spark] SparkQA commented on pull request #34092: [SPARK-36827][CORE] Improve the perf and memory usage of cleaning up stage UI data

SparkQA commented on pull request #34092: URL: https://github.com/apache/spark/pull/34092#issuecomment-926396839 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48098/ -- This is an automated message from the

[GitHub] [spark] AmplabJenkins commented on pull request #34092: [SPARK-36827][CORE] Improve the perf and memory usage of cleaning up stage UI data

AmplabJenkins commented on pull request #34092: URL: https://github.com/apache/spark/pull/34092#issuecomment-926396888 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48098/ --

[GitHub] [spark] AmplabJenkins commented on pull request #34091: [SPARK-36839][INFRA] Add daily build with Hadoop 2 profile in GitHub Actions build

AmplabJenkins commented on pull request #34091: URL: https://github.com/apache/spark/pull/34091#issuecomment-926396364 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143584/ -- This

[GitHub] [spark] MaxGekk closed pull request #34057: [SPARK-36825][SQL] Read/write dataframes with ANSI intervals from/to parquet files

MaxGekk closed pull request #34057: URL: https://github.com/apache/spark/pull/34057 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [spark] SparkQA commented on pull request #34091: [SPARK-36839][INFRA] Add daily build with Hadoop 2 profile in GitHub Actions build

SparkQA commented on pull request #34091: URL: https://github.com/apache/spark/pull/34091#issuecomment-926390902 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48102/ -- This is an automated message from the Apache

[GitHub] [spark] MaxGekk commented on pull request #34057: [SPARK-36825][SQL] Read/write dataframes with ANSI intervals from/to parquet files

MaxGekk commented on pull request #34057: URL: https://github.com/apache/spark/pull/34057#issuecomment-926390400 Merging to master. Thank you, @sarutak @sunchao and @cloud-fan for review. -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [spark] dgd-contributor commented on pull request #33805: [SPARK-36302][SQL] Refactor thirteenth set of 20 query execution errors to use error classes

dgd-contributor commented on pull request #33805: URL: https://github.com/apache/spark/pull/33805#issuecomment-926389935 CC @karenfeng. Could you take another look when you have time ? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] gengliangwang commented on pull request #34092: [SPARK-36827][CORE] Improve the perf and memory usage of cleaning up stage UI data

gengliangwang commented on pull request #34092: URL: https://github.com/apache/spark/pull/34092#issuecomment-926385613 @mridulm Thanks for the review. This one is ready now. As I observe regression after applying the changes when using LevelDB, this PR is to optimize InMemoryStore only.

[GitHub] [spark] SparkQA removed a comment on pull request #34091: [SPARK-36839][INFRA] Add daily build with Hadoop 2 profile in GitHub Actions build

SparkQA removed a comment on pull request #34091: URL: https://github.com/apache/spark/pull/34091#issuecomment-926326364 **[Test build #143584 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143584/testReport)** for PR 34091 at commit

[GitHub] [spark] SparkQA commented on pull request #34091: [SPARK-36839][INFRA] Add daily build with Hadoop 2 profile in GitHub Actions build

SparkQA commented on pull request #34091: URL: https://github.com/apache/spark/pull/34091#issuecomment-926384784 **[Test build #143584 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143584/testReport)** for PR 34091 at commit

[GitHub] [spark] SparkQA commented on pull request #34009: [SPARK-34378][SQL][AVRO] Enhance AvroSerializer validation to allow extra nullable Avro fields

SparkQA commented on pull request #34009: URL: https://github.com/apache/spark/pull/34009#issuecomment-926380423 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48101/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join

AmplabJenkins removed a comment on pull request #34069: URL: https://github.com/apache/spark/pull/34069#issuecomment-926377057 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48095/

[GitHub] [spark] AmplabJenkins commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join

AmplabJenkins commented on pull request #34069: URL: https://github.com/apache/spark/pull/34069#issuecomment-926377057 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48095/ --

[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join

SparkQA commented on pull request #34069: URL: https://github.com/apache/spark/pull/34069#issuecomment-926377045 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48095/ -- This is an automated message from the

[GitHub] [spark] SparkQA commented on pull request #34092: [WIP][SPARK-36827][CORE] Improve the perf and memory usage of cleaning up stage UI data

SparkQA commented on pull request #34092: URL: https://github.com/apache/spark/pull/34092#issuecomment-926375480 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48098/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join

AmplabJenkins removed a comment on pull request #34069: URL: https://github.com/apache/spark/pull/34069#issuecomment-926373023 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48096/

[GitHub] [spark] SparkQA commented on pull request #34091: [SPARK-36839][INFRA] Add daily build with Hadoop 2 profile in GitHub Actions build

SparkQA commented on pull request #34091: URL: https://github.com/apache/spark/pull/34091#issuecomment-926373020 **[Test build #143591 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143591/testReport)** for PR 34091 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join

AmplabJenkins commented on pull request #34069: URL: https://github.com/apache/spark/pull/34069#issuecomment-926373023 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48096/ --

[GitHub] [spark] SparkQA commented on pull request #34069: [SPARK-36823][SQL] Support broadcast nested loop join hint for equi-join

SparkQA commented on pull request #34069: URL: https://github.com/apache/spark/pull/34069#issuecomment-926372976 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48096/ -- This is an automated message from the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34091: [SPARK-36839][INFRA] Add daily build with Hadoop 2 profile in GitHub Actions build

AmplabJenkins removed a comment on pull request #34091: URL: https://github.com/apache/spark/pull/34091#issuecomment-926372336 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48094/

[GitHub] [spark] AmplabJenkins commented on pull request #34091: [SPARK-36839][INFRA] Add daily build with Hadoop 2 profile in GitHub Actions build

AmplabJenkins commented on pull request #34091: URL: https://github.com/apache/spark/pull/34091#issuecomment-926372336 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48094/ --

[GitHub] [spark] SparkQA commented on pull request #34091: [SPARK-36839][INFRA] Add daily build with Hadoop 2 profile in GitHub Actions build

SparkQA commented on pull request #34091: URL: https://github.com/apache/spark/pull/34091#issuecomment-926372263 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48094/ -- This is an automated message from the

[GitHub] [spark] AmplabJenkins removed a comment on pull request #34077: [SPARK-36829][SQL] Refactor NULL check for collectionOperators

AmplabJenkins removed a comment on pull request #34077: URL: https://github.com/apache/spark/pull/34077#issuecomment-926370365 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48097/

[GitHub] [spark] SparkQA commented on pull request #34038: [SPARK-36797][SQL] Union should resolve nested columns as top-level columns

SparkQA commented on pull request #34038: URL: https://github.com/apache/spark/pull/34038#issuecomment-926371144 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48100/ -- This is an automated message from the Apache

[GitHub] [spark] gengliangwang commented on a change in pull request #34092: [WIP][SPARK-36827][CORE] Improve the perf and memory usage of cleaning up stage UI data

gengliangwang commented on a change in pull request #34092: URL: https://github.com/apache/spark/pull/34092#discussion_r715321250 ## File path: core/src/main/scala/org/apache/spark/status/AppStatusListener.scala ## @@ -1253,44 +1254,46 @@ private[spark] class

[GitHub] [spark] SparkQA commented on pull request #34089: [SPARK-36837][BUILD] Upgrade Kafka to 3.0.0

SparkQA commented on pull request #34089: URL: https://github.com/apache/spark/pull/34089#issuecomment-926370883 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48099/ -- This is an automated message from the Apache

[GitHub] [spark] AmplabJenkins commented on pull request #34077: [SPARK-36829][SQL] Refactor NULL check for collectionOperators

AmplabJenkins commented on pull request #34077: URL: https://github.com/apache/spark/pull/34077#issuecomment-926370365 Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48097/ --

[GitHub] [spark] SparkQA commented on pull request #34077: [SPARK-36829][SQL] Refactor NULL check for collectionOperators

SparkQA commented on pull request #34077: URL: https://github.com/apache/spark/pull/34077#issuecomment-926370348 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48097/ -- This is an automated message from the

[GitHub] [spark] attilapiros commented on a change in pull request #33936: [SPARK-36693][REPL] Implement spark-shell idle timeouts

attilapiros commented on a change in pull request #33936: URL: https://github.com/apache/spark/pull/33936#discussion_r715326783 ## File path: repl/src/main/scala-2.12/org/apache/spark/repl/SparkILoop.scala ## @@ -105,6 +108,13 @@ class SparkILoop(in0: Option[BufferedReader],

[GitHub] [spark] gengliangwang commented on pull request #34092: [WIP][SPARK-36827][CORE] Improve the perf and memory usage of cleaning up stage UI data

gengliangwang commented on pull request #34092: URL: https://github.com/apache/spark/pull/34092#issuecomment-926368948 @mridulm there is something wrong with the benchmark. I rerun the updated benchmark and find that after changes the LevelDB is slower. I am considering making the code

[GitHub] [spark] mridulm commented on pull request #34092: [WIP][SPARK-36827][CORE] Improve the perf and memory usage of cleaning up stage UI data

mridulm commented on pull request #34092: URL: https://github.com/apache/spark/pull/34092#issuecomment-926367597 Did not notice the change was still WIP ! But the fix looks good to me - though I am not sure if it is sufficient to fix the issue -- This is an automated message from the

<    1   2   3   4   5