khalidmammadov opened a new pull request, #37988:
URL: https://github.com/apache/spark/pull/37988
### What changes were proposed in this pull request?
It's part of the Pyspark docstrings improvement series
(https://github.com/apache/spark/pull/37592,
https://github.com/apache/spark/pull/
AmplabJenkins commented on PR #37988:
URL: https://github.com/apache/spark/pull/37988#issuecomment-1257145678
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
khalidmammadov commented on PR #37988:
URL: https://github.com/apache/spark/pull/37988#issuecomment-1257156072
@srowen @itholic @HyukjinKwon Please review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above t
HyukjinKwon opened a new pull request, #37989:
URL: https://github.com/apache/spark/pull/37989
### What changes were proposed in this pull request?
This PR is a followup of https://github.com/apache/spark/pull/37533 that
works around the test failure by explicitly checking the element
HyukjinKwon commented on code in PR #37989:
URL: https://github.com/apache/spark/pull/37989#discussion_r979395539
##
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala:
##
@@ -4469,7 +4469,7 @@ class DAGSchedulerSuite extends SparkFunSuite with
TempLocalSpar
HyukjinKwon commented on PR #37989:
URL: https://github.com/apache/spark/pull/37989#issuecomment-1257175536
cc @mridulm FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon commented on PR #37988:
URL: https://github.com/apache/spark/pull/37988#issuecomment-1257175625
Thanks for finishing this work, @khalidmammadov.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL abov
cchantep commented on PR #36030:
URL: https://github.com/apache/spark/pull/36030#issuecomment-1257182398
Closing because nobody review it timely?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to t
roczei commented on PR #37679:
URL: https://github.com/apache/spark/pull/37679#issuecomment-1257184837
Hi @cloud-fan,
All build issues have been fixed and all of your feedbacks have been
implemented. Latest state:
```
$ bin/spark-shell --conf
spark.sql.catalog.spark_catalog
EvgenyZamyatin commented on PR #37967:
URL: https://github.com/apache/spark/pull/37967#issuecomment-1257192550
@zhengruifeng Hi!
Could you please review my changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use th
HyukjinKwon commented on code in PR #37989:
URL: https://github.com/apache/spark/pull/37989#discussion_r979395539
##
core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala:
##
@@ -4469,7 +4469,7 @@ class DAGSchedulerSuite extends SparkFunSuite with
TempLocalSpar
HyukjinKwon commented on PR #37985:
URL: https://github.com/apache/spark/pull/37985#issuecomment-1257193303
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon closed pull request #37985: [SPARK-40548][BUILD] Upgrade rocksdbjni
from 7.5.3 to 7.6.0
URL: https://github.com/apache/spark/pull/37985
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to th
khalidmammadov commented on PR #37988:
URL: https://github.com/apache/spark/pull/37988#issuecomment-1257199499
> Thanks for finishing this work, @khalidmammadov.
Happy to help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to Gi
lvshaokang commented on PR #37986:
URL: https://github.com/apache/spark/pull/37986#issuecomment-1257206608
@MaxGekk Thanks for you review. I'm already addressing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
ivoson commented on PR #37268:
URL: https://github.com/apache/spark/pull/37268#issuecomment-1257212450
> Mostly looks good. We need to update the docs like:
https://github.com/apache/spark/blob/master/docs/configuration.md#stage-level-scheduling-overview
It says "the current implementation
attilapiros opened a new pull request, #37990:
URL: https://github.com/apache/spark/pull/37990
### What changes were proposed in this pull request?
Bump kubernetes-client version from 5.12.3 to 6.1.1 and clean up all the
deprecations.
### Why are the changes needed?
attilapiros commented on PR #37990:
URL: https://github.com/apache/spark/pull/37990#issuecomment-1257240658
The `inNamespace` calls are added because of the [namespace
changes](https://github.com/fabric8io/kubernetes-client/blob/master/doc/MIGRATION-v6.md#namespace-changes)
and to have more
attilapiros commented on code in PR #37990:
URL: https://github.com/apache/spark/pull/37990#discussion_r979435775
##
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/SparkKubernetesClientFactory.scala:
##
@@ -115,7 +115,10 @@ private[spark] object Spa
attilapiros commented on code in PR #37990:
URL: https://github.com/apache/spark/pull/37990#discussion_r979435892
##
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/K8sSubmitOps.scala:
##
@@ -144,14 +134,13 @@ private[spark] class K8SSparkSubm
attilapiros commented on code in PR #37990:
URL: https://github.com/apache/spark/pull/37990#discussion_r979436243
##
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleManager.scala:
##
@@ -168,23 +171,19 @@ private[spark
bjornjorgensen opened a new pull request, #37991:
URL: https://github.com/apache/spark/pull/37991
### What changes were proposed in this pull request?
Upgrade protobuf-python from 4.21.5 to 4.21.6
### Why are the changes needed?
[CVE-2022-1941](https://nvd.nist.gov/vuln/detai
srowen commented on PR #37988:
URL: https://github.com/apache/spark/pull/37988#issuecomment-1257243173
Merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To u
srowen closed pull request #37988: [SPARK-40142][PYTHON][SQL][FOLLOW-UP] Make
pyspark.sql.functions examples self-contained (FINAL)
URL: https://github.com/apache/spark/pull/37988
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub a
attilapiros commented on code in PR #37990:
URL: https://github.com/apache/spark/pull/37990#discussion_r979437063
##
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/submit/K8sSubmitOpSuite.scala:
##
@@ -101,18 +114,19 @@ class K8sSubmitOpSuite extend
attilapiros commented on code in PR #37990:
URL: https://github.com/apache/spark/pull/37990#discussion_r979439689
##
resource-managers/kubernetes/core/pom.xml:
##
@@ -75,6 +75,11 @@
test
+
Review Comment:
https://github.com/fabric8io/kubernetes-client/blo
attilapiros commented on code in PR #37990:
URL: https://github.com/apache/spark/pull/37990#discussion_r979439689
##
resource-managers/kubernetes/core/pom.xml:
##
@@ -75,6 +75,11 @@
test
+
Review Comment:
https://github.com/fabric8io/kubernetes-client/blo
attilapiros commented on code in PR #37990:
URL: https://github.com/apache/spark/pull/37990#discussion_r979439689
##
resource-managers/kubernetes/core/pom.xml:
##
@@ -75,6 +75,11 @@
test
+
Review Comment:
https://github.com/fabric8io/kubernetes-client/blo
AmplabJenkins commented on PR #37991:
URL: https://github.com/apache/spark/pull/37991#issuecomment-1257250279
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
bjornjorgensen commented on PR #37991:
URL: https://github.com/apache/spark/pull/37991#issuecomment-1257271472
cc @grundprinzip
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comme
itholic commented on PR #37988:
URL: https://github.com/apache/spark/pull/37988#issuecomment-1257289652
Thanks for your efforts to finish this work!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go t
srowen commented on PR #37989:
URL: https://github.com/apache/spark/pull/37989#issuecomment-1257299340
Merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To u
srowen closed pull request #37989: [SPARK-40096][CORE][TESTS][FOLLOW-UP]
Explicitly check the element and length
URL: https://github.com/apache/spark/pull/37989
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL abov
zhengruifeng closed pull request #37923: [SPARK-40334][PS] Implement
`GroupBy.prod`
URL: https://github.com/apache/spark/pull/37923
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comme
zhengruifeng commented on PR #37923:
URL: https://github.com/apache/spark/pull/37923#issuecomment-1257315237
Merged into master, thank you @ayudovin for working on it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use th
nkronenfeld commented on PR #36613:
URL: https://github.com/apache/spark/pull/36613#issuecomment-1257319350
I haven't done anything on the branch because I was waiting for comments -
but as far as I know, no one even looked at it. Am I missing something for it
to get considered in the firs
nkronenfeld commented on PR #36613:
URL: https://github.com/apache/spark/pull/36613#issuecomment-1257319840
also, I don't see a button to re-open it - does anyone know where that is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to Gi
zhengruifeng commented on code in PR #37978:
URL: https://github.com/apache/spark/pull/37978#discussion_r979482634
##
python/pyspark/pandas/series.py:
##
@@ -6610,6 +6610,78 @@ def compare(
)
return DataFrame(internal)
+# todo: 1, support array-like 'valu
zhengruifeng commented on code in PR #37978:
URL: https://github.com/apache/spark/pull/37978#discussion_r979482727
##
python/pyspark/pandas/series.py:
##
@@ -6610,6 +6610,78 @@ def compare(
)
return DataFrame(internal)
+# todo: 1, support array-like 'valu
github-actions[bot] commented on PR #36829:
URL: https://github.com/apache/spark/pull/36829#issuecomment-1257322956
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
github-actions[bot] closed pull request #36301: [SPARK-21697][SQL] NPE &
ExceptionInInitializerError trying to load UDF from HDFS
URL: https://github.com/apache/spark/pull/36301
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and
github-actions[bot] commented on PR #36265:
URL: https://github.com/apache/spark/pull/36265#issuecomment-1257322967
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
github-actions[bot] closed pull request #36208: [SPARK-38911][CORE] Fix the
potential resource profile id mess up issue
URL: https://github.com/apache/spark/pull/36208
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
U
github-actions[bot] closed pull request #36180: [SPARK-38887][SQL] Support
switch inner join side for sort merge join
URL: https://github.com/apache/spark/pull/36180
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
github-actions[bot] closed pull request #36151: WIP: [SPARK-27998] [SQL] Add
support for double-quoted named expressions
URL: https://github.com/apache/spark/pull/36151
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
github-actions[bot] closed pull request #36874: [SPARK-39475][SQL] Pull out
complex join keys for shuffled join
URL: https://github.com/apache/spark/pull/36874
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
github-actions[bot] closed pull request #36128: [SPARK-3][SQL] Pushdown
scalar-subquery filter to FileSourceScan
URL: https://github.com/apache/spark/pull/36128
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
github-actions[bot] closed pull request #36088: [SPARK-38805][SHUFFLE]
Automatically remove an expired indexFilePath from the ESS shuffleIndexCache or
the PBS indexCache to save memory.
URL: https://github.com/apache/spark/pull/36088
--
This is an automated message from the Apache Git Servic
github-actions[bot] commented on PR #35858:
URL: https://github.com/apache/spark/pull/35858#issuecomment-1257323009
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
github-actions[bot] closed pull request #35927: [WIP] Simplify the rule of
auto-generated alias name
URL: https://github.com/apache/spark/pull/35927
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to t
github-actions[bot] commented on PR #35845:
URL: https://github.com/apache/spark/pull/35845#issuecomment-1257323023
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
github-actions[bot] commented on PR #35867:
URL: https://github.com/apache/spark/pull/35867#issuecomment-1257323002
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
github-actions[bot] commented on PR #35808:
URL: https://github.com/apache/spark/pull/35808#issuecomment-1257323034
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
github-actions[bot] commented on PR #35764:
URL: https://github.com/apache/spark/pull/35764#issuecomment-1257323067
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
github-actions[bot] commented on PR #35799:
URL: https://github.com/apache/spark/pull/35799#issuecomment-1257323057
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
github-actions[bot] commented on PR #35763:
URL: https://github.com/apache/spark/pull/35763#issuecomment-1257323076
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
github-actions[bot] commented on PR #35806:
URL: https://github.com/apache/spark/pull/35806#issuecomment-1257323038
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
bersprockets commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r979487585
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##
@@ -291,7 +298,8 @@ object RewriteDistinctAggregates exte
HyukjinKwon commented on code in PR #37991:
URL: https://github.com/apache/spark/pull/37991#discussion_r979489138
##
dev/requirements.txt:
##
@@ -48,4 +48,4 @@ black==22.6.0
# Spark Connect
grpcio==1.48.1
-protobuf==4.21.5
\ No newline at end of file
+protobuf==4.21.6
Revie
bersprockets commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r979489753
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##
@@ -213,7 +213,16 @@ object RewriteDistinctAggregates ext
Kwafoor closed pull request #37951: [SPARK-40506]Spark Streaming metrics name
doesn't need application name
URL: https://github.com/apache/spark/pull/37951
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
mridulm commented on PR #37989:
URL: https://github.com/apache/spark/pull/37989#issuecomment-1257374954
This is very interesting behavior !
Thanks for fixing this @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHu
weixiuli commented on code in PR #37922:
URL: https://github.com/apache/spark/pull/37922#discussion_r979507631
##
core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:
##
@@ -2543,16 +2541,13 @@ private[spark] class DAGScheduler(
shuffleIdToMapStage.filter {
HeartSaVioR commented on PR #37285:
URL: https://github.com/apache/spark/pull/37285#issuecomment-1257400388
We can close this now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific com
HeartSaVioR closed pull request #37285: [POC][PYTHON][SS] Arbitrary stateful
processing in Structured Streaming with Python
URL: https://github.com/apache/spark/pull/37285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use t
HeartSaVioR commented on PR #37935:
URL: https://github.com/apache/spark/pull/37935#issuecomment-1257401965
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
LuciferYang commented on PR #37979:
URL: https://github.com/apache/spark/pull/37979#issuecomment-1257402939
thanks @wangyum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
LuciferYang commented on PR #37976:
URL: https://github.com/apache/spark/pull/37976#issuecomment-1257403015
thanks @wangyum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HeartSaVioR closed pull request #37935: [SPARK-40492][SS] Do maintenance before
streaming StateStore unload
URL: https://github.com/apache/spark/pull/37935
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
HeartSaVioR commented on PR #37935:
URL: https://github.com/apache/spark/pull/37935#issuecomment-1257403721
Thanks @chaoqin-li1123 for the contribution! I merged this to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub
zhengruifeng opened a new pull request, #37992:
URL: https://github.com/apache/spark/pull/37992
### What changes were proposed in this pull request?
Make `ddof` in `DataFrame.sem` and `Series.sem` accept arbitary integers
### Why are the changes needed?
for API coverage
beliefer commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r979537901
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##
@@ -218,9 +218,16 @@ object RewriteDistinctAggregates extends
grundprinzip commented on PR #37710:
URL: https://github.com/apache/spark/pull/37710#issuecomment-1257437750
Ack, I will regenerate the protos and update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above t
HyukjinKwon commented on PR #37978:
URL: https://github.com/apache/spark/pull/37978#issuecomment-1257439757
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
HyukjinKwon closed pull request #37978: [SPARK-40330][PS] Implement
`Series.searchsorted`
URL: https://github.com/apache/spark/pull/37978
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
zhengruifeng commented on PR #37978:
URL: https://github.com/apache/spark/pull/37978#issuecomment-1257440961
@HyukjinKwon @itholic Thanks for the reviews!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above t
grundprinzip opened a new pull request, #37993:
URL: https://github.com/apache/spark/pull/37993
### What changes were proposed in this pull request?
This patch cleans up the generated proto files from the initial Spark
Connect import. The previous files had a Databricks specific g
beliefer commented on PR #37825:
URL: https://github.com/apache/spark/pull/37825#issuecomment-1257447790
It seems a little complex.
I have an idea to simplify the binary expressions in other optimizer rule.
Please reference `SimplifyBinaryComparison`.
--
This is an automated mess
HyukjinKwon commented on PR #37993:
URL: https://github.com/apache/spark/pull/37993#issuecomment-1257456288
(Should probably need a separate JIRA for this)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above
lvshaokang commented on PR #37986:
URL: https://github.com/apache/spark/pull/37986#issuecomment-1257478094
@MaxGekk I have addressed, please take a look, thk!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL abo
cloud-fan commented on PR #36700:
URL: https://github.com/apache/spark/pull/36700#issuecomment-1257481093
sorry I missed this PR. @ulysses-you can you do a rebase?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
UR
ulysses-you opened a new pull request, #36700:
URL: https://github.com/apache/spark/pull/36700
### What changes were proposed in this pull request?
Remove all TPCH with stats golden files.
### Why are the changes needed?
It's a dead golden files since we have no s
cloud-fan commented on code in PR #36265:
URL: https://github.com/apache/spark/pull/36265#discussion_r979562710
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##
@@ -2594,6 +2601,31 @@ class Analyzer(override val catalogManager:
CatalogMan
cloud-fan commented on PR #37982:
URL: https://github.com/apache/spark/pull/37982#issuecomment-1257487700
all tests passed: https://github.com/peter-toth/spark/runs/8514875267
merging to 3.3, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the mess
cloud-fan closed pull request #37982: [SPARK-38717][SQL][3.3] Handle Hive's
bucket spec case preserving behaviour
URL: https://github.com/apache/spark/pull/37982
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL abo
cloud-fan commented on code in PR #35789:
URL: https://github.com/apache/spark/pull/35789#discussion_r979569276
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala:
##
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundati
cloud-fan commented on code in PR #37679:
URL: https://github.com/apache/spark/pull/37679#discussion_r979572317
##
sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala:
##
@@ -1932,6 +1932,13 @@ private[sql] object QueryExecutionErrors extends
Quer
cloud-fan commented on code in PR #37679:
URL: https://github.com/apache/spark/pull/37679#discussion_r979572820
##
sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala:
##
@@ -148,13 +148,19 @@ private[sql] class SharedState(
val externalCatalog = SharedS
Ngone51 commented on code in PR #37268:
URL: https://github.com/apache/spark/pull/37268#discussion_r979579541
##
core/src/main/scala/org/apache/spark/resource/ResourceProfileManager.scala:
##
@@ -59,35 +59,65 @@ private[spark] class ResourceProfileManager(sparkConf:
SparkConf,
beliefer commented on code in PR #35789:
URL: https://github.com/apache/spark/pull/35789#discussion_r979579903
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala:
##
@@ -0,0 +1,303 @@
+/*
+ * Licensed to the Apache Software Foundatio
cloud-fan commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r979582134
##
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##
@@ -218,9 +218,16 @@ object RewriteDistinctAggregates extend
amaliujia opened a new pull request, #37994:
URL: https://github.com/apache/spark/pull/37994
### What changes were proposed in this pull request?
Implement an approach to testing the proto to Scala conversion with a DSL
according to the proposal in
[spark-connect-testing-
amaliujia commented on PR #37994:
URL: https://github.com/apache/spark/pull/37994#issuecomment-1257516316
@cloud-fan @HyukjinKwon @@grundprinzip
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to th
zhengruifeng opened a new pull request, #37995:
URL: https://github.com/apache/spark/pull/37995
### What changes were proposed in this pull request?
to clean the intermediate cached datasets created in
`AttachDistributedSequenceExec`
1, persist the input dataset on the python side;
HyukjinKwon commented on code in PR #37994:
URL: https://github.com/apache/spark/pull/37994#discussion_r979588104
##
connect/src/main/scala/org/apache/spark/sql/connect/package.scala:
##
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
HyukjinKwon commented on code in PR #37994:
URL: https://github.com/apache/spark/pull/37994#discussion_r979588691
##
connect/src/test/scala/org/apache/spark/sql/connect/planner/SparkConnectProtoSuite.scala:
##
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (
HyukjinKwon commented on code in PR #37994:
URL: https://github.com/apache/spark/pull/37994#discussion_r979588835
##
connect/src/test/scala/org/apache/spark/sql/connect/planner/SparkConnectProtoSuite.scala:
##
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (
grundprinzip commented on PR #37993:
URL: https://github.com/apache/spark/pull/37993#issuecomment-1257533571
Created [SPARK-40557] to track.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the sp
mskapilks opened a new pull request, #37996:
URL: https://github.com/apache/spark/pull/37996
### What changes were proposed in this pull request?
Currently we allow only a specific pattern in bloom creation side plan
(consecutive Filter/Project/Scan nodes) with added column pr
99 matches
Mail list logo