pkotikalapudi commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-2135858753
> hey guys, which version will this land in?
We have to get reviews and approvals from PMC members and our Sheperd
(@HeartSaVioR ) before set a timeline on when it can be
stym06 commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-2132173624
hey guys, which version will this land in?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
pkotikalapudi commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-2106438054
@jerrypeng, @HeartSaVioR, I have made changes so that the DRA can support
multi queries
([ref](https://lists.apache.org/thread/836j9ckxw13mlf5m8joh8zp9t7ffyhp0)).
I have
pkotikalapudi commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1597752703
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -669,6 +753,27 @@ private[spark] class ExecutorAllocationManager(
private val
pkotikalapudi commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1593285827
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -340,6 +385,45 @@ private[spark] class ExecutorAllocationManager(
}
}
+
tgravescs commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1583032683
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -340,6 +385,45 @@ private[spark] class ExecutorAllocationManager(
}
}
+ /**
pkotikalapudi commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1578138588
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -916,8 +1040,13 @@ private[spark] class ExecutorAllocationManager(
pkotikalapudi commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1578115234
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -669,6 +753,27 @@ private[spark] class ExecutorAllocationManager(
private val
pkotikalapudi commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1577078911
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -340,6 +385,45 @@ private[spark] class ExecutorAllocationManager(
}
}
+
pkotikalapudi commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1577058026
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -340,6 +385,45 @@ private[spark] class ExecutorAllocationManager(
}
}
+
pkotikalapudi commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1577025511
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -96,6 +96,30 @@ import org.apache.spark.util.{Clock, SystemClock,
ThreadUtils,
jerrypeng commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1566272683
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -669,6 +753,27 @@ private[spark] class ExecutorAllocationManager(
private val
jerrypeng commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1566270204
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -340,6 +385,45 @@ private[spark] class ExecutorAllocationManager(
}
}
+ /**
jerrypeng commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1566267014
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -916,8 +1040,13 @@ private[spark] class ExecutorAllocationManager(
jerrypeng commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1566259719
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -340,6 +385,45 @@ private[spark] class ExecutorAllocationManager(
}
}
+ /**
jerrypeng commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1566255030
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -96,6 +96,30 @@ import org.apache.spark.util.{Clock, SystemClock,
ThreadUtils, Utils}
HeartSaVioR commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1561043366
##
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##
@@ -601,6 +601,12 @@ package object config {
.booleanConf
pkotikalapudi commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1548738209
##
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##
@@ -630,6 +636,12 @@ package object config {
.doubleConf
pkotikalapudi commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1548737819
##
core/src/main/scala/org/apache/spark/internal/config/package.scala:
##
@@ -601,6 +601,12 @@ package object config {
.booleanConf
pkotikalapudi commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1548737021
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -340,6 +366,39 @@ private[spark] class ExecutorAllocationManager(
}
}
+
pkotikalapudi commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1548736814
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -133,6 +133,16 @@ private[spark] class ExecutorAllocationManager(
private val
pkotikalapudi commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1548712982
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -185,6 +195,12 @@ private[spark] class ExecutorAllocationManager(
* If not,
pkotikalapudi commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1548620204
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -669,6 +728,27 @@ private[spark] class ExecutorAllocationManager(
private val
tgravescs commented on code in PR #42352:
URL: https://github.com/apache/spark/pull/42352#discussion_r1546387115
##
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:
##
@@ -669,6 +728,27 @@ private[spark] class ExecutorAllocationManager(
private val
pkotikalapudi commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1965668989
> If you'd like to revive this PR, please reopen it and ask a committer to
remove the Stale tag!
@rdblue can you please re-open the PR and remove the stale tag. I think
github-actions[bot] closed pull request #42352: [WIP][SPARK-24815] [CORE]
Trigger Interval based DRA for Structured Streaming
URL: https://github.com/apache/spark/pull/42352
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and
pkotikalapudi opened a new pull request, #42352:
URL: https://github.com/apache/spark/pull/42352
### What changes were proposed in this pull request?
- Initial Implementation to DRA changes to work for structured streaming
applications
- Design doc:
pkotikalapudi commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1958809916
Thanks for subscribing and voting Krystal. Please request engineers in
adobe to vote for this in the same manner. I will bump up the voting thread
again to see if PMC has any
krymitch commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1955238640
Thanks you @pkotikalapudi. I subscribed, and then messaged dev-thread.35716
and it returned all of our comments, only it is not showing my comment in the
UI inline on the same thread.
pkotikalapudi commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1954918297
we need a PMC member to shepherd/review/merge the effort. I didn't see any
response from spark PMC members yet.
>I don't know how to vote on email thread. Last time I've send
krymitch commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1954917208
@vitgorbunov Agreed. Same here. Looks like there are 7 thumbs up for this
PR. Hopefully thumbs up on this PR count since many may not have official
Apache login or may not have the time
vitgorbunov commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1954876934
I don't know how to vote on email thread. Last time I've send the message
from my mail client and it wasn't displayed on web.
I think it's fair to count thumb ups on this PR as
pkotikalapudi commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1901976588
Thanks for the support @krymitch , here is the voting [email
thread](https://lists.apache.org/thread/rlwqrw6ddxdkbvkp78kpd0zgvglgbbp8).
I will ask my team to do the same as
krymitch commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1901136777
@pkotikalapudi please share new voting thread here or in old thread. A few
of us over at Adobe would like to add our vote, since this work will support a
few projects we are currently
mentasm commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1895168960
It may be my corporate proxy blocking google docs. Thanks for the pdf link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
pkotikalapudi commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1895009757
[Dynamic Resource Allocation for Structured
streaming.pdf](https://github.com/apache/spark/files/13959367/SPIP_.Dynamic.Resource.Allocation.for.Structured.streaming.pdf)
I
pkotikalapudi commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1894982472
what kind of error do you get? I had people comment on it in the past, are
you behind any vpn?
or if you are ok sharing your email, I can send an invite to the document
--
mentasm commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1894941470
@pkotikalapudi For me the google docs link to the design doc no longer works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
mentasm commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1894518160
I will happily add my experiences to that email thread as a consumer of SSS
if I can work out how. Our prod environment runs around 40 SSS apps and
traffic is determined by banking
krymitch commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1894134932
Hi @mridulm. We appreciate you support on this. DRA is essential to auto
scaling up and back down. Can you please confirm if this proposal was ever
dropped in the dev list for
pkotikalapudi commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1893936508
@mentasm , yeah we have an [email
thread](https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo) going
for a long time. Mich was gracious enough to do a review, but
pky-c commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1893108571
We have also encountered this issue.!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
mentasm commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1893080905
I am also very interested in this code making it into spark. We run a lot
of structured streaming and have had to resort to static sized cluster due to
the lack of proper scaling code
github-actions[bot] closed pull request #42352: [WIP][SPARK-24815] [CORE]
Trigger Interval based DRA for Structured Streaming
URL: https://github.com/apache/spark/pull/42352
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and
github-actions[bot] commented on PR #42352:
URL: https://github.com/apache/spark/pull/42352#issuecomment-1865306284
We're closing this PR because it hasn't been updated in a while. This isn't
a judgement on the merit of the PR in any way. It's just a way of keeping the
PR queue manageable.
45 matches
Mail list logo
