Github user dvogelbacher commented on the issue:
https://github.com/apache/spark/pull/21993
@gatorsmile yes, I found that workaround. Very useful :)
I think it would still be good to handle this better by default. I can see
that introducing such an arbitrary configuration param
Github user dvogelbacher commented on the issue:
https://github.com/apache/spark/pull/21993
@HyukjinKwon can you help with finding reviewers for this PR?
---
-
To unsubscribe, e-mail: reviews-unsubscr
GitHub user dvogelbacher opened a pull request:
https://github.com/apache/spark/pull/21993
[SPARK-24983][SQL] Add configuration for maximum number of leaf expressions
in collapsed project nodes
## What changes were proposed in this pull request?
Add a configuration option
Github user dvogelbacher commented on the issue:
https://github.com/apache/spark/pull/21366
Agree with @mccheah on not blocking this on a design doc. This PR strictly
improves the management of executor states in k8s compared to how it was done
before. So we really should get
Github user dvogelbacher commented on a diff in the pull request:
https://github.com/apache/spark/pull/21366#discussion_r194183023
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshotsStoreImpl.scala
---
@@ -0,0
Github user dvogelbacher commented on the issue:
https://github.com/apache/spark/pull/21366
good catch @mccheah those kind of things weren't handled by the first
approach.
I like the new one it's basically declarative in saying `state=snapshot
from k8s` (plus optimizing
Github user dvogelbacher commented on a diff in the pull request:
https://github.com/apache/spark/pull/21366#discussion_r194180839
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleManager.scala
---
@@ -0,0
Github user dvogelbacher commented on a diff in the pull request:
https://github.com/apache/spark/pull/21366#discussion_r194179846
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshotsStoreImpl.scala
---
@@ -0,0
Github user dvogelbacher commented on a diff in the pull request:
https://github.com/apache/spark/pull/21366#discussion_r194177422
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala
---
@@ -49,48 +54,58
Github user dvogelbacher commented on a diff in the pull request:
https://github.com/apache/spark/pull/21366#discussion_r194176787
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala
---
@@ -105,14 +120,15
Github user dvogelbacher commented on the issue:
https://github.com/apache/spark/pull/21366
@foxish did you have a chance to take a look?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user dvogelbacher commented on a diff in the pull request:
https://github.com/apache/spark/pull/21366#discussion_r192200162
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleEventHandler.scala
---
@@ -0,0
Github user dvogelbacher commented on a diff in the pull request:
https://github.com/apache/spark/pull/21366#discussion_r192191070
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleEventHandler.scala
---
@@ -0,0
Github user dvogelbacher commented on a diff in the pull request:
https://github.com/apache/spark/pull/21366#discussion_r192187376
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleEventHandler.scala
---
@@ -0,0
Github user dvogelbacher commented on a diff in the pull request:
https://github.com/apache/spark/pull/21366#discussion_r192197340
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala
---
@@ -0,0 +1,120
Github user dvogelbacher commented on a diff in the pull request:
https://github.com/apache/spark/pull/21366#discussion_r192187397
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleEventHandler.scala
---
@@ -0,0
Github user dvogelbacher commented on a diff in the pull request:
https://github.com/apache/spark/pull/21366#discussion_r192182970
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala
---
@@ -0,0 +1,120
Github user dvogelbacher commented on a diff in the pull request:
https://github.com/apache/spark/pull/21366#discussion_r192189992
--- Diff:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleEventHandler.scala
---
@@ -0,0
Github user dvogelbacher commented on the issue:
https://github.com/apache/spark/pull/20553
Can we call this `spark.kubernetes.executor.request.cores`, similar to
`spark.kubernetes.executor.limit.cores` ?
I think this would make the naming a bit less confusing
Github user dvogelbacher commented on the issue:
https://github.com/apache/spark/pull/20943
@felixcheung can you review this?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
GitHub user dvogelbacher opened a pull request:
https://github.com/apache/spark/pull/20943
requesting memory + memory overhead when creating kubernetes pods forâ¦
## What changes were proposed in this pull request?
Kubernetes driver and executor pods should request `memory
Github user dvogelbacher commented on the issue:
https://github.com/apache/spark/pull/20943
jenkins test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user dvogelbacher commented on the issue:
https://github.com/apache/spark/pull/20779
LGTM as well, thanks for making the PR @kiszk
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
Github user dvogelbacher commented on a diff in the pull request:
https://github.com/apache/spark/pull/20779#discussion_r173484471
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
---
@@ -307,4 +309,47 @@ class WholeStageCodegenSuite
Github user dvogelbacher commented on the issue:
https://github.com/apache/spark/pull/20779
Note though, that executing this also sometimes slays the compiler (and
sometimes it doesn't and leads to the above error):
```
That entry seems to have slain the compiler. Shall I
Github user dvogelbacher commented on the issue:
https://github.com/apache/spark/pull/20779
I was able to reproduce just now without changing the value of the constant
(i.e., with unmodified code from master):
> â spark git:(master) ./bin/spark-shell
18/03/09 11:11
26 matches
Mail list logo
