[GitHub] spark issue #21511: [SPARK-24491][Kubernetes] Configuration support for requ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21511
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21511: [SPARK-24491][Kubernetes] Configuration support for requ...

[alexmilowski]

Github user alexmilowski commented on the issue:

https://github.com/apache/spark/pull/21511
  
Hello all,

I've been thinking about trying to make this more generic given I just ran 
into a hostPath/volume issue for executors which is similar. I took a look at 
SPARK-24434 and that seems to be likely the right path. In the end, GPU, 
volumes, etc. are all just aspects that need to go into the pod description so 
the scheduler can choose the right node and associate the right resources.

I will try to contribute to SPARK-24434 and this pull request might not be 
necessary afterwards.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21511: [SPARK-24491][Kubernetes] Configuration support for requ...

[foxish]

Github user foxish commented on the issue:

https://github.com/apache/spark/pull/21511
  
+1 agreed with @liyinan926. Let's be more careful what we add to the 
configuration options. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21511: [SPARK-24491][Kubernetes] Configuration support for requ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21511
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21511: [SPARK-24491][Kubernetes] Configuration support for requ...

[liyinan926]

Github user liyinan926 commented on the issue:

https://github.com/apache/spark/pull/21511
  
Agreed. We might want to see and wait for the design for SPARK-24434. 
Seriously we already have a lot of config options for specific things to 
customize for the pods, and we are running into the problem of config option 
explosion if we keep adding more. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21511: [SPARK-24491][Kubernetes] Configuration support for requ...

[mccheah]

Github user mccheah commented on the issue:

https://github.com/apache/spark/pull/21511
  
I think we might want to defer this feature until we have figured out 
SPARK-24434 - as already mentioned, arbitrary resource types could be desired 
outside of just GPUs. I think once we get a better understanding of what we 
want to do for SPARK-24434 we could decide on what classes of things we want to 
make configurable through spark-submit and what should only be configurable in 
the pod spec. @skonto @liyinan926 for thoughts.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21511: [SPARK-24491][Kubernetes] Configuration support for requ...

[shaneknapp]

Github user shaneknapp commented on the issue:

https://github.com/apache/spark/pull/21511
  
On Fri, Jun 8, 2018 at 10:36 AM, Ilan Filonenko 
wrote:

> *@ifilonenko* requested changes on this pull request.
>
> Thank you for this work. Would be nice to see some integration tests, but
> I don't believe GPUs are available in our test cluster. @shaneknapp
>  to comment?
>
that is correct -- we currently have no GPUs in the cluster.

that being said, we do have plan to add one or two GPUs in the near future
but they (initially) will be reserved for riselab-specific builds.  the two
big problems that we have when it comes to deploying them are:

1) 8 of the 10 build nodes are blade-style servers (4 servers in a 2u
mount) and don't have physical room.  the only two we have that can
physically house them are the current amp-jenkins-staging-worker nodes, and
they can hold 2 each...  but due to how the power cables plug in on most of
them, we are again limited to a couple of specific GPU types (power cables
on the top of the card vs being on the back).

2) GPUs are expensive...  :)

i'm more than happy to revisit k8s tests running w/GPUs once we get them
installed in to the workers and figure out our lab's usage stats.



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21511: [SPARK-24491][Kubernetes] Configuration support for requ...

[mccheah]

Github user mccheah commented on the issue:

https://github.com/apache/spark/pull/21511
  
> But I am concerned about it setting a precedent for having to change 
Spark's source code every time you want to request a new resource type. I am 
very familiar with the space of machine learning accelerators, and trust me, 
there's a lot of cool new things coming down the pipeline.

@galv this might fall under the more general question of how to give 
arbitrary configurations to the pod we make in the driver and executors - see 
https://issues.apache.org/jira/browse/SPARK-24434.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21511: [SPARK-24491][Kubernetes] Configuration support for requ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21511
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #21511: [SPARK-24491][Kubernetes] Configuration support for requ...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/21511
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org