Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

be >>>> opinionated on how to set the cpu request and limit and even try to >>>> determine this automatically? >>>> >>>> >>>> >>>> And I completely agree with your answer Kimoon, we should provide >>>

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

ld provide >>> sensible defaults and make it configurable, as Yinan’s PR does. >>> >>> The only remaining question would then be what a sensible default for >>> *spark.kubernetes.executor.cores >>> *would be. Seeing that I wanted more than 1 and Yi

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

oes. >> >> The only remaining question would then be what a sensible default for >> *spark.kubernetes.executor.cores >> *would be. Seeing that I wanted more than 1 and Yinan wants less, >> leaving it at 1 night be best. >> >> >> >> Thanks, >&

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

r > *spark.kubernetes.executor.cores > *would be. Seeing that I wanted more than 1 and Yinan wants less, leaving > it at 1 night be best. > > > > Thanks, > > David > > > > *From: *Kimoon Kim > *Date: *Friday, March 30, 2018 at 4:28 PM > *To: *Yinan Li

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

ubernetes] Resource requests and limits for Driver and Executor Pods I see. Good to learn the interaction between spark.task.cpus and spark.executor.cores. But am I right to say that PR #20553 can be still used as an additional knob on top of those two? Say a user wants 1.5 core per executor f

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

I see. Good to learn the interaction between spark.task.cpus and spark.executor.cores. But am I right to say that PR #20553 can be still used as an additional knob on top of those two? Say a user wants 1.5 core per executor from Kubernetes, not the rounded up integer value 2? > A relevant question

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

PR #20553 is more for allowing users to use a fractional value for cpu requests. The existing spark.executor.cores is sufficient for specifying more than one cpus. > One way to solve this could be to request more than 1 core from Kubernetes per task. Th

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

> Instead of requesting `[driver,executor].memory`, we should just request `[driver,executor].memory + [driver,executor].memoryOverhead `. I think this case is a bit clearer than the CPU case, so I went ahead and filed an issue with more details a

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

CPU share that is available. From: Yinan Li Date: Thursday, March 29, 2018 at 11:08 PM To: David Vogelbacher Cc: "dev@spark.apache.org" Subject: Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods Hi David, Regarding cpu limit, in Spark 2.3, we d

Re: [Kubernetes] Resource requests and limits for Driver and Executor Pods

Hi David, Regarding cpu limit, in Spark 2.3, we do have the following config properties to specify cpu limit for the driver and executors. See http://spark.apache.org/docs/latest/running-on-kubernetes.html. spark.kubernetes.driver.limit.cores spark.kubernetes.executor.limit.cores On Thu, Mar 29,

[Kubernetes] Resource requests and limits for Driver and Executor Pods

Hi, At the moment driver and executor pods are created using the following requests and limits: CPUMemory Request[driver,executor].cores[driver,executor].memory LimitUnlimited (but can be specified using spark.[driver,executor].cores)[driver,executor].memory + [driver,executor].memoryOverh