viirya commented on pull request #32136: URL: https://github.com/apache/spark/pull/32136#issuecomment-858770342
> And to ensure we're on the same page - for the `RequiredResourceLocation`, how would you provide the PVC info there? IIUC, you want to put the PVC info there, right? Yea, I have not tried yet as we are still in discussion phase. But my idea is to retrieve PVC info from scheduler backend (k8s) when it retrieves executor info. I guess it doesn't return such info now. So in state store rdd, when preparing preferred locations, it queries scheduler backend (if it is k8s) to get PVC info and fill into `RequiredResourceLocation`. `RequiredResourceLocation` should be general enough. So the resource requirement might be a map like resource -> resource id. At task scheduler side, during scheduling task set, it looks at required resources to meet task requirement in the location. Sounds okay? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
