xwu99 commented on pull request #33941:
URL: https://github.com/apache/spark/pull/33941#issuecomment-1067466661
> It think it would be best to backup and discuss what exactly we want to
target for this PR. These are the things I'm thinking:
>
> * Support a basic reuse policy (perhaps EXEC_CORES_EQUAL since I think
that was your original use case) and allow user to specify their own. ie
config that perhaps load all policies, like the spark.plugins config. User can
give the policy a name and reference it by name.
> * I think we can keep the reuse policy specified at the application level
rather then stage level to keep logic easier for now. We can always add an
interface to specify at stage level later.
For those two I agree to keep it simple at first to address the most common
scenarios. I would check the other comments in the coming weeks.
> * Do we want to allow reusing without dynamic allocation? We would have
to throw at runtime if we discovered they tried to specify resource profile
that wouldn't be compatible. We could keep this PR simple and not support that
at first.
Since ResourceProfiles are only supported with dynamic allocation right now.
(SparkException("ResourceProfiles are only supported on YARN and Kubernetes
with dynamic allocation enabled."). Maybe a seperate PR to address that.
> does that sound ok?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
