tgravescs commented on pull request #34672:
URL: https://github.com/apache/spark/pull/34672#issuecomment-975646985
So my first reaction is: you have a 3rd party shuffle manager that is an
external shuffle service because it supports dynamic allocation, then why is it
failing... is it because you didn't override something, or because you
couldn't override something?
In this case it's creating a ExternalBlockStoreClient, which I think isn't
setup to be overridden. I think it comes down to we just haven't really added
support to allow this. I definitely agree that we should have this and that
3rd party shuffle services should be able to use dynamic allocation but I would
like to see it done more officially then this.
Either something like @attilapiros mention - which I definitely like the
idea of asking the shuffle manager what it supports - or user overrides this
class somehow.
I need to go refresh my memory with all the various shuffle stuff that went
in. we had some other shuffle work in progress that seems to have stalled as
well.
Maybe I'm missing why this needs to go in quickly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
