tgravescs commented on pull request #31896: URL: https://github.com/apache/spark/pull/31896#issuecomment-804125940
so I would like to see more design details on this. I get that this adds a plugin for spark submit, but there are lots of other code in here that does stuff specific to cluster manager. It also seems to assume that that plugin supports various options. I think we need to clearly define what specifically it will and will not handle and make sure that is documented. Some basic things are just like does it support both client and cluster deploy modes, which options does it support, how does it deal with dependencies, how does it deal with security, does it work with PySpark, sparkR, spark-shell, spark-submit, etc. I think to do this properly we would need more APIs between these so it can properly figure out what to do. kind of minor but I think this should also be referred to as cluster manager plugin or something like that so we know what the plugin is for. if we are expecting users to implement why is this private[spark]? private[spark] trait SparkSubmitPlugin we would want this a developerapi unstable and made public. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
