Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/19954
> I don't think they are independent as architecturally they make sense
together and represent a single concern: enabling use of remote dependencies
through init-containers. Missing any one of the three makes the feature
unusable. I would also argue that it won't necessarily make review easier as
reviewers need to mentally connect them together to make sense of each change
set. If the general conclusion is that we should first refactor the code to
achieve a better abstraction instead of getting this feature into 2.3, I can
buy that. But I don't think we should tackle this as three components.
It is true that introducing each of those components individually doesn't
make the feature usable, but these parts are independent in the sense that they
each have different classes and different sections of code. Furthermore the
architectural considerations are different for each of them. There's some
architectural connection between the submission client and the driver
configuring the driver and the executor pods though given that we're trying to
share code. But the init-container itself can be contributed as a completely
separate push.
Basically I'm wondering if we can reduce the diff in any way. Making the
code itself simpler as we've been discussing is certainly the primary way.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
