Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19954
> We discuss this a bit more in here
It's nice that there is some documentation somewhere, but that
documentation doesn't really seem to address my comments. For one example, it
only explicitly talks about the driver - which sort of makes sense because the
document is about submission. But why aren't orchestrators used when starting
executors too? It seems there's similar code baked into another class instead.
What I'm asking is for this to be documented properly so that someone who
didn't write the code has enough information to know that it's working as it
should. Right now I don't see what some of these abstractions are for at all -
for example, as far as I can see, the orchestrator can be replaced by a method
call instead of being a completely separate type; it's not really abstracting
anything. Look at where it's used:
```
val configurationStepsOrchestrator = new DriverConfigOrchestrator(...)
Utils.tryWithResource(SparkKubernetesClientFactory.createKubernetesClient(/*
another long list of arguments */)0) { kubernetesClient =>
val client = new Client(
configurationStepsOrchestrator.getAllConfigurationSteps(),
/* another long list of arguments */
```
So aside from not really being able to infer the structure of how these
things work, the current abstraction seems to be creating a lot of constructors
and methods with long lists of arguments, which is another thing that hurts the
readability of the code.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
