Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/20910
For all reviewers - this change is very large. Github's interpretation of
the diff also doesn't present the changes in the most easily consumed manner.
To account for this, the pull request is best reviewed and understood
commit by commit. Each commit roughly translates one component from the old
architecture to the new architecture. The changes are incrementally built as
follows:
1. We begin by presenting the [base building
blocks](https://github.com/apache/spark/pull/20910/commits/f1b8c08f90b3d3a689e0206d8f8c71c14bb91899)
in [this commit]().
2. Migrate [basic driver
configurations](https://github.com/apache/spark/pull/20910/commits/c3460ae49a3206a4abd1e386caeacca249cb2e4e).
3. Migrate [mounting Kubernetes credentials in the
driver](https://github.com/apache/spark/pull/20910/commits/4036d724268a8e9915f32838bacaff76c0754ada).
4. Migrate [creation of the driver
service](https://github.com/apache/spark/pull/20910/commits/2936aa5d7c98b15dcb37ecf8022aa7313062fb81).
5. [Remove dependency
resolution](https://github.com/apache/spark/pull/20910/commits/d2751b6696522bd775d6d9907574ebbe6aea6434)
as it is subsumed by `BasicDriverFeatureStep`.
6. Migrate [mounting user-provided driver
secrets](https://github.com/apache/spark/pull/20910/commits/430fbb2a7d6559fd6c32a80a77e9d553df419223).
7. Wire steps 2-6 above to [build the entire driver
spec](https://github.com/apache/spark/pull/20910/commits/fd3e8e6eae52af46c9eafe569890ae225a11c596),
composing the incremental steps accordingly. Removes all of the remaining old
driver-construction code that wasn't removed in steps 2-6.
8. Move all of the executor construction to [use the new
architecture](https://github.com/apache/spark/pull/20910/commits/67e9ca165282d05c3b3ada46827bfd0756e5d1c3)
as well.
We can alternatively create multiple pull requests to merge this change
incrementally, but each intermediate pull request would likely be broken in the
K8s functionality. To ensure that master is never in a broken unusable state
for K8s, we unfortunately need to merge the entire change at once.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
