> Sorry, but what are those again? So far all the benefits are already > provided by spark-submit...
1. Retries of init-containers are automatically supported by k8s through pod restart policies. For this point, sorry I'm not sure how spark-submit achieves this. 2. The ability to use credentials that are not shared with the main containers. 3. Not only the user code, but Spark internal code like Executor won't be run if the init-container fails. 4. Easier to build tooling around k8s events/status of the init-container in case of failures as it's doing exactly one thing: downloading dependencies. There could be others that I'm not aware of. On Wed, Jan 10, 2018 at 2:21 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > On Wed, Jan 10, 2018 at 2:16 PM, Yinan Li <liyinan...@gmail.com> wrote: > > but we can not rule out the benefits init-containers bring either. > > Sorry, but what are those again? So far all the benefits are already > provided by spark-submit... > > > Again, I would suggest we look at this more thoroughly post 2.3. > > Actually, one of the reasons why I brought this up is that we should > remove init containers from 2.3 unless they're really required for > something. > > Simplifying the code is not the only issue. The init container support > introduces a whole lot of user-visible behavior - like config options > and the execution of a completely separate container that the user can > customize. If removed later, that could be considered a breaking > change. > > So if we ship 2.3 without init containers and add them later if > needed, it's a much better world than flipping that around. > > -- > Marcelo >
