Thanks for this discussion everyone. It has been very useful in getting an overall understanding here. I think in general, consensus is that this change doesn't introduce behavioral changes, and it's definitely an advantage to reuse the constructs that Spark provides to us.
Moving on to a different question here - of pushing this through to Spark. The init containers have been tested over the past two Spark releases by external users and integration testing - and this would be a fundamental change to that behavior. We should work on getting enough test coverage and confidence here. We can start by getting a PR going perhaps, and start augmenting the integration testing to ensure that there are no surprises - with/without credentials, accessing GCS, S3 etc as well. When we get enough confidence and test coverage, let's merge this in. Does that sound like a reasonable path forward? On Wed, Jan 10, 2018 at 2:53 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > On Wed, Jan 10, 2018 at 2:51 PM, Matt Cheah <mch...@palantir.com> wrote: > > those sidecars may perform side effects that are undesirable if the main > Spark application failed because dependencies weren’t available > > If the contract is that the Spark driver pod does not have an init > container, and the driver handles its own dependencies, then by > definition that situation cannot exist. > > -- > Marcelo > -- Anirudh Ramanathan
