I would like to discuss if we can introduce a change here - maybe it’s also possible to get that into 2.4.4?
I’m keen to contribute that change. Please let me know your thoughts on this. King Regards Roland Johann Software Developer/Data Engineer phenetic GmbH Lütticher Straße 10, 50674 Köln, Germany Mobil: +49 172 365 26 46 Mail: roland.joh...@phenetic.io Web: phenetic.io Handelsregister: Amtsgericht Köln (HRB 92595) Geschäftsführer: Roland Johann, Uwe Reimann > Am 13.08.2019 um 21:15 schrieb Roland Johann <roland.joh...@phenetic.io>: > > Hi all, > > K8S resource manager dumps the config map to > /opt/spark/conf/spark-defaults.conf and passes it to spark submit twice: > • via env var SPARK_CONF_DIR=/opt/spark/conf/ > • argument --properties-file /opt/spark/conf/spark-defaults.conf > This prevents definition of user defined properties, such as from secrets - > currently it seems to be the only possibility to define spark config > properties from k8s secrets. > > Is the current implementation intent? If yes, what are the reasons behind > that decision? > > Thanks and kind regards > > Roland Johann > Software Developer/Data Engineer > > phenetic GmbH > Lütticher Straße 10, 50674 Köln, Germany > > Mobil: +49 172 365 26 46 > Mail: roland.joh...@phenetic.io <mailto:roland.joh...@phenetic.io> > Web: phenetic.io <http://phenetic.io/> > > Handelsregister: Amtsgericht Köln (HRB 92595) > Geschäftsführer: Roland Johann, Uwe Reimann > > >
