jlpedrosa commented on issue #24962: [SPARK-28149][K8S] Added variable to disable negative DNS caching URL: https://github.com/apache/spark/pull/24962#issuecomment-515381057 @erikerlandson I'd say step 1) should be and standard good practice to keep them in your own repos (not even sure if there are in public "official" repos for the spark images, quick google does not seem to found official ones). Point 5) was clearly miss explained by me, k8s best practices would say that if you change your code, you should upload a new version of the image (instead of loading it from hdfs). The file can be mounted just with the template and pushing the config map. Image 5 should contain only the generated jar with the job, not the file. So an existing image working (made with steps 1) and 5)) can be modified by creating the security file, a map with that file and running spark submit with the right set of options. As you can see on #24702 Initially I proposed to go the approach @vanzin suggested (which I agree with). Taking aside the "changing existing behaviour", personally I think that default behaviour of java of caching forever DNS resolution failures is a bit undesirable choice by default. I don't think that is done by any other programming language by default. I'd suggest to add a new config property to spark so it is actionable for all schedulers, and keep the JVM default behaviour (as it was). But I am open to implement it in any way you guys consider, just let me know the approach.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
