does anyone know why all spark settings end up being system properties, and where this is done?
for example when i pass "--conf spark.foo=bar" into spark-submit then System.getProperty("spark.foo") will be equal to "bar" i grepped the spark codebase for System.setProperty or System.setProperties and i see it being used in some places but never for all spark settings. we are running into some weird side effects because of this since we use typesafe config which has system properties as overrides so we see them pop up there again unexpectedly.