durgaswaroop commented on issue #5617: [SPARK-7037] [CORE] Inconsistent behavior for non-spark config properties in spark-shell and spark-submit URL: https://github.com/apache/spark/pull/5617#issuecomment-571147738 I'm not entirely sure if this is related to this PR, but I am getting warnings with non-spark configuration. I'm using pyspark 2.4.3 with spark 2.4.3 and I'm trying to add configuration for a file system scheme. I'm adding the configuration as ``` SparkSession.builder \ .appName(..) \ .master(..) \ .config('fs.oci.client.hostname', '..') \ .config('fs.oci.client.auth.pemfilepath'. '..') \ .. ``` And with those, I get warnings as follows: ``` Warning: Ignoring non-spark config property: namespace=.... Warning: Ignoring non-spark config property: fs.oci.client.hostname=.... Warning: Ignoring non-spark config property: fs.oci.client.auth.fingerprint=.... Warning: Ignoring non-spark config property: fs.oci.client.auth.tenantId=.... Warning: Ignoring non-spark config property: fs.oci.client.auth.pemfilepath=.... ``` How to set non-spark config in pyspark?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
