HyukjinKwon edited a comment on pull request #31151: URL: https://github.com/apache/spark/pull/31151#issuecomment-759129842
> I'm against make this change unless we decide in general to change our naming rules for configs and think it should be decided on by the devs and documented appropriately This i agree. PySpark has the same issue too and I have been tracking this issue: SQL: - `spark.sql.pyspark.jvmStacktrace.enabled` - `spark.sql.execution.arrow.pyspark.*` - `spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled` CORE - `spark.executor.pyspark.memory` - `spark.pyspark.driver.python` - `spark.pyspark.python` - `spark.kubernetes.pyspark.pythonVersion` - `spark.python.daemon.module` - `spark.python.worker.*` - `spark.python.use.daemon` - `spark.python.task.killTimeout` - `spark.python.authenticate.socketTimeout` - `spark.python.profile.*` We should also merge both configurations `spark.executor.pyspark.memory` and `spark.python.worker.memory` (SPARK-26679). It would be great if we have have a general naming to keep. Then we can deprecate and change the name. The current naming is kind of confusing. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
