PerilousApricot commented on pull request #34903: URL: https://github.com/apache/spark/pull/34903#issuecomment-994196438
@HyukjinKwon dunno if this is the right person to ask, but since you're the pyspark guy, are you amenable to having a `spark-env.py` that gets loaded from the pyspark context before the java gateway gets started? PySpark is at a disadvantage compared with other languages because the config hooks happen so late in the process. It would be nice to set configs in java_gateway.py before the JVM is started, compared with now where pyspark initialization happens very late -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
