ueshin opened a new pull request, #42328: URL: https://github.com/apache/spark/pull/42328
### What changes were proposed in this pull request? Adds memory limits for Python UDTF analyzer. - `spark.sql.analyzer.pythonUDTF.analyzeInPython.memory` (`None` by default) > The amount of memory to be allocated to PySpark for Python UDTF analyzer, in MiB unless otherwise specified. If set, PySpark memory for Python UDTF analyzer will be limited to this amount. If not set, Spark will not limit Python's memory use and it is up to the application to avoid exceeding the overhead memory space shared with other non-JVM processes.\nNote: Windows does not support resource limiting and actual resource is not limited on MacOS. ### Why are the changes needed? Python UDTF analyzer should be able to set a memory limit. ### Does this PR introduce _any_ user-facing change? Users will be able to set the memory limit for Python UDTF analyzer. ### How was this patch tested? Existing tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
