Does multiprocessing do a fork, or an exec of a whole new python process? If it's exec then the memory consumption is quite drasticly higher, and this should default to 1 or 2. If it's a fork and we can take advantage of COW to reduce memory then I guess this is okay.
Still probably worth a note in UPDATING.md about this new setting and how people should tweak it based on their scheduler node size. [ Full content available at: https://github.com/apache/incubator-airflow/pull/3830 ] This message was relayed via gitbox.apache.org for [email protected]
