The container size is determined at compile time when creating the DAG. There is black box user code running inside a task and so Tez cannot determine a good size since it does not know whats running.
When tasks actually run in a container then Tez has APIs by which it tries to divide a tasks memory resources among inputs/outputs and processor. At that point inputs may use data size to make an ask for a % of the memory. *From:* Grandl Robert [mailto:[email protected]] *Sent:* Tuesday, September 09, 2014 6:38 PM *To:* [email protected] *Subject:* container size Hi guys, It seems the size of a container allocated to a task in Tez is static ? Please correct me if I am wrong. Is there a way such that Tez is dynamically choosing the task container size based on various things ? (like a map or reduce vertex, size of data to process, etc.) At least for the tpcds/h queries I am running it looks containers are fixed size. Thanks, Robert -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
