Hi Fabio The behavior that Tez induces on a cluster for a MapReduce-like job may be vastly different to what MapReduce does today: - Tez can do splits calculation on the cluster and makes use of information such as available cluster resources to decide how many tasks to run - Tez does container reuse across tasks. There are multiple factors that can affect reuse. The kind of workload, where tasks run, how long the task ran, how much common locality do the different tasks have. At this point, depending on the locality-scheduling related flags, Tez waits for a certain delay before assigning a task to a rack-local container or an off-rack container. This is assuming that the RM has not provided a matching container within the timeframe and even then, a newly allocated container may not be used as there is a penalty for launching a new JVM. - Tez also keeps containers around (configurable) so as to use them later for tasks that have not yet been scheduled or future DAGs to be run within the same Application Master. - Lastly, Tez in case of Hive/Pig, has vastly complex and dynamic DAGs. Unlike MapReduce, the resource needs are not known completely upfront. Also, resource asks may also change depending on how much data is actually being processed.
Had a couple of questions to ask: - When you say run a trace of a map-reduce job against SLS+Tez, could you clarify what that means? - I have not yet had a chance to look at SLS or the MR AM simulator in detail. What does writing a Tez-specific simulator entail? Let us know if you are planning to look at implementing some of the missing simulation pieces for Tez. thanks — Hitesh On Nov 14, 2014, at 5:46 PM, Fabio <[email protected]> wrote: > Thanks for the reply, actually what I was planning to do is to generate > artificial traces of map-reduce jobs and run them against SLS and SLS+Tez to > analyze the differences. > I asked here directly since I am pretty sure that in the Hadoop mailing list > they were going to tell me to ask you about it since that class is > app-specific, so I thought someone here may have already written an AM > simulator class for Tez. I doubt the SLS developers will do it any soon > (unless Tez will become part of the Hadoop code). > Anyway, no problem. Some tests in a small cluster will do the job ;) > > Thanks again > > Fabio > > On 11/14/2014 07:54 PM, Hitesh Shah wrote: >> Hello Fabio >> >> We do not have a job trace file generated by Tez and therefore no simulator >> that can re-run the trace. We do store some historical data for the job but >> the level of tooling around it is pretty minimal. >> >> — Hitesh >> >> On Nov 14, 2014, at 3:29 AM, Fabio <[email protected]> wrote: >> >>> With SLS (Yarn Scheduler Load Simulator) I can test a MR job trace against >>> different schedulers, but to do so I see one has to specify >>> "yarn.sls.am.type.mapreduce" that is "The AMSimulator implementation for >>> MapReduce-like applications. Users can specify implementations for other >>> type of applications.". As far as I understand this class is a simulator of >>> the AM, so I suppose that if I want to execute a job trace as if it is run >>> on top of Tez, I should implement this class in order to simulate a Tez AM. >>> Is this correct? A of today is there already some implementation of this? >>> >>> Thanks in advance >>> >>> Fabio >> >
