Thanks for reaching out to use XuanCao. Sounds like you are planning to use new custom statistics that a new custom Vertex can utilize to make better decisions. Are you planning to contribute these custom statistics changes to tez? Or are are we trying to solidify and enhance the Tez Statistics API so that applications (pig/hive/scope/etc) can provide their own custom implementations? This sounds like a great addition if I understand correctly. There are some request for similar (TEZ-1167, TEZ-764) so we can try to find a way to implement this that the whole community can benefit.
Regards, jeagles On Tue, Mar 6, 2018 at 4:22 PM, Xuan Cao <cao...@yahoo.com.invalid> wrote: > > Hi, > > > > Thisis Xuan with Microsoft. We are looking at the Tez statistics model > trying toextend it to support our custom workloads. Here are some > observations of thefew statistics objects in the current codebase: > > 1. InputStatistics,OutputStatictics and VertexStatiscs are interfaces > in tez-api. > > 2. TaskStatistics is a classin tez-runtime-internals. > > 3. VertexStatiscsImpl is aninner class of VertexImpl implementing > VertexStatistics in tez-dag. > > 4. IOStatistics is in a classin tez-runtime-internals and > IOStaticsticsImpl is a static inner class ofVertexImpl extending > IOStatistics but implementing InputStatistics andOutputStatistics. > > > > Ourintention is to extend VertexStatisticsImpl so that we can aggregate > the custompayload in TaskStatistics. But the object model here seems to be > ratherconfusing and not consistent. We are planning to do some work in this > area toimprove it, but not sure whether there is any work going on right > now and thestatus of it? > > > > Regards, > > XuanCao > > > >
