Hello Yunqi,
If you are looking at just HDFS access patterns, you can probably just look at
the HDFS audit logs to see who is reading/writing from/to HDFS.
If you are looking for code, there are a couple of places where Tez will access
HDFS:
- In the tasks, MRInput and MROutput are used to read and write from/to
HDFS. These are usually the tasks in the root or leaf vertices. Intermediate
vertices do not usually access HDFS unless there is a join happening across
different datasets. You can look at the .dot file generated for a given DAG to
see which vertices have additional inputs/outputs attached.
- The split generation layer ( MRInputAMSplit* or the various InputFormat
classes in Hadoop ) also read from HDFS to find the locations of the dataset
being processed.
Hope that helps.
thanks
— HItesh
On Jul 29, 2015, at 6:55 PM, Yunqi Zhang <[email protected]> wrote:
> Hi all,
>
> I'm trying to study the data accesses generated by Tez. In particular, I
> would like to record all the data accesses (i.e., in terms of HDFS blocks)
> generated by Tez. Which folder should I look into and where should I start
> with?
>
> I have a setup on Tez + Hive, and the databases in Hive are stored on HDFS.
>
> Thank you!
>
> Yunqi
