[jira] [Comment Edited] (TEZ-4098) tez-tools improvements: log-split, swimlane

[Jira]

    [ 
https://issues.apache.org/jira/browse/TEZ-4098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991421#comment-16991421
 ] 

László Bodor edited comment on TEZ-4098 at 12/9/19 9:47 AM:
------------------------------------------------------------

thanks [~rajesh.balamohan] for the idea, I just simply appended hostname to 
container folder's name, like this:
{code}
├── container_e10_1575565459633_0004_01_000001_vc0525.halxg.cloudera.com_8041
│   ├── container-localizer-syslog
│   ├── dag_1575565459633_0004_1-tez-dag.pb.txt
│   ├── dag_1575565459633_0004_1.dot
│   ├── prelaunch.err
│   ├── prelaunch.out
│   ├── stderr
│   ├── stdout
│   ├── syslog
│   ├── syslog_dag_1575565459633_0004_1
│   └── syslog_dag_1575565459633_0004_1_post
├── container_e10_1575565459633_0004_01_000002_vc0528.halxg.cloudera.com_8041
│   ├── container-localizer-syslog
│   ├── prelaunch.err
│   ├── prelaunch.out
│   ├── stderr
│   ├── stdout
│   ├── syslog
│   └── syslog_attempt_1575565459633_0004_1_00_000000_0
├── container_e10_1575565459633_0004_01_000003_vc0536.halxg.cloudera.com_8041
│   ├── container-localizer-syslog
│   ├── prelaunch.err
│   ├── prelaunch.out
│   ├── stderr
│   ├── stdout
│   ├── syslog
│   └── syslog_attempt_1575565459633_0004_1_00_000000_1
├── container_e10_1575565459633_0004_01_000004_vc0526.halxg.cloudera.com_8041
│   ├── container-localizer-syslog
│   ├── prelaunch.err
│   ├── prelaunch.out
│   ├── stderr
│   ├── stdout
│   ├── syslog
│   └── syslog_attempt_1575565459633_0004_1_00_000000_2
└── container_e10_1575565459633_0004_01_000005_vc0529.halxg.cloudera.com_8041
    ├── container-localizer-syslog
    ├── prelaunch.err
    ├── prelaunch.out
    ├── stderr
    ├── stdout
    ├── syslog
    └── syslog_attempt_1575565459633_0004_1_00_000000_3
{code}

(maybe you meant to introduce a top level folder stucture for machine names, 
that could work as well, not sure which is better)


was (Author: abstractdog):
thanks [~rajesh.balamohan] for the idea, I just simply appended hostname to 
container folder's name, like this:
{code}
├── container_e10_1575565459633_0004_01_000001_vc0525.halxg.cloudera.com_8041
│   ├── container-localizer-syslog
│   ├── dag_1575565459633_0004_1-tez-dag.pb.txt
│   ├── dag_1575565459633_0004_1.dot
│   ├── prelaunch.err
│   ├── prelaunch.out
│   ├── stderr
│   ├── stdout
│   ├── syslog
│   ├── syslog_dag_1575565459633_0004_1
│   └── syslog_dag_1575565459633_0004_1_post
├── container_e10_1575565459633_0004_01_000002_vc0528.halxg.cloudera.com_8041
│   ├── container-localizer-syslog
│   ├── prelaunch.err
│   ├── prelaunch.out
│   ├── stderr
│   ├── stdout
│   ├── syslog
│   └── syslog_attempt_1575565459633_0004_1_00_000000_0
├── container_e10_1575565459633_0004_01_000003_vc0536.halxg.cloudera.com_8041
│   ├── container-localizer-syslog
│   ├── prelaunch.err
│   ├── prelaunch.out
│   ├── stderr
│   ├── stdout
│   ├── syslog
│   └── syslog_attempt_1575565459633_0004_1_00_000000_1
├── container_e10_1575565459633_0004_01_000004_vc0526.halxg.cloudera.com_8041
│   ├── container-localizer-syslog
│   ├── prelaunch.err
│   ├── prelaunch.out
│   ├── stderr
│   ├── stdout
│   ├── syslog
│   └── syslog_attempt_1575565459633_0004_1_00_000000_2
└── container_e10_1575565459633_0004_01_000005_vc0529.halxg.cloudera.com_8041
    ├── container-localizer-syslog
    ├── prelaunch.err
    ├── prelaunch.out
    ├── stderr
    ├── stdout
    ├── syslog
    └── syslog_attempt_1575565459633_0004_1_00_000000_3
{code}

> tez-tools improvements: log-split, swimlane
> -------------------------------------------
>
>                 Key: TEZ-4098
>                 URL: https://issues.apache.org/jira/browse/TEZ-4098
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: László Bodor
>            Assignee: László Bodor
>            Priority: Major
>         Attachments: TEZ-4098.01.patch, TEZ-4098.02.patch
>
>
> While using tez-tools for analyzing application logs, I'm about to improve 
> them a little bit. Details will be added here to the description.
> 1. Support swimlane.sh to consume local file
> 2. Create a log splitter, which is able to split the aggregated log file into 
> separate container directories, like below:
> {code}
> ├── container_e02_1572948601374_0004_01_000001
> │   ├── container-localizer-syslog
> │   ├── dag_1572948601374_0004_1.dot
> │   ├── prelaunch.err
> │   ├── prelaunch.out
> │   ├── stderr
> │   ├── stdout
> │   ├── syslog
> │   ├── syslog_dag_1572948601374_0004_1
> │   └── syslog_dag_1572948601374_0004_1_post
> ├── container_e02_1572948601374_0004_01_000002
> │   ├── prelaunch.err
> │   ├── prelaunch.out
> │   ├── stderr
> │   ├── stdout
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)