[
https://issues.apache.org/jira/browse/TEZ-3915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446517#comment-16446517
]
Harish Jaiprakash commented on TEZ-3915:
----------------------------------------
[~jeagles], sorry for committing it early. I did look at TEZ-3914 and
incorporate the CodedInputStream/CodedOutputStream reuse.
Some more context on this patch:
We found that ATSv15 does not scale on read, and way to go forward is to use
ATSv2. ATSv2 still needs some design changes before we can get it integrated
fully. For lot of debugging we rely on this data. And decided to create a
format which logs to hdfs, we can download this file or mount it as a hive
table and analyze the data in it. Adding the reader into it will also allow
people to use it independently.
I made a mistake by not sharing the design details with the community to start
with. I'll share details on bigger patches earlier so that a discussion can be
had before implementing something like that. Sorry about that.
> Create protobuf based history event logger.
> -------------------------------------------
>
> Key: TEZ-3915
> URL: https://issues.apache.org/jira/browse/TEZ-3915
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Harish Jaiprakash
> Assignee: Harish Jaiprakash
> Priority: Major
> Fix For: 0.9.next
>
> Attachments: TEZ-3915.01.patch, TEZ-3915.02.patch, TEZ-3915.03.patch,
> TEZ-3915.04.patch, TEZ-3915.05.patch, TEZ-3915.06.patch, TEZ-3915.07.patch
>
>
> A protobuf based history event logger, to log directly into hdfs. Implement a
> reader api also, to get the events from the files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
