[
https://issues.apache.org/jira/browse/YARN-5814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656598#comment-15656598
]
Bingxue Qiu commented on YARN-5814:
-----------------------------------
I have uploaded the design. It contains our ideas about druid writer, reader
and schema. Please feel free to give your suggestions. Thanks
> Add druid as storage backend in YARN Timeline Service
> ------------------------------------------------------
>
> Key: YARN-5814
> URL: https://issues.apache.org/jira/browse/YARN-5814
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: ATSv2
> Affects Versions: 3.0.0-alpha2
> Reporter: Bingxue Qiu
> Attachments: Add-Druid-in-YARN-Timeline-Service.pdf
>
>
> h3. Introduction
> I propose to add druid as storage backend in YARN Timeline Service.
> We run more than 6000 applications and generate 450 million metrics daily in
> Alibaba Clusters with thousands of nodes. We need to collect and store
> meta/events/metrics data, online analyze the utilization reports of various
> dimensions and display the trends of allocation/usage resources for cluster
> by joining and aggregating data. It helps us to manage and optimize the
> cluster by tracking resource utilization.
> To achieve our goal we have changed to use druid as the storage instead of
> HBase and have achieved sub-second OLAP performance in our production
> environment for few months.
> h3. Analysis
> Currently YARN Timeline Service only supports aggregating metrics at a) flow
> level by FlowRunCoprocessor and b) application level metrics aggregating by
> AppLevelTimelineCollector, offline (time-based periodic) aggregation for
> flows/users/queues for reporting and analysis is planned but not yet
> implemented. YARN Timeline Service chooses Apache HBase as the primary
> storage backend. As we all know that HBase doesn't fit for OLAP.
> For arbitrary exploration of data,such as online analyze the utilization
> reports of various dimensions(Queue,Flow,Users,Application,CPU,Memory) by
> joining and aggregating data, Druid's custom column format enables ad-hoc
> queries without pre-computation. The format also enables fast scans on
> columns, which is important for good aggregation performance.
> To achieve our goal that support to online analyze the utilization reports of
> various dimensions, display the variation trends of allocation/usage
> resources for cluster, and arbitrary exploration of data, we propose to add
> druid storage and implement DruidWriter /DruidReader in YARN Timeline Service.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
