[
https://issues.apache.org/jira/browse/YARN-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935568#comment-13935568
]
Zhijie Shen commented on YARN-1717:
-----------------------------------
Billie, thanks for your metrics. I've don some simple calculation myself. In
long term, if a cluster has x entities written per second, no matter how long
the ttl is, the number of entities to delete per second should be x on average.
Therefore, let's say throughput of put requests is 100 entities/sec, the number
of entities to delete per second will be 100 as well. Given we do the deletion
every 5 minutes, we have 30,000 entities to delete per round. According to your
measurement, it will take less than 8 sec to complete the deletion. The
deletion will delay put request, but every 5 mins, it just happens for 8 secs,
i.e., 2.67%. It sounds good to me.
+1 for the patch. Will commit it.
> Enable offline deletion of entries in leveldb timeline store
> ------------------------------------------------------------
>
> Key: YARN-1717
> URL: https://issues.apache.org/jira/browse/YARN-1717
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Billie Rinaldi
> Assignee: Billie Rinaldi
> Attachments: YARN-1717.1.patch, YARN-1717.10.patch,
> YARN-1717.11.patch, YARN-1717.2.patch, YARN-1717.3.patch, YARN-1717.4.patch,
> YARN-1717.5.patch, YARN-1717.6-extra.patch, YARN-1717.6.patch,
> YARN-1717.7.patch, YARN-1717.8.patch, YARN-1717.9.patch
>
>
> The leveldb timeline store implementation needs the following:
> * better documentation of its internal structures
> * internal changes to enable deleting entities
> ** never overwrite existing primary filter entries
> ** add hidden reverse pointers to related entities
--
This message was sent by Atlassian JIRA
(v6.2#6252)
