[
https://issues.apache.org/jira/browse/HDFS-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-5092:
------------------------------
Description:
The initial {{cacheReport}} patch at HDFS-5051 does frequent full reports of DN
cache state. Better would be a scheme similar to how block reports are
currently done: send incremental cache reports on every heartbeat (seconds),
and full reports on a longer time scale (minutes to hours). This should reduce
network traffic and allow us to make incremental reports even faster.
As per discussion on HDFS-5051, we should also roll-up the following review
comments:
- Remove gen stamp and length from {{cacheReport}}, unnecessary until we do
auto-caching of appended data
- Only jitter full cache reports, similar to how full block reports are jittered
- On DN startup, skip all cache reports until the cache is populated. The NN
can just assume the DN cache is empty in the meantime.
was:We should send incremental cache reports as part of DN heartbeats,
similar to how we do incremental block reports. Then we would only need to
send full cache reports rarely (again similar to full block reports).
Assignee: Andrew Wang
Summary: Add support for incremental cache reports (was: piggyback
incremental cache reports on DN heartbeats)
> Add support for incremental cache reports
> -----------------------------------------
>
> Key: HDFS-5092
> URL: https://issues.apache.org/jira/browse/HDFS-5092
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
> Reporter: Colin Patrick McCabe
> Assignee: Andrew Wang
> Priority: Minor
>
> The initial {{cacheReport}} patch at HDFS-5051 does frequent full reports of
> DN cache state. Better would be a scheme similar to how block reports are
> currently done: send incremental cache reports on every heartbeat (seconds),
> and full reports on a longer time scale (minutes to hours). This should
> reduce network traffic and allow us to make incremental reports even faster.
> As per discussion on HDFS-5051, we should also roll-up the following review
> comments:
> - Remove gen stamp and length from {{cacheReport}}, unnecessary until we do
> auto-caching of appended data
> - Only jitter full cache reports, similar to how full block reports are
> jittered
> - On DN startup, skip all cache reports until the cache is populated. The NN
> can just assume the DN cache is empty in the meantime.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
