[
https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752682#comment-15752682
]
Li Lu commented on YARN-4061:
-----------------------------
I went through the new design doc in HBASE-17018 and I think it's mostly good.
As we discussed in the weekly sync meeting, one thing we may want to sort out
here is how to handle the case when the collectors started up and the HBase
cluster was down. From my point of view, the most conservative approach is to
assume the HBase cluster was always BAD upon start up. However, the problem is
we have to spool the very first writes anyways. Can we have a "PROBING" state
in the coordinator, where we may tolerate slightly longer submission time, to
let the spooling mutator firstly probe the state of the HBase cluster? Also,
this probing process may happen before the first write ever comes, so that we
can do out-of-band probing?
Another my question is on the idempotent write requirements. Moving my comments
from google doc to here:
bq. The spooling mutator itself guarantees an "at least once" semantic? One
thing I'd like to discuss here is about the write timestamp of each timeline
writes. I'm not familiar with the HBase code, but are we generating one unique
timestamp for each write when we actually write them to HBase? If this is the
case, replaying timeline writes may generate different timestamp and those
repeated writes may not be idempotent in timeline's perspective?
> [Fault tolerance] Fault tolerant writer for timeline v2
> -------------------------------------------------------
>
> Key: YARN-4061
> URL: https://issues.apache.org/jira/browse/YARN-4061
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Li Lu
> Assignee: Joep Rottinghuis
> Labels: YARN-5355
> Attachments: FaulttolerantwriterforTimelinev2.pdf
>
>
> We need to build a timeline writer that can be resistant to backend storage
> down time and timeline collector failures.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
