[
https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15529989#comment-15529989
]
Joep Rottinghuis commented on YARN-4061:
----------------------------------------
I'm going to write up some requirements for what we'd want from a spooling
BufferedMutator and open an HBase jira for the same. We get a type safe
BufferedMutator from a shared connection in
o.a.h.yarn.server.timelineservice.storage.common.BaseTable#getBufferedMutator
In HBase the BufferedMutatorImpl is created up in
HConnectionImplementation#getBufferedMutator, a static class inside
ConnectionManager. It implements ClusterConnection which extends (the
deprecated) HConnection, which in turn extends the Connection interface. While
we can pass BufferedMutatorParams, but have no other way to inject our own
implementation.
This means we'll have to either wrap the return implementation and override and
delegate to the methods, or have some modifications in HBase in order to be
able to create a different implementation.
BufferedMutatorParams has a (setter) method #listener which allows us to pass a
BufferedMutator.ExceptionListener, this gives a glimmer of hope to be able to
capture exceptions and possibly direct the BufferedMutator wrapper class to
(temporarily) spool mutations to a FileSystem implementation. This could be
HDFS, local filesystem, s3, gcs, or whatever the user wants to configure as a
perfix path.
> [Fault tolerance] Fault tolerant writer for timeline v2
> -------------------------------------------------------
>
> Key: YARN-4061
> URL: https://issues.apache.org/jira/browse/YARN-4061
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Li Lu
> Assignee: Li Lu
> Labels: YARN-5355
> Attachments: FaulttolerantwriterforTimelinev2.pdf
>
>
> We need to build a timeline writer that can be resistant to backend storage
> down time and timeline collector failures.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
