[
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushil Ks updated YARN-9016:
----------------------------
Description:
h1. Document Store for ATSv2
The Document Store for ATSv2 is a framework for plugging in any
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB,
ElasticSearch etc.
* Supports multiple Document Store Vendors like CosmosDB, ElasticSearch,
MongoDB etc by just adding new configurations properties and writing Document
Store reader and writer clients.
* Currently has support for CosmosDB.
* All writes are Async and buffered, latest document would be flushed to the
store either if the document buffer gets full or periodically at every flush
interval in background without adding any additional latency to the running
jobs..
* All the REST API's of Timeline Reader Server are supported.
h4.
*How to enable?*
Add the flowing properties under *yarn-site.xml*
{code:java}
<!-- config required for ATSv2 to use DocumentStore-->
<property>
<name>yarn.timeline-service.writer.class</name>
<value>org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl</value>
</property>
<property>
<name>yarn.timeline-service.reader.class</name>
<value>org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl</value>
</property>
<property>
<name>yarn.timeline-service.documentstore.db-name</name>
<value>YOUR_DATABASE_NAME</value> <!-- default is timeline_service -->
</property>{code}
h3. *Creating DB and Collections for storing documents*
This is similar to HBase *TimelineSchemaCreator* the
following command needs to be executed once for setting up the database and
collections for storing documents.
{code:java}
hadoop
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
h3. *Azure CosmosDB*
To use Azure CosmosDB as a DocumentStore for ATSv2, the additional
properties under *yarn-site.xml* is required..
{code:java}
<!-- config required for using Azure CosmosDB as a DocumentStore for ATSv2 -->
<property>
<name>yarn.timeline-service.store-type</name>
<value>COSMOS_DB</value>
</property>
<property>
<name>yarn.timeline-service.cosmos-db.endpoint</name>
<value>http://YOUR_AZURE_COSMOS_DB_URL:443/</value>
</property>
<property>
<name>yarn.timeline-service.cosmos-db.masterkey</name>
<value>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL</value>
</property>
{code}
*Testing locally*
In order to test the Azure CosmosDB as a DocumentStore locally,
install the emulator from
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and
start it locally. Set the endpoint and master key under *yarn-site.xml* as
mentioned above and run any example job like DistributedShell etc. Later you
can check the data explorer UI of Azure CosmosDB locally to query the documents
or even launch the *TimelineReader* locally and fetch/query the data from REST
API's.
was:
h1. Document Store for ATSv2
The Document Store for ATSv2 is a framework for plugging in any
Document Store as a backend for ATSv2 i.e Azure CosmosDB , MongoDB,
ElasticSearch etc.
* Supports multiple Document Store Vendors like CosmosDB, ElasticSearch,
MongoDB etc by just adding new configurations properties and writing Document
Store reader and writer clients.
* Currently has support for CosmosDB.
* All writes are Async and buffered, latest document would be flushed to the
store either if the document buffer gets full or periodically at every flush
interval in background without adding any additional latency to the running
jobs..
* All the REST API's of Timeline Reader Server are supported.
h4.
*How to enable?*
Add the flowing properties under *yarn-site.xml*
{code:java}
<!-- config required for ATSv2 to use DocumentStore-->
<property>
<name>yarn.timeline-service.writer.class</name>
<value>org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl</value>
</property>
<property>
<name>yarn.timeline-service.reader.class</name>
<value>org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl</value>
</property>
<property>
<name>yarn.timeline-service.documentstore.db-name</name>
<value>YOUR_DATABASE_NAME</value> <!-- default is timeline_service -->
</property>{code}
h3. *Creating DB and Collections for storing documents*
This is similar to HBase *TimelineSchemaCreator* the
following command needs to be executed once for setting up the database and
collections for storing documents.
{code:java}
hadoop
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
h3. *Azure CosmosDB*
To use Azure CosmosDB as a DocumentStore for ATSv2, the additional
properties under *yarn-site.xml* is required..
{code:java}
<!-- config required for using Azure CosmosDB as a DocumentStore for ATSv2 -->
<property>
<name>yarn.timeline-service.store-type</name>
<value>COSMOS_DB</value>
</property>
<property>
<name>yarn.timeline-service.cosmos-db.endpoint</name>
<value>http://YOUR_AZURE_COSMOS_DB_URL:443/</value>
</property>
<property>
<name>yarn.timeline-service.cosmos-db.masterkey</name>
<value>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL</value>
</property>
{code}
*Testing locally*
In order to test the Azure CosmosDB as a DocumentStore locally,
install the emulator from
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and
start it locally. Set the endpoint and master key under *yarn-site.xml* as
mentioned above and run any example job like DistributedShell etc. Later you
can check the data explorer UI of Azure CosmosDB locally to query the documents
or even launch the *TimelineReader* locally and fetch/query the data from REST
API's.
> DocumentStore as a backend for ATSv2
> ------------------------------------
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: ATSv2
> Reporter: Sushil Ks
> Assignee: Sushil Ks
> Priority: Major
> Attachments: YARN-9016.001.patch
>
>
> h1. Document Store for ATSv2
> The Document Store for ATSv2 is a framework for plugging in
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB ,
> MongoDB, ElasticSearch etc.
> * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch,
> MongoDB etc by just adding new configurations properties and writing Document
> Store reader and writer clients.
> * Currently has support for CosmosDB.
> * All writes are Async and buffered, latest document would be flushed to the
> store either if the document buffer gets full or periodically at every flush
> interval in background without adding any additional latency to the running
> jobs..
> * All the REST API's of Timeline Reader Server are supported.
> h4.
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
> <!-- config required for ATSv2 to use DocumentStore-->
> <property>
> <name>yarn.timeline-service.writer.class</name>
>
> <value>org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl</value>
> </property>
> <property>
> <name>yarn.timeline-service.reader.class</name>
> <value>org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl</value>
> </property>
> <property>
> <name>yarn.timeline-service.documentstore.db-name</name>
> <value>YOUR_DATABASE_NAME</value> <!-- default is timeline_service -->
> </property>{code}
> h3. *Creating DB and Collections for storing documents*
> This is similar to HBase *TimelineSchemaCreator* the
> following command needs to be executed once for setting up the database and
> collections for storing documents.
> {code:java}
> hadoop
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3. *Azure CosmosDB*
> To use Azure CosmosDB as a DocumentStore for ATSv2, the additional
> properties under *yarn-site.xml* is required..
> {code:java}
> <!-- config required for using Azure CosmosDB as a DocumentStore for ATSv2 -->
> <property>
> <name>yarn.timeline-service.store-type</name>
> <value>COSMOS_DB</value>
> </property>
> <property>
> <name>yarn.timeline-service.cosmos-db.endpoint</name>
> <value>http://YOUR_AZURE_COSMOS_DB_URL:443/</value>
> </property>
> <property>
> <name>yarn.timeline-service.cosmos-db.masterkey</name>
> <value>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL</value>
> </property>
> {code}
>
> *Testing locally*
> In order to test the Azure CosmosDB as a DocumentStore
> locally, install the emulator from
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and
> start it locally. Set the endpoint and master key under *yarn-site.xml* as
> mentioned above and run any example job like DistributedShell etc. Later you
> can check the data explorer UI of Azure CosmosDB locally to query the
> documents or even launch the *TimelineReader* locally and fetch/query the
> data from REST API's.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
