[
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16867197#comment-16867197
]
Akira Ajisaka commented on YARN-9016:
-------------------------------------
Hi [~Sushil-K-S] and [~vrushalic], org.json:json is included via
azure-documentdb:1.16.2 and it is JSON licensed. The JSON License is now in
category X, so would you please remove the json library?
https://apache.org/legal/resolved.html#category-x
I filed HADOOP-16381 for tracking this issue.
> DocumentStore as a backend for ATSv2
> ------------------------------------
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: ATSv2
> Reporter: Sushil Ks
> Assignee: Sushil Ks
> Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9016.001.patch, YARN-9016.002.patch,
> YARN-9016.003.patch, YARN-9016.004.patch
>
>
> h1. Document Store for ATSv2
> The Document Store for ATSv2 is a framework for plugging in
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB ,
> MongoDB, ElasticSearch etc.
> * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch,
> MongoDB etc by just adding new configurations properties and writing Document
> Store reader and writer clients.
> * Currently has support for CosmosDB.
> * All writes are Async and buffered, latest document would be flushed to the
> store either if the document buffer gets full or periodically at every flush
> interval in background without adding any additional latency to the running
> jobs..
> * All the REST API's of Timeline Reader Server are supported.
> h3. *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
> <!-- config required for ATSv2 to use DocumentStore-->
> <property>
> <name>yarn.timeline-service.writer.class </name>
>
> <value>org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl</value>
> </property>
> <property>
> <name>yarn.timeline-service.reader.class </name>
> <value>org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl</value>
> </property>
> <property>
> <name>yarn.timeline-service.document-store.db-name</name>
> <value>YOUR_DATABASE_NAME</value> <!-- default is timeline_service -->
> </property>{code}
> h3. *Creating DB and Collections for storing documents*
> The following config needs to be set inside
> *yarn-site.xml* for creating the database and collections for storing
> documents.
> {code:java}
> <!-- Using schema creator class for DocumentStore-->
> <property>
> <name>yarn.timeline-service.schema-creator.class </name>
>
> <value>org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator</value>
> </property>{code}
> Running the schema creator tool to create the necessary
> collections.
> {code:java}
> bin/hadoop
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
> h3. *Azure CosmosDB*
> To use Azure CosmosDB as a DocumentStore for ATSv2, the additional
> properties under *yarn-site.xml* is required..
> {code:java}
> <!-- config required for using Azure CosmosDB as a DocumentStore for ATSv2 -->
> <property>
> <name>yarn.timeline-service.document-store-type</name>
> <value>COSMOS_DB</value>
> </property>
> <property>
> <name>yarn.timeline-service.document-store.cosmos-db.endpoint</name>
> <value>http://YOUR_AZURE_COSMOS_DB_URL:443/</value>
> </property>
> <property>
> <name>yarn.timeline-service.document-store.cosmos-db.masterkey</name>
> <value>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL</value>
> </property>
> {code}
>
> *Testing locally*
> In order to test the Azure CosmosDB as a DocumentStore
> locally, install the emulator from
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and
> start it locally. Set the endpoint and master key under *yarn-site.xml* as
> mentioned above and run any example job like DistributedShell etc. Later you
> can check the data explorer UI of Azure CosmosDB locally to query the
> documents or even launch the *TimelineReader* locally to fetch/query the data
> from REST API's.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
