[jira] [Updated] (YARN-10896) RM fail over is not reporting the nodes DECOMMISSIONED

[Sushil Ks (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-10896:
-
Description: 
Whenever we add the host entries into the exclude file in order to DECOMMISSION 
the Nodemanager, we would issue the *yarn rmadmin -refreshNodes* command to 
transition the nodes from RUNNING to DECOMMISSIONED state. However if the fail 
over to standby resource manager happens and the exclude file has the list of 
hosts to be disallowed, then these disallowed nodes are never seen through the 
Cluster Metrics on the new active resource manager. 

Whatever host entries that are present in the exclude files are being listed in 
the Cluster Metrics whenever resource manager is restarted, i.e as part of the 
service init of *NodeListManager* , however during fail over this info is lost. 
Hence this patch tries to set the  *DECOMMISSIONED* nodes inside the RM Context 
so that its available through Cluster Metrics whenever we issue the *yarn 
rmadmin -refreshNodes* command.

  was:
Whenever we add the host entries into the exclude file in order to DECOMMISSION 
the Nodemanager, we would issue the *yarn rmadmin -refreshNodes* command to 
transition the nodes from RUNNING to DECOMMISSIONED state. However if the fail 
over to standby resource manager happens and the exclude file has the list of 
hosts to be disallowed, then these disallowed nodes are never seen through the 
Cluster Metrics on the new active resource manager. 

Whatever host entries that are present in the exclude files are being listed in 
the Cluster Metrics whenever resource manager is restarted, i.e as part of the 
service init of *NodeListManager* , however during fail over this info is lost. 
Hence this patch tries to set the  * *DECOMMISSIONED* nodes inside the RM 
Context so that its available through Cluster Metrics whenever we issue the 
**yarn rmadmin -refreshNodes* command.


> RM fail over is not reporting the nodes DECOMMISSIONED 
> ---
>
> Key: YARN-10896
> URL: https://issues.apache.org/jira/browse/YARN-10896
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-10896.001.patch
>
>
> Whenever we add the host entries into the exclude file in order to 
> DECOMMISSION the Nodemanager, we would issue the *yarn rmadmin -refreshNodes* 
> command to transition the nodes from RUNNING to DECOMMISSIONED state. However 
> if the fail over to standby resource manager happens and the exclude file has 
> the list of hosts to be disallowed, then these disallowed nodes are never 
> seen through the Cluster Metrics on the new active resource manager. 
> Whatever host entries that are present in the exclude files are being listed 
> in the Cluster Metrics whenever resource manager is restarted, i.e as part of 
> the service init of *NodeListManager* , however during fail over this info is 
> lost. Hence this patch tries to set the  *DECOMMISSIONED* nodes inside the RM 
> Context so that its available through Cluster Metrics whenever we issue the 
> *yarn rmadmin -refreshNodes* command.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10896) RM fail over is not reporting the nodes DECOMMISSIONED

[Sushil Ks (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-10896:
-
Description: 
Whenever we add the host entries into the exclude file in order to DECOMMISSION 
the Nodemanager, we would issue the *yarn rmadmin -refreshNodes* command to 
transition the nodes from RUNNING to DECOMMISSIONED state. However if the fail 
over to standby resource manager happens and the exclude file has the list of 
hosts to be disallowed, then these disallowed nodes are never seen through the 
Cluster Metrics on the new active resource manager. 

Whatever host entries that are present in the exclude files are being listed in 
the Cluster Metrics whenever resource manager is restarted, i.e as part of the 
service init of *NodeListManager* , however during fail over this info is lost. 
Hence this patch tries to set the  * *DECOMMISSIONED* nodes inside the RM 
Context so that its available through Cluster Metrics whenever we issue the 
**yarn rmadmin -refreshNodes* command.

  was:
Whenever we add the host entries into the exclude file in order to DECOMMISSION 
the Nodemanager, we would issue the *yarn rmadmin -refreshNodes* command to 
transition the nodes from RUNNING to DECOMMISSIONED state. However if the fail 
over to standby resource manager happens and the exclude file has the list of 
hosts to be disallowed, then these disallowed nodes are never seen through the 
Cluster Metrics on the new active resource manager. 

Whatever host entries that are present in the exclude files are being listed in 
the Cluster Metrics whenever resource manager is restarted, i.e as part of the 
service init of *NodeListManager* , however during fail over this info is lost. 
Hence this patch tries to set the  ** DECOMMISSIONED nodes inside the RM 
Context so that its available through Cluster Metrics whenever we issue the 
*yarn rmadmin -refreshNodes* command.


> RM fail over is not reporting the nodes DECOMMISSIONED 
> ---
>
> Key: YARN-10896
> URL: https://issues.apache.org/jira/browse/YARN-10896
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-10896.001.patch
>
>
> Whenever we add the host entries into the exclude file in order to 
> DECOMMISSION the Nodemanager, we would issue the *yarn rmadmin -refreshNodes* 
> command to transition the nodes from RUNNING to DECOMMISSIONED state. However 
> if the fail over to standby resource manager happens and the exclude file has 
> the list of hosts to be disallowed, then these disallowed nodes are never 
> seen through the Cluster Metrics on the new active resource manager. 
> Whatever host entries that are present in the exclude files are being listed 
> in the Cluster Metrics whenever resource manager is restarted, i.e as part of 
> the service init of *NodeListManager* , however during fail over this info is 
> lost. Hence this patch tries to set the  * *DECOMMISSIONED* nodes inside the 
> RM Context so that its available through Cluster Metrics whenever we issue 
> the **yarn rmadmin -refreshNodes* command.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10896) RM fail over is not reporting the nodes DECOMMISSIONED

[Sushil Ks (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-10896:
-
Summary: RM fail over is not reporting the nodes DECOMMISSIONED   (was: RM 
fail over would not report the DECOMMISSIONED nodes )

> RM fail over is not reporting the nodes DECOMMISSIONED 
> ---
>
> Key: YARN-10896
> URL: https://issues.apache.org/jira/browse/YARN-10896
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
>
> Whenever we add the host entries into the exclude file in order to 
> DECOMMISSION the Nodemanager, we would issue the *yarn rmadmin -refreshNodes* 
> command to transition the nodes from RUNNING to DECOMMISSIONED state. However 
> if the fail over to standby resource manager happens and the exclude file has 
> the list of hosts to be disallowed, then these disallowed nodes are never 
> seen through the Cluster Metrics on the new active resource manager. 
> Whatever host entries that are present in the exclude files are being listed 
> in the Cluster Metrics whenever resource manager is restarted, i.e as part of 
> the service init of *NodeListManager* , however during fail over this info is 
> lost. Hence this patch tries to set the  ** DECOMMISSIONED nodes inside the 
> RM Context so that its available through Cluster Metrics whenever we issue 
> the *yarn rmadmin -refreshNodes* command.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10896) RM fail over would not report the DECOMMISSIONED nodes

[Sushil Ks (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-10896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-10896:
-
Summary: RM fail over would not report the DECOMMISSIONED nodes   (was: RM 
fail over is not reporting the DECOMMISSIONED nodes through Cluster Metrics)

> RM fail over would not report the DECOMMISSIONED nodes 
> ---
>
> Key: YARN-10896
> URL: https://issues.apache.org/jira/browse/YARN-10896
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
>
> Whenever we add the host entries into the exclude file in order to 
> DECOMMISSION the Nodemanager, we would issue the *yarn rmadmin -refreshNodes* 
> command to transition the nodes from RUNNING to DECOMMISSIONED state. However 
> if the fail over to standby resource manager happens and the exclude file has 
> the list of hosts to be disallowed, then these disallowed nodes are never 
> seen through the Cluster Metrics on the new active resource manager. 
> Whatever host entries that are present in the exclude files are being listed 
> in the Cluster Metrics whenever resource manager is restarted, i.e as part of 
> the service init of *NodeListManager* , however during fail over this info is 
> lost. Hence this patch tries to set the  ** DECOMMISSIONED nodes inside the 
> RM Context so that its available through Cluster Metrics whenever we issue 
> the *yarn rmadmin -refreshNodes* command.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-10896) RM fail over is not reporting the DECOMMISSIONED nodes through Cluster Metrics

[Sushil Ks (Jira)]

Sushil Ks created YARN-10896:


 Summary: RM fail over is not reporting the DECOMMISSIONED nodes 
through Cluster Metrics
 Key: YARN-10896
 URL: https://issues.apache.org/jira/browse/YARN-10896
 Project: Hadoop YARN
  Issue Type: Bug
  Components: resourcemanager
Reporter: Sushil Ks
Assignee: Sushil Ks


Whenever we add the host entries into the exclude file in order to DECOMMISSION 
the Nodemanager, we would issue the *yarn rmadmin -refreshNodes* command to 
transition the nodes from RUNNING to DECOMMISSIONED state. However if the fail 
over to standby resource manager happens and the exclude file has the list of 
hosts to be disallowed, then these disallowed nodes are never seen through the 
Cluster Metrics on the new active resource manager. 

Whatever host entries that are present in the exclude files are being listed in 
the Cluster Metrics whenever resource manager is restarted, i.e as part of the 
service init of *NodeListManager* , however during fail over this info is lost. 
Hence this patch tries to set the  ** DECOMMISSIONED nodes inside the RM 
Context so that its available through Cluster Metrics whenever we issue the 
*yarn rmadmin -refreshNodes* command.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10356) Consider node labels also for centralized O scheduling

[Sushil Ks (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161823#comment-17161823
 ] 

Sushil Ks commented on YARN-10356:
--

Hi [~bibinchundatt], I will start working on this.

> Consider node labels also for centralized O scheduling
> --
>
> Key: YARN-10356
> URL: https://issues.apache.org/jira/browse/YARN-10356
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin Chundatt
>Assignee: Sushil Ks
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-10356) Consider node labels also for centralized O scheduling

[Sushil Ks (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-10356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks reassigned YARN-10356:


Assignee: Sushil Ks

> Consider node labels also for centralized O scheduling
> --
>
> Key: YARN-10356
> URL: https://issues.apache.org/jira/browse/YARN-10356
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin Chundatt
>Assignee: Sushil Ks
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-10315) Avoid sending RMNodeResourceupdate event if resource is same

[Sushil Ks (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161747#comment-17161747
 ] 

Sushil Ks commented on YARN-10315:
--

[~bibinchundatt], I have fixed the white space errors and uploaded a new patch. 

> Avoid sending RMNodeResourceupdate event if resource is same
> 
>
> Key: YARN-10315
> URL: https://issues.apache.org/jira/browse/YARN-10315
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin Chundatt
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-10315.001.patch, YARN-10315.002.patch
>
>
> When the node is in DECOMMISSIONING state the RMNodeResourceUpdateEvent is 
> send for every heartbeat . Which will result in scheduler resource update.
> Avoid sending the same.
>  Scheduler node resource update iterates through all the queues for resource 
> update which is costly..



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-10315) Avoid sending RMNodeResourceupdate event if resource is same

[Sushil Ks (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-10315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-10315:
-
Attachment: YARN-10315.002.patch

> Avoid sending RMNodeResourceupdate event if resource is same
> 
>
> Key: YARN-10315
> URL: https://issues.apache.org/jira/browse/YARN-10315
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin Chundatt
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-10315.001.patch, YARN-10315.002.patch
>
>
> When the node is in DECOMMISSIONING state the RMNodeResourceUpdateEvent is 
> send for every heartbeat . Which will result in scheduler resource update.
> Avoid sending the same.
>  Scheduler node resource update iterates through all the queues for resource 
> update which is costly..



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Description: 
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h3. *How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class 
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class  
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      The following config needs to be set inside 
*yarn-site.xml* for creating the database and collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class 
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
            Running the schema creator tool to create the necessary collections.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-service.document-store.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.document-store.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally to fetch/query the data from REST 
API's.   
                   

      

  was:
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h3. *How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class 
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class  
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      The following config needs to be set inside 
*yarn-site.xml* for creating the database and collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class 
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
            Running the schema creator tool to create the necessary collections.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-service.document-store.cosmos-db.endpoint
   http:

[jira] [Commented] (YARN-10315) Avoid sending RMNodeResoureupdate event if resource is same

[Sushil Ks (Jira)]

[ 
https://issues.apache.org/jira/browse/YARN-10315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135406#comment-17135406
 ] 

Sushil Ks commented on YARN-10315:
--

Hi [~bibinchundatt], I am working on a patch for this.

> Avoid sending RMNodeResoureupdate event if resource is same
> ---
>
> Key: YARN-10315
> URL: https://issues.apache.org/jira/browse/YARN-10315
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Bibin Chundatt
>Assignee: Sushil Ks
>Priority: Major
>
> When the node is in DECOMMISSIONING state the RMNodeResourceUpdateEvent is 
> send for every heartbeat . Which will result in scheduler resource update.
> Avoid sending the same.
>  Scheduler node resource update iterates through all the queues for resource 
> update which is costly..



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-10315) Avoid sending RMNodeResoureupdate event if resource is same

[Sushil Ks (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-10315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks reassigned YARN-10315:


Assignee: Sushil Ks

> Avoid sending RMNodeResoureupdate event if resource is same
> ---
>
> Key: YARN-10315
> URL: https://issues.apache.org/jira/browse/YARN-10315
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Bibin Chundatt
>Assignee: Sushil Ks
>Priority: Major
>
> When the node is in DECOMMISSIONING state the RMNodeResourceUpdateEvent is 
> send for every heartbeat . Which will result in scheduler resource update.
> Avoid sending the same.
>  Scheduler node resource update iterates through all the queues for resource 
> update which is costly..



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-10315) Avoid sending RMNodeResoureupdate event if resource is same

[Sushil Ks (Jira)]

 [ 
https://issues.apache.org/jira/browse/YARN-10315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks reassigned YARN-10315:


Assignee: Sushil Ks

> Avoid sending RMNodeResoureupdate event if resource is same
> ---
>
> Key: YARN-10315
> URL: https://issues.apache.org/jira/browse/YARN-10315
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Bibin Chundatt
>Assignee: Sushil Ks
>Priority: Major
>
> When the node is in DECOMMISSIONING state the RMNodeResourceUpdateEvent is 
> send for every heartbeat . Which will result in scheduler resource update.
> Avoid sending the same.
>  Scheduler node resource update iterates through all the queues for resource 
> update which is costly..



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16867202#comment-16867202
 ] 

Sushil Ks commented on YARN-9016:
-

Hi [~aajisaka], thanks for pointing this out, I will be working on HADOOP-16381 
for resolving this.

> DocumentStore as a backend for ATSv2
> 
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-9016.001.patch, YARN-9016.002.patch, 
> YARN-9016.003.patch, YARN-9016.004.patch
>
>
> h1. Document Store for ATSv2
>                The Document Store for ATSv2 is a framework for plugging in 
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , 
> MongoDB, ElasticSearch etc.
>  * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
> MongoDB etc by just adding new configurations properties and writing Document 
> Store reader and writer clients.
>  * Currently has support for CosmosDB.
>  * All writes are Async and buffered, latest document would be flushed to the 
> store either if the document buffer gets full or periodically at every flush 
> interval in background without adding any additional latency to the running 
> jobs..
>  * All the REST API's of Timeline Reader Server are supported.
> h3. *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.writer.class 
>  
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
> 
> 
>yarn.timeline-service.reader.class  
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
> 
>  
>yarn.timeline-service.document-store.db-name   
>YOUR_DATABASE_NAME  
> {code}
> h3. *Creating DB and Collections for storing documents*
>                       The following config needs to be set inside 
> *yarn-site.xml* for creating the database and collections for storing 
> documents.
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class 
>  
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
>             Running the schema creator tool to create the necessary 
> collections.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
> h3.  *Azure CosmosDB* 
>        To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
> properties under *yarn-site.xml* is required..
> {code:java}
> 
>   
>yarn.timeline-service.document-store-type  
>COSMOS_DB
> 
> 
>yarn.timeline-service.document-store.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
> 
> 
>yarn.timeline-service.document-store.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
> 
> {code}
>  
>    *Testing locally*
>                In order to test the Azure CosmosDB as a DocumentStore 
> locally, install the emulator from 
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
> start it locally. Set the endpoint and master key under *yarn-site.xml* as 
> mentioned above and run any example job like DistributedShell etc. Later you 
> can check the data explorer UI of Azure CosmosDB locally to query the 
> documents or even launch the *TimelineReader* locally to fetch/query the data 
> from REST API's.   
>                    
>       



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Description: 
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h3. *How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class 
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class  
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      The following config needs to be set inside 
*yarn-site.xml* for creating the database and collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class 
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
            Running the schema creator tool to create the necessary collections.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-service.document-store.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.document-store.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally to fetch/query the data from REST 
API's.   
                   

      

  was:
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class 
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class  
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      The following config needs to be set inside 
*yarn-site.xml* for creating the database and collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class 
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
            Running the schema creator tool to create the necessary collections.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-service.document-store.cosmos-d

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Description: 
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      The following config needs to be set inside 
*yarn-site.xml* for creating the database and collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class</name>
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
            Running the schema creator tool to create the necessary collections.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-service.document-store.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.document-store.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally and fetch/query the data from REST 
API's.   
                   

      

  was:
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class</name>
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class</name>   
  
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      The following config needs to be set inside 
*yarn-site.xml* for creating the database and collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class</name>
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
            Running the schema creator tool to create the necessary collections.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Description: 
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class 
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class  
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      The following config needs to be set inside 
*yarn-site.xml* for creating the database and collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class 
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
            Running the schema creator tool to create the necessary collections.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-service.document-store.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.document-store.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally to fetch/query the data from REST 
API's.   
                   

      

  was:
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class 
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class  
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      The following config needs to be set inside 
*yarn-site.xml* for creating the database and collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class 
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
            Running the schema creator tool to create the necessary collections.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-service.document-store.cosmo

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Description: 
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class 
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class  
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      The following config needs to be set inside 
*yarn-site.xml* for creating the database and collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class 
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
            Running the schema creator tool to create the necessary collections.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-service.document-store.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.document-store.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally and fetch/query the data from REST 
API's.   
                   

      

  was:
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class 
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class  
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      The following config needs to be set inside 
*yarn-site.xml* for creating the database and collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class</name>
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
            Running the schema creator tool to create the necessary collections.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-service.docume

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Description: 
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class 
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class  
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      The following config needs to be set inside 
*yarn-site.xml* for creating the database and collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class</name>
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
            Running the schema creator tool to create the necessary collections.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-service.document-store.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.document-store.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally and fetch/query the data from REST 
API's.   
                   

      

  was:
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
 



   yarn.timeline-service.reader.class 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      The following config needs to be set inside 
*yarn-site.xml* for creating the database and collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class</name>
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
            Running the schema creator tool to create the necessary collections.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-s

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Description: 
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
 



   yarn.timeline-service.reader.class 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      The following config needs to be set inside 
*yarn-site.xml* for creating the database and collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class</name>
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
            Running the schema creator tool to create the necessary collections.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-service.document-store.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.document-store.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally and fetch/query the data from REST 
API's.   
                   

      

  was:
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      The following config needs to be set inside 
*yarn-site.xml* for creating the database and collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class</name>
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
            Running the schema creator tool to create the necessary collections.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-ser

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Description: 
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class</name>
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class</name>   
  
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      The following config needs to be set inside 
*yarn-site.xml* for creating the database and collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class</name>
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
            Running the schema creator tool to create the necessary collections.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-service.document-store.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.document-store.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally and fetch/query the data from REST 
API's.   
                   

      

  was:
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class&amp;lt;/name>
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class&amp;lt;/name> 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      The following config needs to be set inside 
*yarn-site.xml* for creating the database and collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
            Command to invoke Timeline Schema Creation.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Description: 
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class</name>
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class</name> 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      The following config needs to be set inside 
*yarn-site.xml* for creating the database and collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class
 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
            Command to invoke Timeline Schema Creation.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-service.document-store.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.document-store.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally and fetch/query the data from REST 
API's.   
                   

      

  was:
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class</name>
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class</name> 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      This is similar to HBase *TimelineSchemaCreator* the 
following command needs to be executed once for setting up the database and 
collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS


hadoop 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-service.document-store.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yar

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Description: 
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class</name>
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class</name> 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      This is similar to HBase *TimelineSchemaCreator* the 
following command needs to be executed once for setting up the database and 
collections for storing documents.
{code:java}


 yarn.timeline-service.schema-creator.class
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS


hadoop 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-service.document-store.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.document-store.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally and fetch/query the data from REST 
API's.   
                   

      

  was:
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class</name>
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class</name> 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      This is similar to HBase *TimelineSchemaCreator* the 
following command needs to be executed once for setting up the database and 
collections for storing documents.
{code:java}
hadoop 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-service.document-store.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.document-store.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as 

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Description: 
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class</name>
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class</name> 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.document-store.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      This is similar to HBase *TimelineSchemaCreator* the 
following command needs to be executed once for setting up the database and 
collections for storing documents.
{code:java}
hadoop 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.document-store-type  
   COSMOS_DB



   yarn.timeline-service.document-store.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.document-store.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally and fetch/query the data from REST 
API's.   
                   

      

  was:
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.documentstore.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      This is similar to HBase *TimelineSchemaCreator* the 
following command needs to be executed once for setting up the database and 
collections for storing documents.
{code:java}
hadoop 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.store-type  
   COSMOS_DB



   yarn.timeline-service.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Attachment: YARN-9016.004.patch

> DocumentStore as a backend for ATSv2
> 
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9016.001.patch, YARN-9016.002.patch, 
> YARN-9016.003.patch, YARN-9016.004.patch
>
>
> h1. Document Store for ATSv2
>                The Document Store for ATSv2 is a framework for plugging in 
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , 
> MongoDB, ElasticSearch etc.
>  * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
> MongoDB etc by just adding new configurations properties and writing Document 
> Store reader and writer clients.
>  * Currently has support for CosmosDB.
>  * All writes are Async and buffered, latest document would be flushed to the 
> store either if the document buffer gets full or periodically at every flush 
> interval in background without adding any additional latency to the running 
> jobs..
>  * All the REST API's of Timeline Reader Server are supported.
> h4.  
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.writer.class
>  
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
> 
> 
>yarn.timeline-service.reader.class 
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
> 
>  
>yarn.timeline-service.documentstore.db-name   
>YOUR_DATABASE_NAME  
> {code}
> h3. *Creating DB and Collections for storing documents*
>                       This is similar to HBase *TimelineSchemaCreator* the 
> following command needs to be executed once for setting up the database and 
> collections for storing documents.
> {code:java}
> hadoop 
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3.  *Azure CosmosDB* 
>        To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
> properties under *yarn-site.xml* is required..
> {code:java}
> 
>   
>yarn.timeline-service.store-type  
>COSMOS_DB
> 
> 
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
> 
> 
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
> 
> {code}
>  
>    *Testing locally*
>                In order to test the Azure CosmosDB as a DocumentStore 
> locally, install the emulator from 
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
> start it locally. Set the endpoint and master key under *yarn-site.xml* as 
> mentioned above and run any example job like DistributedShell etc. Later you 
> can check the data explorer UI of Azure CosmosDB locally to query the 
> documents or even launch the *TimelineReader* locally and fetch/query the 
> data from REST API's.   
>                    
>       



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781937#comment-16781937
 ] 

Sushil Ks commented on YARN-9150:
-

Hi [~vrushalic], I have uploaded a new patch to resolve the conflicts with 
*YarnConfiguration.java:2690* , kindly review it. The find-bugs error seems to 
be unrelated to my patch.

> Making TimelineSchemaCreator support different backends for Timeline Schema 
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150-branch-2.001.patch, 
> YARN-9150-branch-2.002.patch, YARN-9150-branch-2.003.patch, 
> YARN-9150-branch-2.004.patch, YARN-9150-branch-2.patch, YARN-9150.001.patch, 
> YARN-9150.002.patch, jenkins_build.png
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
>     The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9150:

Attachment: YARN-9150-branch-2.004.patch

> Making TimelineSchemaCreator support different backends for Timeline Schema 
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150-branch-2.001.patch, 
> YARN-9150-branch-2.002.patch, YARN-9150-branch-2.003.patch, 
> YARN-9150-branch-2.004.patch, YARN-9150-branch-2.patch, YARN-9150.001.patch, 
> YARN-9150.002.patch, jenkins_build.png
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
>     The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776137#comment-16776137
 ] 

Sushil Ks commented on YARN-9016:
-

Hi [~vrushalic], [~rohithsharma], I have updated the patch to trunk 
incorporating the YARN-9150 changes, kindly review it. I don't see any test 
failures, however Jenkins is timing out while running all the tests.

> DocumentStore as a backend for ATSv2
> 
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9016.001.patch, YARN-9016.002.patch, 
> YARN-9016.003.patch
>
>
> h1. Document Store for ATSv2
>                The Document Store for ATSv2 is a framework for plugging in 
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , 
> MongoDB, ElasticSearch etc.
>  * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
> MongoDB etc by just adding new configurations properties and writing Document 
> Store reader and writer clients.
>  * Currently has support for CosmosDB.
>  * All writes are Async and buffered, latest document would be flushed to the 
> store either if the document buffer gets full or periodically at every flush 
> interval in background without adding any additional latency to the running 
> jobs..
>  * All the REST API's of Timeline Reader Server are supported.
> h4.  
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.writer.class
>  
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
> 
> 
>yarn.timeline-service.reader.class 
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
> 
>  
>yarn.timeline-service.documentstore.db-name   
>YOUR_DATABASE_NAME  
> {code}
> h3. *Creating DB and Collections for storing documents*
>                       This is similar to HBase *TimelineSchemaCreator* the 
> following command needs to be executed once for setting up the database and 
> collections for storing documents.
> {code:java}
> hadoop 
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3.  *Azure CosmosDB* 
>        To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
> properties under *yarn-site.xml* is required..
> {code:java}
> 
>   
>yarn.timeline-service.store-type  
>COSMOS_DB
> 
> 
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
> 
> 
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
> 
> {code}
>  
>    *Testing locally*
>                In order to test the Azure CosmosDB as a DocumentStore 
> locally, install the emulator from 
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
> start it locally. Set the endpoint and master key under *yarn-site.xml* as 
> mentioned above and run any example job like DistributedShell etc. Later you 
> can check the data explorer UI of Azure CosmosDB locally to query the 
> documents or even launch the *TimelineReader* locally and fetch/query the 
> data from REST API's.   
>                    
>       



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Attachment: YARN-9016.003.patch

> DocumentStore as a backend for ATSv2
> 
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9016.001.patch, YARN-9016.002.patch, 
> YARN-9016.003.patch
>
>
> h1. Document Store for ATSv2
>                The Document Store for ATSv2 is a framework for plugging in 
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , 
> MongoDB, ElasticSearch etc.
>  * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
> MongoDB etc by just adding new configurations properties and writing Document 
> Store reader and writer clients.
>  * Currently has support for CosmosDB.
>  * All writes are Async and buffered, latest document would be flushed to the 
> store either if the document buffer gets full or periodically at every flush 
> interval in background without adding any additional latency to the running 
> jobs..
>  * All the REST API's of Timeline Reader Server are supported.
> h4.  
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.writer.class
>  
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
> 
> 
>yarn.timeline-service.reader.class 
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
> 
>  
>yarn.timeline-service.documentstore.db-name   
>YOUR_DATABASE_NAME  
> {code}
> h3. *Creating DB and Collections for storing documents*
>                       This is similar to HBase *TimelineSchemaCreator* the 
> following command needs to be executed once for setting up the database and 
> collections for storing documents.
> {code:java}
> hadoop 
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3.  *Azure CosmosDB* 
>        To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
> properties under *yarn-site.xml* is required..
> {code:java}
> 
>   
>yarn.timeline-service.store-type  
>COSMOS_DB
> 
> 
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
> 
> 
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
> 
> {code}
>  
>    *Testing locally*
>                In order to test the Azure CosmosDB as a DocumentStore 
> locally, install the emulator from 
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
> start it locally. Set the endpoint and master key under *yarn-site.xml* as 
> mentioned above and run any example job like DistributedShell etc. Later you 
> can check the data explorer UI of Azure CosmosDB locally to query the 
> documents or even launch the *TimelineReader* locally and fetch/query the 
> data from REST API's.   
>                    
>       



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Attachment: YARN-9016.002.patch

> DocumentStore as a backend for ATSv2
> 
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9016.001.patch, YARN-9016.002.patch
>
>
> h1. Document Store for ATSv2
>                The Document Store for ATSv2 is a framework for plugging in 
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , 
> MongoDB, ElasticSearch etc.
>  * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
> MongoDB etc by just adding new configurations properties and writing Document 
> Store reader and writer clients.
>  * Currently has support for CosmosDB.
>  * All writes are Async and buffered, latest document would be flushed to the 
> store either if the document buffer gets full or periodically at every flush 
> interval in background without adding any additional latency to the running 
> jobs..
>  * All the REST API's of Timeline Reader Server are supported.
> h4.  
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.writer.class
>  
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
> 
> 
>yarn.timeline-service.reader.class 
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
> 
>  
>yarn.timeline-service.documentstore.db-name   
>YOUR_DATABASE_NAME  
> {code}
> h3. *Creating DB and Collections for storing documents*
>                       This is similar to HBase *TimelineSchemaCreator* the 
> following command needs to be executed once for setting up the database and 
> collections for storing documents.
> {code:java}
> hadoop 
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3.  *Azure CosmosDB* 
>        To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
> properties under *yarn-site.xml* is required..
> {code:java}
> 
>   
>yarn.timeline-service.store-type  
>COSMOS_DB
> 
> 
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
> 
> 
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
> 
> {code}
>  
>    *Testing locally*
>                In order to test the Azure CosmosDB as a DocumentStore 
> locally, install the emulator from 
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
> start it locally. Set the endpoint and master key under *yarn-site.xml* as 
> mentioned above and run any example job like DistributedShell etc. Later you 
> can check the data explorer UI of Azure CosmosDB locally to query the 
> documents or even launch the *TimelineReader* locally and fetch/query the 
> data from REST API's.   
>                    
>       



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16763347#comment-16763347
 ] 

Sushil Ks commented on YARN-9150:
-

Hi [~vrushalic], I uploaded the patch for *branch2* which is same as trunk, 
however the unit test failures are not related to my patch. Might be some issue 
with the build. The below stack trace is common for all the test failures.


{code:java}
   /usr/bin/mvn --batch-mode 
-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/yetus-m2/hadoop-branch-2-patch-1
 -Ptest-patch -Pparallel-tests -P!shelltest -Pnative -Drequire.fuse 
-Drequire.openssl -Drequire.snappy -Drequire.valgrind -Drequire.test.libhadoop 
-Pyarn-ui clean test -fae
[INFO] Scanning for projects...
[INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/felix/maven-bundle-plugin/2.5.0/maven-bundle-plugin-2.5.0.pom
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[WARNING] 
'dependencyManagement.dependencies.dependency.(groupId:artifactId:type:classifier)'
 must be unique: com.microsoft.azure:azure-storage:jar -> version 7.0.0 vs 
5.4.0 @ org.apache.hadoop:hadoop-project:2.10.0-SNAPSHOT, 
/testptch/hadoop/hadoop-project/pom.xml, line 1157, column 19
[ERROR] Unresolveable build extension: Plugin 
org.apache.felix:maven-bundle-plugin:2.5.0 or one of its dependencies could not 
be resolved: Failed to read artifact descriptor for 
org.apache.felix:maven-bundle-plugin:jar:2.5.0 @
@
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project 
org.apache.hadoop:hadoop-yarn-server-timelineservice:2.10.0-SNAPSHOT 
(/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/pom.xml)
 has 1 error
[ERROR] Unresolveable build extension: Plugin 
org.apache.felix:maven-bundle-plugin:2.5.0 or one of its dependencies could not 
be resolved: Failed to read artifact descriptor for 
org.apache.felix:maven-bundle-plugin:jar:2.5.0: Could not transfer artifact 
org.apache.felix:maven-bundle-plugin:pom:2.5.0 from/to central 
(https://repo.maven.apache.org/maven2): Received fatal alert: protocol_version 
-> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException{code}

> Making TimelineSchemaCreator support different backends for Timeline Schema 
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150-branch-2.001.patch, 
> YARN-9150-branch-2.002.patch, YARN-9150-branch-2.003.patch, 
> YARN-9150-branch-2.patch, YARN-9150.001.patch, YARN-9150.002.patch, 
> jenkins_build.png
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
>     The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9150:

Attachment: YARN-9150-branch-2.003.patch

> Making TimelineSchemaCreator support different backends for Timeline Schema 
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150-branch-2.001.patch, 
> YARN-9150-branch-2.002.patch, YARN-9150-branch-2.003.patch, 
> YARN-9150-branch-2.patch, YARN-9150.001.patch, YARN-9150.002.patch, 
> jenkins_build.png
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
>     The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-6105) Support for new REST end point /clusterids

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks reassigned YARN-6105:
---

Assignee: Sushil Ks

> Support for new REST end point /clusterids
> --
>
> Key: YARN-6105
> URL: https://issues.apache.org/jira/browse/YARN-6105
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Sushil Ks
>Priority: Major
>
> As discussed in YARN-5378 and YARN-6095, it is required to have */clusterids* 
> that returns list of clusterids that back end has is useful. 
> Use case : In cloud, clusters are arbitrarily spin up and destroyed. Each 
> cluster has its own clusterId which UI never knows about it. To all those 
> newly spin up cluster, same ATS server has been used. And sam web UI has been 
> used. Admin can select the clusterId and navigate to any pages. So, it is 
> worth to list ClusterId's from ATS



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9150:

Attachment: YARN-9150-branch-2.002.patch

> Making TimelineSchemaCreator support different backends for Timeline Schema 
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150-branch-2.001.patch, 
> YARN-9150-branch-2.002.patch, YARN-9150-branch-2.patch, YARN-9150.001.patch, 
> YARN-9150.002.patch, jenkins_build.png
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
>     The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9150:

Attachment: YARN-9150-branch-2.001.patch

> Making TimelineSchemaCreator support different backends for Timeline Schema 
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150-branch-2.001.patch, YARN-9150-branch-2.patch, 
> YARN-9150.001.patch, YARN-9150.002.patch, jenkins_build.png
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
>     The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-7271) Add a yarn application cost calculation framework in TimelineService v2

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755587#comment-16755587
 ] 

Sushil Ks edited comment on YARN-7271 at 1/30/19 2:26 AM:
--

Hi [~vrushalic], have went through this -YARN-415- briefly and have some ideas, 
would like to discuss with you in our call today for your suggestions/feedback 
on this to take it forward.


was (Author: sushil-k-s):
Hi [~vrushalic], have went through this 
[YARN-415|https://jira.apache.org/jira/browse/YARN-415] briefly and have some 
ideas, would like to discuss with you in our call today for your 
suggestions/feedback on this to take it forward.

> Add a yarn application cost calculation framework in TimelineService v2
> ---
>
> Key: YARN-7271
> URL: https://issues.apache.org/jira/browse/YARN-7271
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Sushil Ks
>Priority: Major
>
> Timeline Service v2 captures information about a yarn application. From this 
> info, we would like to calculate the "cost" of an yarn application. This 
> would be rolled up to the flow level  as well (and user and queue level 
> eventually).
> We need a way to accept machine cost (TCO per day) and enable this 
> calculation. This will help in chargeback for yarn apps. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7271) Add a yarn application cost calculation framework in TimelineService v2

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755587#comment-16755587
 ] 

Sushil Ks commented on YARN-7271:
-

Hi [~vrushalic], have went through this 
[YARN-415|https://jira.apache.org/jira/browse/YARN-415] briefly and have some 
ideas, would like to discuss with you in our call today for your 
suggestions/feedback on this to take it forward.

> Add a yarn application cost calculation framework in TimelineService v2
> ---
>
> Key: YARN-7271
> URL: https://issues.apache.org/jira/browse/YARN-7271
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Sushil Ks
>Priority: Major
>
> Timeline Service v2 captures information about a yarn application. From this 
> info, we would like to calculate the "cost" of an yarn application. This 
> would be rolled up to the flow level  as well (and user and queue level 
> eventually).
> We need a way to accept machine cost (TCO per day) and enable this 
> calculation. This will help in chargeback for yarn apps. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-7271) Add a yarn application cost calculation framework in TimelineService v2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks reassigned YARN-7271:
---

Assignee: Sushil Ks

> Add a yarn application cost calculation framework in TimelineService v2
> ---
>
> Key: YARN-7271
> URL: https://issues.apache.org/jira/browse/YARN-7271
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Sushil Ks
>Priority: Major
>
> Timeline Service v2 captures information about a yarn application. From this 
> info, we would like to calculate the "cost" of an yarn application. This 
> would be rolled up to the flow level  as well (and user and queue level 
> eventually).
> We need a way to accept machine cost (TCO per day) and enable this 
> calculation. This will help in chargeback for yarn apps. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-6095) create a REST API that returns the clusters for a given app id

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks reassigned YARN-6095:
---

Assignee: Sushil Ks

> create a REST API that returns the clusters for a given app id
> --
>
> Key: YARN-6095
> URL: https://issues.apache.org/jira/browse/YARN-6095
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Sushil Ks
>Priority: Major
>
> It would be good to have a timeline service REST endpoint that can return the 
> list of clusters for a given app id. This becomes possible after YARN-5378 is 
> in.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5336) Limit the flow name size & consider cleanup for hex chars

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16753677#comment-16753677
 ] 

Sushil Ks commented on YARN-5336:
-

Hi [~vrushalic], Thanks for reviewing, I have updated the patch. Kindly review 
it.

> Limit the flow name size & consider cleanup for hex chars
> -
>
> Key: YARN-5336
> URL: https://issues.apache.org/jira/browse/YARN-5336
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Sushil Ks
>Priority: Major
>  Labels: YARN-5355
> Attachments: YARN-5336.001.patch, YARN-5336.002.patch, 
> YARN-5336.003.patch
>
>
> As recommended by [~jrottinghuis] , need to add in some limit (default and 
> configurable) for accepting key values to be written to the backend.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9150:

Description: 
h3. Currently the TimelineSchemaCreator has a concrete implementation for 
creating Timeline Schema's only for HBase, Hence creating this JIRA for 
supporting multiple back-ends that ATSv2 can support.

*Usage:*

   Add the following property in *yarn-site.xml*
{code:java}


 yarn.timeline-service.schema-creator.class
 
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 


{code}
    The Command needed to run the TimelineSchemaCreator need not be changed i.e 
the below existing command can be used irrespective of the backend configured.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
-create
{code}
 

 

  was:
h3. Currently the TimelineSchemaCreator has a concrete implementation for 
creating Timeline Schema's only for HBase, Hence creating this JIRA for 
supporting multiple back-ends that ATSv2 can support.

*Usage:*

   Add the following property in *yarn-site.xml*
{code:java}


 yarn.timeline-service.schema-creator.class
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 


{code}

     The Command needed to run the TimelineSchemaCreator need not be changed 
i.e the below existing command can be used irrespective of the backend 
configured.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
-create
{code}
 

 


> Making TimelineSchemaCreator support different backends for Timeline Schema 
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150-branch-2.patch, YARN-9150.001.patch, 
> YARN-9150.002.patch, jenkins_build.png
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
>     The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5336) Limit the flow name size & consider cleanup for hex chars

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744973#comment-16744973
 ] 

Sushil Ks commented on YARN-5336:
-

Hi [~vrushalic], I have resolved your comments in the new patch. Kindly review 
it.

> Limit the flow name size & consider cleanup for hex chars
> -
>
> Key: YARN-5336
> URL: https://issues.apache.org/jira/browse/YARN-5336
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Sushil Ks
>Priority: Major
>  Labels: YARN-5355
> Attachments: YARN-5336.001.patch, YARN-5336.002.patch
>
>
> As recommended by [~jrottinghuis] , need to add in some limit (default and 
> configurable) for accepting key values to be written to the backend.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5336) Limit the flow name size & consider cleanup for hex chars

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-5336:

Attachment: (was: YARN-5336.002.patch)

> Limit the flow name size & consider cleanup for hex chars
> -
>
> Key: YARN-5336
> URL: https://issues.apache.org/jira/browse/YARN-5336
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Sushil Ks
>Priority: Major
>  Labels: YARN-5355
> Attachments: YARN-5336.001.patch
>
>
> As recommended by [~jrottinghuis] , need to add in some limit (default and 
> configurable) for accepting key values to be written to the backend.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5336) Limit the flow name size & consider cleanup for hex chars

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-5336:

Attachment: YARN-5336.002.patch

> Limit the flow name size & consider cleanup for hex chars
> -
>
> Key: YARN-5336
> URL: https://issues.apache.org/jira/browse/YARN-5336
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Sushil Ks
>Priority: Major
>  Labels: YARN-5355
> Attachments: YARN-5336.001.patch, YARN-5336.002.patch
>
>
> As recommended by [~jrottinghuis] , need to add in some limit (default and 
> configurable) for accepting key values to be written to the backend.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744758#comment-16744758
 ] 

Sushil Ks commented on YARN-9150:
-

Hi [~vrushalic], have attached the patch for branch2. Kindly review it.

> Making TimelineSchemaCreator support different backends for Timeline Schema 
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150-branch-2.patch, YARN-9150.001.patch, 
> YARN-9150.002.patch, jenkins_build.png
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
>      The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9150:

Attachment: YARN-9150-branch-2.patch

> Making TimelineSchemaCreator support different backends for Timeline Schema 
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150-branch-2.patch, YARN-9150.001.patch, 
> YARN-9150.002.patch, jenkins_build.png
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
>      The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743573#comment-16743573
 ] 

Sushil Ks commented on YARN-9150:
-

Sure, Thanks [~vrushalic].

> Making TimelineSchemaCreator support different backends for Timeline Schema 
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch, YARN-9150.002.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
>      The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16741792#comment-16741792
 ] 

Sushil Ks commented on YARN-9150:
-

Hi [~vrushalic], I resubmitted the same patch and everything looks fine from 
Jenkins side now.

> Making TimelineSchemaCreator support different backends for Timeline Schema 
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch, YARN-9150.002.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
>      The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9150:

Description: 
h3. Currently the TimelineSchemaCreator has a concrete implementation for 
creating Timeline Schema's only for HBase, Hence creating this JIRA for 
supporting multiple back-ends that ATSv2 can support.

*Usage:*

   Add the following property in *yarn-site.xml*
{code:java}


 yarn.timeline-service.schema-creator.class
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 


{code}

     The Command needed to run the TimelineSchemaCreator need not be changed 
i.e the below existing command can be used irrespective of the backend 
configured.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
-create
{code}
 

 

  was:
h3. Currently the TimelineSchemaCreator has a concrete implementation for 
creating Timeline Schema's only for HBase, Hence creating this JIRA for 
supporting multiple back-ends that ATSv2 can support.

*Usage:*

   Add the following property in *yarn-site.xml*
{code:java}


 yarn.timeline-service.schema-creator.class 
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 


{code}

     The Command needed to run the TimelineSchemaCreator need not be changed 
i.e the below existing command can be used irrespective of the backend 
configured.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
-create
{code}
 

 


> Making TimelineSchemaCreator support different backends for Timeline Schema 
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch, YARN-9150.002.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
>      The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9150:

Description: 
h3. Currently the TimelineSchemaCreator has a concrete implementation for 
creating Timeline Schema's only for HBase, Hence creating this JIRA for 
supporting multiple back-ends that ATSv2 can support.

*Usage:*

   Add the following property in *yarn-site.xml*
{code:java}


 yarn.timeline-service.schema-creator.class 
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 


{code}

     The Command needed to run the TimelineSchemaCreator need not be changed 
i.e the below existing command can be used irrespective of the backend 
configured.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
-create
{code}
 

 

  was:
h3. Currently the TimelineSchemaCreator has a concrete implementation for 
creating Timeline Schema's only for HBase, Hence creating this JIRA for 
supporting multiple back-ends that ATSv2 can support.

*Usage:*

   Add the following property in *yarn-site.xml*
{code:java}


 yarn.timeline-service.schema-creator.class
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 


{code}

     The Command needed to run the TimelineSchemaCreator need not be changed 
i.e the below existing command can be used irrespective of the backend 
configured.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
-create
{code}
 

 


> Making TimelineSchemaCreator support different backends for Timeline Schema 
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch, YARN-9150.002.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class 
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
>      The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9150:

Attachment: (was: YARN-9150.002.patch)

> Making TimelineSchemaCreator support different backends for Timeline Schema 
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
>      The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9150:

Attachment: YARN-9150.002.patch

> Making TimelineSchemaCreator support different backends for Timeline Schema 
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
>      The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5336) Limit the flow name size & consider cleanup for hex chars

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16729033#comment-16729033
 ] 

Sushil Ks commented on YARN-5336:
-

Hi [~vrushalic], kindly review the patch.

> Limit the flow name size & consider cleanup for hex chars
> -
>
> Key: YARN-5336
> URL: https://issues.apache.org/jira/browse/YARN-5336
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Vrushali C
>Assignee: Sushil Ks
>Priority: Major
>  Labels: YARN-5355
> Attachments: YARN-5336.001.patch
>
>
> As recommended by [~jrottinghuis] , need to add in some limit (default and 
> configurable) for accepting key values to be written to the backend.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9150:

Description: 
h3. Currently the TimelineSchemaCreator has a concrete implementation for 
creating Timeline Schema's only for HBase, Hence creating this JIRA for 
supporting multiple back-ends that ATSv2 can support.

*Usage:*

   Add the following property in *yarn-site.xml*
{code:java}


 yarn.timeline-service.schema-creator.class
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 


{code}

     The Command needed to run the TimelineSchemaCreator need not be changed 
i.e the below existing command can be used irrespective of the backend 
configured.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
-create
{code}
 

 

  was:
h3. Currently the TimelineSchemaCreator has a concrete implementation for 
creating Timeline Schema's only for HBase, Hence creating this JIRA for 
supporting multiple back-ends that ATSv2 can support.

*Usage:*

   Add the following property in *yarn-site.xml*
{code:java}


 yarn.timeline-service.schema-creator.class
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 


{code}
**
     The Command needed to run the TimelineSchemaCreator need not be changed 
i.e the below existing command can be used irrespective of the backend 
configured.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
-create
{code}
 

 


> Making TimelineSchemaCreator support different backends for Timeline Schema 
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
>      The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9150:

Summary: Making TimelineSchemaCreator support different backends for 
Timeline Schema Creation in ATSv2  (was: Making TimelineSchemaCreator to 
support different backends for Timeline Schema Creation in ATSv2)

> Making TimelineSchemaCreator support different backends for Timeline Schema 
> Creation in ATSv2
> -
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
> **
>      The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9150) Making TimelineSchemaCreator to support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16725084#comment-16725084
 ] 

Sushil Ks commented on YARN-9150:
-

Hi  [~rohithsharma] and [~vrushalic], Kindly review this patch. Have created 
this JIRA for making *TimelineSchemaCreator* support multiple back-ends as 
discussed when you reviewed the 
[YARN-9016|https://issues.apache.org/jira/browse/YARN-9016] .

Note sure if the -1 for *compile* and *javac* tests posted from Jenkins above 
are related to my patch.

> Making TimelineSchemaCreator to support different backends for Timeline 
> Schema Creation in ATSv2
> 
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
> **
>      The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator to support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9150:

Description: 
h3. Currently the TimelineSchemaCreator has a concrete implementation for 
creating Timeline Schema's only for HBase, Hence creating this JIRA for 
supporting multiple back-ends that ATSv2 can support.

*Usage:*

   Add the following property in *yarn-site.xml*
{code:java}


 yarn.timeline-service.schema-creator.class
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 


{code}
**
     The Command needed to run the TimelineSchemaCreator need not be changed 
i.e the below existing command can be used irrespective of the backend 
configured.
{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
-create
{code}
 

 

  was:
h3. Currently the TimelineSchemaCreator has a concrete implementation for 
creating Timeline Schema's only for HBase, Hence creating this JIRA for 
supporting multiple back-ends that ATSv2 can support.

*Usage:*

   Add the following property in *yarn-site.xml*
{code:java}


 yarn.timeline-service.schema-creator.class</name>
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 


{code}
**
     The Command needed to run the TimelineSchemaCreator need not be changed 
i.e the below existing command can be used irrespective of the backend 
configured.


{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
-create
{code}
 

 


> Making TimelineSchemaCreator to support different backends for Timeline 
> Schema Creation in ATSv2
> 
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
> **
>      The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator to support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9150:

Description: 
h3. Currently the TimelineSchemaCreator has a concrete implementation for 
creating Timeline Schema's only for HBase, Hence creating this JIRA for 
supporting multiple back-ends that ATSv2 can support.

*Usage:*

   Add the following property in *yarn-site.xml*
{code:java}


 yarn.timeline-service.schema-creator.class</name>
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 


{code}
**
     The Command needed to run the TimelineSchemaCreator need not be changed 
i.e the below existing command can be used irrespective of the backend 
configured.


{code:java}
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
-create
{code}
 

 

  was:
h3. Currently the TimelineSchemaCreator has a concrete implementation for 
creating Timeline Schema's only for HBase, Hence creating this JIRA for 
supporting multiple back-ends that ATSv2 can support.

*Usage:*

   **  Add the following property in *yarn-site.xml*
{code:java}


 yarn.timeline-service.schema-creator.class</name>
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 


{code}
h3.  

*Running*
     The Command needed to run the TimelineSchemaCreator need not be changed 
i.e the below existing command can be used irrespective of the backend 
configured.
 **
 bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
-create


> Making TimelineSchemaCreator to support different backends for Timeline 
> Schema Creation in ATSv2
> 
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class</name>
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
> **
>      The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> {code:java}
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator to support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9150:

Description: 
h3. Currently the TimelineSchemaCreator has a concrete implementation for 
creating Timeline Schema's only for HBase, Hence creating this JIRA for 
supporting multiple back-ends that ATSv2 can support.

*Usage:*

 **  Add the following property in *yarn-site.xml*
{code:java}


 yarn.timeline-service.schema-creator.class</name>
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 


{code}
h3.  

*Running*
     The Command needed to run the TimelineSchemaCreator need not be changed 
i.e the below existing command can be used irrespective of the backend 
configured.
 **
 bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
-create

  was:
h3. Currently the TimelineSchemaCreator has a concrete implementation for 
creating Timeline Schema's only for HBase, Hence creating this JIRA for 
supporting multiple back-ends that ATSv2 can support.

*Usage:*

   **   Add the following property in *yarn-site.xml*
{code:java}


 yarn.timeline-service.schema-creator.class</name>
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 


{code}
h3. 
*Running*
    The Command needed to run the TimelineSchemaCreator need not be changed i.e 
the below existing command can be used irrespective of the backend configured.
**
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
-create


> Making TimelineSchemaCreator to support different backends for Timeline 
> Schema Creation in ATSv2
> 
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>  **  Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class&lt;/name>
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
> h3.  
> *Running*
>      The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
>  **
>  bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator to support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9150:

Description: 
h3. Currently the TimelineSchemaCreator has a concrete implementation for 
creating Timeline Schema's only for HBase, Hence creating this JIRA for 
supporting multiple back-ends that ATSv2 can support.

*Usage:*

   **  Add the following property in *yarn-site.xml*
{code:java}


 yarn.timeline-service.schema-creator.class</name>
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 


{code}
h3.  

*Running*
     The Command needed to run the TimelineSchemaCreator need not be changed 
i.e the below existing command can be used irrespective of the backend 
configured.
 **
 bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
-create

  was:
h3. Currently the TimelineSchemaCreator has a concrete implementation for 
creating Timeline Schema's only for HBase, Hence creating this JIRA for 
supporting multiple back-ends that ATSv2 can support.

*Usage:*

 **  Add the following property in *yarn-site.xml*
{code:java}


 yarn.timeline-service.schema-creator.class</name>
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 


{code}
h3.  

*Running*
     The Command needed to run the TimelineSchemaCreator need not be changed 
i.e the below existing command can be used irrespective of the backend 
configured.
 **
 bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
-create


> Making TimelineSchemaCreator to support different backends for Timeline 
> Schema Creation in ATSv2
> 
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9150.001.patch
>
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    **  Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class</name>
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
> h3.  
> *Running*
>      The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
>  **
>  bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9150) Making TimelineSchemaCreator to support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9150:

Description: 
h3. Currently the TimelineSchemaCreator has a concrete implementation for 
creating Timeline Schema's only for HBase, Hence creating this JIRA for 
supporting multiple back-ends that ATSv2 can support.

*Usage:*

   **   Add the following property in *yarn-site.xml*
{code:java}


 yarn.timeline-service.schema-creator.class</name>
 YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 


{code}
h3. 
*Running*
    The Command needed to run the TimelineSchemaCreator need not be changed i.e 
the below existing command can be used irrespective of the backend configured.
**
bin/hadoop 
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
-create

> Making TimelineSchemaCreator to support different backends for Timeline 
> Schema Creation in ATSv2
> 
>
> Key: YARN-9150
> URL: https://issues.apache.org/jira/browse/YARN-9150
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
>
> h3. Currently the TimelineSchemaCreator has a concrete implementation for 
> creating Timeline Schema's only for HBase, Hence creating this JIRA for 
> supporting multiple back-ends that ATSv2 can support.
> *Usage:*
>    **   Add the following property in *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.schema-creator.class</name>
>  YOUR_TIMELINE_SCHEMA_CREATOR_CLASS 
> 
> {code}
> h3. 
> *Running*
>     The Command needed to run the TimelineSchemaCreator need not be changed 
> i.e the below existing command can be used irrespective of the backend 
> configured.
> **
> bin/hadoop 
> org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator 
> -create



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-9150) Making TimelineSchemaCreator to support different backends for Timeline Schema Creation in ATSv2

[Sushil Ks (JIRA)]

Sushil Ks created YARN-9150:
---

 Summary: Making TimelineSchemaCreator to support different 
backends for Timeline Schema Creation in ATSv2
 Key: YARN-9150
 URL: https://issues.apache.org/jira/browse/YARN-9150
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: ATSv2
Reporter: Sushil Ks
Assignee: Sushil Ks






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16707379#comment-16707379
 ] 

Sushil Ks commented on YARN-9016:
-

Hi [~rohithsharma], Thanks for reviewing the patch :) please find my comments 
below,

1. Since initially *TimelineSchemaCreator* was specific to HBase didn't 
consider of changing it, now it makes sense to create an interface or abstract 
class with the same name. So the *TimelineSchemaCreator* can be moved to the 
*hadoop-yarn-server-timelineservice* module and each backend can have their 
specific implementation, what do you think?

2 & 3,  sure will add *document-store* for readability.

Also not sure why some unit tests are getting skipped while building, am seeing 
the following error from build machine
{code:java}
[WARNING] The requested profile "parallel-tests" could not be activated because 
it does not exist.
[WARNING] The requested profile "yarn-ui" could not be activated because it 
does not exist.
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M1:test (default-test) on 
project hadoop-yarn-server-resourcemanager: There was a timeout or other error 
in the fork -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the 
command{code}

Sure, let me know if there are any more comments :)

 

> DocumentStore as a backend for ATSv2
> 
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9016.001.patch
>
>
> h1. Document Store for ATSv2
>                The Document Store for ATSv2 is a framework for plugging in 
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , 
> MongoDB, ElasticSearch etc.
>  * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
> MongoDB etc by just adding new configurations properties and writing Document 
> Store reader and writer clients.
>  * Currently has support for CosmosDB.
>  * All writes are Async and buffered, latest document would be flushed to the 
> store either if the document buffer gets full or periodically at every flush 
> interval in background without adding any additional latency to the running 
> jobs..
>  * All the REST API's of Timeline Reader Server are supported.
> h4.  
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.writer.class
>  
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
> 
> 
>yarn.timeline-service.reader.class 
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
> 
>  
>yarn.timeline-service.documentstore.db-name   
>YOUR_DATABASE_NAME  
> {code}
> h3. *Creating DB and Collections for storing documents*
>                       This is similar to HBase *TimelineSchemaCreator* the 
> following command needs to be executed once for setting up the database and 
> collections for storing documents.
> {code:java}
> hadoop 
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3.  *Azure CosmosDB* 
>        To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
> properties under *yarn-site.xml* is required..
> {code:java}
> 
>   
>yarn.timeline-service.store-type  
>COSMOS_DB
> 
> 
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
> 
> 
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
> 
> {code}
>  
>    *Testing locally*
>                In order to test the Azure CosmosDB as a DocumentStore 
> locally, install the emulator from 
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
> start it locally. Set the endpoint and master key under *yarn-site.xml* as 
> mentioned above and run any example job like DistributedShell etc. Later you 
> can check the data explorer UI of Azure CosmosDB locally to query the 
> documents or even launch the *TimelineReader* locally and fetch/query the 
> data from REST API's.   
>                    
>       



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16694587#comment-16694587
 ] 

Sushil Ks commented on YARN-9016:
-

Hi [~vrushalic] and [~rohithsharma], based on our discussion in the weekly 
ATSv2 call, I have updated the patch. Kindly review it.

> DocumentStore as a backend for ATSv2
> 
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-9016.001.patch
>
>
> h1. Document Store for ATSv2
>                The Document Store for ATSv2 is a framework for plugging in 
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , 
> MongoDB, ElasticSearch etc.
>  * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
> MongoDB etc by just adding new configurations properties and writing Document 
> Store reader and writer clients.
>  * Currently has support for CosmosDB.
>  * All writes are Async and buffered, latest document would be flushed to the 
> store either if the document buffer gets full or periodically at every flush 
> interval in background without adding any additional latency to the running 
> jobs..
>  * All the REST API's of Timeline Reader Server are supported.
> h4.  
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.writer.class
>  
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
> 
> 
>yarn.timeline-service.reader.class 
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
> 
>  
>yarn.timeline-service.documentstore.db-name   
>YOUR_DATABASE_NAME  
> {code}
> h3. *Creating DB and Collections for storing documents*
>                       This is similar to HBase *TimelineSchemaCreator* the 
> following command needs to be executed once for setting up the database and 
> collections for storing documents.
> {code:java}
> hadoop 
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3.  *Azure CosmosDB* 
>        To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
> properties under *yarn-site.xml* is required..
> {code:java}
> 
>   
>yarn.timeline-service.store-type  
>COSMOS_DB
> 
> 
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
> 
> 
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
> 
> {code}
>  
>    *Testing locally*
>                In order to test the Azure CosmosDB as a DocumentStore 
> locally, install the emulator from 
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
> start it locally. Set the endpoint and master key under *yarn-site.xml* as 
> mentioned above and run any example job like DistributedShell etc. Later you 
> can check the data explorer UI of Azure CosmosDB locally to query the 
> documents or even launch the *TimelineReader* locally and fetch/query the 
> data from REST API's.   
>                    
>       



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Attachment: (was: YARN-9016.001.patch)

> DocumentStore as a backend for ATSv2
> 
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
>
> h1. Document Store for ATSv2
>                The Document Store for ATSv2 is a framework for plugging in 
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , 
> MongoDB, ElasticSearch etc.
>  * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
> MongoDB etc by just adding new configurations properties and writing Document 
> Store reader and writer clients.
>  * Currently has support for CosmosDB.
>  * All writes are Async and buffered, latest document would be flushed to the 
> store either if the document buffer gets full or periodically at every flush 
> interval in background without adding any additional latency to the running 
> jobs..
>  * All the REST API's of Timeline Reader Server are supported.
> h4.  
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.writer.class
>  
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
> 
> 
>yarn.timeline-service.reader.class 
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
> 
>  
>yarn.timeline-service.documentstore.db-name   
>YOUR_DATABASE_NAME  
> {code}
> h3. *Creating DB and Collections for storing documents*
>                       This is similar to HBase *TimelineSchemaCreator* the 
> following command needs to be executed once for setting up the database and 
> collections for storing documents.
> {code:java}
> hadoop 
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3.  *Azure CosmosDB* 
>        To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
> properties under *yarn-site.xml* is required..
> {code:java}
> 
>   
>yarn.timeline-service.store-type  
>COSMOS_DB
> 
> 
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
> 
> 
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
> 
> {code}
>  
>    *Testing locally*
>                In order to test the Azure CosmosDB as a DocumentStore 
> locally, install the emulator from 
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
> start it locally. Set the endpoint and master key under *yarn-site.xml* as 
> mentioned above and run any example job like DistributedShell etc. Later you 
> can check the data explorer UI of Azure CosmosDB locally to query the 
> documents or even launch the *TimelineReader* locally and fetch/query the 
> data from REST API's.   
>                    
>       



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Attachment: (was: YARN-9016.001.patch)

> DocumentStore as a backend for ATSv2
> 
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
>
> h1. Document Store for ATSv2
>                The Document Store for ATSv2 is a framework for plugging in 
> any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , 
> MongoDB, ElasticSearch etc.
>  * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
> MongoDB etc by just adding new configurations properties and writing Document 
> Store reader and writer clients.
>  * Currently has support for CosmosDB.
>  * All writes are Async and buffered, latest document would be flushed to the 
> store either if the document buffer gets full or periodically at every flush 
> interval in background without adding any additional latency to the running 
> jobs..
>  * All the REST API's of Timeline Reader Server are supported.
> h4.  
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.writer.class
>  
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
> 
> 
>yarn.timeline-service.reader.class 
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
> 
>  
>yarn.timeline-service.documentstore.db-name   
>YOUR_DATABASE_NAME  
> {code}
> h3. *Creating DB and Collections for storing documents*
>                       This is similar to HBase *TimelineSchemaCreator* the 
> following command needs to be executed once for setting up the database and 
> collections for storing documents.
> {code:java}
> hadoop 
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3.  *Azure CosmosDB* 
>        To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
> properties under *yarn-site.xml* is required..
> {code:java}
> 
>   
>yarn.timeline-service.store-type  
>COSMOS_DB
> 
> 
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
> 
> 
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
> 
> {code}
>  
>    *Testing locally*
>                In order to test the Azure CosmosDB as a DocumentStore 
> locally, install the emulator from 
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
> start it locally. Set the endpoint and master key under *yarn-site.xml* as 
> mentioned above and run any example job like DistributedShell etc. Later you 
> can check the data explorer UI of Azure CosmosDB locally to query the 
> documents or even launch the *TimelineReader* locally and fetch/query the 
> data from REST API's.   
>                    
>       



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Description: 
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.documentstore.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      This is similar to HBase *TimelineSchemaCreator* the 
following command needs to be executed once for setting up the database and 
collections for storing documents.
{code:java}
hadoop 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.store-type  
   COSMOS_DB



   yarn.timeline-service.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally and fetch/query the data from REST 
API's.   
                   

      

  was:
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.documentstore.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      This is similar to HBase *TimelineSchemaCreator* the 
following command needs to be executed once for setting up the database and 
collections for storing documents.
{code:java}
hadoop 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.store-type  
   COSMOS_DB



   yarn.timeline-service.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Attachment: (was: YARN-9016.001.patch)

> DocumentStore as a backend for ATSv2
> 
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
>
> h1. Document Store for ATSv2
>                The Document Store for ATSv2 is a framework for plugging in 
> any Document Store as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
> ElasticSearch etc.
>  * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
> MongoDB etc by just adding new configurations properties and writing Document 
> Store reader and writer clients.
>  * Currently has support for CosmosDB.
>  * All writes are Async and buffered, latest document would be flushed to the 
> store either if the document buffer gets full or periodically at every flush 
> interval in background without adding any additional latency to the running 
> jobs..
>  * All the REST API's of Timeline Reader Server are supported.
> h4.  
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.writer.class
>  
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
> 
> 
>yarn.timeline-service.reader.class 
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
> 
>  
>yarn.timeline-service.documentstore.db-name   
>YOUR_DATABASE_NAME  
> {code}
> h3. *Creating DB and Collections for storing documents*
>                       This is similar to HBase *TimelineSchemaCreator* the 
> following command needs to be executed once for setting up the database and 
> collections for storing documents.
> {code:java}
> hadoop 
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3.  *Azure CosmosDB* 
>        To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
> properties under *yarn-site.xml* is required..
> {code:java}
> 
>   
>yarn.timeline-service.store-type  
>COSMOS_DB
> 
> 
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
> 
> 
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
> 
> {code}
>  
>    *Testing locally*
>                In order to test the Azure CosmosDB as a DocumentStore 
> locally, install the emulator from 
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
> start it locally. Set the endpoint and master key under *yarn-site.xml* as 
> mentioned above and run any example job like DistributedShell etc. Later you 
> can check the data explorer UI of Azure CosmosDB locally to query the 
> documents or even launch the *TimelineReader* locally and fetch/query the 
> data from REST API's.   
>                    
>       



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Attachment: (was: YARN-9016.001.patch)

> DocumentStore as a backend for ATSv2
> 
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
>
> h1. Document Store for ATSv2
>                The Document Store for ATSv2 is a framework for plugging in 
> any Document Store as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
> ElasticSearch etc.
>  * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
> MongoDB etc by just adding new configurations properties and writing Document 
> Store reader and writer clients.
>  * Currently has support for CosmosDB.
>  * All writes are Async and buffered, latest document would be flushed to the 
> store either if the document buffer gets full or periodically at every flush 
> interval in background without adding any additional latency to the running 
> jobs..
>  * All the REST API's of Timeline Reader Server are supported.
> h4.  
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.writer.class
>  
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
> 
> 
>yarn.timeline-service.reader.class 
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
> 
>  
>yarn.timeline-service.documentstore.db-name   
>YOUR_DATABASE_NAME  
> {code}
> h3. *Creating DB and Collections for storing documents*
>                       This is similar to HBase *TimelineSchemaCreator* the 
> following command needs to be executed once for setting up the database and 
> collections for storing documents.
> {code:java}
> hadoop 
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3.  *Azure CosmosDB* 
>        To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
> properties under *yarn-site.xml* is required..
> {code:java}
> 
>   
>yarn.timeline-service.store-type  
>COSMOS_DB
> 
> 
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
> 
> 
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
> 
> {code}
>  
>    *Testing locally*
>                In order to test the Azure CosmosDB as a DocumentStore 
> locally, install the emulator from 
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
> start it locally. Set the endpoint and master key under *yarn-site.xml* as 
> mentioned above and run any example job like DistributedShell etc. Later you 
> can check the data explorer UI of Azure CosmosDB locally to query the 
> documents or even launch the *TimelineReader* locally and fetch/query the 
> data from REST API's.   
>                    
>       



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Attachment: YARN-9016.001.patch

> DocumentStore as a backend for ATSv2
> 
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
>
> h1. Document Store for ATSv2
>                The Document Store for ATSv2 is a framework for plugging in 
> any Document Store as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
> ElasticSearch etc.
>  * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
> MongoDB etc by just adding new configurations properties and writing Document 
> Store reader and writer clients.
>  * Currently has support for CosmosDB.
>  * All writes are Async and buffered, latest document would be flushed to the 
> store either if the document buffer gets full or periodically at every flush 
> interval in background without adding any additional latency to the running 
> jobs..
>  * All the REST API's of Timeline Reader Server are supported.
> h4.  
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.writer.class
>  
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
> 
> 
>yarn.timeline-service.reader.class 
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
> 
>  
>yarn.timeline-service.documentstore.db-name   
>YOUR_DATABASE_NAME  
> {code}
> h3. *Creating DB and Collections for storing documents*
>                       This is similar to HBase *TimelineSchemaCreator* the 
> following command needs to be executed once for setting up the database and 
> collections for storing documents.
> {code:java}
> hadoop 
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3.  *Azure CosmosDB* 
>        To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
> properties under *yarn-site.xml* is required..
> {code:java}
> 
>   
>yarn.timeline-service.store-type  
>COSMOS_DB
> 
> 
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
> 
> 
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
> 
> {code}
>  
>    *Testing locally*
>                In order to test the Azure CosmosDB as a DocumentStore 
> locally, install the emulator from 
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
> start it locally. Set the endpoint and master key under *yarn-site.xml* as 
> mentioned above and run any example job like DistributedShell etc. Later you 
> can check the data explorer UI of Azure CosmosDB locally to query the 
> documents or even launch the *TimelineReader* locally and fetch/query the 
> data from REST API's.   
>                    
>       



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Description: 
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.documentstore.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      This is similar to HBase *TimelineSchemaCreator* the 
following command needs to be executed once for setting up the database and 
collections for storing documents.
{code:java}
hadoop 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.store-type  
   COSMOS_DB



   yarn.timeline-service.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally and fetch/query the data from REST 
API's.   
                   

      

  was:
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class</name>
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class</name> 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.documentstore.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      This is similar to HBase *TimelineSchemaCreator* the 
following command needs to be executed once for setting up the database and 
collections for storing documents.
{code:java}
hadoop 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.store-type  
   COSMOS_DB



   yarn.timeline-service.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Description: 
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class</name>
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class</name> 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.documentstore.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      This is similar to HBase *TimelineSchemaCreator* the 
following command needs to be executed once for setting up the database and 
collections for storing documents.
{code:java}
hadoop 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.store-type  
   COSMOS_DB



   yarn.timeline-service.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally and fetch/query the data from REST 
API's.   
                   

      

  was:
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.documentstore.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      **This is similar to HBase *TimelineSchemaCreator* the 
following command needs to be executed once for setting up the database and 
collections for storing documents.
{code:java}
hadoop 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.store-type  
   COSMOS_DB



   yarn.timeline-service.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yar

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Description: 
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*
{code:java}


 yarn.timeline-service.writer.class
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.documentstore.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      **This is similar to HBase *TimelineSchemaCreator* the 
following command needs to be executed once for setting up the database and 
collections for storing documents.
{code:java}
hadoop 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}

  
   yarn.timeline-service.store-type  
   COSMOS_DB



   yarn.timeline-service.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally and fetch/query the data from REST 
API's.   
                   

      

  was:
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*


{code:java}


 yarn.timeline-service.writer.class
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.documentstore.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      **This is similar to HBase *TimelineSchemaCreator* the 
following command needs to be executed once for setting up the database and 
collections for storing documents.
{code:java}
hadoop 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}


  
   yarn.timeline-service.store-type  
   COSMOS_DB



   yarn.timeline-service.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 

 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* 

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Attachment: (was: YARN-9016.patch)

> DocumentStore as a backend for ATSv2
> 
>
> Key: YARN-9016
> URL: https://issues.apache.org/jira/browse/YARN-9016
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: ATSv2
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
>
> h1. Document Store for ATSv2
>                The Document Store for ATSv2 is a framework for plugging in 
> any Document Store as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
> ElasticSearch etc.
>  * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
> MongoDB etc by just adding new configurations properties and writing Document 
> Store reader and writer clients.
>  * Currently has support for CosmosDB.
>  * All writes are Async and buffered, latest document would be flushed to the 
> store either if the document buffer gets full or periodically at every flush 
> interval in background without adding any additional latency to the running 
> jobs..
>  * All the REST API's of Timeline Reader Server are supported.
> h4.  
> *How to enable?*
> Add the flowing properties under *yarn-site.xml*
> {code:java}
> 
> 
>  yarn.timeline-service.writer.class
>  
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl
> 
> 
>yarn.timeline-service.reader.class 
> org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl
> 
>  
>yarn.timeline-service.documentstore.db-name   
>YOUR_DATABASE_NAME  
> {code}
> h3. *Creating DB and Collections for storing documents*
>                       **This is similar to HBase *TimelineSchemaCreator* the 
> following command needs to be executed once for setting up the database and 
> collections for storing documents.
> {code:java}
> hadoop 
> org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
> {code}
> h3.  *Azure CosmosDB* 
>        To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
> properties under *yarn-site.xml* is required..
> {code:java}
> 
>   
>yarn.timeline-service.store-type  
>COSMOS_DB
> 
> 
>yarn.timeline-service.cosmos-db.endpoint
>http://YOUR_AZURE_COSMOS_DB_URL:443/
> 
> 
>yarn.timeline-service.cosmos-db.masterkey
>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
> 
> {code}
>  
>    *Testing locally*
>                In order to test the Azure CosmosDB as a DocumentStore 
> locally, install the emulator from 
> [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
> start it locally. Set the endpoint and master key under *yarn-site.xml* as 
> mentioned above and run any example job like DistributedShell etc. Later you 
> can check the data explorer UI of Azure CosmosDB locally to query the 
> documents or even launch the *TimelineReader* locally and fetch/query the 
> data from REST API's.   
>                    
>       



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-9016:

Description: 
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*


{code:java}


 yarn.timeline-service.writer.class
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.documentstore.db-name   
   YOUR_DATABASE_NAME  
{code}
h3. *Creating DB and Collections for storing documents*

                      **This is similar to HBase *TimelineSchemaCreator* the 
following command needs to be executed once for setting up the database and 
collections for storing documents.
{code:java}
hadoop 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
h3.  *Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..
{code:java}


  
   yarn.timeline-service.store-type  
   COSMOS_DB



   yarn.timeline-service.cosmos-db.endpoint
   http://YOUR_AZURE_COSMOS_DB_URL:443/



   yarn.timeline-service.cosmos-db.masterkey
   YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL

{code}
 

 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally and fetch/query the data from REST 
API's.   
                   

      

  was:
h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*

 
{code:java}


 yarn.timeline-service.writer.class
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.documentstore.db-name   
   YOUR_DATABASE_NAME  

 {code}
h5.  
h3. *Creating DB and Collections for storing documents*

           *           *This is similar to HBase *TimelineSchemaCreator* the 
following command needs to be executed once for setting up the database and 
collections for storing documents.
{code:java}
hadoop 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
h3.  

*Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..

 

{{
 
 yarn.timeline-service.store-type
 COSMOS_DB
 }}
 {{
 yarn.timeline-service.cosmos-db.endpoint
 
[http://YOUR_AZURE_COSMOS_DB_URL:443/|http://your_azure_cosmos_db_url:443/]
 }}
 {{
 yarn.timeline-service.cosmos-db.masterkey
 YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
 }}

 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoin

[jira] [Created] (YARN-9016) DocumentStore as a backend for ATSv2

[Sushil Ks (JIRA)]

Sushil Ks created YARN-9016:
---

 Summary: DocumentStore as a backend for ATSv2
 Key: YARN-9016
 URL: https://issues.apache.org/jira/browse/YARN-9016
 Project: Hadoop YARN
  Issue Type: New Feature
  Components: ATSv2
Reporter: Sushil Ks
Assignee: Sushil Ks


h1. Document Store for ATSv2

               The Document Store for ATSv2 is a framework for plugging in any 
Document Store as a backend for ATSv2 i.e Azure CosmosDB , MongoDB, 
ElasticSearch etc.
 * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, 
MongoDB etc by just adding new configurations properties and writing Document 
Store reader and writer clients.
 * Currently has support for CosmosDB.
 * All writes are Async and buffered, latest document would be flushed to the 
store either if the document buffer gets full or periodically at every flush 
interval in background without adding any additional latency to the running 
jobs..
 * All the REST API's of Timeline Reader Server are supported.

h4.  

*How to enable?*

Add the flowing properties under *yarn-site.xml*

 
{code:java}


 yarn.timeline-service.writer.class
 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl



   yarn.timeline-service.reader.class 
org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl


 
   yarn.timeline-service.documentstore.db-name   
   YOUR_DATABASE_NAME  

 {code}
h5.  
h3. *Creating DB and Collections for storing documents*

           *           *This is similar to HBase *TimelineSchemaCreator* the 
following command needs to be executed once for setting up the database and 
collections for storing documents.
{code:java}
hadoop 
org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator
{code}
h3.  

*Azure CosmosDB* 

       To use Azure CosmosDB as a DocumentStore for ATSv2, the additional 
properties under *yarn-site.xml* is required..

 

{{
 
 yarn.timeline-service.store-type
 COSMOS_DB
 }}
 {{
 yarn.timeline-service.cosmos-db.endpoint
 
[http://YOUR_AZURE_COSMOS_DB_URL:443/|http://your_azure_cosmos_db_url:443/]
 }}
 {{
 yarn.timeline-service.cosmos-db.masterkey
 YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL
 }}

 
   *Testing locally*
               In order to test the Azure CosmosDB as a DocumentStore locally, 
install the emulator from 
[here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and 
start it locally. Set the endpoint and master key under *yarn-site.xml* as 
mentioned above and run any example job like DistributedShell etc. Later you 
can check the data explorer UI of Azure CosmosDB locally to query the documents 
or even launch the *TimelineReader* locally and fetch/query the data from REST 
API's.   
                   

      



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8856) TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663296#comment-16663296
 ] 

Sushil Ks commented on YARN-8856:
-

Hi [~rohithsharma], Kindly review the patch.

> TestTimelineReaderWebServicesHBaseStorage tests failing with 
> NoClassDefFoundError
> -
>
> Key: YARN-8856
> URL: https://issues.apache.org/jira/browse/YARN-8856
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Jason Lowe
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-8856.001.patch
>
>
> TestTimelineReaderWebServicesHBaseStorage has been failing in nightly builds 
> with NoClassDefFoundError in the tests.  Sample error and stacktrace to 
> follow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8856) TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-8856:

Attachment: (was: YARN-8856.001.patch)

> TestTimelineReaderWebServicesHBaseStorage tests failing with 
> NoClassDefFoundError
> -
>
> Key: YARN-8856
> URL: https://issues.apache.org/jira/browse/YARN-8856
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Jason Lowe
>Assignee: Sushil Ks
>Priority: Major
>
> TestTimelineReaderWebServicesHBaseStorage has been failing in nightly builds 
> with NoClassDefFoundError in the tests.  Sample error and stacktrace to 
> follow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8856) TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16646798#comment-16646798
 ] 

Sushil Ks commented on YARN-8856:
-

HI [~rohithsharma], as discussed in our weekly ATSv2 call, will be mocking the 
metrics in order to skip loading metrics while running hbase tests.

> TestTimelineReaderWebServicesHBaseStorage tests failing with 
> NoClassDefFoundError
> -
>
> Key: YARN-8856
> URL: https://issues.apache.org/jira/browse/YARN-8856
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Jason Lowe
>Assignee: Sushil Ks
>Priority: Major
>
> TestTimelineReaderWebServicesHBaseStorage has been failing in nightly builds 
> with NoClassDefFoundError in the tests.  Sample error and stacktrace to 
> follow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-8856) TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks reassigned YARN-8856:
---

Assignee: Sushil Ks

> TestTimelineReaderWebServicesHBaseStorage tests failing with 
> NoClassDefFoundError
> -
>
> Key: YARN-8856
> URL: https://issues.apache.org/jira/browse/YARN-8856
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Jason Lowe
>Assignee: Sushil Ks
>Priority: Major
>
> TestTimelineReaderWebServicesHBaseStorage has been failing in nightly builds 
> with NoClassDefFoundError in the tests.  Sample error and stacktrace to 
> follow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8270) Adding JMX Metrics for Timeline Collector and Reader

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-8270:

Description: This Jira is for emitting JMX Metrics for ATS v2 Timeline 
Collector and Timeline Reader, basically for Timeline Collector it tries to 
capture success and failure latencies for *putEntities* and *putEntitiesAsync*  
from *TimelineCollectorWebService* , similarly all the API's success and 
failure latencies for fetching TimelineEntities from 
*TimelineReaderWebServices*. This would actually help in monitoring and 
measuring performance for ATSv2 at scale.  (was: This Jira is for emitting JMX 
Metrics for ATS v2 Timeline Collector and Timeline Reader, basically for 
Timeline Collector it tries to capture success, failure and latencies for 
*putEntities* and *putEntitiesAsync*  from *TimelineCollectorWebService* and 
all the API's success, failure and latencies for fetching TimelineEntities from 
*TimelineReaderWebServices*. This would actually help in monitoring and 
measuring performance for ATSv2 at scale.)

> Adding JMX Metrics for Timeline Collector and Reader
> 
>
> Key: YARN-8270
> URL: https://issues.apache.org/jira/browse/YARN-8270
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2, timelineserver
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-8270.001.patch, YARN-8270.002.patch
>
>
> This Jira is for emitting JMX Metrics for ATS v2 Timeline Collector and 
> Timeline Reader, basically for Timeline Collector it tries to capture success 
> and failure latencies for *putEntities* and *putEntitiesAsync*  from 
> *TimelineCollectorWebService* , similarly all the API's success and failure 
> latencies for fetching TimelineEntities from *TimelineReaderWebServices*. 
> This would actually help in monitoring and measuring performance for ATSv2 at 
> scale.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8270) Adding JMX Metrics for Timeline Collector and Reader

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-8270:

Attachment: (was: YARN-8270.002.patch)

> Adding JMX Metrics for Timeline Collector and Reader
> 
>
> Key: YARN-8270
> URL: https://issues.apache.org/jira/browse/YARN-8270
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2, timelineserver
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-8270.001.patch
>
>
> This Jira is for emitting JMX Metrics for ATS v2 Timeline Collector and 
> Timeline Reader, basically for Timeline Collector it tries to capture 
> success, failure and latencies for *putEntities* and *putEntitiesAsync*  from 
> *TimelineCollectorWebService* and all the API's success, failure and 
> latencies for fetching TimelineEntities from *TimelineReaderWebServices*. 
> This would actually help in monitoring and measuring performance for ATSv2 at 
> scale.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8270) Adding JMX Metrics for Timeline Collector and Reader

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522094#comment-16522094
 ] 

Sushil Ks commented on YARN-8270:
-

Thanks [~haibochen] for reviewing, I have resolved your comments in the new 
patch, kindly review it.

> Adding JMX Metrics for Timeline Collector and Reader
> 
>
> Key: YARN-8270
> URL: https://issues.apache.org/jira/browse/YARN-8270
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2, timelineserver
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-8270.001.patch
>
>
> This Jira is for emitting JMX Metrics for ATS v2 Timeline Collector and 
> Timeline Reader, basically for Timeline Collector it tries to capture 
> success, failure and latencies for *putEntities* and *putEntitiesAsync*  from 
> *TimelineCollectorWebService* and all the API's success, failure and 
> latencies for fetching TimelineEntities from *TimelineReaderWebServices*. 
> This would actually help in monitoring and measuring performance for ATSv2 at 
> scale.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8270) Adding JMX Metrics for Timeline Collector and Reader

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-8270:

Attachment: (was: YARN-8270.001.patch)

> Adding JMX Metrics for Timeline Collector and Reader
> 
>
> Key: YARN-8270
> URL: https://issues.apache.org/jira/browse/YARN-8270
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2, timelineserver
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
>
> This Jira is for emitting JMX Metrics for ATS v2 Timeline Collector and 
> Timeline Reader, basically for Timeline Collector it tries to capture 
> success, failure and latencies for *putEntities* and *putEntitiesAsync*  from 
> *TimelineCollectorWebService* and all the API's success, failure and 
> latencies for fetching TimelineEntities from *TimelineReaderWebServices*. 
> This would actually help in monitoring and measuring performance for ATSv2 at 
> scale.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8270) Adding JMX Metrics for Timeline Collector and Reader

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-8270:

Attachment: (was: YARN-8270.001.patch)

> Adding JMX Metrics for Timeline Collector and Reader
> 
>
> Key: YARN-8270
> URL: https://issues.apache.org/jira/browse/YARN-8270
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2, timelineserver
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
>
> This Jira is for emitting JMX Metrics for ATS v2 Timeline Collector and 
> Timeline Reader, basically for Timeline Collector it tries to capture 
> success, failure and latencies for *putEntities* and *putEntitiesAsync*  from 
> *TimelineCollectorWebService* and all the API's success, failure and 
> latencies for fetching TimelineEntities from *TimelineReaderWebServices*. 
> This would actually help in monitoring and measuring performance for ATSv2 at 
> scale.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8270) Adding JMX Metrics for Timeline Collector and Reader

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-8270:

Attachment: YARN-8270.001.patch

> Adding JMX Metrics for Timeline Collector and Reader
> 
>
> Key: YARN-8270
> URL: https://issues.apache.org/jira/browse/YARN-8270
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2, timelineserver
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-8270.001.patch
>
>
> This Jira is for emitting JMX Metrics for ATS v2 Timeline Collector and 
> Timeline Reader, basically for Timeline Collector it tries to capture 
> success, failure and latencies for *putEntities* and *putEntitiesAsync*  from 
> *TimelineCollectorWebService* and all the API's success, failure and 
> latencies for fetching TimelineEntities from *TimelineReaderWebServices*. 
> This would actually help in monitoring and measuring performance for ATSv2 at 
> scale.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8270) Adding JMX Metrics for Timeline Collector and Reader

[Sushil Ks (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sushil Ks updated YARN-8270:

Attachment: (was: YARN-8270.001.patch)

> Adding JMX Metrics for Timeline Collector and Reader
> 
>
> Key: YARN-8270
> URL: https://issues.apache.org/jira/browse/YARN-8270
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2, timelineserver
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
>
> This Jira is for emitting JMX Metrics for ATS v2 Timeline Collector and 
> Timeline Reader, basically for Timeline Collector it tries to capture 
> success, failure and latencies for *putEntities* and *putEntitiesAsync*  from 
> *TimelineCollectorWebService* and all the API's success, failure and 
> latencies for fetching TimelineEntities from *TimelineReaderWebServices*. 
> This would actually help in monitoring and measuring performance for ATSv2 at 
> scale.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-8270) Adding JMX Metrics for Timeline Collector and Reader

[Sushil Ks (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16480124#comment-16480124
 ] 

Sushil Ks edited comment on YARN-8270 at 6/1/18 3:31 PM:
-

Hi Haibo, thanks for reviewing please find my comments below,

1) Yes the collector metric is an aggregate for all collectors on the same node 
i.e on all NM and RM
 2) Do you mean separate metrics by *TimelineEntityType*? like 
YARN_APPLICATION, YARN_APPLICATION_ATTEMPT etc.
 3) My bad its actually private, sure makes sense will make it *update*Metrics()

4) Will add double check inside getInstance() for thread safety.


was (Author: sushil-k-s):
Hi Haibo, thanks for reviewing please find my comments below,

1) Yes the collector metric is an aggregate for all collectors on the same node 
i.e on all NM and RM
2) Do you mean separate metrics by *TimelineEntityType*? like YARN_APPLICATION, 
YARN_APPLICATION_ATTEMPT etc.
3) My bad its actually private, sure makes sense will make it *update*Metrics()

4) I made use of *double checked locking pattern*  for getInstance() is it 
still not thread-safe?

> Adding JMX Metrics for Timeline Collector and Reader
> 
>
> Key: YARN-8270
> URL: https://issues.apache.org/jira/browse/YARN-8270
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: ATSv2, timelineserver
>Reporter: Sushil Ks
>Assignee: Sushil Ks
>Priority: Major
> Attachments: YARN-8270.001.patch
>
>
> This Jira is for emitting JMX Metrics for ATS v2 Timeline Collector and 
> Timeline Reader, basically for Timeline Collector it tries to capture 
> success, failure and latencies for *putEntities* and *putEntitiesAsync*  from 
> *TimelineCollectorWebService* and all the API's success, failure and 
> latencies for fetching TimelineEntities from *TimelineReaderWebServices*. 
> This would actually help in monitoring and measuring performance for ATSv2 at 
> scale.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org