[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on metrics and log

2022-11-06 Thread Madhan Neethiraj (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-4330:

Fix Version/s: (was: 2.3.0)

> Add Kafka topics lag information on metrics and log
> ---
>
> Key: ATLAS-4330
> URL: https://issues.apache.org/jira/browse/ATLAS-4330
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.1.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
>  Labels: kafka, metrics, topic-lag
> Fix For: 3.0.0
>
> Attachments: image-2021-06-07-15-03-28-166.png
>
>
> In metrics page, in addition to other Notification details. Lag information 
> of kafka consumer topics will be useful to check the current backed up 
> messages in Atlas: 
> !image-2021-06-07-15-03-28-166.png!
> It will also be useful to print the lag information periodically in atlas 
> application.log on a frequency that can be made configurable (print lag every 
> 1 hour, 5min or so).
> *atlas.notification.consumer.topic.report.frequency.seconds=60*
> {code:java}
> 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: 
> topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 
> 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: 
> topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on metrics and log

2021-08-24 Thread Madhan Neethiraj (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Madhan Neethiraj updated ATLAS-4330:

Fix Version/s: (was: 2.2.0)
   2.3.0

> Add Kafka topics lag information on metrics and log
> ---
>
> Key: ATLAS-4330
> URL: https://issues.apache.org/jira/browse/ATLAS-4330
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.1.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
>  Labels: kafka, metrics, topic-lag
> Fix For: 3.0.0, 2.3.0
>
> Attachments: image-2021-06-07-15-03-28-166.png
>
>
> In metrics page, in addition to other Notification details. Lag information 
> of kafka consumer topics will be useful to check the current backed up 
> messages in Atlas: 
> !image-2021-06-07-15-03-28-166.png!
> It will also be useful to print the lag information periodically in atlas 
> application.log on a frequency that can be made configurable (print lag every 
> 1 hour, 5min or so).
> *atlas.notification.consumer.topic.report.frequency.seconds=60*
> {code:java}
> 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: 
> topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 
> 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: 
> topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on metrics and log

2021-06-07 Thread Sarath Subramanian (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-4330:
--
Labels: kafka metrics topic-lag  (was: )

> Add Kafka topics lag information on metrics and log
> ---
>
> Key: ATLAS-4330
> URL: https://issues.apache.org/jira/browse/ATLAS-4330
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.1.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
>  Labels: kafka, metrics, topic-lag
> Fix For: 3.0.0, 2.2.0
>
> Attachments: image-2021-06-07-15-03-28-166.png
>
>
> In metrics page, in addition to other Notification details. Lag information 
> of kafka consumer topics will be useful to check the current backed up 
> messages in Atlas: 
> !image-2021-06-07-15-03-28-166.png!
> It will also be useful to print the lag information periodically in atlas 
> application.log on a frequency that can be made configurable (print lag every 
> 1 hour, 5min or so).
> *atlas.notification.consumer.topic.report.frequency.seconds=60*
> {code:java}
> 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: 
> topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 
> 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: 
> topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on metrics and log

2021-06-07 Thread Sarath Subramanian (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-4330:
--
Description: 
In metrics page, in addition to other Notification details. Lag information of 
kafka consumer topics will be useful to check the current backed up messages in 
Atlas: 

!image-2021-06-07-15-03-28-166.png!

It will also be useful to print the lag information periodically in atlas 
application.log on a frequency that can be made configurable (print lag every 1 
hour, 5min or so).

*atlas.notification.consumer.topic.report.frequency.seconds=60*
{code:java}
2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: 
topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 
2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0, 
currentOffset=1118, endOffset=1118, lag=0
{code}

  was:
In metrics page, in addition to other Notification details. Lag information of 
kafka consumer topics will be useful to check the current backup up messages in 
Atlas: 

!image-2021-06-07-15-03-28-166.png!

It will also be useful to print the lag information periodically in atlas 
application.log on a frequency that can be made configurable (print lag every 1 
hour, 5min or so).

*atlas.notification.consumer.topic.report.frequency.seconds=60*
{code:java}
2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: 
topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 
2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0, 
currentOffset=1118, endOffset=1118, lag=0
{code}


> Add Kafka topics lag information on metrics and log
> ---
>
> Key: ATLAS-4330
> URL: https://issues.apache.org/jira/browse/ATLAS-4330
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.1.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: image-2021-06-07-15-03-28-166.png
>
>
> In metrics page, in addition to other Notification details. Lag information 
> of kafka consumer topics will be useful to check the current backed up 
> messages in Atlas: 
> !image-2021-06-07-15-03-28-166.png!
> It will also be useful to print the lag information periodically in atlas 
> application.log on a frequency that can be made configurable (print lag every 
> 1 hour, 5min or so).
> *atlas.notification.consumer.topic.report.frequency.seconds=60*
> {code:java}
> 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: 
> topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 
> 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: 
> topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on metrics and log

2021-06-07 Thread Sarath Subramanian (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-4330:
--
Description: 
In metrics page, in addition to other Notification details. Lag information of 
kafka consumer topics will be useful to check the current backup up messages in 
Atlas: 

!image-2021-06-07-15-03-28-166.png!

It will also be useful to print the lag information periodically in atlas 
application.log on a frequency that can be made configurable (print lag every 1 
hour, 5min or so).

*atlas.notification.consumer.topic.report.frequency.seconds=60*
{code:java}
2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: 
topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 
2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0, 
currentOffset=1118, endOffset=1118, lag=0
{code}

  was:
In metrics page, in addition to other Notification details. Lag information of 
kafka consumer topics will be useful to check the current backup up messages in 
Atlas: 

!image-2021-06-07-15-03-28-166.png!

It will also be useful to print the lag information periodically in atlas 
application.log on a frequency that can be made configurable (print lag every 1 
hour, 5min or so).

 

*atlas.notification.consumer.topic.report.frequency.seconds=60*

 

 
{code:java}
2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: 
topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 
2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0, 
currentOffset=1118, endOffset=1118, lag=0
{code}


> Add Kafka topics lag information on metrics and log
> ---
>
> Key: ATLAS-4330
> URL: https://issues.apache.org/jira/browse/ATLAS-4330
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.1.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: image-2021-06-07-15-03-28-166.png
>
>
> In metrics page, in addition to other Notification details. Lag information 
> of kafka consumer topics will be useful to check the current backup up 
> messages in Atlas: 
> !image-2021-06-07-15-03-28-166.png!
> It will also be useful to print the lag information periodically in atlas 
> application.log on a frequency that can be made configurable (print lag every 
> 1 hour, 5min or so).
> *atlas.notification.consumer.topic.report.frequency.seconds=60*
> {code:java}
> 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: 
> topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 
> 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: 
> topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on metrics and log

2021-06-07 Thread Sarath Subramanian (Jira) 


 [ 
https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sarath Subramanian updated ATLAS-4330:
--
Description: 
In metrics page, in addition to other Notification details. Lag information of 
kafka consumer topics will be useful to check the current backup up messages in 
Atlas: 

!image-2021-06-07-15-03-28-166.png!

It will also be useful to print the lag information periodically in atlas 
application.log on a frequency that can be made configurable (print lag every 1 
hour, 5min or so).

 

*atlas.notification.consumer.topic.report.frequency.seconds=60*

 

 
{code:java}
2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: 
topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 
2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0, 
currentOffset=1118, endOffset=1118, lag=0
{code}

  was:
In metrics page, in addition to other Notification details. Lag information of 
kafka consumer topics will be useful to check the current backup up messages in 
Atlas:

 

!image-2021-06-07-15-03-28-166.png!


> Add Kafka topics lag information on metrics and log
> ---
>
> Key: ATLAS-4330
> URL: https://issues.apache.org/jira/browse/ATLAS-4330
> Project: Atlas
>  Issue Type: Improvement
>  Components:  atlas-core
>Affects Versions: 2.1.0
>Reporter: Sarath Subramanian
>Assignee: Sarath Subramanian
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: image-2021-06-07-15-03-28-166.png
>
>
> In metrics page, in addition to other Notification details. Lag information 
> of kafka consumer topics will be useful to check the current backup up 
> messages in Atlas: 
> !image-2021-06-07-15-03-28-166.png!
> It will also be useful to print the lag information periodically in atlas 
> application.log on a frequency that can be made configurable (print lag every 
> 1 hour, 5min or so).
>  
> *atlas.notification.consumer.topic.report.frequency.seconds=60*
>  
>  
> {code:java}
> 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: 
> topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 
> 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: 
> topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)