[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on metrics and log
[ https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated ATLAS-4330: Fix Version/s: (was: 2.3.0) > Add Kafka topics lag information on metrics and log > --- > > Key: ATLAS-4330 > URL: https://issues.apache.org/jira/browse/ATLAS-4330 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Sarath Subramanian >Assignee: Sarath Subramanian >Priority: Major > Labels: kafka, metrics, topic-lag > Fix For: 3.0.0 > > Attachments: image-2021-06-07-15-03-28-166.png > > > In metrics page, in addition to other Notification details. Lag information > of kafka consumer topics will be useful to check the current backed up > messages in Atlas: > !image-2021-06-07-15-03-28-166.png! > It will also be useful to print the lag information periodically in atlas > application.log on a frequency that can be made configurable (print lag every > 1 hour, 5min or so). > *atlas.notification.consumer.topic.report.frequency.seconds=60* > {code:java} > 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: > topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 > 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: > topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0 > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on metrics and log
[ https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Madhan Neethiraj updated ATLAS-4330: Fix Version/s: (was: 2.2.0) 2.3.0 > Add Kafka topics lag information on metrics and log > --- > > Key: ATLAS-4330 > URL: https://issues.apache.org/jira/browse/ATLAS-4330 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Sarath Subramanian >Assignee: Sarath Subramanian >Priority: Major > Labels: kafka, metrics, topic-lag > Fix For: 3.0.0, 2.3.0 > > Attachments: image-2021-06-07-15-03-28-166.png > > > In metrics page, in addition to other Notification details. Lag information > of kafka consumer topics will be useful to check the current backed up > messages in Atlas: > !image-2021-06-07-15-03-28-166.png! > It will also be useful to print the lag information periodically in atlas > application.log on a frequency that can be made configurable (print lag every > 1 hour, 5min or so). > *atlas.notification.consumer.topic.report.frequency.seconds=60* > {code:java} > 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: > topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 > 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: > topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on metrics and log
[ https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4330: -- Labels: kafka metrics topic-lag (was: ) > Add Kafka topics lag information on metrics and log > --- > > Key: ATLAS-4330 > URL: https://issues.apache.org/jira/browse/ATLAS-4330 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Sarath Subramanian >Assignee: Sarath Subramanian >Priority: Major > Labels: kafka, metrics, topic-lag > Fix For: 3.0.0, 2.2.0 > > Attachments: image-2021-06-07-15-03-28-166.png > > > In metrics page, in addition to other Notification details. Lag information > of kafka consumer topics will be useful to check the current backed up > messages in Atlas: > !image-2021-06-07-15-03-28-166.png! > It will also be useful to print the lag information periodically in atlas > application.log on a frequency that can be made configurable (print lag every > 1 hour, 5min or so). > *atlas.notification.consumer.topic.report.frequency.seconds=60* > {code:java} > 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: > topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 > 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: > topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on metrics and log
[ https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4330: -- Description: In metrics page, in addition to other Notification details. Lag information of kafka consumer topics will be useful to check the current backed up messages in Atlas: !image-2021-06-07-15-03-28-166.png! It will also be useful to print the lag information periodically in atlas application.log on a frequency that can be made configurable (print lag every 1 hour, 5min or so). *atlas.notification.consumer.topic.report.frequency.seconds=60* {code:java} 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0 {code} was: In metrics page, in addition to other Notification details. Lag information of kafka consumer topics will be useful to check the current backup up messages in Atlas: !image-2021-06-07-15-03-28-166.png! It will also be useful to print the lag information periodically in atlas application.log on a frequency that can be made configurable (print lag every 1 hour, 5min or so). *atlas.notification.consumer.topic.report.frequency.seconds=60* {code:java} 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0 {code} > Add Kafka topics lag information on metrics and log > --- > > Key: ATLAS-4330 > URL: https://issues.apache.org/jira/browse/ATLAS-4330 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Sarath Subramanian >Assignee: Sarath Subramanian >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: image-2021-06-07-15-03-28-166.png > > > In metrics page, in addition to other Notification details. Lag information > of kafka consumer topics will be useful to check the current backed up > messages in Atlas: > !image-2021-06-07-15-03-28-166.png! > It will also be useful to print the lag information periodically in atlas > application.log on a frequency that can be made configurable (print lag every > 1 hour, 5min or so). > *atlas.notification.consumer.topic.report.frequency.seconds=60* > {code:java} > 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: > topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 > 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: > topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on metrics and log
[ https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4330: -- Description: In metrics page, in addition to other Notification details. Lag information of kafka consumer topics will be useful to check the current backup up messages in Atlas: !image-2021-06-07-15-03-28-166.png! It will also be useful to print the lag information periodically in atlas application.log on a frequency that can be made configurable (print lag every 1 hour, 5min or so). *atlas.notification.consumer.topic.report.frequency.seconds=60* {code:java} 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0 {code} was: In metrics page, in addition to other Notification details. Lag information of kafka consumer topics will be useful to check the current backup up messages in Atlas: !image-2021-06-07-15-03-28-166.png! It will also be useful to print the lag information periodically in atlas application.log on a frequency that can be made configurable (print lag every 1 hour, 5min or so). *atlas.notification.consumer.topic.report.frequency.seconds=60* {code:java} 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0 {code} > Add Kafka topics lag information on metrics and log > --- > > Key: ATLAS-4330 > URL: https://issues.apache.org/jira/browse/ATLAS-4330 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Sarath Subramanian >Assignee: Sarath Subramanian >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: image-2021-06-07-15-03-28-166.png > > > In metrics page, in addition to other Notification details. Lag information > of kafka consumer topics will be useful to check the current backup up > messages in Atlas: > !image-2021-06-07-15-03-28-166.png! > It will also be useful to print the lag information periodically in atlas > application.log on a frequency that can be made configurable (print lag every > 1 hour, 5min or so). > *atlas.notification.consumer.topic.report.frequency.seconds=60* > {code:java} > 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: > topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 > 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: > topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-4330) Add Kafka topics lag information on metrics and log
[ https://issues.apache.org/jira/browse/ATLAS-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarath Subramanian updated ATLAS-4330: -- Description: In metrics page, in addition to other Notification details. Lag information of kafka consumer topics will be useful to check the current backup up messages in Atlas: !image-2021-06-07-15-03-28-166.png! It will also be useful to print the lag information periodically in atlas application.log on a frequency that can be made configurable (print lag every 1 hour, 5min or so). *atlas.notification.consumer.topic.report.frequency.seconds=60* {code:java} 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0 {code} was: In metrics page, in addition to other Notification details. Lag information of kafka consumer topics will be useful to check the current backup up messages in Atlas: !image-2021-06-07-15-03-28-166.png! > Add Kafka topics lag information on metrics and log > --- > > Key: ATLAS-4330 > URL: https://issues.apache.org/jira/browse/ATLAS-4330 > Project: Atlas > Issue Type: Improvement > Components: atlas-core >Affects Versions: 2.1.0 >Reporter: Sarath Subramanian >Assignee: Sarath Subramanian >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: image-2021-06-07-15-03-28-166.png > > > In metrics page, in addition to other Notification details. Lag information > of kafka consumer topics will be useful to check the current backup up > messages in Atlas: > !image-2021-06-07-15-03-28-166.png! > It will also be useful to print the lag information periodically in atlas > application.log on a frequency that can be made configurable (print lag every > 1 hour, 5min or so). > > *atlas.notification.consumer.topic.report.frequency.seconds=60* > > > {code:java} > 2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: > topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 > 2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: > topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)