[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17559154#comment-17559154 ] zhengsicheng commented on HBASE-26981: -- PR HBASE-27146 Prepare to merge to online version and run for a while > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: 20220624-144119.html, image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, > image-2022-06-24-14-48-26-451.png, image-2022-06-24-14-57-10-424.png, jstack, > rs_jstack_202206241458.log > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558378#comment-17558378 ] Duo Zhang commented on HBASE-26981: --- I can not recall the full logic of hbase-1.x but IIRC, we used to rely on zookeeper to test whether a table exists, or is disabled? On 2.x, all the traffic will go to meta. > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: 20220624-144119.html, image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, > image-2022-06-24-14-48-26-451.png, image-2022-06-24-14-57-10-424.png, jstack, > rs_jstack_202206241458.log > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558373#comment-17558373 ] zhengsicheng commented on HBASE-26981: -- [~zhangduo] The PR for HBASE-27146 does reduce the MetaCellComparator.compare PR is available > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: 20220624-144119.html, image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, > image-2022-06-24-14-48-26-451.png, image-2022-06-24-14-57-10-424.png, jstack, > rs_jstack_202206241458.log > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558371#comment-17558371 ] zhengsicheng commented on HBASE-26981: -- [~zhangduo] I did an extreme test, the client concurrently fetches a non-existing table with 90 threads, the CPU usage of hbase-1.1.6 is about 25%, and the usage of hbase-2.3.4 is more than 90%. Why do you do this test because the online business may have taken the table offline but the task of operating hbase has not stopped. But in this scenario, the CPU of hbase-1.1.6 is within the normal range, but the CPU usage of hbase-2.3.4 is too high. > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: 20220624-144119.html, image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, > image-2022-06-24-14-48-26-451.png, image-2022-06-24-14-57-10-424.png, jstack, > rs_jstack_202206241458.log > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558364#comment-17558364 ] zhengsicheng commented on HBASE-26981: -- top -H -p rs_pid !image-2022-06-24-14-57-10-424.png! new jstack [^rs_jstack_202206241458.log] > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: 20220624-144119.html, image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, > image-2022-06-24-14-48-26-451.png, image-2022-06-24-14-57-10-424.png, jstack > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558356#comment-17558356 ] Duo Zhang commented on HBASE-26981: --- What about the qps? Do we have higher qps then before? Checked the flame graph, the result is good. There is no CellUtil.cloneRow any more, and it seems that the CPU time spend in MetaCellComparator.compare is also reduced? So then you may need to use [~huaxiangsun]'s solution, use the meta read replica feature to distribute the load to multiple region servers. And out of interest, what is the typical work load on your cluister? Usually, hbase clients should cache the location locally, so they do not need to request meta too much. Thanks. > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: 20220624-144119.html, image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, > image-2022-06-24-14-48-26-451.png, image-2022-06-24-14-57-10-424.png, jstack > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558350#comment-17558350 ] zhengsicheng commented on HBASE-26981: -- [~zhangduo] Still getting high CPU， new flame graph [^20220624-144119.html] !image-2022-06-24-14-48-26-451.png! > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: 20220624-144119.html, image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, > image-2022-06-24-14-48-26-451.png, jstack > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558285#comment-17558285 ] zhengsicheng commented on HBASE-26981: -- Very good, thanks you [~zhangduo] I will test it to see the effect > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, jstack > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558206#comment-17558206 ] Huaxiang Sun commented on HBASE-26981: -- [~zhengsicheng] We enabled the LoadBalance mode on some clusters' hbase clients, we observed meta scan requests are well balanced across meta replica regions. More data will be available in the coming couple months. > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, jstack > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558088#comment-17558088 ] Duo Zhang commented on HBASE-26981: --- The PR for HBASE-27146 is ready, please have a try if you have interest. Thanks. > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, jstack > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558057#comment-17558057 ] zhengsicheng commented on HBASE-26981: -- Hi [~huaxiaolong] meta replica LoadBalance mode ，How's the effect？ > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, jstack > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556942#comment-17556942 ] Huaxiang Sun commented on HBASE-26981: -- Meta scan is cpu heavy operation. We identified this issue in 2020. We developed meta replica LoadBalance mode to load balance meta scan across replica regions. [https://hbase.apache.org/book.html#async.wal.replication.meta?] > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, jstack > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556906#comment-17556906 ] Duo Zhang commented on HBASE-26981: --- Filed HBASE-27146 for avoiding CellUtil.cloneRow > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, jstack > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17555862#comment-17555862 ] zhengsicheng commented on HBASE-26981: -- [~bbeaudreault] Hi Bryan Beaudreault this problem is still relatively difficult, and in the special attack > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, jstack > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17540333#comment-17540333 ] Andrew Kyle Purtell commented on HBASE-26981: - Unscheduling from upcoming releases until there is a patch available. > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Major > Fix For: 2.6.0, 3.0.0-alpha-3 > > Attachments: image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, jstack > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17539566#comment-17539566 ] Bryan Beaudreault commented on HBASE-26981: --- Hi [~zhengsicheng] , thanks for your work here. Just curious if you plan to work on a solution soon? We are also seeing this issue > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Assignee: zhengsicheng >Priority: Major > Attachments: image-2022-04-27-20-24-33-252.png, > image-2022-04-27-20-45-09-227.png, image-2022-04-28-09-59-40-567.png, jstack > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-28-09-59-40-567.png! > > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528870#comment-17528870 ] ramkrishna.s.vasudevan commented on HBASE-26981: Thanks for the analysis [~zhengsicheng] > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: image-2022-04-27-20-22-17-089.png, > image-2022-04-27-20-24-33-252.png, image-2022-04-27-20-45-09-227.png, jstack > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-27-20-22-17-089.png! > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528787#comment-17528787 ] Duo Zhang commented on HBASE-26981: --- OK, so MetaCellComparatator is the problem. We need to copy byte array while comparing... Thanks for providing the analysis. Will take a look soon. [~anoopsamjohn] FYI. > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: image-2022-04-27-20-22-17-089.png, > image-2022-04-27-20-24-33-252.png, image-2022-04-27-20-45-09-227.png, jstack > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-27-20-22-17-089.png! > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528775#comment-17528775 ] zhengsicheng commented on HBASE-26981: -- [~zhangduo] Thanks for the analysis > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: image-2022-04-27-20-22-17-089.png, > image-2022-04-27-20-24-33-252.png, image-2022-04-27-20-45-09-227.png, jstack > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-27-20-22-17-089.png! > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (HBASE-26981) The CPU usage of the regionserver node where the meta table is located is too high
[ https://issues.apache.org/jira/browse/HBASE-26981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528765#comment-17528765 ] zhengsicheng commented on HBASE-26981: -- Analyze CPU consumption by flame graph org.apache.hadoop.hbase.CellUtil#cloneRow !image-2022-04-27-20-45-09-227.png! > The CPU usage of the regionserver node where the meta table is located is too > high > -- > > Key: HBASE-26981 > URL: https://issues.apache.org/jira/browse/HBASE-26981 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: 2.3.4 >Reporter: zhengsicheng >Priority: Major > Attachments: image-2022-04-27-20-22-17-089.png, > image-2022-04-27-20-24-33-252.png, image-2022-04-27-20-45-09-227.png, jstack > > > When the read and write pressure is high, the CPU usage of the meta table > node is too high > !image-2022-04-27-20-22-17-089.png! > !image-2022-04-27-20-24-33-252.png! -- This message was sent by Atlassian Jira (v8.20.7#820007)
