[
https://issues.apache.org/jira/browse/HBASE-28663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Manning updated HBASE-28663:
----------------------------------
Description:
If you run the {{CanaryTool}} in region mode until it reaches the configured
timeout, the logs and sink results will show that it can continue executing and
scanning for 10 seconds.
This is because the RegionTasks have already been submitted to an
ExecutorService which continues execution after timeout, and the Monitor
continues execution on a separate thread.
The 10 second delay in shutdown is seen, in hbase 2.x at least, because
{{runMonitor}} will close the {{Connection}} and that process
([code|https://github.com/apache/hbase/blob/e865c852c0e9a1e9b55b9d1512d379072d3e7a7b/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L1054-L1094])
will lead to {{ConnectionImplementation#close}}
([code|https://github.com/apache/hbase/blob/e865c852c0e9a1e9b55b9d1512d379072d3e7a7b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L2272-L2300])
and inside {{shutdownPools}} we will potentially wait the full 10 seconds of
{{awaitTermination}} if client operations are in progress.
The scenario can be improved by simply interrupting the monitor thread, as we
will often be in an {{invokeAll}} call in a {{sniff}} method. The {{invokeAll}}
method is blocking, and interrupting the monitor in this call will interrupt
the client threads and generally shutdown properly and timely. However, we can
be more robust by also watching for a shutdown signal in the various tasks such
as {{RegionTask}} so any remaining tasks will drain quickly and without errors.
{code:java}
2024-06-12 02:57:14 [Time-limited test] ERROR tool.Canary(1076): The monitor is
running too long (1140098) after timeout limit:1140000 will be killed itself !!
2024-06-12 02:57:14 [Time-limited test] INFO
client.ConnectionImplementation(2039): Closing master protocol: MasterService
2024-06-12 02:57:14 [pool-3-thread-4] ERROR tool.Canary(353): Read from
REGION1. on serverName=REGIONSERVER-1, columnFamily=0 failed
java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException:
Task
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@54f2a9a4
rejected from java.util.concurrent.ThreadPoolExecutor@2d3d204d[Shutting down,
pool size = 7, active threads = 7, queued tasks = 0, completed tasks = 180094]
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:199)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:271)
at
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:440)
at
org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:314)
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:612)
at
org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.readColumnFamily(CanaryTool.java:565)
at
org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.read(CanaryTool.java:609)
at
org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:503)
at
org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:471)
[... repeats for 10 seconds and tens of thousands of regions ... ]
2024-06-12 02:57:16 [pool-3-thread-11] ERROR tool.Canary(353): Read from
REGION10000. on serverName=REGIONSERVER-2, columnFamily=0 failed
java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException:
Task
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@d08d21f
rejected from java.util.concurrent.ThreadPoolExecutor@2d3d204d[Shutting down,
pool size = 3, active threads = 3, queued tasks = 0, completed tasks = 180098]
[...]
2024-06-12 02:57:24 [pool-3-thread-11] ERROR tool.Canary(353): Read from
REGION42000. on serverName=REGIONSERVER-3, columnFamily=0 failed
java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException:
Task
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@38e7a5a1
rejected from java.util.concurrent.ThreadPoolExecutor@2d3d204d[Terminated,
pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 180101]
2024-06-12T02:57:24.202Z, java.io.InterruptedIOException
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:294)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:255)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:53)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:271)
at
org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:440)
at
org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:314)
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:612)
at
org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.readColumnFamily(CanaryTool.java:565)
at
org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.read(CanaryTool.java:609)
at
org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:503)
at
org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:471)
2024-06-12 02:57:24 [pool-3-thread-8] WARN
client.ConnectionImplementation(428): Checking master connection
org.apache.hadoop.hbase.ipc.StoppedRpcClientException: Call to
address=HMASTER-1 failed on local exception:
org.apache.hadoop.hbase.ipc.StoppedRpcClientException at
sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:206) at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:391)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:92)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:425)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:420)
at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:114) at
org.apache.hadoop.hbase.ipc.Call.setException(Call.java:129) at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:446)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:332)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:92)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:595)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java)
at
org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1235)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.lambda$masterConnectionStateSupplier$1(ConnectionImplementation.java:421)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:2025)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1441)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.getMaster(ConnectionImplementation.java:1433)
at
org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:57)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:101)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3020)
at
org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:579)
at org.apache.hadoop.hbase.client.HTable.getDescriptor(HTable.java:242) at
org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.read(CanaryTool.java:593) at
org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:503) at
org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:471)
2024-06-12 02:57:24 [pool-3-thread-7] ERROR tool.Canary(345): Read from
REGION43000. on serverName=REGIONSERVER-4 failed
org.apache.hadoop.hbase.DoNotRetryIOException: Connection was closed while
trying to get master
at
org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionImplementation.java:1381)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1445)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.getMaster(ConnectionImplementation.java:1433)
at
org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:57)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:101)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3020)
at
org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:579)
at org.apache.hadoop.hbase.client.HTable.getDescriptor(HTable.java:242)
at
org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.read(CanaryTool.java:593)
at
org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:503)
at
org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:471){code}
was:
If you run the {{CanaryTool}} in region mode until it reaches the configured
timeout, the logs and sink results will show that it can continue executing and
scanning for 10 seconds.
This is because the RegionTasks have already been submitted to an
ExecutorService which continues execution after timeout, and the Monitor
continues execution on a separate thread.
The 10 seconds is seen in hbase 2.x, at least, because {{runMonitor}} will
close the {{Connection}} and that process
([code|https://github.com/apache/hbase/blob/e865c852c0e9a1e9b55b9d1512d379072d3e7a7b/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L1054-L1094])
will lead to {{ConnectionImplementation#close}}
([code|https://github.com/apache/hbase/blob/e865c852c0e9a1e9b55b9d1512d379072d3e7a7b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L2272-L2300])
and inside {{shutdownPools}} we will potentially wait the full 10 seconds of
{{awaitTermination}} if client operations are in progress.
The scenario can be improved by simply interrupting the monitor thread, as we
will often be in an {{invokeAll}} call in a {{sniff}} method, which will
interrupt the client threads and generally shutdown properly and timely.
However, we could be more robust by also watching for a shutdown signal in the
various tasks such as {{{}RegionTask{}}}.
> CanaryTool continues executing and scanning after timeout
> ---------------------------------------------------------
>
> Key: HBASE-28663
> URL: https://issues.apache.org/jira/browse/HBASE-28663
> Project: HBase
> Issue Type: Bug
> Components: canary
> Affects Versions: 2.0.0, 3.0.0
> Reporter: David Manning
> Assignee: David Manning
> Priority: Minor
> Labels: pull-request-available
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> If you run the {{CanaryTool}} in region mode until it reaches the configured
> timeout, the logs and sink results will show that it can continue executing
> and scanning for 10 seconds.
> This is because the RegionTasks have already been submitted to an
> ExecutorService which continues execution after timeout, and the Monitor
> continues execution on a separate thread.
> The 10 second delay in shutdown is seen, in hbase 2.x at least, because
> {{runMonitor}} will close the {{Connection}} and that process
> ([code|https://github.com/apache/hbase/blob/e865c852c0e9a1e9b55b9d1512d379072d3e7a7b/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/CanaryTool.java#L1054-L1094])
> will lead to {{ConnectionImplementation#close}}
> ([code|https://github.com/apache/hbase/blob/e865c852c0e9a1e9b55b9d1512d379072d3e7a7b/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L2272-L2300])
> and inside {{shutdownPools}} we will potentially wait the full 10 seconds of
> {{awaitTermination}} if client operations are in progress.
> The scenario can be improved by simply interrupting the monitor thread, as we
> will often be in an {{invokeAll}} call in a {{sniff}} method. The
> {{invokeAll}} method is blocking, and interrupting the monitor in this call
> will interrupt the client threads and generally shutdown properly and timely.
> However, we can be more robust by also watching for a shutdown signal in the
> various tasks such as {{RegionTask}} so any remaining tasks will drain
> quickly and without errors.
>
> {code:java}
> 2024-06-12 02:57:14 [Time-limited test] ERROR tool.Canary(1076): The monitor
> is running too long (1140098) after timeout limit:1140000 will be killed
> itself !!
> 2024-06-12 02:57:14 [Time-limited test] INFO
> client.ConnectionImplementation(2039): Closing master protocol: MasterService
> 2024-06-12 02:57:14 [pool-3-thread-4] ERROR tool.Canary(353): Read from
> REGION1. on serverName=REGIONSERVER-1, columnFamily=0 failed
> java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException:
> Task
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@54f2a9a4
> rejected from java.util.concurrent.ThreadPoolExecutor@2d3d204d[Shutting
> down, pool size = 7, active threads = 7, queued tasks = 0, completed tasks =
> 180094]
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:199)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:271)
> at
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:440)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:314)
> at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:612)
> at
> org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.readColumnFamily(CanaryTool.java:565)
> at
> org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.read(CanaryTool.java:609)
> at
> org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:503)
> at
> org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:471)
> [... repeats for 10 seconds and tens of thousands of regions ... ]
> 2024-06-12 02:57:16 [pool-3-thread-11] ERROR tool.Canary(353): Read from
> REGION10000. on serverName=REGIONSERVER-2, columnFamily=0 failed
> java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException:
> Task
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@d08d21f
> rejected from java.util.concurrent.ThreadPoolExecutor@2d3d204d[Shutting
> down, pool size = 3, active threads = 3, queued tasks = 0, completed tasks =
> 180098]
> [...]
> 2024-06-12 02:57:24 [pool-3-thread-11] ERROR tool.Canary(353): Read from
> REGION42000. on serverName=REGIONSERVER-3, columnFamily=0 failed
> java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException:
> Task
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@38e7a5a1
> rejected from java.util.concurrent.ThreadPoolExecutor@2d3d204d[Terminated,
> pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 180101]
> 2024-06-12T02:57:24.202Z, java.io.InterruptedIOException
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:294)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:255)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:53)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:191)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:271)
> at
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:440)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:314)
> at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:612)
> at
> org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.readColumnFamily(CanaryTool.java:565)
> at
> org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.read(CanaryTool.java:609)
> at
> org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:503)
> at
> org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:471)
> 2024-06-12 02:57:24 [pool-3-thread-8] WARN
> client.ConnectionImplementation(428): Checking master connection
> org.apache.hadoop.hbase.ipc.StoppedRpcClientException: Call to
> address=HMASTER-1 failed on local exception:
> org.apache.hadoop.hbase.ipc.StoppedRpcClientException at
> sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:206) at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:391)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:92)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:425)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:420)
> at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:114) at
> org.apache.hadoop.hbase.ipc.Call.setException(Call.java:129) at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:446)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:332)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:92)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:595)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceState.isMasterRunning(ConnectionImplementation.java:1235)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.lambda$masterConnectionStateSupplier$1(ConnectionImplementation.java:421)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.isKeepAliveMasterConnectedAndRunning(ConnectionImplementation.java:2025)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1441)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.getMaster(ConnectionImplementation.java:1433)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:57)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:101)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3020)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:579)
> at org.apache.hadoop.hbase.client.HTable.getDescriptor(HTable.java:242) at
> org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.read(CanaryTool.java:593)
> at
> org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:503)
> at
> org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:471)
> 2024-06-12 02:57:24 [pool-3-thread-7] ERROR tool.Canary(345): Read from
> REGION43000. on serverName=REGIONSERVER-4 failed
> org.apache.hadoop.hbase.DoNotRetryIOException: Connection was closed while
> trying to get master
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionImplementation.java:1381)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1445)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.getMaster(ConnectionImplementation.java:1433)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:57)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:101)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3020)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:579)
> at org.apache.hadoop.hbase.client.HTable.getDescriptor(HTable.java:242)
> at
> org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.read(CanaryTool.java:593)
> at
> org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:503)
> at
> org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.call(CanaryTool.java:471){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
