boystray opened a new issue #5193:
URL: https://github.com/apache/skywalking/issues/5193
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [*] Question or discussion
- [ ] Bug
- [ ] Requirement
- [ ] Feature or performance improvement
___
### Question
- What do you want to know?
why happen OutOfMemoryError? after i setup skywalking , it will report
OutOfMemoryError then close database,so report "Table not found " repeatly
___
### Bug
- Which version of SkyWalking, OS and JRE?
SkyWalking 7.0.0
OS centos
JRE java version "1.8.0_201"
- Which company or project?
- What happen?
If possible, provide a way for reproducing the error. e.g. demo application,
component version.
the following is log :
2020-07-28 00:10:29,429 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=46s390ms165µs522ns).
2020-07-28 00:11:31,458 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=1m27s892ms59µs253ns).
2020-07-28 00:13:01,647 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=1m31s852ms149µs819ns).
2020-07-28 00:13:58,866 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=52s710ms358µs61ns).
2020-07-28 00:15:50,978 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=1m6s346ms629µs613ns).
2020-07-28 00:17:17,712 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=2m5s581ms830µs94ns).
2020-07-28 00:19:09,280 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=1m47s879ms900µs736ns).
2020-07-28 00:22:17,587 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=2m53s786ms663µs590ns).
2020-07-28 00:24:46,488 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=2m29s268ms732µs70ns).
2020-07-28 00:26:29,663 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=1m35s575ms543µs221ns).
2020-07-28 00:29:24,010 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=2m39s809ms458µs759ns).
2020-07-28 00:31:59,025 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=2m53s387ms823µs928ns).
2020-07-28 00:32:57,759 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=1m27s722ms938µs807ns).
2020-07-28 00:35:23,575 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=56s134ms700µs105ns).
2020-07-28 00:36:06,795 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=58s804ms398µs297ns).
2020-07-28 00:37:21,913 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=56s657ms234µs568ns).
2020-07-28 00:39:08,709 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=1m54s88ms733µs513ns).
2020-07-28 00:40:10,867 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=1m2s299ms190µs742ns).
2020-07-28 00:40:56,785 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=56s944ms320µs926ns).
2020-07-28 00:42:17,899 -
org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler
- 70 [grpcServerPool-1-thread-15] ERROR [] - CANCELLED: cancelled before
receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half
close
at io.grpc.Status.asRuntimeException(Status.java:524)
~[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
[grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.26.0.jar:1.26.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_201]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
2020-07-28 00:44:49,781 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=54s177ms771µs729ns).
2020-07-28 00:45:28,278 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=52s43ms72µs340ns).
2020-07-28 00:48:05,943 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=46s440ms226µs401ns).
2020-07-28 00:49:18,380 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=1m7s775ms831µs295ns).
2020-07-28 00:50:04,087 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=57s714ms452µs850ns).
2020-07-28 00:50:57,452 -
org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler
- 70 [grpcServerPool-1-thread-10] ERROR [] - CANCELLED: cancelled before
receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half
close
at io.grpc.Status.asRuntimeException(Status.java:524)
~[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
[grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.26.0.jar:1.26.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_201]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
2020-07-28 00:52:06,622 -
org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler
- 70 [grpcServerPool-1-thread-8] ERROR [] - CANCELLED: cancelled before
receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half
close
at io.grpc.Status.asRuntimeException(Status.java:524)
~[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
[grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.26.0.jar:1.26.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_201]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
2020-07-28 00:53:52,384 -
org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler
- 70 [grpcServerPool-1-thread-9] ERROR [] - CANCELLED: cancelled before
receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half
close
at io.grpc.Status.asRuntimeException(Status.java:524)
~[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
[grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.26.0.jar:1.26.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_201]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
2020-07-28 00:55:01,390 -
org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler
- 70 [grpcServerPool-1-thread-16] ERROR [] - CANCELLED: cancelled before
receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half
close
at io.grpc.Status.asRuntimeException(Status.java:524)
~[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
[grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.26.0.jar:1.26.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_201]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
2020-07-28 00:56:42,333 -
org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler
- 70 [grpcServerPool-1-thread-9] ERROR [] - CANCELLED: cancelled before
receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half
close
at io.grpc.Status.asRuntimeException(Status.java:524)
~[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
[grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.26.0.jar:1.26.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_201]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
2020-07-28 00:58:15,634 -
org.apache.skywalking.oap.server.receiver.trace.provider.handler.v6.grpc.TraceSegmentReportServiceHandler
- 70 [grpcServerPool-1-thread-17] ERROR [] - CANCELLED: cancelled before
receiving half close
io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half
close
at io.grpc.Status.asRuntimeException(Status.java:524)
~[grpc-api-1.26.0.jar:1.26.0]
at
io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:275)
[grpc-stub-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closedInternal(ServerCallImpl.java:353)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:341)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:867)
[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[grpc-core-1.26.0.jar:1.26.0]
at
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[grpc-core-1.26.0.jar:1.26.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_201]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
2020-07-28 01:00:30,849 -
org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer - 96
[pool-9-thread-1] WARN [] - History of segment delete failure
2020-07-28 01:00:56,985 -
org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2ServiceInstanceInventoryCacheDAO
- 90 [pool-10-thread-1] ERROR [] - The database has been closed [90098-196]
org.apache.skywalking.oap.server.library.client.jdbc.JDBCClientException:
The database has been closed [90098-196]
at
org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient.executeQuery(JDBCHikariCPClient.java:124)
~[library-client-7.0.0.jar:7.0.0]
at
org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2ServiceInstanceInventoryCacheDAO.loadLastUpdate(H2ServiceInstanceInventoryCacheDAO.java:76)
~[storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at
org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer.updateServiceInstanceInventory(CacheUpdateTimer.java:101)
~[server-core-7.0.0.jar:7.0.0]
at
org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer.update(CacheUpdateTimer.java:65)
~[server-core-7.0.0.jar:7.0.0]
at
org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer.lambda$start$0(CacheUpdateTimer.java:56)
~[server-core-7.0.0.jar:7.0.0]
at
org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33)
[apm-util-7.0.0.jar:7.0.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_201]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[?:1.8.0_201]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_201]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_201]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_201]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: org.h2.jdbc.JdbcSQLException: The database has been closed
[90098-196]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:179)
~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:155)
~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:144)
~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Database.checkPowerOff(Database.java:524)
~[h2-1.4.196.jar:1.4.196]
at org.h2.command.Command.executeQuery(Command.java:227)
~[h2-1.4.196.jar:1.4.196]
at
org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:111)
~[h2-1.4.196.jar:1.4.196]
at
com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
~[HikariCP-3.1.0.jar:?]
at
com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
~[HikariCP-3.1.0.jar:?]
at
org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient.executeQuery(JDBCHikariCPClient.java:115)
~[library-client-7.0.0.jar:7.0.0]
... 12 more
2020-07-28 01:05:54,986 - com.zaxxer.hikari.pool.HikariPool - 766
[HikariPool-1 housekeeper] WARN [] - HikariPool-1 - Thread starvation or clock
leap detected (housekeeper delta=46s764ms522µs627ns).
2020-07-28 01:04:04,940 -
org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2NetworkAddressInventoryCacheDAO
- 81 [pool-10-thread-1] ERROR [] - HikariPool-1 - Connection is not available,
request timed out after 39558ms.
org.apache.skywalking.oap.server.library.client.jdbc.JDBCClientException:
HikariPool-1 - Connection is not available, request timed out after 39558ms.
at
org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient.getConnection(JDBCHikariCPClient.java:73)
~[library-client-7.0.0.jar:7.0.0]
at
org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient.getConnection(JDBCHikariCPClient.java:60)
~[library-client-7.0.0.jar:7.0.0]
at
org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2NetworkAddressInventoryCacheDAO.loadLastUpdate(H2NetworkAddressInventoryCacheDAO.java:66)
~[storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at
org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer.updateNetAddressInventory(CacheUpdateTimer.java:125)
~[server-core-7.0.0.jar:7.0.0]
at
org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer.update(CacheUpdateTimer.java:66)
~[server-core-7.0.0.jar:7.0.0]
at
org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer.lambda$start$0(CacheUpdateTimer.java:56)
~[server-core-7.0.0.jar:7.0.0]
at
org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33)
[apm-util-7.0.0.jar:7.0.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_201]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[?:1.8.0_201]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[?:1.8.0_201]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[?:1.8.0_201]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_201]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_201]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 -
Connection is not available, request timed out after 39558ms.
at
com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:676)
~[HikariCP-3.1.0.jar:?]
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:190)
~[HikariCP-3.1.0.jar:?]
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:155)
~[HikariCP-3.1.0.jar:?]
at
com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
~[HikariCP-3.1.0.jar:?]
at
org.apache.skywalking.oap.server.library.client.jdbc.hikaricp.JDBCHikariCPClient.getConnection(JDBCHikariCPClient.java:69)
~[library-client-7.0.0.jar:7.0.0]
... 13 more
2020-07-28 01:03:38,008 -
org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO - 78
[DataCarrier.H2_ASYNCHRONOUS_BATCH_PERSISTENT.BulkConsumePool.0.Thread] ERROR
[] - The database has been closed [90098-196]
org.h2.jdbc.JdbcSQLException: The database has been closed [90098-196]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:179)
~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:155)
~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:144)
~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.getTransaction(Session.java:1636)
~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.getStatementSavepoint(Session.java:1646)
~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.setSavepoint(Session.java:825)
~[h2-1.4.196.jar:1.4.196]
at org.h2.command.Command.executeUpdate(Command.java:254)
~[h2-1.4.196.jar:1.4.196]
at
org.h2.jdbc.JdbcPreparedStatement.execute(JdbcPreparedStatement.java:207)
~[h2-1.4.196.jar:1.4.196]
at
com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
~[HikariCP-3.1.0.jar:?]
at
com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java)
~[HikariCP-3.1.0.jar:?]
at
org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLExecutor.invoke(SQLExecutor.java:53)
~[storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at
org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO.synchronous(H2BatchDAO.java:75)
[storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at
org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO$H2BatchConsumer.consume(H2BatchDAO.java:106)
[storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at
org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.consume(MultipleChannelsConsumer.java:79)
[apm-datacarrier-7.0.0.jar:7.0.0]
at
org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:50)
[apm-datacarrier-7.0.0.jar:7.0.0]
2020-07-28 01:07:40,986 -
org.apache.skywalking.oap.server.core.storage.PersistenceTimer - 122
[pool-8-thread-1] ERROR [] - Java heap space
java.lang.OutOfMemoryError: Java heap space
2020-07-28 01:07:40,986 -
org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO - 78
[DataCarrier.H2_ASYNCHRONOUS_BATCH_PERSISTENT.BulkConsumePool.0.Thread] ERROR
[] - The database has been closed [90098-196]
org.h2.jdbc.JdbcSQLException: The database has been closed [90098-196]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:179)
~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:155)
~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:144)
~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.getTransaction(Session.java:1636)
~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.getStatementSavepoint(Session.java:1646)
~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.setSavepoint(Session.java:825)
~[h2-1.4.196.jar:1.4.196]
at org.h2.command.Command.executeUpdate(Command.java:254)
~[h2-1.4.196.jar:1.4.196]
at
org.h2.jdbc.JdbcPreparedStatement.execute(JdbcPreparedStatement.java:207)
~[h2-1.4.196.jar:1.4.196]
at
com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
~[HikariCP-3.1.0.jar:?]
at
com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java)
~[HikariCP-3.1.0.jar:?]
at
org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLExecutor.invoke(SQLExecutor.java:53)
~[storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at
org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO.synchronous(H2BatchDAO.java:75)
[storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at
org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO$H2BatchConsumer.consume(H2BatchDAO.java:106)
[storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at
org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.consume(MultipleChannelsConsumer.java:79)
[apm-datacarrier-7.0.0.jar:7.0.0]
at
org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:50)
[apm-datacarrier-7.0.0.jar:7.0.0]
2020-07-28 01:06:53,781 -
org.apache.skywalking.oap.server.core.remote.client.RemoteClientManager - 147
[pool-3-thread-1] ERROR [] - Java heap space
java.lang.OutOfMemoryError: Java heap space
2020-07-28 01:07:40,988 -
org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO - 78
[DataCarrier.H2_ASYNCHRONOUS_BATCH_PERSISTENT.BulkConsumePool.0.Thread] ERROR
[] - The database has been closed [90098-196]
org.h2.jdbc.JdbcSQLException: The database has been closed [90098-196]
at org.h2.message.DbException.getJdbcSQLException(DbException.java:345)
~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:179)
~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:155)
~[h2-1.4.196.jar:1.4.196]
at org.h2.message.DbException.get(DbException.java:144)
~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.getTransaction(Session.java:1636)
~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.getStatementSavepoint(Session.java:1646)
~[h2-1.4.196.jar:1.4.196]
at org.h2.engine.Session.setSavepoint(Session.java:825)
~[h2-1.4.196.jar:1.4.196]
at org.h2.command.Command.executeUpdate(Command.java:254)
~[h2-1.4.196.jar:1.4.196]
at
org.h2.jdbc.JdbcPreparedStatement.execute(JdbcPreparedStatement.java:207)
~[h2-1.4.196.jar:1.4.196]
at
com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
~[HikariCP-3.1.0.jar:?]
at
com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java)
~[HikariCP-3.1.0.jar:?]
at
org.apache.skywalking.oap.server.storage.plugin.jdbc.SQLExecutor.invoke(SQLExecutor.java:53)
~[storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at
org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO.synchronous(H2BatchDAO.java:75)
[storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at
org.apache.skywalking.oap.server.storage.plugin.jdbc.h2.dao.H2BatchDAO$H2BatchConsumer.consume(H2BatchDAO.java:106)
[storage-jdbc-hikaricp-plugin-7.0.0.jar:7.0.0]
at
org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.consume(MultipleChannelsConsumer.java:79)
[apm-datacarrier-7.0.0.jar:7.0.0]
at
org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:50)
[apm-datacarrier-7.0.0.jar:7.0.0]
___
### Requirement or improvement
- Please describe about your requirements or improvement suggestions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
