zhujian86 commented on issue #5586:
URL:
https://github.com/apache/dolphinscheduler/issues/5586#issuecomment-853520385
```
[WARN] 2021-06-02 08:52:52.194
org.apache.dolphinscheduler.common.utils.OSUtils:[442] - current cpu load
average 17.28 is too high or available memory 1.91G is too low, under
max.cpuload.avg=8.0 and reserved.memory=0.3G
[INFO] 2021-06-02 08:52:52.408
org.apache.dolphinscheduler.server.master.MasterServer:[164] - master server is
stopping ..., cause : i was judged to death, release resources and stop myself
[INFO] 2021-06-02 08:52:52.411
org.apache.dolphinscheduler.server.master.runner.MasterExecThread:[925] -
process:101 end, state :RUNNING_EXECUTION
[WARN] 2021-06-02 08:52:52.423
org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159]
- worker 192.168.100.129:1234 current cpu load average 20.37 is too high or
available memory 1.57G is too low
[INFO] 2021-06-02 08:52:53.481
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[218] -
currentNodes : [192.168.100.129:1234, 192.168.100.124:1234]
[INFO] 2021-06-02 08:52:53.481
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[254] -
master node : /dolphinscheduler/nodes/master/192.168.100.130:5678 added.
[INFO] 2021-06-02 08:52:53.780
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[221] -
worker group node : /dolphinscheduler/nodes/worker/default/192.168.100.129:1234
down.
[INFO] 2021-06-02 08:52:55.150
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[254] -
master node : /dolphinscheduler/nodes/master/192.168.100.129:5678 added.
[INFO] 2021-06-02 08:52:55.206
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[221] -
worker group node : /dolphinscheduler/nodes/worker/default/192.168.100.130:1234
down.
[INFO] 2021-06-02 08:52:55.336
org.apache.dolphinscheduler.server.master.runner.MasterTaskExecThread:[102] -
task :mf_sale_bill_d id:248, process id:101, exec thread completed
[INFO] 2021-06-02 08:52:55.473
org.apache.dolphinscheduler.remote.NettyRemotingClient:[403] - netty client
closed
[INFO] 2021-06-02 08:52:55.474
org.apache.dolphinscheduler.server.master.runner.MasterSchedulerService:[111] -
master schedule service stopped...
[INFO] 2021-06-02 08:52:55.501
org.apache.dolphinscheduler.remote.NettyRemotingServer:[237] - netty server
closed
[INFO] 2021-06-02 08:52:55.614
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[215] -
worker group node : /dolphinscheduler/nodes/worker/default/192.168.100.129:1234
added.
[INFO] 2021-06-02 08:52:56.058
org.apache.dolphinscheduler.server.master.registry.MasterRegistry:[118] -
master node : 192.168.100.129:5678 unRegistry to ZK.
[INFO] 2021-06-02 08:52:56.058
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[218] -
currentNodes : [192.168.100.129:1234]
[INFO] 2021-06-02 08:52:56.304
org.apache.dolphinscheduler.server.master.registry.MasterRegistry:[120] -
heartbeat executor shutdown
[INFO] 2021-06-02 08:52:56.304
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[215] -
worker group node : /dolphinscheduler/nodes/worker/default/192.168.100.130:1234
added.
[INFO] 2021-06-02 08:52:56.309
org.apache.dolphinscheduler.remote.NettyRemotingClient:[403] - netty client
closed
[INFO] 2021-06-02 08:52:56.311
org.apache.dolphinscheduler.service.log.LogClientService:[59] - logger client
closed
[ERROR] 2021-06-02 08:52:56.311
org.apache.dolphinscheduler.server.utils.ProcessUtils:[440] - task instance
work dir is empty
[ERROR] 2021-06-02 08:52:56.312
org.apache.dolphinscheduler.server.utils.ProcessUtils:[449] - kill yarn job
failure
java.lang.RuntimeException: task instance work dir is empty
at
org.apache.dolphinscheduler.server.utils.ProcessUtils.killYarnJob(ProcessUtils.java:441)
at
org.apache.dolphinscheduler.server.master.zk.ZKMasterClient.failoverWorker(ZKMasterClient.java:339)
at
org.apache.dolphinscheduler.server.master.zk.ZKMasterClient.failoverServerWhenDown(ZKMasterClient.java:186)
at
org.apache.dolphinscheduler.server.master.zk.ZKMasterClient.removeZKNodePath(ZKMasterClient.java:164)
at
org.apache.dolphinscheduler.server.master.zk.ZKMasterClient.handleWorkerEvent(ZKMasterClient.java:242)
at
org.apache.dolphinscheduler.server.master.zk.ZKMasterClient.dataChanged(ZKMasterClient.java:132)
at
org.apache.dolphinscheduler.service.zk.ZookeeperCachedOperator.lambda$registerListener$0(ZookeeperCachedOperator.java:48)
at
org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:760)
at
org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:754)
at
org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:100)
at
org.apache.curator.shaded.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at
org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:92)
at
org.apache.curator.framework.recipes.cache.TreeCache.callListeners(TreeCache.java:753)
at
org.apache.curator.framework.recipes.cache.TreeCache.access$1900(TreeCache.java:75)
at
org.apache.curator.framework.recipes.cache.TreeCache$4.run(TreeCache.java:865)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[INFO] 2021-06-02 08:52:56.445
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[218] -
currentNodes : [192.168.100.129:1234]
[INFO] 2021-06-02 08:52:56.446
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[258] -
master node : /dolphinscheduler/nodes/master/192.168.100.130:5678 down.
[INFO] 2021-06-02 08:52:56.552
org.apache.dolphinscheduler.server.master.zk.ZKMasterClient:[344] - end
worker[192.168.100.130:1234] failover ...
[INFO] 2021-06-02 08:52:56.942
org.apache.curator.framework.imps.CuratorFrameworkImpl:[955] -
backgroundOperationsLoop exiting
[INFO] 2021-06-02 08:52:57.277
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[221] -
worker group node : /dolphinscheduler/nodes/worker/default/192.168.100.124:1234
down.
[INFO] 2021-06-02 08:52:57.383 org.apache.zookeeper.ZooKeeper:[693] -
Session: 0x10008a2bbda0006 closed
[INFO] 2021-06-02 08:52:57.383 org.apache.zookeeper.ClientCnxn:[522] -
EventThread shut down for session: 0x10008a2bbda0006
[INFO] 2021-06-02 08:52:57.389 org.quartz.core.QuartzScheduler:[666] -
Scheduler DolphinScheduler_$_hadoop1021622450691533 shutting down.
[INFO] 2021-06-02 08:52:57.389 org.quartz.core.QuartzScheduler:[585] -
Scheduler DolphinScheduler_$_hadoop1021622450691533 paused.
[WARN] 2021-06-02 08:52:57.427
org.apache.dolphinscheduler.server.master.dispatch.host.LowerWeightHostManager:[159]
- worker 192.168.100.129:1234 current cpu load average 20.37 is too high or
available memory 1.57G is too low
[INFO] 2021-06-02 08:52:57.499 com.alibaba.druid.pool.DruidDataSource:[1928]
- {dataSource-1} closed
[INFO] 2021-06-02 08:52:57.500 org.quartz.core.QuartzScheduler:[740] -
Scheduler DolphinScheduler_$_hadoop1021622450691533 shutdown complete.
[INFO] 2021-06-02 08:52:57.500
org.apache.dolphinscheduler.service.quartz.QuartzExecutors:[171] - Quartz
service stopped, and halt all tasks
[INFO] 2021-06-02 08:52:57.500
org.apache.dolphinscheduler.server.master.MasterServer:[182] - Quartz service
stopped
[INFO] 2021-06-02 08:52:57.802
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[221] -
worker group node : /dolphinscheduler/nodes/worker/default/192.168.100.130:1234
down.
[ERROR] 2021-06-02 08:52:58.297
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[230] -
WorkerGroupListener capture data change and get data failed
org.mybatis.spring.MyBatisSystemException: nested exception is
org.apache.ibatis.exceptions.PersistenceException:
### Error updating database. Cause:
org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain
JDBC Connection; nested exception is
com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at
Wed Jun 02 08:52:57 CST 2021
### The error may exist in
org/apache/dolphinscheduler/dao/mapper/AlertMapper.java (best guess)
### The error may involve
org.apache.dolphinscheduler.dao.mapper.AlertMapper.insert
### The error occurred while executing an update
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed
to obtain JDBC Connection; nested exception is
com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at
Wed Jun 02 08:52:57 CST 2021
at
org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:78)
at
org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440)
at com.sun.proxy.$Proxy84.insert(Unknown Source)
at
org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:271)
at
com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:58)
at
com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:61)
at com.sun.proxy.$Proxy108.insert(Unknown Source)
at
org.apache.dolphinscheduler.dao.AlertDao.saveTaskTimeoutAlert(AlertDao.java:135)
at
org.apache.dolphinscheduler.dao.AlertDao.sendServerStopedAlert(AlertDao.java:102)
at
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager$WorkerGroupNodeListener.dataChanged(ServerNodeManager.java:225)
at
org.apache.dolphinscheduler.service.zk.AbstractListener.childEvent(AbstractListener.java:32)
at
org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:760)
at
org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:754)
at
org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:100)
at
org.apache.curator.shaded.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at
org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:92)
at
org.apache.curator.framework.recipes.cache.TreeCache.callListeners(TreeCache.java:753)
at
org.apache.curator.framework.recipes.cache.TreeCache.access$1900(TreeCache.java:75)
at
org.apache.curator.framework.recipes.cache.TreeCache$4.run(TreeCache.java:865)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ibatis.exceptions.PersistenceException:
### Error updating database. Cause:
org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain
JDBC Connection; nested exception is
com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at
Wed Jun 02 08:52:57 CST 2021
### The error may exist in
org/apache/dolphinscheduler/dao/mapper/AlertMapper.java (best guess)
### The error may involve
org.apache.dolphinscheduler.dao.mapper.AlertMapper.insert
### The error occurred while executing an update
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed
to obtain JDBC Connection; nested exception is
com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at
Wed Jun 02 08:52:57 CST 2021
at
org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
at
org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:199)
at
org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:184)
at sun.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:426)
... 22 common frames omitted
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed
to obtain JDBC Connection; nested exception is
com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at
Wed Jun 02 08:52:57 CST 2021
at
org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82)
at
org.mybatis.spring.transaction.SpringManagedTransaction.openConnection(SpringManagedTransaction.java:80)
at
org.mybatis.spring.transaction.SpringManagedTransaction.getConnection(SpringManagedTransaction.java:67)
at
org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:336)
at
com.baomidou.mybatisplus.core.executor.MybatisSimpleExecutor.prepareStatement(MybatisSimpleExecutor.java:93)
at
com.baomidou.mybatisplus.core.executor.MybatisSimpleExecutor.doUpdate(MybatisSimpleExecutor.java:53)
at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117)
at
org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:197)
... 27 common frames omitted
Caused by: com.alibaba.druid.pool.DataSourceClosedException: dataSource
already closed at Wed Jun 02 08:52:57 CST 2021
at
com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1429)
at
com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1326)
at
com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1306)
at
com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1296)
at
com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:109)
at
org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:158)
at
org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:116)
at
org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79)
... 34 common frames omitted
[INFO] 2021-06-02 08:52:58.298
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[254] -
master node : /dolphinscheduler/nodes/master/192.168.100.130:5678 added.
[INFO] 2021-06-02 08:52:58.303
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[215] -
worker group node : /dolphinscheduler/nodes/worker/default/192.168.100.124:1234
added.
[INFO] 2021-06-02 08:52:58.844
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[218] -
currentNodes : [192.168.100.129:1234]
[INFO] 2021-06-02 08:52:58.844
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[221] -
worker group node : /dolphinscheduler/nodes/worker/default/192.168.100.124:1234
down.
[ERROR] 2021-06-02 08:52:58.867
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager:[230] -
WorkerGroupListener capture data change and get data failed
org.mybatis.spring.MyBatisSystemException: nested exception is
org.apache.ibatis.exceptions.PersistenceException:
### Error updating database. Cause:
org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain
JDBC Connection; nested exception is
com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at
Wed Jun 02 08:52:57 CST 2021
### The error may exist in
org/apache/dolphinscheduler/dao/mapper/AlertMapper.java (best guess)
### The error may involve
org.apache.dolphinscheduler.dao.mapper.AlertMapper.insert
### The error occurred while executing an update
### Cause: org.springframework.jdbc.CannotGetJdbcConnectionException: Failed
to obtain JDBC Connection; nested exception is
com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at
Wed Jun 02 08:52:57 CST 2021
at
org.mybatis.spring.MyBatisExceptionTranslator.translateExceptionIfPossible(MyBatisExceptionTranslator.java:78)
at
org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:440)
at com.sun.proxy.$Proxy84.insert(Unknown Source)
at
org.mybatis.spring.SqlSessionTemplate.insert(SqlSessionTemplate.java:271)
at
com.baomidou.mybatisplus.core.override.MybatisMapperMethod.execute(MybatisMapperMethod.java:58)
at
com.baomidou.mybatisplus.core.override.MybatisMapperProxy.invoke(MybatisMapperProxy.java:61)
at com.sun.proxy.$Proxy108.insert(Unknown Source)
at
org.apache.dolphinscheduler.dao.AlertDao.saveTaskTimeoutAlert(AlertDao.java:135)
at
org.apache.dolphinscheduler.dao.AlertDao.sendServerStopedAlert(AlertDao.java:102)
at
org.apache.dolphinscheduler.server.master.registry.ServerNodeManager$WorkerGroupNodeListener.dataChanged(ServerNodeManager.java:225)
at
org.apache.dolphinscheduler.service.zk.AbstractListener.childEvent(AbstractListener.java:32)
at
org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:760)
at
org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:754)
at
org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:100)
at
org.apache.curator.shaded.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
at
org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:92)
at
org.apache.curator.framework.recipes.cache.TreeCache.callListeners(TreeCache.java:753)
at
org.apache.curator.framework.recipes.cache.TreeCache.access$1900(TreeCache.java:75)
at
org.apache.curator.framework.recipes.cache.TreeCache$4.run(TreeCache.java:865)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ibatis.exceptions.PersistenceException:
### Error updating database. Cause:
org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain
JDBC Connection; nested exception is
com.alibaba.druid.pool.DataSourceClosedException: dataSource already closed at
Wed Jun 02 08:52:57 CST 2021
### The error may exist in
org/apache/dolphinscheduler/dao/mapper/AlertMapper.java (best guess)
### The error may involve
org.apache.dolphinscheduler.dao.mapper.AlertMapper.insert
### The error occurred while executing an update
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
