shenhuaxin opened a new issue #9257:
URL: https://github.com/apache/dubbo/issues/9257


   <!-- If you need to report a security issue please visit 
https://github.com/apache/dubbo/security/policy -->
   
   - [ ] I have searched the [issues](https://github.com/apache/dubbo/issues) 
of this repository and believe that this is not a duplicate.
   
   ## Ask your question here
   依赖版本
   zookeeper: 
   ```java
           <dependency>
               <groupId>org.apache.dubbo</groupId>
               <artifactId>dubbo-spring-boot-starter</artifactId>
               <version>2.7.8</version>
           </dependency>
   
           <dependency>
               <groupId>org.apache.curator</groupId>
               <artifactId>curator-recipes</artifactId>
               <version>2.10.0</version>
           </dependency>
   ```
   
   遇到的错误日志如下:
   ```java
   2021-11-11 09:49:36 | siac | * | main-SendThread(10.135.101.101:2181) | INFO 
 | org.apache.zookeeper.ClientCnxn |  | ClientCnxn.java:1096 | Client session 
timed out, have not heard from server in 26676ms for sessionid 
0x17a78d8154236ec, closing socket connection and attempting reconnect
   2021-11-11 09:49:36 | siac | * | main-EventThread | INFO  | 
o.a.curator.framework.state.ConnectionStateManager |  | 
ConnectionStateManager.java:228 | State change: SUSPENDED
   2021-11-11 09:49:37 | siac | * | main-SendThread(10.135.101.101:2181) | INFO 
 | org.apache.zookeeper.ClientCnxn |  | ClientCnxn.java:975 | Opening socket 
connection to server 10.135.101.101/10.135.101.101:2181. Will not attempt to 
authenticate using SASL (unknown error)
   2021-11-11 09:49:37 | siac | * | main-SendThread(10.135.101.101:2181) | INFO 
 | org.apache.zookeeper.ClientCnxn |  | ClientCnxn.java:852 | Socket connection 
established to 10.135.101.101/10.135.101.101:2181, initiating session
   2021-11-11 09:49:37 | siac | * | main-SendThread(10.135.101.101:2181) | INFO 
 | org.apache.zookeeper.ClientCnxn |  | ClientCnxn.java:1235 | Session 
establishment complete on server 10.135.101.101/10.135.101.101:2181, sessionid 
= 0x17a78d8154236ec, negotiated timeout = 40000
   2021-11-11 09:49:37 | siac | * | main-EventThread | INFO  | 
o.a.curator.framework.state.ConnectionStateManager |  | 
ConnectionStateManager.java:228 | State change: RECONNECTED
   2021-11-11 09:49:37 | siac | * | main-SendThread(10.135.101.101:2181) | INFO 
 | org.apache.zookeeper.ClientCnxn |  | ClientCnxn.java:1098 | Unable to read 
additional data from server sessionid 0x17a78d8154236ec, likely server has 
closed socket, closing socket connection and attempting reconnect
   2021-11-11 09:49:38 | siac | * | Curator-ConnectionStateManager-0 | INFO  | 
o.a.dubbo.registry.zookeeper.ZookeeperRegistry |  | ZookeeperRegistry.java:304 
|  [DUBBO] Fetching the latest urls of 
[consumer://10.135.101.104/com.authority.service.IAuthorityService?application=*&category=providers,configurators,routers&check=false&dubbo=2.0.2&init=false&interface=com.authority.service.IAuthorityService&lazy=true&logger=slf4j&metadata-type=remote&methods=getRoleInfoByRoleEnName,updateRole,addResources,getAuthorithByUsername,deleteResource,getOperateLogList,addLog,disableRole,getUsefulRoles,getResourcesDetail,batchInsertAuthUserRole,getRoles,powerList,getResourcesTree,getUserByUsername,getResourcesList,getRole,getAuthTreeByRoleId,selectBdList,getUserInfoByRoleId,getRoleByUsername,modifyResource,getUserInfoByRoleEnName,powerRedis,userRole,addRole,getRolesByResKey&pid=948&qos.enable=false&release=2.7.8&side=consumer&sticky=false&timeout=10000&timestamp=1636446226827,
 consumer://10.135.101.
 
104/com.rpc.goods.service.IGoodsService?application=*&category=providers,configurators,routers&check=false&dubbo=2.0.2&init=false&interface=com.rpc.goods.service.IGoodsService&lazy=true&logger=slf4j&metadata-type=remote&methods=queryGoodsLogPage,queryGoodsPage,getCategoryList,queryGoodsList,getCategoryTree&pid=948&qos.enable=false&release=2.7.8&revision=1.1.0&side=consumer&sticky=false&timeout=10000&timestamp=1636446227578,
 
consumer://10.135.101.104/com.rpc.goods.service.ICategoryService?application=*&category=providers,configurators,routers&check=false&dubbo=2.0.2&init=false&interface=com.rpc.goods.service.ICategoryService&lazy=true&logger=slf4j&metadata-type=remote&methods=listSelect,deleteById,searchFirstCategoryDetail,list,searchSecondCategoryDetail,saveCategoryInfo,queryAttr,queryAttrById,saveSecondCategory,updateById&pid=948&qos.enable=false&release=2.7.8&revision=1.1.0&side=consumer&sticky=false&timeout=10000&timestamp=1636446227716,
 consumer://10.135.101.104/com.rpc.goods.se
 
rvice.GoodsManagerService?application=*&category=providers,configurators,routers&check=false&dubbo=2.0.2&init=false&interface=com.rpc.goods.service.GoodsManagerService&lazy=true&logger=slf4j&metadata-type=remote&methods=updateGoods,pullOffShelves,reject,deleteGoods,approve,addGoods,queryGoods,executeGoodsJob,applyReviewGoods,getGoodsDetail,putOnShelves&pid=948&qos.enable=false&release=2.7.8&revision=1.1.0&side=consumer&sticky=false&timeout=10000&timestamp=1636446227537,
 
consumer://10.135.101.104/com.rpc.coupon.service.ICouponModelServ?application=*&category=providers,configurators,routers&check=false&dubbo=2.0.2&init=false&interface=com.rpc.coupon.service.ICouponModelServ&lazy=true&logger=slf4j&metadata-type=remote&methods=getCouponModelListByCouponSeq,addCouponModel,getMatchedCouponModel,getCouponModelByCouponSeq,getAllCouponModelsByCouponSeq,getAllCouponModels&pid=948&qos.enable=false&release=2.7.8&side=consumer&sticky=false&timeout=10000&timestamp=1636446227674,
 consumer://10.135
 
.101.104/com.sso.service.IOrgSrv?application=*&category=providers,configurators,routers&check=false&dubbo=2.0.2&init=false&interface=com.sso.service.IOrgSrv&lazy=true&logger=slf4j&metadata-type=remote&methods=orgTreeFilter,getOrgInfoById,updateOrgInfo,getOrgInfo,getAllProvince,orgTree,getAreaByCity,getCityByProvince,selectCityByOrg,addOrgInfo,getAllOrg,getOrgInfoByCode&pid=948&qos.enable=false&release=2.7.8&side=consumer&sticky=false&timeout=10000&timestamp=1636446227482,
 
consumer://10.135.101.104/com.activity.dubboService.IActivityService?application=*&category=providers,configurators,routers&check=false&dubbo=2.0.2&init=false&interface=com.activity.dubboService.IActivityService&lazy=true&logger=slf4j&metadata-type=remote&methods=getMmpCouponList,updateAllUpComingActivityStatus,getAllUpComingActivityByActivityType,getOrderShareActivityByOrderAmount,offerByThirdCoupons,getAllForthComingActivityByActivityType,offerByThirdCoupon,queryShopActivitySettingById,smsActivityBrandCodeToUser,
 
queryShopActivitySettingDetailByCondition,getActivityCouponList,getRedEnvelopeActivity,updateShopActivitySetting,getUserRewardsDetail,queryActivityListByCondition,queryShopActivitySettingByCondition,updateAllForthComingActivityStatus,getOrderShareActivityDetail&pid=948&qos.enable=false&release=2.7.8&side=consumer&sticky=false&timeout=10000&timestamp=1636446227636,
 
consumer://10.135.101.104/com.authority.service.UserSynService?application=*&category=providers,configurators,routers&check=false&dubbo=2.0.2&init=false&interface=com.authority.service.UserSynService&lazy=true&logger=slf4j&metadata-type=remote&methods=queryLoginUserInfo,queryUserInfoDetail,queryUserInfoList,synUserInfo,getSynUserInfo&pid=948&qos.enable=false&release=2.7.8&side=consumer&sticky=false&timeout=10000&timestamp=1636446227130,
 
consumer://10.135.101.104/com.rpc.goods.service.GoodsUserService?application=*&category=providers,configurators,routers&check=false&dubbo=2.0.2&init=false&interface=com.rpc.goods.service.Go
 
odsUserService&lazy=true&logger=slf4j&metadata-type=remote&methods=selectByUserName,queryUserInfoDetail,save,selectById,updateByIdSelective&pid=948&qos.enable=false&release=2.7.8&revision=1.1.0&side=consumer&sticky=false&timeout=10000&timestamp=1636446227230],
 dubbo version: 2.7.8, current host: 10.135.101.104
   ```
   
   
   ```java
   2021-11-11 09:49:40 | siac | * | Curator-Framework-0 | INFO  | 
o.a.curator.framework.state.ConnectionStateManager |  | 
ConnectionStateManager.java:228 | State change: LOST
   2021-11-11 09:49:41 | siac | * | Curator-Framework-0 | INFO  | 
o.apache.curator.framework.recipes.cache.TreeCache |  | TreeCache.java:469 | 
Unknown event CuratorEventImpl{type=WATCHED, resultCode=-4, path='null', 
name='null', children=null, context=null, stat=null, data=null, 
watchedEvent=WatchedEvent state:Disconnected type:None path:null, aclList=null}
   2021-11-11 09:49:41 | siac | * | Curator-Framework-0 | INFO  | 
o.apache.curator.framework.recipes.cache.TreeCache |  | TreeCache.java:469 | 
Unknown event CuratorEventImpl{type=WATCHED, resultCode=-4, path='null', 
name='null', children=null, context=null, stat=null, data=null, 
watchedEvent=WatchedEvent state:Disconnected type:None path:null, aclList=null}
   ```
   
   ```java
   2021-11-11 09:49:48 | siac | * | DubboRegistryRetryTimer-thread-1 | INFO  | 
o.apache.dubbo.registry.retry.FailedSubscribedTask |  | 
AbstractRetryTask.java:121 |  [DUBBO] retry subscribe : 
consumer://10.135.101.104/com.authority.service.IAuthorityService?application=*&category=providers,configurators,routers&check=false&dubbo=2.0.2&init=false&interface=com.authority.service.IAuthorityService&lazy=true&logger=slf4j&metadata-type=remote&methods=getRoleInfoByRoleEnName,updateRole,addResources,getAuthorithByUsername,deleteResource,getOperateLogList,addLog,disableRole,getUsefulRoles,getResourcesDetail,batchInsertAuthUserRole,getRoles,powerList,getResourcesTree,getUserByUsername,getResourcesList,getRole,getAuthTreeByRoleId,selectBdList,getUserInfoByRoleId,getRoleByUsername,modifyResource,getUserInfoByRoleEnName,powerRedis,userRole,addRole,getRolesByResKey&pid=948&qos.enable=false&release=2.7.8&side=consumer&sticky=false&timeout=10000&timestamp=1636446226827,
 dubbo version: 2.7.8, current
  host: 10.135.101.104
   2021-11-11 09:49:48 | siac | * | Curator-Framework-0 | INFO  | 
o.apache.curator.framework.recipes.cache.TreeCache |  | TreeCache.java:469 | 
Unknown event CuratorEventImpl{type=WATCHED, resultCode=-4, path='null', 
name='null', children=null, context=null, stat=null, data=null, 
watchedEvent=WatchedEvent state:Disconnected type:None path:null, aclList=null}
   ```
   
   ```java
   2021-11-11 09:49:47 | siac | * | main-SendThread(10.135.101.101:2181) | WARN 
 | org.apache.zookeeper.ClientCnxn |  | ClientCnxn.java:1102 | Session 
0x17a78d8154236ec for server 10.135.101.101/10.135.101.101:2181, unexpected 
error, closing socket connection and attempting reconnect
   java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
        at 
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:117)
        at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   2021-11-11 09:49:47 | siac | * | Curator-ConnectionStateManager-0 | WARN  | 
o.a.d.r.zookeeper.curator.CuratorZookeeperClient |  | 
CuratorZookeeperClient.java:376 |  [DUBBO] Curator zookeeper connection 
recovered from connection lose, reuse the old session 17a78d8154236ec, dubbo 
version: 2.7.8, current host: 10.135.101.104
   2021-11-11 09:49:47 | siac | * | Curator-ConnectionStateManager-0 | WARN  | 
o.a.dubbo.registry.zookeeper.ZookeeperRegistry |  | ZookeeperRegistry.java:89 | 
 [DUBBO] Trying to fetch the latest urls, in case there're provider changes 
during connection loss.
    Since ephemeral ZNode will not get deleted for a connection lose, there's 
no need to re-register url of this instance., dubbo version: 2.7.8, current 
host: 10.135.101.104
   ```
   
   ```java
   2021-11-11 10:23:46 | siac | * | main-EventThread | ERROR | 
o.a.curator.framework.imps.CuratorFrameworkImpl |  | 
CuratorFrameworkImpl.java:557 | Background operation retry gave up
   org.apache.zookeeper.KeeperException$ConnectionLossException: 
KeeperErrorCode = ConnectionLoss
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at 
org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:728)
        at 
org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:516)
        at 
org.apache.curator.framework.imps.GetChildrenBuilderImpl$2.processResult(GetChildrenBuilderImpl.java:166)
        at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:593)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   ```
   
   
   内存溢出的直接原因在:
   
![image](https://user-images.githubusercontent.com/23735480/141311992-bd37ab7c-4ee0-4dfa-95d1-c6303d90d7d3.png)
   
   绝大多数的 ClientCnxn$Packet 的 AsyncCallback cb 为 GetChildrenBuilderImpl。
   
   存活对象:
   
![image](https://user-images.githubusercontent.com/23735480/141312261-da9770c7-a093-4d96-9b2d-7e4ff090b558.png)
   
   
   
   
   遇到过两次这种问题。
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]



---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to