xuet0ng edited a comment on issue #2190: [Agent] can not connect to the collector maybe cause memory leak URL: https://github.com/apache/incubator-skywalking/issues/2190#issuecomment-456293895 - original jar - start & connect collector success ``` ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 1252: 1 216 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 2978: 1 40 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 3361: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 3362: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 4125: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 4126: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 4127: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 4128: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 4129: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 5484: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 5485: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 5486: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 5487: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 5488: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent ➜ skywalking-agent ➜ skywalking-agent ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 1246: 1 216 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 2985: 1 40 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 3372: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 3373: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 4130: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 4131: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 4132: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 4133: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 4134: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 5485: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 5486: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 5487: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 5488: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 5489: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ``` - disconnect network & wait for 3 mins ``` ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 1246: 1 216 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 2992: 1 40 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 3387: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 3388: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 4152: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 4153: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 4154: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 4155: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 4156: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 5511: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 5512: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 5513: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 5514: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 5515: 1 16 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 882: 2 432 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 2017: 2 80 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 2352: 2 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 2353: 2 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 2691: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 2692: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 2693: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 2694: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 2695: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 3407: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 3408: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 3409: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 3410: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 3411: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent ➜ skywalking-agent ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 732: 3 648 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 1676: 3 120 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 1876: 3 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 1877: 3 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 2159: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 2160: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 2161: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 2162: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 2163: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 2714: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 2715: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 2716: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 2717: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 2718: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 663: 4 864 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 1437: 4 160 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 1578: 4 128 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 1579: 4 128 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 1910: 4 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 1911: 4 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 1912: 4 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 1913: 4 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 1914: 4 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 2408: 4 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 2409: 4 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 2410: 4 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 2411: 4 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 2412: 4 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 601: 5 1080 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 1290: 5 200 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 1458: 5 160 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 1459: 5 160 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 1727: 5 120 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 1728: 5 120 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 1729: 5 120 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 1730: 5 120 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 1731: 5 120 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 2090: 5 80 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 2091: 5 80 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 2092: 5 80 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 2093: 5 80 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 2094: 5 80 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 559: 6 1296 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 1186: 6 240 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 1343: 6 192 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 1344: 6 192 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 1554: 6 144 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 1555: 6 144 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 1556: 6 144 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 1557: 6 144 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 1558: 6 144 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 1942: 6 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 1943: 6 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 1944: 6 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 1945: 6 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 1946: 6 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl 524: 7 1512 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 1098: 7 280 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 1267: 7 224 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 1268: 7 224 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 1441: 7 168 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 1442: 7 168 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 1443: 7 168 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 1444: 7 168 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 1445: 7 168 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 1820: 7 112 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 1821: 7 112 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 1822: 7 112 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 1823: 7 112 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 1824: 7 112 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ``` - modified code **if reconnect & old channel shutdownNow** - start & connect collector success ``` ➜ skywalking-agent jmap -histo:live 20262 | grep ManagedChannelImpl 1241: 1 216 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 2978: 1 40 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 3365: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 3366: 1 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 4126: 1 24 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 ``` - disconnect network & wait for 3 mins ``` ➜ skywalking-agent jmap -histo:live 20956 | grep ManagedChannelImpl 732: 3 648 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 1675: 3 120 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 1874: 3 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 1875: 3 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 2150: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 2151: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 2152: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 2153: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 2154: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 2708: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 2709: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 2710: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 2711: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 2712: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent jmap -histo:live 20956 | grep ManagedChannelImpl 878: 2 432 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 1999: 2 80 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 2338: 2 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 2339: 2 64 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 2681: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 2682: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 2683: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 2684: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 2685: 2 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 3399: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 3400: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 3401: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 3402: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 3403: 2 32 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ➜ skywalking-agent jmap -histo:live 20956 | grep ManagedChannelImpl 732: 3 648 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl 1675: 3 120 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl 1874: 3 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1 1875: 3 96 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry 2150: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3 2151: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6 2152: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl 2153: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1 2154: 3 72 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl 2706: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor 2707: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2 2708: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5 2709: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer 2710: 3 48 org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel ``` - 中文了...这是本地电脑上的验证结果.. 1. 不修改...一直能链接上collector..链接3分钟...ManagedChannelImpl的对象数一直是1 2. 不修改...先链接collector..再断网..断网3分钟...ManagedChannelImpl的对象数7...中途几次查看对象数目一直在飙升 3. 修改...先链接collector..再断网..断网3分钟...ManagedChannelImpl的对象数3...中途几次查看对象数目1,2,3,2,3...有shutdown关闭释放的迹象，但可能存在不shutdown慢或者shutdown失败（猜测）...
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
