[
https://issues.apache.org/jira/browse/CASSANDRA-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019976#comment-14019976
]
Jacek Furmankiewicz commented on CASSANDRA-7361:
------------------------------------------------
I have a tar.gz of the heapdump (2.3 GB). I am not able to attach it to JIRA.
Let me know if there is anywhere we could upload it for you.
> Cassandra locks up in full GC, unable to free memory
> ----------------------------------------------------
>
> Key: CASSANDRA-7361
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7361
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu, RedHat, JDK 1.7
> Reporter: Jacek Furmankiewicz
> Priority: Blocker
>
> We have a long running batch load process, which runs for many hours.
> Massive amount of writes, in large mutation batches (we increase the thrift
> frame size to 45 MB).
> Everything goes well, but after about 3 hrs of processing everything locks
> up. We start getting NoHostsAvailable exceptions on the Java application side
> (with Astyanax as our driver), eventually socket timeouts.
> Looking at Cassandra, we can see that it is using nearly the full 8GB of heap
> and unable to free it. It spends most of its time in full GC, but the amount
> of memory does not go down.
> Here is a long sample from jstat to show this over an extended time period
> e.g.
> {quote}
> -bash-4.1$ $JAVA_HOME/bin/jstat -gc 3728 5s
> S0C S1C S0U S1U EC EU OC OU PC
> PU YGC YGCT FGC FGCT GCT
> 81920.0 81920.0 0.0 4097.0 655360.0 150419.0 7569408.0 7202990.3 43280.0
> 25855.8 5271 268.378 4940 9877.726 10146.104
> 81920.0 81920.0 0.0 4097.0 655360.0 150657.2 7569408.0 7202990.3 43280.0
> 25855.8 5271 268.378 4940 9877.726 10146.104
> 81920.0 81920.0 0.0 4097.0 655360.0 150708.5 7569408.0 7202990.3 43280.0
> 25855.8 5271 268.378 4941 9877.950 10146.328
> 81920.0 81920.0 0.0 4097.0 655360.0 150920.4 7569408.0 7202990.2 43280.0
> 25855.8 5271 268.378 4942 9878.085 10146.463
> 81920.0 81920.0 0.0 4097.0 655360.0 150930.7 7569408.0 7202990.2 43280.0
> 25855.8 5271 268.378 4942 9878.085 10146.463
> 81920.0 81920.0 0.0 4097.0 655360.0 150998.2 7569408.0 7202990.2 43280.0
> 25855.8 5271 268.378 4942 9878.085 10146.463
> 81920.0 81920.0 0.0 4097.0 655360.0 151335.6 7569408.0 7202990.2 43280.0
> 25855.8 5271 268.378 4943 9878.253 10146.630
> 81920.0 81920.0 0.0 4097.0 655360.0 151483.3 7569408.0 7202990.2 43280.0
> 25855.8 5271 268.378 4944 9878.389 10146.766
> 81920.0 81920.0 0.0 4097.0 655360.0 151750.4 7569408.0 7202990.2 43280.0
> 25855.8 5271 268.378 4944 9878.389 10146.766
> 81920.0 81920.0 0.0 4097.0 655360.0 151752.5 7569408.0 7202990.2 43280.0
> 25855.8 5271 268.378 4944 9878.389 10146.766
> 81920.0 81920.0 0.0 4097.0 655360.0 151997.4 7569408.0 7202990.2 43280.0
> 25855.8 5271 268.378 4945 9878.593 10146.970
> 81920.0 81920.0 0.0 4097.0 655360.0 152063.0 7569408.0 7202988.6 43280.0
> 25855.8 5271 268.378 4946 9878.732 10147.110
> 81920.0 81920.0 0.0 4097.0 655360.0 152256.2 7569408.0 7202988.6 43280.0
> 25855.8 5271 268.378 4946 9878.732 10147.110
> 81920.0 81920.0 0.0 4097.0 655360.0 152276.2 7569408.0 7202988.6 43280.0
> 25855.8 5271 268.378 4946 9878.732 10147.110
> 81920.0 81920.0 0.0 4097.0 655360.0 152276.2 7569408.0 7202988.6 43280.0
> 25855.8 5271 268.378 4947 9878.968 10147.345
> 81920.0 81920.0 0.0 4097.0 655360.0 152525.3 7569408.0 7202857.5 43280.0
> 25855.8 5271 268.378 4948 9879.107 10147.484
> 81920.0 81920.0 0.0 4097.0 655360.0 152673.0 7569408.0 7202857.5 43280.0
> 25855.8 5271 268.378 4948 9879.107 10147.484
> 81920.0 81920.0 0.0 4097.0 655360.0 152855.5 7569408.0 7202857.5 43280.0
> 25855.8 5271 268.378 4948 9879.107 10147.484
> 81920.0 81920.0 0.0 4097.0 655360.0 152871.3 7569408.0 7202857.5 43280.0
> 25855.8 5271 268.378 4949 9879.275 10147.653
> 81920.0 81920.0 0.0 4097.0 655360.0 153116.2 7569408.0 7202857.0 43280.0
> 25855.8 5271 268.378 4950 9879.415 10147.793
> 81920.0 81920.0 0.0 4097.0 655360.0 153191.6 7569408.0 7202857.0 43280.0
> 25855.8 5271 268.378 4950 9879.415 10147.793
> 81920.0 81920.0 0.0 4097.0 655360.0 153454.3 7569408.0 7202857.0 43280.0
> 25855.8 5271 268.378 4950 9879.415 10147.793
> 81920.0 81920.0 0.0 4097.0 655360.0 153528.0 7569408.0 7202857.0 43280.0
> 25855.8 5271 268.378 4951 9879.583 10147.961
> 81920.0 81920.0 0.0 4097.0 655360.0 153766.1 7569408.0 7202857.0 43280.0
> 25855.8 5271 268.378 4952 9879.721 10148.099
> 81920.0 81920.0 0.0 4097.0 655360.0 153791.9 7569408.0 7202857.0 43280.0
> 25855.8 5271 268.378 4952 9879.721 10148.099
> 81920.0 81920.0 0.0 4097.0 655360.0 153855.1 7569408.0 7202857.0 43280.0
> 25855.8 5271 268.378 4952 9879.721 10148.099
> 81920.0 81920.0 0.0 4097.0 655360.0 154114.8 7569408.0 7202857.0 43280.0
> 25855.8 5271 268.378 4953 9879.953 10148.331
> 81920.0 81920.0 0.0 4097.0 655360.0 154118.8 7569408.0 7202856.8 43280.0
> 25855.8 5271 268.378 4954 9880.097 10148.474
> 81920.0 81920.0 0.0 4097.0 655360.0 154382.7 7569408.0 7202856.8 43280.0
> 25855.8 5271 268.378 4954 9880.097 10148.474
> 81920.0 81920.0 0.0 4097.0 655360.0 154511.8 7569408.0 7202856.8 43280.0
> 25855.8 5271 268.378 4954 9880.097 10148.474
> 81920.0 81920.0 0.0 4097.0 655360.0 154699.2 7569408.0 7202856.8 43280.0
> 25855.8 5271 268.378 4955 9880.273 10148.650
> 81920.0 81920.0 0.0 4097.0 655360.0 154707.4 7569408.0 7202856.7 43280.0
> 25855.8 5271 268.378 4956 9880.273 10148.650
> 81920.0 81920.0 0.0 4097.0 655360.0 154985.2 7569408.0 7202856.7 43280.0
> 25855.8 5271 268.378 4956 9880.414 10148.792
> 81920.0 81920.0 0.0 4097.0 655360.0 155006.1 7569408.0 7202856.7 43280.0
> 25855.8 5271 268.378 4956 9880.414 10148.792
> 81920.0 81920.0 0.0 4097.0 655360.0 155050.8 7569408.0 7202856.7 43280.0
> 25855.8 5271 268.378 4957 9880.585 10148.963
> 81920.0 81920.0 0.0 4097.0 655360.0 155233.1 7569408.0 7202856.7 43280.0
> 25855.8 5271 268.378 4957 9880.585 10148.963
> 81920.0 81920.0 0.0 4097.0 655360.0 155536.7 7569408.0 7202856.7 43280.0
> 25855.8 5271 268.378 4958 9880.730 10149.108
> 81920.0 81920.0 0.0 4097.0 655360.0 156764.1 7569408.0 7202856.7 43280.0
> 25855.8 5271 268.378 4958 9880.730 10149.108
> 81920.0 81920.0 0.0 4097.0 655360.0 156829.3 7569408.0 7202856.7 43280.0
> 25855.8 5271 268.378 4959 9880.963 10149.341
> 81920.0 81920.0 0.0 4097.0 655360.0 157015.2 7569408.0 7202856.6 43280.0
> 25855.8 5271 268.378 4959 9880.963 10149.341
> 81920.0 81920.0 0.0 4097.0 655360.0 157089.1 7569408.0 7202856.6 43280.0
> 25855.8 5271 268.378 4960 9881.104 10149.481
> 81920.0 81920.0 0.0 4097.0 655360.0 157353.1 7569408.0 7202856.6 43280.0
> 25855.8 5271 268.378 4960 9881.104 10149.481
> 81920.0 81920.0 0.0 4097.0 655360.0 157368.8 7569408.0 7202856.6 43280.0
> 25855.8 5271 268.378 4961 9881.279 10149.656
> 81920.0 81920.0 0.0 4097.0 655360.0 157608.5 7569408.0 7202856.6 43280.0
> 25855.8 5271 268.378 4961 9881.279 10149.656
> 81920.0 81920.0 0.0 4097.0 655360.0 157684.2 7569408.0 7202856.6 43280.0
> 25855.8 5271 268.378 4962 9881.422 10149.800
> 81920.0 81920.0 0.0 4097.0 655360.0 157922.4 7569408.0 7202856.6 43280.0
> 25855.8 5271 268.378 4962 9881.422 10149.800
> 81920.0 81920.0 0.0 4097.0 655360.0 157959.9 7569408.0 7202856.6 43280.0
> 25855.8 5271 268.378 4963 9881.600 10149.977
> 81920.0 81920.0 0.0 4097.0 655360.0 158004.8 7569408.0 7202856.5 43280.0
> 25855.8 5271 268.378 4963 9881.600 10149.977
> 81920.0 81920.0 0.0 4097.0 655360.0 158197.6 7569408.0 7202856.5 43280.0
> 25855.8 5271 268.378 4964 9881.742 10150.120
> 81920.0 81920.0 0.0 4097.0 655360.0 158209.8 7569408.0 7202856.5 43280.0
> 25855.8 5271 268.378 4964 9881.742 10150.120
> 81920.0 81920.0 0.0 4097.0 655360.0 158473.8 7569408.0 7202856.5 43280.0
> 25855.8 5271 268.378 4965 9881.920 10150.297
> 81920.0 81920.0 0.0 4097.0 655360.0 158473.8 7569408.0 7202856.5 43280.0
> 25855.8 5271 268.378 4965 9881.920 10150.297
> 81920.0 81920.0 0.0 4097.0 655360.0 158769.7 7569408.0 7202856.5 43280.0
> 25855.8 5271 268.378 4966 9882.063 10150.441
> 81920.0 81920.0 0.0 4097.0 655360.0 158796.6 7569408.0 7202856.5 43280.0
> 25855.8 5271 268.378 4966 9882.063 10150.441
> 81920.0 81920.0 0.0 4097.0 655360.0 159076.4 7569408.0 7202856.5 43280.0
> 25855.8 5271 268.378 4966 9882.063 10150.441
> 81920.0 81920.0 0.0 4097.0 655360.0 159080.7 7569408.0 7202856.4 43280.0
> 25855.8 5271 268.378 4967 9882.250 10150.628
> 81920.0 81920.0 0.0 4097.0 655360.0 159190.8 7569408.0 7202856.4 43280.0
> 25855.8 5271 268.378 4968 9882.394 10150.772
> 81920.0 81920.0 0.0 4097.0 655360.0 159391.8 7569408.0 7202856.4 43280.0
> 25855.8 5271 268.378 4968 9882.394 10150.772
> 81920.0 81920.0 0.0 4097.0 655360.0 159697.5 7569408.0 7202856.4 43280.0
> 25855.8 5271 268.378 4968 9882.394 10150.772
> 81920.0 81920.0 0.0 4097.0 655360.0 159737.1 7569408.0 7202856.3 43280.0
> 25855.8 5271 268.378 4969 9882.636 10151.014
> 81920.0 81920.0 0.0 4097.0 655360.0 159737.1 7569408.0 7202856.3 43280.0
> 25855.8 5271 268.378 4970 9882.782 10151.159
> 81920.0 81920.0 0.0 4097.0 655360.0 159958.2 7569408.0 7202856.3 43280.0
> 25855.8 5271 268.378 4970 9882.782 10151.159
> 81920.0 81920.0 0.0 4097.0 655360.0 159985.0 7569408.0 7202856.3 43280.0
> 25855.8 5271 268.378 4970 9882.782 10151.159
> 81920.0 81920.0 0.0 4097.0 655360.0 160249.0 7569408.0 7202856.3 43280.0
> 25855.8 5271 268.378 4971 9882.974 10151.352
> 81920.0 81920.0 0.0 4097.0 655360.0 160249.0 7569408.0 7202856.3 43280.0
> 25855.8 5271 268.378 4972 9883.121 10151.499
> 81920.0 81920.0 0.0 4097.0 655360.0 160361.3 7569408.0 7202856.3 43280.0
> 25855.8 5271 268.378 4972 9883.121 10151.499
> 81920.0 81920.0 0.0 4097.0 655360.0 160638.2 7569408.0 7202856.3 43280.0
> 25855.8 5271 268.378 4972 9883.121 10151.499
> 81920.0 81920.0 0.0 4097.0 655360.0 160894.9 7569408.0 7202856.3 43280.0
> 25855.8 5271 268.378 4973 9883.368 10151.746
> 81920.0 81920.0 0.0 4097.0 655360.0 160920.7 7569408.0 7202856.3 43280.0
> 25855.8 5271 268.378 4974 9883.519 10151.897
> 81920.0 81920.0 0.0 4097.0 655360.0 160920.7 7569408.0 7202856.3 43280.0
> 25855.8 5271 268.378 4974 9883.519 10151.897
> 81920.0 81920.0 0.0 4097.0 655360.0 161139.5 7569408.0 7202856.3 43280.0
> 25855.8 5271 268.378 4975 9883.519 10151.897
> 81920.0 81920.0 0.0 4097.0 655360.0 161168.4 7569408.0 7202856.1 43280.0
> 25855.8 5271 268.378 4975 9883.735 10152.113
> 81920.0 81920.0 0.0 4097.0 655360.0 161427.2 7569408.0 7202856.1 43280.0
> 25855.8 5271 268.378 4976 9883.884 10152.262
> 81920.0 81920.0 0.0 4097.0 655360.0 161446.2 7569408.0 7202856.1 43280.0
> 25855.8 5271 268.378 4976 9883.884 10152.262
> 81920.0 81920.0 0.0 4097.0 655360.0 161490.9 7569408.0 7202856.1 43280.0
> 25855.8 5271 268.378 4976 9883.884 10152.262
> 81920.0 81920.0 0.0 4097.0 655360.0 161673.2 7569408.0 7202856.1 43280.0
> 25855.8 5271 268.378 4977 9884.066 10152.444
> 81920.0 81920.0 0.0 4097.0 655360.0 162002.3 7569408.0 7202856.1 43280.0
> 25855.8 5271 268.378 4978 9884.217 10152.595
> 81920.0 81920.0 0.0 4097.0 655360.0 162023.6 7569408.0 7202856.1 43280.0
> 25855.8 5271 268.378 4978 9884.217 10152.595
> 81920.0 81920.0 0.0 4097.0 655360.0 162086.0 7569408.0 7202856.1 43280.0
> 25855.8 5271 268.378 4978 9884.217 10152.595
> 81920.0 81920.0 0.0 4097.0 655360.0 162271.9 7569408.0 7202856.1 43280.0
> 25855.8 5271 268.378 4979 9884.405 10152.782
> 81920.0 81920.0 0.0 4097.0 655360.0 162537.0 7569408.0 7202856.1 43280.0
> 25855.8 5271 268.378 4980 9884.557 10152.935
> 81920.0 81920.0 0.0 4097.0 655360.0 162696.2 7569408.0 7202856.1 43280.0
> 25855.8 5271 268.378 4980 9884.557 10152.935
> 81920.0 81920.0 0.0 4097.0 655360.0 162696.2 7569408.0 7202856.1 43280.0
> 25855.8 5271 268.378 4980 9884.557 10152.935
> 81920.0 81920.0 0.0 4097.0 655360.0 162740.8 7569408.0 7202856.1 43280.0
> 25855.8 5271 268.378 4981 9884.748 10153.125
> 81920.0 81920.0 0.0 4097.0 655360.0 163009.5 7569408.0 7202856.1 43280.0
> 25855.8 5271 268.378 4982 9884.901 10153.279
> 81920.0 81920.0 0.0 4097.0 655360.0 163285.1 7569408.0 7202856.1 43280.0
> 25855.8 5271 268.378 4982 9884.901 10153.279
> 81920.0 81920.0 0.0 4097.0 655360.0 163285.1 7569408.0 7202856.1 43280.0
> 25855.8 5271 268.378 4982 9884.901 10153.279
> 81920.0 81920.0 0.0 4097.0 655360.0 163397.4 7569408.0 7202856.1 43280.0
> 25855.8 5271 268.378 4983 9885.093 10153.471
> 81920.0 81920.0 0.0 4097.0 655360.0 163590.2 7569408.0 7202855.9 43280.0
> 25855.8 5271 268.378 4984 9885.251 10153.628
> 81920.0 81920.0 0.0 4097.0 655360.0 165038.0 7569408.0 7202855.9 43280.0
> 25855.8 5271 268.378 4984 9885.251 10153.628
> 81920.0 81920.0 0.0 4097.0 655360.0 165142.5 7569408.0 7202855.9 43280.0
> 25855.8 5271 268.378 4984 9885.251 10153.628
> 81920.0 81920.0 0.0 4097.0 655360.0 165142.5 7569408.0 7202855.9 43280.0
> 25855.8 5271 268.378 4985 9885.435 10153.813
> 81920.0 81920.0 0.0 4097.0 655360.0 165391.7 7569408.0 7202855.8 43280.0
> 25855.8 5271 268.378 4986 9885.588 10153.966
> 81920.0 81920.0 0.0 4097.0 655360.0 165401.9 7569408.0 7202855.8 43280.0
> 25855.8 5271 268.378 4986 9885.588 10153.966
> 81920.0 81920.0 0.0 4097.0 655360.0 165665.9 7569408.0 7202855.8 43280.0
> 25855.8 5271 268.378 4986 9885.588 10153.966
> 81920.0 81920.0 0.0 4097.0 655360.0 165667.9 7569408.0 7202855.8 43280.0
> 25855.8 5271 268.378 4987 9885.848 10154.226
> 81920.0 81920.0 0.0 4097.0 655360.0 165712.6 7569408.0 7202855.8 43280.0
> 25855.8 5271 268.378 4988 9886.002 10154.379
> 81920.0 81920.0 0.0 4097.0 655360.0 165971.0 7569408.0 7202855.8 43280.0
> 25855.8 5271 268.378 4988 9886.002 10154.379
> 81920.0 81920.0 0.0 4097.0 655360.0 166254.9 7569408.0 7202855.8 43280.0
> 25855.8 5271 268.378 4988 9886.002 10154.379
> 81920.0 81920.0 0.0 4097.0 655360.0 166257.1 7569408.0 7202855.8 43280.0
> 25855.8 5271 268.378 4989 9886.256 10154.633
> 81920.0 81920.0 0.0 4097.0 655360.0 166257.1 7569408.0 7202855.7 43280.0
> 25855.8 5271 268.378 4989 9886.256 10154.633
> 81920.0 81920.0 0.0 4097.0 655360.0 166496.5 7569408.0 7202855.7 43280.0
> 25855.8 5271 268.378 4990 9886.412 10154.789
> 81920.0 81920.0 0.0 4097.0 655360.0 166843.1 7569408.0 7202855.7 43280.0
> 25855.8 5271 268.378 4990 9886.412 10154.789
> 81920.0 81920.0 0.0 4097.0 655360.0 166852.0 7569408.0 7202855.7 43280.0
> 25855.8 5271 268.378 4991 9886.605 10154.983
> 81920.0 81920.0 0.0 4097.0 655360.0 166852.0 7569408.0 7202855.6 43280.0
> 25855.8 5271 268.378 4991 9886.605 10154.983
> 81920.0 81920.0 0.0 4097.0 655360.0 166908.3 7569408.0 7202855.6 43280.0
> 25855.8 5271 268.378 4992 9886.762 10155.140
> 81920.0 81920.0 0.0 4097.0 655360.0 167122.1 7569408.0 7202855.6 43280.0
> 25855.8 5271 268.378 4992 9886.762 10155.140
> 81920.0 81920.0 0.0 4097.0 655360.0 167375.4 7569408.0 7202855.6 43280.0
> 25855.8 5271 268.378 4993 9887.023 10155.401
> 81920.0 81920.0 0.0 4097.0 655360.0 167375.4 7569408.0 7202855.6 43280.0
> 25855.8 5271 268.378 4993 9887.023 10155.401
> 81920.0 81920.0 0.0 4097.0 655360.0 167377.6 7569408.0 7202855.6 43280.0
> 25855.8 5271 268.378 4994 9887.179 10155.557
> 81920.0 81920.0 0.0 4097.0 655360.0 167683.0 7569408.0 7202855.6 43280.0
> 25855.8 5271 268.378 4994 9887.179 10155.557
> 81920.0 81920.0 0.0 4097.0 655360.0 167948.1 7569408.0 7202855.6 43280.0
> 25855.8 5271 268.378 4994 9887.179 10155.557
> 81920.0 81920.0 0.0 4097.0 655360.0 167968.7 7569408.0 7202855.6 43280.0
> 25855.8 5271 268.378 4995 9887.376 10155.754
> 81920.0 81920.0 0.0 4097.0 655360.0 168034.2 7569408.0 7202855.6 43280.0
> 25855.8 5271 268.378 4996 9887.534 10155.911
> 81920.0 81920.0 0.0 4097.0 655360.0 168094.6 7569408.0 7202855.6 43280.0
> 25855.8 5271 268.378 4996 9887.534 10155.911
> 81920.0 81920.0 0.0 4097.0 655360.0 168270.8 7569408.0 7202855.6 43280.0
> 25855.8 5271 268.378 4996 9887.534 10155.911
> 81920.0 81920.0 0.0 4097.0 655360.0 168561.5 7569408.0 7202855.6 43280.0
> 25855.8 5271 268.378 4997 9887.808 10156.185
> 81920.0 81920.0 0.0 4097.0 655360.0 168627.3 7569408.0 7202855.5 43280.0
> 25855.8 5271 268.378 4998 9887.968 10156.345
> 81920.0 81920.0 0.0 4097.0 655360.0 168641.1 7569408.0 7202855.5 43280.0
> 25855.8 5271 268.378 4998 9887.968 10156.345
> 81920.0 81920.0 0.0 4097.0 655360.0 168859.9 7569408.0 7202855.5 43280.0
> 25855.8 5271 268.378 4998 9887.968 10156.345
> 81920.0 81920.0 0.0 4097.0 655360.0 169143.9 7569408.0 7202855.5 43280.0
> 25855.8 5271 268.378 4999 9888.170 10156.547
> 81920.0 81920.0 0.0 4097.0 655360.0 169152.9 7569408.0 7202855.4 43280.0
> 25855.8 5271 268.378 5000 9888.330 10156.708
> 81920.0 81920.0 0.0 4097.0 655360.0 169152.9 7569408.0 7202855.4 43280.0
> 25855.8 5271 268.378 5000 9888.330 10156.708
> 81920.0 81920.0 0.0 4097.0 655360.0 169373.9 7569408.0 7202855.4 43280.0
> 25855.8 5271 268.378 5000 9888.330 10156.708
> 81920.0 81920.0 0.0 4097.0 655360.0 169639.0 7569408.0 7202855.4 43280.0
> 25855.8 5271 268.378 5001 9888.591 10156.968
> 81920.0 81920.0 0.0 4097.0 655360.0 169744.2 7569408.0 7202855.4 43280.0
> 25855.8 5271 268.378 5002 9888.754 10157.131
> 81920.0 81920.0 0.0 4097.0 655360.0 169822.8 7569408.0 7202855.4 43280.0
> 25855.8 5271 268.378 5002 9888.754 10157.131
> 81920.0 81920.0 0.0 4097.0 655360.0 169888.3 7569408.0 7202855.4 43280.0
> 25855.8 5271 268.378 5002 9888.754 10157.131
> 81920.0 81920.0 0.0 4097.0 655360.0 170125.7 7569408.0 7202855.4 43280.0
> 25855.8 5271 268.378 5003 9888.946 10157.323
> 81920.0 81920.0 0.0 4097.0 655360.0 170399.8 7569408.0 7202855.3 43280.0
> 25855.8 5271 268.378 5004 9889.106 10157.483
> 81920.0 81920.0 0.0 4097.0 655360.0 170399.8 7569408.0 7202855.3 43280.0
> 25855.8 5271 268.378 5004 9889.106 10157.483
> 81920.0 81920.0 0.0 4097.0 655360.0 170420.7 7569408.0 7202855.3 43280.0
> 25855.8 5271 268.378 5004 9889.106 10157.483
> 81920.0 81920.0 0.0 4097.0 655360.0 170651.4 7569408.0 7202855.3 43280.0
> 25855.8 5271 268.378 5005 9889.301 10157.679
> 81920.0 81920.0 0.0 4097.0 655360.0 170916.8 7569408.0 7202855.2 43280.0
> 25855.8 5271 268.378 5006 9889.464 10157.842
> 81920.0 81920.0 0.0 4097.0 655360.0 170923.6 7569408.0 7202855.2 43280.0
> 25855.8 5271 268.378 5006 9889.464 10157.842
> 81920.0 81920.0 0.0 4097.0 655360.0 170925.6 7569408.0 7202855.2 43280.0
> 25855.8 5271 268.378 5006 9889.464 10157.842
> 81920.0 81920.0 0.0 4097.0 655360.0 171009.8 7569408.0 7202855.2 43280.0
> 25855.8 5271 268.378 5007 9889.660 10158.037
> 81920.0 81920.0 0.0 4097.0 655360.0 171249.8 7569408.0 7202855.2 43280.0
> 25855.8 5271 268.378 5008 9889.823 10158.201
> 81920.0 81920.0 0.0 4097.0 655360.0 171518.8 7569408.0 7202855.2 43280.0
> 25855.8 5271 268.378 5008 9889.823 10158.201
> 81920.0 81920.0 0.0 4097.0 655360.0 171530.6 7569408.0 7202855.2 43280.0
> 25855.8 5271 268.378 5008 9889.823 10158.201
> 81920.0 81920.0 0.0 4097.0 655360.0 171549.2 7569408.0 7202855.2 43280.0
> 25855.8 5271 268.378 5009 9890.024 10158.402
> 81920.0 81920.0 0.0 4097.0 655360.0 173035.0 7569408.0 7202850.4 43280.0
> 25855.8 5271 268.378 5010 9890.190 10158.568
> 81920.0 81920.0 0.0 4097.0 655360.0 173367.8 7569408.0 7202850.4 43280.0
> 25855.8 5271 268.378 5010 9890.190 10158.568
> 81920.0 81920.0 0.0 4097.0 655360.0 173376.8 7569408.0 7202850.4 43280.0
> 25855.8 5271 268.378 5010 9890.190 10158.568
> 81920.0 81920.0 0.0 4097.0 655360.0 173395.5 7569408.0 7202850.4 43280.0
> 25855.8 5271 268.378 5011 9890.456 10158.834
> 81920.0 81920.0 0.0 4097.0 655360.0 173451.8 7569408.0 7202850.3 43280.0
> 25855.8 5271 268.378 5011 9890.456 10158.834
> 81920.0 81920.0 0.0 4097.0 655360.0 173933.5 7569408.0 7202850.3 43280.0
> 25855.8 5271 268.378 5012 9890.622 10158.999
> 81920.0 81920.0 0.0 4097.0 655360.0 173969.7 7569408.0 7202850.3 43280.0
> 25855.8 5271 268.378 5012 9890.622 10158.999
> 81920.0 81920.0 0.0 4097.0 655360.0 173988.3 7569408.0 7202850.3 43280.0
> 25855.8 5271 268.378 5013 9890.820 10159.197
> 81920.0 81920.0 0.0 4097.0 655360.0 174053.8 7569408.0 7202850.2 43280.0
> 25855.8 5271 268.378 5013 9890.820 10159.197
> 81920.0 81920.0 0.0 4097.0 655360.0 174286.1 7569408.0 7202850.2 43280.0
> 25855.8 5271 268.378 5014 9890.986 10159.363
> 81920.0 81920.0 0.0 4097.0 655360.0 174551.0 7569408.0 7202850.2 43280.0
> 25855.8 5271 268.378 5014 9890.986 10159.363
> 81920.0 81920.0 0.0 4097.0 655360.0 174572.0 7569408.0 7202850.2 43280.0
> 25855.8 5271 268.378 5015 9890.986 10159.363
> 81920.0 81920.0 0.0 4097.0 655360.0 174592.9 7569408.0 7202850.1 43280.0
> 25855.8 5271 268.378 5015 9891.185 10159.562
> 81920.0 81920.0 0.0 4097.0 655360.0 174637.4 7569408.0 7202850.1 43280.0
> 25855.8 5271 268.378 5016 9891.350 10159.728
> 81920.0 81920.0 0.0 4097.0 655360.0 175049.8 7569408.0 7202850.1 43280.0
> 25855.8 5271 268.378 5016 9891.350 10159.728
> 81920.0 81920.0 0.0 4097.0 655360.0 175130.7 7569408.0 7202850.1 43280.0
> 25855.8 5271 268.378 5016 9891.350 10159.728
> 81920.0 81920.0 0.0 4097.0 655360.0 175168.4 7569408.0 7202850.1 43280.0
> 25855.8 5271 268.378 5017 9891.554 10159.932
> 81920.0 81920.0 0.0 4097.0 655360.0 175182.2 7569408.0 7202850.1 43280.0
> 25855.8 5271 268.378 5018 9891.721 10160.098
> 81920.0 81920.0 0.0 4097.0 655360.0 175402.9 7569408.0 7202850.1 43280.0
> 25855.8 5271 268.378 5018 9891.721 10160.098
> 81920.0 81920.0 0.0 4097.0 655360.0 175723.5 7569408.0 7202850.1 43280.0
> 25855.8 5271 268.378 5018 9891.721 10160.098
> 81920.0 81920.0 0.0 4097.0 655360.0 175742.2 7569408.0 7202850.1 43280.0
> 25855.8 5271 268.378 5019 9891.921 10160.298
> 81920.0 81920.0 0.0 4097.0 655360.0 175778.9 7569408.0 7202850.0 43280.0
> 25855.8 5271 268.378 5020 9892.089 10160.466
> 81920.0 81920.0 0.0 4097.0 655360.0 175823.6 7569408.0 7202850.0 43280.0
> 25855.8 5271 268.378 5020 9892.089 10160.466
> 81920.0 81920.0 0.0 4097.0 655360.0 176315.6 7569408.0 7202850.0 43280.0
> 25855.8 5271 268.378 5020 9892.089 10160.466
> 81920.0 81920.0 0.0 4097.0 655360.0 176351.6 7569408.0 7202850.0 43280.0
> 25855.8 5271 268.378 5021 9892.291 10160.669
> 81920.0 81920.0 0.0 4097.0 655360.0 176351.6 7569408.0 7202849.9 43280.0
> 25855.8 5271 268.378 5022 9892.460 10160.838
> 81920.0 81920.0 0.0 4097.0 655360.0 176370.6 7569408.0 7202849.9 43280.0
> 25855.8 5271 268.378 5022 9892.460 10160.838
> 81920.0 81920.0 0.0 4097.0 655360.0 176846.1 7569408.0 7202849.9 43280.0
> 25855.8 5271 268.378 5022 9892.460 10160.838
> 81920.0 81920.0 0.0 4097.0 655360.0 176865.2 7569408.0 7202849.9 43280.0
> 25855.8 5271 268.378 5023 9892.665 10161.043
> 81920.0 81920.0 0.0 4097.0 655360.0 176865.2 7569408.0 7202849.9 43280.0
> 25855.8 5271 268.378 5024 9892.834 10161.212
> 81920.0 81920.0 0.0 4097.0 655360.0 176942.8 7569408.0 7202849.9 43280.0
> 25855.8 5271 268.378 5024 9892.834 10161.212
> 81920.0 81920.0 0.0 4097.0 655360.0 177025.1 7569408.0 7202849.9 43280.0
> 25855.8 5271 268.378 5024 9892.834 10161.212
> 81920.0 81920.0 0.0 4097.0 655360.0 177437.4 7569408.0 7202849.9 43280.0
> 25855.8 5271 268.378 5025 9893.039 10161.416
> 81920.0 81920.0 0.0 4097.0 655360.0 177468.4 7569408.0 7202849.9 43280.0
> 25855.8 5271 268.378 5026 9893.213 10161.590
> 81920.0 81920.0 0.0 4097.0 655360.0 177472.6 7569408.0 7202849.9 43280.0
> 25855.8 5271 268.378 5026 9893.213 10161.590
> 81920.0 81920.0 0.0 4097.0 655360.0 177556.8 7569408.0 7202849.9 43280.0
> 25855.8 5271 268.378 5026 9893.213 10161.590
> 81920.0 81920.0 0.0 4097.0 655360.0 178032.6 7569408.0 7202849.9 43280.0
> 25855.8 5271 268.378 5027 9893.454 10161.832
> 81920.0 81920.0 0.0 4097.0 655360.0 178059.4 7569408.0 7202849.8 43280.0
> 25855.8 5271 268.378 5028 9893.624 10162.001
> 81920.0 81920.0 0.0 4097.0 655360.0 178059.4 7569408.0 7202849.8 43280.0
> 25855.8 5271 268.378 5028 9893.624 10162.001
> 81920.0 81920.0 0.0 4097.0 655360.0 178059.4 7569408.0 7202849.8 43280.0
> 25855.8 5271 268.378 5028 9893.624 10162.001
> 81920.0 81920.0 0.0 4097.0 655360.0 178300.8 7569408.0 7202849.8 43280.0
> 25855.8 5271 268.378 5029 9893.901 10162.279
> 81920.0 81920.0 0.0 4097.0 655360.0 178630.4 7569408.0 7202849.7 43280.0
> 25855.8 5271 268.378 5030 9894.075 10162.453
> 81920.0 81920.0 0.0 4097.0 655360.0 178638.6 7569408.0 7202849.7 43280.0
> 25855.8 5271 268.378 5030 9894.075 10162.453
> 81920.0 81920.0 0.0 4097.0 655360.0 178638.6 7569408.0 7202849.7 43280.0
> 25855.8 5271 268.378 5030 9894.075 10162.453
> 81920.0 81920.0 0.0 4097.0 655360.0 178734.5 7569408.0 7202849.7 43280.0
> 25855.8 5271 268.378 5031 9894.294 10162.672
> 81920.0 81920.0 0.0 4097.0 655360.0 179272.2 7569408.0 7202849.7 43280.0
> 25855.8 5271 268.378 5032 9894.468 10162.845
> 81920.0 81920.0 0.0 4097.0 655360.0 179308.5 7569408.0 7202849.7 43280.0
> 25855.8 5271 268.378 5032 9894.468 10162.845
> 81920.0 81920.0 0.0 4097.0 655360.0 179308.5 7569408.0 7202849.7 43280.0
> 25855.8 5271 268.378 5032 9894.468 10162.845
> 81920.0 81920.0 0.0 4097.0 655360.0 179308.5 7569408.0 7202849.7 43280.0
> 25855.8 5271 268.378 5033 9894.681 10163.058
> 81920.0 81920.0 0.0 4097.0 655360.0 179784.0 7569408.0 7202849.6 43280.0
> 25855.8 5271 268.378 5033 9894.681 10163.058
> 81920.0 81920.0 0.0 4097.0 655360.0 179804.6 7569408.0 7202849.6 43280.0
> 25855.8 5271 268.378 5034 9894.854 10163.231
> 81920.0 81920.0 0.0 4097.0 655360.0 181033.4 7569408.0 7202849.6 43280.0
> 25855.8 5271 268.378 5034 9894.854 10163.231
> 81920.0 81920.0 0.0 4097.0 655360.0 181033.4 7569408.0 7202849.6 43280.0
> 25855.8 5271 268.378 5035 9895.135 10163.513
> 81920.0 81920.0 0.0 4097.0 655360.0 181052.0 7569408.0 7202849.5 43280.0
> 25855.8 5271 268.378 5035 9895.135 10163.513
> 81920.0 81920.0 0.0 4097.0 655360.0 181509.0 7569408.0 7202849.5 43280.0
> 25855.8 5271 268.378 5036 9895.311 10163.689
> 81920.0 81920.0 0.0 4097.0 655360.0 181608.5 7569408.0 7202849.5 43280.0
> 25855.8 5271 268.378 5036 9895.311 10163.689
> 81920.0 81920.0 0.0 4097.0 655360.0 181610.7 7569408.0 7202849.5 43280.0
> 25855.8 5271 268.378 5037 9895.311 10163.689
> 81920.0 81920.0 0.0 4097.0 655360.0 181640.3 7569408.0 7202849.5 43280.0
> 25855.8 5271 268.378 5037 9895.539 10163.917
> 81920.0 81920.0 0.0 4097.0 655360.0 182116.0 7569408.0 7202849.5 43280.0
> 25855.8 5271 268.378 5038 9895.716 10164.093
> 81920.0 81920.0 0.0 4097.0 655360.0 182207.3 7569408.0 7202849.5 43280.0
> 25855.8 5271 268.378 5038 9895.716 10164.093
> 81920.0 81920.0 0.0 4097.0 655360.0 182281.0 7569408.0 7202849.5 43280.0
> 25855.8 5271 268.378 5038 9895.716 10164.093
> 81920.0 81920.0 0.0 4097.0 655360.0 182281.0 7569408.0 7202849.5 43280.0
> 25855.8 5271 268.378 5039 9896.000 10164.378
> 81920.0 81920.0 0.0 4097.0 655360.0 182285.1 7569408.0 7202849.5 43280.0
> 25855.8 5271 268.378 5040 9896.180 10164.558
> 81920.0 81920.0 0.0 4097.0 655360.0 182844.7 7569408.0 7202849.5 43280.0
> 25855.8 5271 268.378 5040 9896.180 10164.558
> 81920.0 81920.0 0.0 4097.0 655360.0 182872.0 7569408.0 7202849.5 43280.0
> 25855.8 5271 268.378 5040 9896.180 10164.558
> 81920.0 81920.0 0.0 4097.0 655360.0 182872.0 7569408.0 7202849.5 43280.0
> 25855.8 5271 268.378 5041 9896.463 10164.841
> 81920.0 81920.0 0.0 4097.0 655360.0 182872.0 7569408.0 7202849.3 43280.0
> 25855.8 5271 268.378 5042 9896.641 10165.019
> 81920.0 81920.0 0.0 4097.0 655360.0 183413.3 7569408.0 7202849.3 43280.0
> 25855.8 5271 268.378 5042 9896.641 10165.019
> 81920.0 81920.0 0.0 4097.0 655360.0 183443.4 7569408.0 7202849.3 43280.0
> 25855.8 5271 268.378 5042 9896.641 10165.019
> 81920.0 81920.0 0.0 4097.0 655360.0 183464.8 7569408.0 7202849.3 43280.0
> 25855.8 5271 268.378 5043 9896.857 10165.235
> 81920.0 81920.0 0.0 4097.0 655360.0 183476.6 7569408.0 7202849.2 43280.0
> 25855.8 5271 268.378 5044 9897.033 10165.411
> 81920.0 81920.0 0.0 4097.0 655360.0 183733.3 7569408.0 7202849.2 43280.0
> 25855.8 5271 268.378 5044 9897.033 10165.411
> 81920.0 81920.0 0.0 4097.0 655360.0 184045.6 7569408.0 7202849.2 43280.0
> 25855.8 5271 268.378 5044 9897.033 10165.411
> 81920.0 81920.0 0.0 4097.0 655360.0 184055.9 7569408.0 7202849.2 43280.0
> 25855.8 5271 268.378 5045 9897.314 10165.691
> 81920.0 81920.0 0.0 4097.0 655360.0 184057.9 7569408.0 7202849.2 43280.0
> 25855.8 5271 268.378 5046 9897.492 10165.869
> 81920.0 81920.0 0.0 4097.0 655360.0 184057.9 7569408.0 7202849.2 43280.0
> 25855.8 5271 268.378 5046 9897.492 10165.869
> 81920.0 81920.0 0.0 4097.0 655360.0 184618.0 7569408.0 7202849.2 43280.0
> 25855.8 5271 268.378 5046 9897.492 10165.869
> 81920.0 81920.0 0.0 4097.0 655360.0 184644.8 7569408.0 7202849.2 43280.0
> 25855.8 5271 268.378 5047 9897.706 10166.084
> 81920.0 81920.0 0.0 4097.0 655360.0 184647.0 7569408.0 7202849.1 43280.0
> 25855.8 5271 268.378 5048 9897.886 10166.264
> 81920.0 81920.0 0.0 4097.0 655360.0 184647.0 7569408.0 7202849.1 43280.0
> 25855.8 5271 268.378 5048 9897.886 10166.264
> 81920.0 81920.0 0.0 4097.0 655360.0 185105.9 7569408.0 7202849.1 43280.0
> 25855.8 5271 268.378 5048 9897.886 10166.264
> 81920.0 81920.0 0.0 4097.0 655360.0 185231.6 7569408.0 7202849.1 43280.0
> 25855.8 5271 268.378 5049 9898.095 10166.472
> 81920.0 81920.0 0.0 4097.0 655360.0 185239.8 7569408.0 7202849.1 43280.0
> 25855.8 5271 268.378 5050 9898.276 10166.654
> 81920.0 81920.0 0.0 4097.0 655360.0 185251.5 7569408.0 7202849.1 43280.0
> 25855.8 5271 268.378 5050 9898.276 10166.654
> 81920.0 81920.0 0.0 4097.0 655360.0 194817.6 7569408.0 7202849.1 43280.0
> 25855.8 5271 268.378 5050 9898.276 10166.654
> 81920.0 81920.0 0.0 4097.0 655360.0 195326.2 7569408.0 7202849.1 43280.0
> 25855.8 5271 268.378 5051 9898.570 10166.948
> 81920.0 81920.0 0.0 4097.0 655360.0 195333.0 7569408.0 7202849.0 43280.0
> 25855.8 5271 268.378 5052 9898.752 10167.130
> 81920.0 81920.0 0.0 4097.0 655360.0 195341.3 7569408.0 7202849.0 43280.0
> 25855.8 5271 268.378 5052 9898.752 10167.130
> 81920.0 81920.0 0.0 4097.0 655360.0 195341.3 7569408.0 7202849.0 43280.0
> 25855.8 5271 268.378 5052 9898.752 10167.130
> 81920.0 81920.0 0.0 4097.0 655360.0 196751.1 7569408.0 7202849.0 43280.0
> 25855.8 5271 268.378 5053 9899.045 10167.423
> 81920.0 81920.0 0.0 4097.0 655360.0 196934.4 7569408.0 7202848.8 43280.0
> 25855.8 5271 268.378 5053 9899.045 10167.423
> 81920.0 81920.0 0.0 4097.0 655360.0 196944.7 7569408.0 7202848.8 43280.0
> 25855.8 5271 268.378 5054 9899.230 10167.608
> 81920.0 81920.0 0.0 4097.0 655360.0 197021.9 7569408.0 7202848.8 43280.0
> 25855.8 5271 268.378 5054 9899.230 10167.608
> 81920.0 81920.0 0.0 4097.0 655360.0 197037.8 7569408.0 7202848.8 43280.0
> 25855.8 5271 268.378 5055 9899.451 10167.829
> 81920.0 81920.0 0.0 4097.0 655360.0 197541.2 7569408.0 7202848.7 43280.0
> 25855.8 5271 268.378 5055 9899.451 10167.829
> 81920.0 81920.0 0.0 4097.0 655360.0 197541.2 7569408.0 7202848.7 43280.0
> 25855.8 5271 268.378 5056 9899.638 10168.015
> 81920.0 81920.0 0.0 4097.0 655360.0 197614.9 7569408.0 7202848.7 43280.0
> 25855.8 5271 268.378 5056 9899.638 10168.015
> 81920.0 81920.0 0.0 4097.0 655360.0 197617.0 7569408.0 7202848.7 43280.0
> 25855.8 5271 268.378 5057 9899.858 10168.235
> 81920.0 81920.0 0.0 4097.0 655360.0 198123.4 7569408.0 7202848.7 43280.0
> 25855.8 5271 268.378 5057 9899.858 10168.235
> 81920.0 81920.0 0.0 4097.0 655360.0 198130.2 7569408.0 7202848.7 43280.0
> 25855.8 5271 268.378 5058 9900.043 10168.420
> 81920.0 81920.0 0.0 4097.0 655360.0 198140.7 7569408.0 7202848.7 43280.0
> 25855.8 5271 268.378 5058 9900.043 10168.420
> 81920.0 81920.0 0.0 4097.0 655360.0 199340.2 7569408.0 7202848.7 43280.0
> 25855.8 5271 268.378 5059 9900.340 10168.718
> 81920.0 81920.0 0.0 4097.0 655360.0 199580.3 7569408.0 7202848.6 43280.0
> 25855.8 5271 268.378 5059 9900.340 10168.718
> 81920.0 81920.0 0.0 4097.0 655360.0 199843.5 7569408.0 7202848.6 43280.0
> 25855.8 5271 268.378 5060 9900.528 10168.905
> 81920.0 81920.0 0.0 4097.0 655360.0 199917.3 7569408.0 7202848.6 43280.0
> 25855.8 5271 268.378 5060 9900.528 10168.905
> 81920.0 81920.0 0.0 4097.0 655360.0 200007.7 7569408.0 7202848.6 43280.0
> 25855.8 5271 268.378 5061 9900.528 10168.905
> 81920.0 81920.0 0.0 4097.0 655360.0 200009.8 7569408.0 7202848.6 43280.0
> 25855.8 5271 268.378 5061 9900.818 10169.196
> 81920.0 81920.0 0.0 4097.0 655360.0 200466.7 7569408.0 7202848.6 43280.0
> 25855.8 5271 268.378 5062 9901.005 10169.383
> 81920.0 81920.0 0.0 4097.0 655360.0 200513.2 7569408.0 7202848.6 43280.0
> 25855.8 5271 268.378 5062 9901.005 10169.383
> 81920.0 81920.0 0.0 4097.0 655360.0 200521.4 7569408.0 7202848.6 43280.0
> 25855.8 5271 268.378 5062 9901.005 10169.383
> 81920.0 81920.0 0.0 4097.0 655360.0 200521.4 7569408.0 7202848.5 43280.0
> 25855.8 5271 268.378 5063 9901.302 10169.680
> 81920.0 81920.0 0.0 4097.0 655360.0 200759.4 7569408.0 7202848.5 43280.0
> 25855.8 5271 268.378 5064 9901.490 10169.868
> 81920.0 81920.0 0.0 4097.0 655360.0 201018.1 7569408.0 7202848.5 43280.0
> 25855.8 5271 268.378 5064 9901.490 10169.868
> 81920.0 81920.0 0.0 4097.0 655360.0 201045.0 7569408.0 7202848.5 43280.0
> 25855.8 5271 268.378 5064 9901.490 10169.868
> 81920.0 81920.0 0.0 4097.0 655360.0 201112.5 7569408.0 7202848.5 43280.0
> 25855.8 5271 268.378 5065 9901.710 10170.087
> 81920.0 81920.0 0.0 4097.0 655360.0 201112.5 7569408.0 7202848.5 43280.0
> 25855.8 5271 268.378 5066 9901.900 10170.277
> 81920.0 81920.0 0.0 4097.0 655360.0 201590.6 7569408.0 7202848.5 43280.0
> 25855.8 5271 268.378 5066 9901.900 10170.277
> 81920.0 81920.0 0.0 4097.0 655360.0 201613.2 7569408.0 7202848.5 43280.0
> 25855.8 5271 268.378 5066 9901.900 10170.277
> 81920.0 81920.0 0.0 4097.0 655360.0 201640.1 7569408.0 7202848.5 43280.0
> 25855.8 5271 268.378 5067 9902.200 10170.578
> 81920.0 81920.0 0.0 4097.0 655360.0 201717.4 7569408.0 7202848.4 43280.0
> 25855.8 5271 268.378 5068 9902.390 10170.768
> 81920.0 81920.0 0.0 4097.0 655360.0 202023.1 7569408.0 7202848.4 43280.0
> 25855.8 5271 268.378 5068 9902.390 10170.768
> 81920.0 81920.0 0.0 4097.0 655360.0 202267.7 7569408.0 7202848.4 43280.0
> 25855.8 5271 268.378 5068 9902.390 10170.768
> 81920.0 81920.0 0.0 4097.0 655360.0 202278.1 7569408.0 7202848.4 43280.0
> 25855.8 5271 268.378 5069 9902.693 10171.071
> 81920.0 81920.0 0.0 4097.0 655360.0 202362.5 7569408.0 7202848.4 43280.0
> 25855.8 5271 268.378 5070 9902.883 10171.261
> 81920.0 81920.0 0.0 4097.0 655360.0 202600.7 7569408.0 7202848.4 43280.0
> 25855.8 5271 268.378 5070 9902.883 10171.261
> 81920.0 81920.0 0.0 4097.0 655360.0 202854.2 7569408.0 7202848.4 43280.0
> 25855.8 5271 268.378 5070 9902.883 10171.261
> 81920.0 81920.0 0.0 4097.0 655360.0 202926.5 7569408.0 7202848.3 43280.0
> 25855.8 5271 268.378 5071 9903.113 10171.491
> 81920.0 81920.0 0.0 4097.0 655360.0 202934.8 7569408.0 7202848.2 43280.0
> 25855.8 5271 268.378 5072 9903.303 10171.681
> 81920.0 81920.0 0.0 4097.0 655360.0 202955.5 7569408.0 7202848.2 43280.0
> 25855.8 5271 268.378 5072 9903.303 10171.681
> 81920.0 81920.0 0.0 4097.0 655360.0 203291.3 7569408.0 7202848.2 43280.0
> 25855.8 5271 268.378 5072 9903.303 10171.681
> 81920.0 81920.0 0.0 4097.0 655360.0 203517.0 7569408.0 7202848.2 43280.0
> 25855.8 5271 268.378 5073 9903.607 10171.985
> 81920.0 81920.0 0.0 4097.0 655360.0 203539.0 7569408.0 7202848.0 43280.0
> 25855.8 5271 268.378 5074 9903.799 10172.177
> 81920.0 81920.0 0.0 4097.0 655360.0 203557.6 7569408.0 7202848.0 43280.0
> 25855.8 5271 268.378 5074 9903.799 10172.177
> 81920.0 81920.0 0.0 4097.0 655360.0 203795.7 7569408.0 7202848.0 43280.0
> 25855.8 5271 268.378 5074 9903.799 10172.177
> 81920.0 81920.0 0.0 4097.0 655360.0 204042.6 7569408.0 7202848.0 43280.0
> 25855.8 5271 268.378 5075 9904.025 10172.403
> 81920.0 81920.0 0.0 4097.0 655360.0 204042.6 7569408.0 7202848.0 43280.0
> 25855.8 5271 268.378 5076 9904.220 10172.598
> 81920.0 81920.0 0.0 4097.0 655360.0 204135.0 7569408.0 7202848.0 43280.0
> 25855.8 5271 268.378 5076 9904.220 10172.598
> 81920.0 81920.0 0.0 4097.0 655360.0 204137.0 7569408.0 7202848.0 43280.0
> 25855.8 5271 268.378 5076 9904.220 10172.598
> 81920.0 81920.0 0.0 4097.0 655360.0 204582.2 7569408.0 7202848.0 43280.0
> 25855.8 5271 268.378 5077 9904.451 10172.829
> 81920.0 81920.0 0.0 4097.0 655360.0 204633.6 7569408.0 7202847.9 43280.0
> 25855.8 5271 268.378 5078 9904.644 10173.022
> 81920.0 81920.0 0.0 4097.0 655360.0 204660.4 7569408.0 7202847.9 43280.0
> 25855.8 5271 268.378 5078 9904.644 10173.022
> 81920.0 81920.0 0.0 4097.0 655360.0 204741.6 7569408.0 7202847.9 43280.0
> 25855.8 5271 268.378 5078 9904.644 10173.022
> 81920.0 81920.0 0.0 4097.0 655360.0 205000.7 7569408.0 7202847.9 43280.0
> 25855.8 5271 268.378 5079 9904.952 10173.329
> 81920.0 81920.0 0.0 4097.0 655360.0 205226.3 7569408.0 7202847.8 43280.0
> 25855.8 5271 268.378 5080 9904.952 10173.329
> 81920.0 81920.0 0.0 4097.0 655360.0 205228.5 7569408.0 7202847.8 43280.0
> 25855.8 5271 268.378 5080 9905.144 10173.521
> 81920.0 81920.0 0.0 4097.0 655360.0 205267.2 7569408.0 7202847.8 43280.0
> 25855.8 5271 268.378 5080 9905.144 10173.521
> 81920.0 81920.0 0.0 4097.0 655360.0 205267.2 7569408.0 7202847.8 43280.0
> 25855.8 5271 268.378 5081 9905.369 10173.746
> 81920.0 81920.0 0.0 4097.0 655360.0 205698.6 7569408.0 7202847.8 43280.0
> 25855.8 5271 268.378 5081 9905.369 10173.746
> 81920.0 81920.0 0.0 4097.0 655360.0 205752.3 7569408.0 7202847.8 43280.0
> 25855.8 5271 268.378 5082 9905.563 10173.940
> 81920.0 81920.0 0.0 4097.0 655360.0 205844.6 7569408.0 7202847.8 43280.0
> 25855.8 5271 268.378 5082 9905.563 10173.940
> 81920.0 81920.0 0.0 4097.0 655360.0 205846.6 7569408.0 7202847.8 43280.0
> 25855.8 5271 268.378 5083 9905.797 10174.174
> 81920.0 81920.0 0.0 4097.0 655360.0 206115.6 7569408.0 7202847.8 43280.0
> 25855.8 5271 268.378 5083 9905.797 10174.174
> 81920.0 81920.0 0.0 4097.0 655360.0 206406.6 7569408.0 7202847.8 43280.0
> 25855.8 5271 268.378 5084 9905.992 10174.369
> 81920.0 81920.0 0.0 4097.0 655360.0 207640.6 7569408.0 7202847.8 43280.0
> 25855.8 5271 268.378 5084 9905.992 10174.369
> 81920.0 81920.0 0.0 4097.0 655360.0 207650.8 7569408.0 7202847.8 43280.0
> 25855.8 5271 268.378 5085 9906.220 10174.598
> 81920.0 81920.0 0.0 4097.0 655360.0 207908.0 7569408.0 7202847.7 43280.0
> 25855.8 5271 268.378 5085 9906.220 10174.598
> 81920.0 81920.0 0.0 4097.0 655360.0 208149.9 7569408.0 7202847.7 43280.0
> 25855.8 5271 268.378 5086 9906.417 10174.794
> 81920.0 81920.0 0.0 4097.0 655360.0 208201.3 7569408.0 7202847.7 43280.0
> 25855.8 5271 268.378 5086 9906.417 10174.794
> 81920.0 81920.0 0.0 4097.0 655360.0 208240.0 7569408.0 7202847.7 43280.0
> 25855.8 5271 268.378 5087 9906.417 10174.794
> 81920.0 81920.0 0.0 4097.0 655360.0 208240.0 7569408.0 7202847.6 43280.0
> 25855.8 5271 268.378 5087 9906.654 10175.031
> 81920.0 81920.0 0.0 4097.0 655360.0 208501.2 7569408.0 7202847.6 43280.0
> 25855.8 5271 268.378 5088 9906.853 10175.230
> 81920.0 81920.0 0.0 4097.0 655360.0 208726.8 7569408.0 7202847.6 43280.0
> 25855.8 5271 268.378 5088 9906.853 10175.230
> 81920.0 81920.0 0.0 4097.0 655360.0 208726.8 7569408.0 7202847.6 43280.0
> 25855.8 5271 268.378 5088 9906.853 10175.230
> 81920.0 81920.0 0.0 4097.0 655360.0 208821.1 7569408.0 7202847.6 43280.0
> 25855.8 5271 268.378 5089 9907.164 10175.542
> 81920.0 81920.0 0.0 4097.0 655360.0 209059.2 7569408.0 7202847.6 43280.0
> 25855.8 5271 268.378 5090 9907.365 10175.743
> 81920.0 81920.0 0.0 4097.0 655360.0 209398.1 7569408.0 7202847.6 43280.0
> 25855.8 5271 268.378 5090 9907.365 10175.743
> 81920.0 81920.0 0.0 4097.0 655360.0 209400.4 7569408.0 7202847.6 43280.0
> 25855.8 5271 268.378 5090 9907.365 10175.743
> 81920.0 81920.0 0.0 4097.0 655360.0 209408.6 7569408.0 7202847.6 43280.0
> 25855.8 5271 268.378 5091 9907.596 10175.973
> 81920.0 81920.0 0.0 4097.0 655360.0 209427.3 7569408.0 7202847.5 43280.0
> 25855.8 5271 268.378 5092 9907.793 10176.171
> 81920.0 81920.0 0.0 4097.0 655360.0 209684.5 7569408.0 7202847.5 43280.0
> 25855.8 5271 268.378 5092 9907.793 10176.171
> 81920.0 81920.0 0.0 4097.0 655360.0 209910.1 7569408.0 7202847.5 43280.0
> 25855.8 5271 268.378 5092 9907.793 10176.171
> 81920.0 81920.0 0.0 4097.0 655360.0 209932.1 7569408.0 7202847.5 43280.0
> 25855.8 5271 268.378 5093 9908.028 10176.405
> 81920.0 81920.0 0.0 4097.0 655360.0 209952.8 7569408.0 7202847.4 43280.0
> 25855.8 5271 268.378 5094 9908.225 10176.603
> 81920.0 81920.0 0.0 4097.0 655360.0 210190.9 7569408.0 7202847.4 43280.0
> 25855.8 5271 268.378 5094 9908.225 10176.603
> 81920.0 81920.0 0.0 4097.0 655360.0 210500.9 7569408.0 7202847.4 43280.0
> 25855.8 5271 268.378 5094 9908.225 10176.603
> 81920.0 81920.0 0.0 4097.0 655360.0 210500.9 7569408.0 7202847.4 43280.0
> 25855.8 5271 268.378 5095 9908.470 10176.848
> 81920.0 81920.0 0.0 4097.0 655360.0 210509.1 7569408.0 7202847.4 43280.0
> 25855.8 5271 268.378 5096 9908.668 10177.046
> 81920.0 81920.0 0.0 4097.0 655360.0 210785.8 7569408.0 7202847.4 43280.0
> 25855.8 5271 268.378 5096 9908.668 10177.046
> 81920.0 81920.0 0.0 4097.0 655360.0 210818.8 7569408.0 7202847.4 43280.0
> 25855.8 5271 268.378 5096 9908.668 10177.046
> 81920.0 81920.0 0.0 4097.0 655360.0 211109.8 7569408.0 7202847.4 43280.0
> 25855.8 5271 268.378 5097 9908.902 10177.279
> 81920.0 81920.0 0.0 4097.0 655360.0 211175.3 7569408.0 7202847.4 43280.0
> 25855.8 5271 268.378 5098 9909.103 10177.481
> 81920.0 81920.0 0.0 4097.0 655360.0 211267.7 7569408.0 7202847.4 43280.0
> 25855.8 5271 268.378 5098 9909.103 10177.481
> 81920.0 81920.0 0.0 4097.0 655360.0 211527.0 7569408.0 7202847.4 43280.0
> 25855.8 5271 268.378 5098 9909.103 10177.481
> 81920.0 81920.0 0.0 4097.0 655360.0 211754.7 7569408.0 7202847.4 43280.0
> 25855.8 5271 268.378 5099 9909.341 10177.718
> 81920.0 81920.0 0.0 4097.0 655360.0 211766.4 7569408.0 7202847.2 43280.0
> 25855.8 5271 268.378 5100 9909.542 10177.920
> 81920.0 81920.0 0.0 4097.0 655360.0 211840.2 7569408.0 7202847.2 43280.0
> 25855.8 5271 268.378 5100 9909.542 10177.920
> 81920.0 81920.0 0.0 4097.0 655360.0 212116.0 7569408.0 7202847.2 43280.0
> 25855.8 5271 268.378 5100 9909.542 10177.920
> 81920.0 81920.0 0.0 4097.0 655360.0 212122.8 7569408.0 7202847.2 43280.0
> 25855.8 5271 268.378 5101 9909.783 10178.161
> 81920.0 81920.0 0.0 4097.0 655360.0 212411.4 7569408.0 7202847.2 43280.0
> 25855.8 5271 268.378 5102 9909.783 10178.161
> 81920.0 81920.0 0.0 4097.0 655360.0 212424.6 7569408.0 7202847.2 43280.0
> 25855.8 5271 268.378 5102 9909.987 10178.365
> 81920.0 81920.0 0.0 4097.0 655360.0 212432.8 7569408.0 7202847.2 43280.0
> 25855.8 5271 268.378 5102 9909.987 10178.365
> 81920.0 81920.0 0.0 4097.0 655360.0 212720.2 7569408.0 7202847.2 43280.0
> 25855.8 5271 268.378 5103 9910.313 10178.690
> 81920.0 81920.0 0.0 4097.0 655360.0 212945.8 7569408.0 7202847.1 43280.0
> 25855.8 5271 268.378 5103 9910.313 10178.690
> 81920.0 81920.0 0.0 4097.0 655360.0 212945.8 7569408.0 7202847.1 43280.0
> 25855.8 5271 268.378 5104 9910.516 10178.893
> 81920.0 81920.0 0.0 4097.0 655360.0 212958.1 7569408.0 7202847.1 43280.0
> 25855.8 5271 268.378 5104 9910.516 10178.893
> 81920.0 81920.0 0.0 4097.0 655360.0 213301.3 7569408.0 7202847.1 43280.0
> 25855.8 5271 268.378 5105 9910.851 10179.229
> 81920.0 81920.0 0.0 4097.0 655360.0 213482.3 7569408.0 7202847.0 43280.0
> 25855.8 5271 268.378 5105 9910.851 10179.229
> 81920.0 81920.0 0.0 4097.0 655360.0 213527.0 7569408.0 7202847.0 43280.0
> 25855.8 5271 268.378 5106 9911.056 10179.433
> 81920.0 81920.0 0.0 4097.0 655360.0 213547.0 7569408.0 7202847.0 43280.0
> 25855.8 5271 268.378 5106 9911.056 10179.433
> 81920.0 81920.0 0.0 4097.0 655360.0 213612.5 7569408.0 7202847.0 43280.0
> 25855.8 5271 268.378 5107 9911.307 10179.684
> 81920.0 81920.0 0.0 4097.0 655360.0 213892.6 7569408.0 7202847.0 43280.0
> 25855.8 5271 268.378 5107 9911.307 10179.684
> 81920.0 81920.0 0.0 4097.0 655360.0 214133.8 7569408.0 7202847.0 43280.0
> 25855.8 5271 268.378 5108 9911.513 10179.891
> 81920.0 81920.0 0.0 4097.0 655360.0 214133.8 7569408.0 7202847.0 43280.0
> 25855.8 5271 268.378 5108 9911.513 10179.891
> 81920.0 81920.0 0.0 4097.0 655360.0 215407.3 7569408.0 7202847.0 43280.0
> 25855.8 5271 268.378 5108 9911.513 10179.891
> 81920.0 81920.0 0.0 4097.0 655360.0 215677.9 7569408.0 7202846.9 43280.0
> 25855.8 5271 268.378 5109 9911.822 10180.199
> 81920.0 81920.0 0.0 4097.0 655360.0 215879.9 7569408.0 7202846.9 43280.0
> 25855.8 5271 268.378 5110 9912.026 10180.404
> 81920.0 81920.0 0.0 4097.0 655360.0 215924.5 7569408.0 7202846.9 43280.0
> 25855.8 5271 268.378 5110 9912.026 10180.404
> 81920.0 81920.0 0.0 4097.0 655360.0 215932.7 7569408.0 7202846.9 43280.0
> 25855.8 5271 268.378 5110 9912.026 10180.404
> 81920.0 81920.0 0.0 4097.0 655360.0 216189.6 7569408.0 7202846.9 43280.0
> 25855.8 5271 268.378 5111 9912.347 10180.725
> 81920.0 81920.0 0.0 4097.0 655360.0 216208.6 7569408.0 7202846.9 43280.0
> 25855.8 5271 268.378 5112 9912.556 10180.934
> 81920.0 81920.0 0.0 4097.0 655360.0 216504.1 7569408.0 7202846.9 43280.0
> 25855.8 5271 268.378 5112 9912.556 10180.934
> 81920.0 81920.0 0.0 4097.0 655360.0 216517.9 7569408.0 7202846.9 43280.0
> 25855.8 5271 268.378 5112 9912.556 10180.934
> {quote}
> This continues even after we shut down our app. Nothing is connected to
> Cassandra any more, yet it is still stuck in full GC and cannot free up
> memory.
> Running nodetool tpstats shows that nothing is pending, all seems OK:
> {quote}
> Pool Name Active Pending Completed Blocked All
> time blocked
> ReadStage 0 0 69555935 0
> 0
> RequestResponseStage 0 0 0 0
> 0
> MutationStage 0 0 73123690 0
> 0
> ReadRepairStage 0 0 0 0
> 0
> ReplicateOnWriteStage 0 0 0 0
> 0
> GossipStage 0 0 0 0
> 0
> CacheCleanupExecutor 0 0 0 0
> 0
> MigrationStage 0 0 46 0
> 0
> MemoryMeter 0 0 1125 0
> 0
> FlushWriter 0 0 824 0
> 30
> ValidationExecutor 0 0 0 0
> 0
> InternalResponseStage 0 0 23 0
> 0
> AntiEntropyStage 0 0 0 0
> 0
> MemtablePostFlusher 0 0 1783 0
> 0
> MiscStage 0 0 0 0
> 0
> PendingRangeCalculator 0 0 1 0
> 0
> CompactionExecutor 0 0 74330 0
> 0
> commitlog_archiver 0 0 0 0
> 0
> HintedHandoff 0 0 0 0
> 0
> Message type Dropped
> RANGE_SLICE 0
> READ_REPAIR 0
> PAGED_RANGE 0
> BINARY 0
> READ 585
> MUTATION 75775
> _TRACE 0
> REQUEST_RESPONSE 0
> COUNTER_MUTATION 0
> {quote}
> We had this happen on 2 separate boxes, one with 2.0.6, the other with 2.0.8.
> Right now this is a total blocker for us. We are unable to process the
> customer data and have to abort in the middle of large processing.
> This is a new customer, so we did not have a chance to see if this occurred
> with 1.1 or 1.2 in the past (we moved to 2.0 recently).
> We have the Cassandra process still running, pls let us know if there is
> anything else we could run to give you more insight.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
