Hi,
I have problem with Cassandra on Dublin release.
I checked Casandra logs and found that gc is done very often and after one of
gc threads were unavailable for
2019-08-13T10:34:32.507+0000: 6065.138: Total time for which application
threads were stopped: 68.5349949 seconds, Stopping threads took: 68.5339197
seconds
2019-08-13T10:34:34.533+0000: 6067.164: Total time for which application
threads were stopped: 0.1177914 seconds, Stopping threads took: 0.1173895
seconds
2019-08-13T10:34:34.534+0000: 6067.165: Total time for which application
threads were stopped: 0.0002319 seconds, Stopping threads took: 0.0001111
seconds
2019-08-13T10:34:34.543+0000: 6067.174: Total time for which application
threads were stopped: 0.0044992 seconds, Stopping threads took: 0.0043041
seconds
2019-08-13T10:34:34.558+0000: 6067.188: Total time for which application
threads were stopped: 0.0062902 seconds, Stopping threads took: 0.0058122
seconds
2019-08-13T10:34:34.647+0000: 6067.278: Total time for which application
threads were stopped: 0.0133881 seconds, Stopping threads took: 0.0130333
seconds
2019-08-13T10:34:34.746+0000: 6067.376: Total time for which application
threads were stopped: 0.0126673 seconds, Stopping threads took: 0.0123053
seconds
2019-08-13T10:34:34.906+0000: 6067.537: Total time for which application
threads were stopped: 0.0004966 seconds, Stopping threads took: 0.0002260
seconds
2019-08-13T10:34:34.954+0000: 6067.585: Total time for which application
threads were stopped: 0.0085233 seconds, Stopping threads took: 0.0082347
seconds
{Heap before GC invocations=53 (full 1):
par new generation total 943744K, used 854623K [0x0000000080000000,
0x00000000c0000000, 0x00000000c0000000)
eden space 838912K, 100% used [0x0000000080000000, 0x00000000b3340000,
0x00000000b3340000)
from space 104832K, 14% used [0x00000000b99a0000, 0x00000000ba8f7c10,
0x00000000c0000000)
to space 104832K, 0% used [0x00000000b3340000, 0x00000000b3340000,
0x00000000b99a0000)
concurrent mark-sweep generation total 1048576K, used 457452K
[0x00000000c0000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 38081K, capacity 38485K, committed 38744K, reserved
1083392K
class space used 4050K, capacity 4152K, committed 4220K, reserved 1048576K
2019-08-13T10:34:34.970+0000: 6067.601: [GC (Allocation Failure)
2019-08-13T10:34:34.970+0000: 6067.601: [ParNew
Desired survivor size 53673984 bytes, new threshold 1 (max 1)
- age 1: 3908880 bytes, 3908880 total
: 854623K->9208K(943744K), 0.0615766 secs] 1312075K->479953K(1992320K),
0.0616971 secs] [Times: user=0.04 sys=0.02, real=0.06 secs]
Heap after GC invocations=54 (full 1):
par new generation total 943744K, used 9208K [0x0000000080000000,
0x00000000c0000000, 0x00000000c0000000)
eden space 838912K, 0% used [0x0000000080000000, 0x0000000080000000,
0x00000000b3340000)
from space 104832K, 8% used [0x00000000b3340000, 0x00000000b3c3e198,
0x00000000b99a0000)
to space 104832K, 0% used [0x00000000b99a0000, 0x00000000b99a0000,
0x00000000c0000000)
concurrent mark-sweep generation total 1048576K, used 470745K
[0x00000000c0000000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 38081K, capacity 38485K, committed 38744K, reserved
1083392K
class space used 4050K, capacity 4152K, committed 4220K, reserved 1048576K
}
I found that sdc-be pod got restarted after "Cassandra down" problem:
2019-08-13T10:33:44.827Z [qtp215145189-13] ERROR
o.o.s.b.f.ComponentsAvailabilityFilter AuditBeginTimestamp=2019-08-13
10:33:44.827Z RequestId=68ad264b-4df8-4525-a93b-d3f841c788a3
ErrorCategory=ERROR ServerIPAddress=10.42.3.50 ServiceName=/version
ErrorCode=500 PartnerName=kube-probe/1.13 auditOn=true
ServerFQDN=dev-sdc-sdc-be-65cd8755f8-hvp2x Components Availability Filter
Failed - ES/Cassandra is DOWN
This problem occurs very often and pod sdc-be keeps restarting.
What is interesting Cassandra default XMX is 512M. After changing it to 2048M
situation still occurs.
Can anyone help with that issue?
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#18535): https://lists.onap.org/g/onap-discuss/message/18535
Mute This Topic: https://lists.onap.org/mt/32851336/21656
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
