[jira] [Updated] (CASSANDRA-13523) StreamReceiveTask: java.lang.OutOfMemoryError: Map failed
[
https://issues.apache.org/jira/browse/CASSANDRA-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Shuler updated CASSANDRA-13523:
---
Environment: Cassandra 2.1.13, Ubuntu 14.04.5 LTS, Docker version 1.9.1,
run as a container, 4 core server with 16GB memory. (was: Ubuntu 14.04.5 LTS,
Docker version 1.9.1, run as a container, 4 core server with 16GB memory.)
> StreamReceiveTask: java.lang.OutOfMemoryError: Map failed
> -
>
> Key: CASSANDRA-13523
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13523
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: Cassandra 2.1.13, Ubuntu 14.04.5 LTS, Docker version
> 1.9.1, run as a container, 4 core server with 16GB memory.
>Reporter: Matthew O'Riordan
>Priority: Major
> Labels: bug, crash
> Fix For: 2.1.13
>
>
> During a nodetool repair -par on one of our keyspaces, Cassandra crashed due
> to what seems like memory exhaustion within the JVM. The machine itself had
> plenty of available memory at the time and did not appear to be under any
> significant load.
> In the system log, before the crash, the following was logged:
> {code}
> ...
> INFO [AntiEntropySessions:55] 2017-05-10 18:18:20,627 RepairJob.java:163 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] requesting merkle trees for
> stats_day_aggregates (to [/54.162.66.114, /54.236.226.76, /52.221.228.170,
> /54.154.35.144, /54.154.96.213, /52.221.217.27])
> INFO [ValidationExecutor:54] 2017-05-10 18:18:20,628
> ColumnFamilyStore.java:905 - Enqueuing flush of stats_day_aggregates: 7018
> (0%) on-heap, 0 (0%) off-heap
> INFO [MemtableFlushWriter:13608] 2017-05-10 18:18:20,628 Memtable.java:347
> - Writing Memtable-stats_day_aggregates@3469792(2.734KiB serialized bytes, 14
> ops, 0%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:13608] 2017-05-10 18:18:20,629 Memtable.java:382
> - Completed flushing
> /var/lib/cassandra/data/ably_production_0_stats/stats_day_aggregates-b6e29201e3d111e5bbf3091830ac5256/ably_production_0_stats-stats_day_aggregates-tmp-ka-43026-Data.db
> (0.000KiB) for commitlog position ReplayPosition(segmentId=1491420635101,
> position=24224955)
> INFO [StreamReceiveTask:638] 2017-05-10 18:18:21,220
> StreamResultFuture.java:180 - [Stream #0008cac1-35ad-11e7-b7e4-091830ac5256]
> Session with /52.203.21.193 is complete
> INFO [StreamReceiveTask:638] 2017-05-10 18:18:21,220
> StreamResultFuture.java:212 - [Stream #0008cac1-35ad-11e7-b7e4-091830ac5256]
> All sessions completed
> INFO [StreamReceiveTask:638] 2017-05-10 18:18:21,221
> StreamingRepairTask.java:96 - [repair #fe7e3320-35ac-11e7-b7e4-091830ac5256]
> streaming task succeed, returning response to /52.221.217.27
> INFO [CqlSlowLog-Writer-thread-0] 2017-05-10 18:18:26,230
> CqlSlowLogWriter.java:151 - Recording statements with duration of 4844 in
> slow log
> INFO [Service Thread] 2017-05-10 18:18:26,233 GCInspector.java:258 - G1 Old
> Generation GC in 4781ms. G1 Eden Space: 131072 -> 0; G1 Old Gen:
> 2774539816 -> 1830851216; G1 Survivor Space: 37748736 -> 0;
> INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,237 RepairSession.java:171 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for
> stats_day_aggregates from /54.162.66.114
> INFO [StreamConnectionEstablisher:1] 2017-05-10 18:18:26,239
> StreamCoordinator.java:209 - [Stream #0293bb60-35ad-11e7-b7e4-091830ac5256,
> ID#0] Beginning stream session with /54.154.226.20
> INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,294 RepairSession.java:171 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for
> stats_day_aggregates from /54.236.226.76
> INFO [Service Thread] 2017-05-10 18:18:26,298 StatusLogger.java:51 - Pool
> NameActive Pending Completed Blocked All Time
> Blocked
> INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,344 RepairSession.java:171 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for
> stats_day_aggregates from /54.154.35.144
> INFO [CqlSlowLog-Writer-thread-0] 2017-05-10 18:18:26,344
> CqlSlowLogWriter.java:151 - Recording statements with duration of 5035 in
> slow log
> WARN [GossipTasks:1] 2017-05-10 18:18:26,344 FailureDetector.java:258 - Not
> marking nodes down due to local pause of 5109502584 > 50
> INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,344 RepairSession.java:171 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for
> stats_day_aggregates from /54.154.96.213
> INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,344 RepairSession.java:171 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for
> stats_day_aggregates from /52.221.228.170
>
[jira] [Updated] (CASSANDRA-13523) StreamReceiveTask: java.lang.OutOfMemoryError: Map failed
[
https://issues.apache.org/jira/browse/CASSANDRA-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Shuler updated CASSANDRA-13523:
---
Fix Version/s: (was: 2.1.13)
2.1.x
> StreamReceiveTask: java.lang.OutOfMemoryError: Map failed
> -
>
> Key: CASSANDRA-13523
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13523
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: Cassandra 2.1.13, Ubuntu 14.04.5 LTS, Docker version
> 1.9.1, run as a container, 4 core server with 16GB memory.
>Reporter: Matthew O'Riordan
>Priority: Major
> Labels: bug, crash
> Fix For: 2.1.x
>
>
> During a nodetool repair -par on one of our keyspaces, Cassandra crashed due
> to what seems like memory exhaustion within the JVM. The machine itself had
> plenty of available memory at the time and did not appear to be under any
> significant load.
> In the system log, before the crash, the following was logged:
> {code}
> ...
> INFO [AntiEntropySessions:55] 2017-05-10 18:18:20,627 RepairJob.java:163 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] requesting merkle trees for
> stats_day_aggregates (to [/54.162.66.114, /54.236.226.76, /52.221.228.170,
> /54.154.35.144, /54.154.96.213, /52.221.217.27])
> INFO [ValidationExecutor:54] 2017-05-10 18:18:20,628
> ColumnFamilyStore.java:905 - Enqueuing flush of stats_day_aggregates: 7018
> (0%) on-heap, 0 (0%) off-heap
> INFO [MemtableFlushWriter:13608] 2017-05-10 18:18:20,628 Memtable.java:347
> - Writing Memtable-stats_day_aggregates@3469792(2.734KiB serialized bytes, 14
> ops, 0%/0% of on/off-heap limit)
> INFO [MemtableFlushWriter:13608] 2017-05-10 18:18:20,629 Memtable.java:382
> - Completed flushing
> /var/lib/cassandra/data/ably_production_0_stats/stats_day_aggregates-b6e29201e3d111e5bbf3091830ac5256/ably_production_0_stats-stats_day_aggregates-tmp-ka-43026-Data.db
> (0.000KiB) for commitlog position ReplayPosition(segmentId=1491420635101,
> position=24224955)
> INFO [StreamReceiveTask:638] 2017-05-10 18:18:21,220
> StreamResultFuture.java:180 - [Stream #0008cac1-35ad-11e7-b7e4-091830ac5256]
> Session with /52.203.21.193 is complete
> INFO [StreamReceiveTask:638] 2017-05-10 18:18:21,220
> StreamResultFuture.java:212 - [Stream #0008cac1-35ad-11e7-b7e4-091830ac5256]
> All sessions completed
> INFO [StreamReceiveTask:638] 2017-05-10 18:18:21,221
> StreamingRepairTask.java:96 - [repair #fe7e3320-35ac-11e7-b7e4-091830ac5256]
> streaming task succeed, returning response to /52.221.217.27
> INFO [CqlSlowLog-Writer-thread-0] 2017-05-10 18:18:26,230
> CqlSlowLogWriter.java:151 - Recording statements with duration of 4844 in
> slow log
> INFO [Service Thread] 2017-05-10 18:18:26,233 GCInspector.java:258 - G1 Old
> Generation GC in 4781ms. G1 Eden Space: 131072 -> 0; G1 Old Gen:
> 2774539816 -> 1830851216; G1 Survivor Space: 37748736 -> 0;
> INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,237 RepairSession.java:171 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for
> stats_day_aggregates from /54.162.66.114
> INFO [StreamConnectionEstablisher:1] 2017-05-10 18:18:26,239
> StreamCoordinator.java:209 - [Stream #0293bb60-35ad-11e7-b7e4-091830ac5256,
> ID#0] Beginning stream session with /54.154.226.20
> INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,294 RepairSession.java:171 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for
> stats_day_aggregates from /54.236.226.76
> INFO [Service Thread] 2017-05-10 18:18:26,298 StatusLogger.java:51 - Pool
> NameActive Pending Completed Blocked All Time
> Blocked
> INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,344 RepairSession.java:171 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for
> stats_day_aggregates from /54.154.35.144
> INFO [CqlSlowLog-Writer-thread-0] 2017-05-10 18:18:26,344
> CqlSlowLogWriter.java:151 - Recording statements with duration of 5035 in
> slow log
> WARN [GossipTasks:1] 2017-05-10 18:18:26,344 FailureDetector.java:258 - Not
> marking nodes down due to local pause of 5109502584 > 50
> INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,344 RepairSession.java:171 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for
> stats_day_aggregates from /54.154.96.213
> INFO [AntiEntropyStage:1] 2017-05-10 18:18:26,344 RepairSession.java:171 -
> [repair #0330beb0-35ad-11e7-b7e4-091830ac5256] Received merkle tree for
> stats_day_aggregates from /52.221.228.170
> INFO [Service Thread] 2017-05-10 18:18:26,345 StatusLogger.java:66 -
> MutationStage 0 0 1199223432 0
> 0
>
