[
https://issues.apache.org/jira/browse/CASSANDRA-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056787#comment-15056787
]
Carlos Scheidecker commented on CASSANDRA-10861:
------------------------------------------------
Michael,
Sure, sorry if I had tried to be brief.
I cannot reproduce this. It only occurs after 1 or 2 days. But it occurs. Once
Tomcat is restarted it is all good. As the API has not changed only the server
and driver, the only errors I have are on the application driver side.
There are no errors on Cassandra side that I could find either on system.log
nor debug.log. This seems to be a driver error.
The only thing I had from the same time on Cassandra debug.log is as follows:
INFO [IndexSummaryManager:1] 2015-12-14 11:49:18,904
IndexSummaryManager.java:257 - Redistributing index summaries
DEBUG [ScheduledTasks:1] 2015-12-14 12:49:15,198 ColumnFamilyStore.java:829 -
Enqueuing flush of size_estimates: 5545698 (1%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:141] 2015-12-14 12:49:15,198 Memtable.java:363 -
Writing Memtable-size_estimates@1082810684(514.892KiB serialized bytes, 52200
ops, 1%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:141] 2015-12-14 12:49:15,226 Memtable.java:396 -
Completed flushing
/var/lib/cassandra/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/ma-486-big-Data.db
(351.674KiB) for commitlog position ReplayPosition(segmentId=1449870554433,
position=15986025)
DEBUG [ScheduledTasks:1] 2015-12-14 12:49:17,758 ColumnFamilyStore.java:829 -
Enqueuing flush of sstable_activity: 9294 (0%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:142] 2015-12-14 12:49:17,758 Memtable.java:363 -
Writing Memtable-sstable_activity@1052341053(1.162KiB serialized bytes, 168
ops, 0%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:142] 2015-12-14 12:49:17,770 Memtable.java:396 -
Completed flushing
/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-484-big-Data.db
(1.297KiB) for commitlog position ReplayPosition(segmentId=1449870554433,
position=15986025)
DEBUG [CompactionExecutor:3412] 2015-12-14 12:49:17,814 CompactionTask.java:146
- Compacting (c2e5d160-a29b-11e5-bbad-cd29e2a8cd4b)
[/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-484-big-Data.db:level=0,
/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-483-big-Data.db:level=0,
/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-482-big-Data.db:level=0,
/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-481-big-Data.db:level=0,
]
DEBUG [CompactionExecutor:3412] 2015-12-14 12:49:17,902 CompactionTask.java:217
- Compacted (c2e5d160-a29b-11e5-bbad-cd29e2a8cd4b) 4 sstables to
[/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-485-big,]
to level=0. 2,555 bytes to 603 (~23% of original) in 88ms = 0.006535MB/s. 0
total partitions merged to 14. Partition merge counts were {1:8, 4:14, }
INFO [IndexSummaryManager:1] 2015-12-14 12:49:18,914
IndexSummaryManager.java:257 - Redistributing index summaries
DEBUG [ScheduledTasks:1] 2015-12-14 13:49:15,200 ColumnFamilyStore.java:829 -
Enqueuing flush of size_estimates: 5545698 (1%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:143] 2015-12-14 13:49:15,200 Memtable.java:363 -
Writing Memtable-size_estimates@647480674(514.892KiB serialized bytes, 52200
ops, 1%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:143] 2015-12-14 13:49:15,220 Memtable.java:396 -
Completed flushing
/var/lib/cassandra/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/ma-487-big-Data.db
(351.674KiB) for commitlog position ReplayPosition(segmentId=1449870554433,
position=20068341)
DEBUG [ScheduledTasks:1] 2015-12-14 13:49:17,756 ColumnFamilyStore.java:829 -
Enqueuing flush of compaction_history: 1147 (0%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:144] 2015-12-14 13:49:17,756 Memtable.java:363 -
Writing Memtable-compaction_history@953834276(0.226KiB serialized bytes, 1 ops,
0%/0% of on/off-heap limit)
DEBUG [ScheduledTasks:1] 2015-12-14 13:49:17,760 ColumnFamilyStore.java:829 -
Enqueuing flush of sstable_activity: 9902 (0%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:143] 2015-12-14 13:49:17,760 Memtable.java:363 -
Writing Memtable-sstable_activity@481319531(1.193KiB serialized bytes, 172 ops,
0%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:144] 2015-12-14 13:49:17,772 Memtable.java:396 -
Completed flushing
/var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/ma-208-big-Data.db
(0.122KiB) for commitlog position ReplayPosition(segmentId=1449870554433,
position=20068341)
DEBUG [MemtableFlushWriter:143] 2015-12-14 13:49:17,772 Memtable.java:396 -
Completed flushing
/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-486-big-Data.db
(1.506KiB) for commitlog position ReplayPosition(segmentId=1449870554433,
position=20068341)
DEBUG [CompactionExecutor:3460] 2015-12-14 13:49:17,860 CompactionTask.java:146
- Compacting (24b13e40-a2a4-11e5-bbad-cd29e2a8cd4b)
[/var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/ma-206-big-Data.db:level=0,
/var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/ma-207-big-Data.db:level=0,
/var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/ma-205-big-Data.db:level=0,
/var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/ma-208-big-Data.db:level=0,
]
DEBUG [CompactionExecutor:3460] 2015-12-14 13:49:17,940 CompactionTask.java:217
- Compacted (24b13e40-a2a4-11e5-bbad-cd29e2a8cd4b) 4 sstables to
[/var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/ma-209-big,]
to level=0. 6,021 bytes to 5,669 (~94% of original) in 80ms = 0.067580MB/s.
0 total partitions merged to 123. Partition merge counts were {1:126, }
INFO [IndexSummaryManager:1] 2015-12-14 13:49:18,920
IndexSummaryManager.java:257 - Redistributing index summaries
Cassandra state was just fine when those occurred. Never seemed to be on the
server side but on the driver side instead.
On my @Repository class here is the connect() method:
private void connect() {
if (session == null) {
String contactpoints =
env.getProperty("cassandra.contactpoints");
PlainTextAuthProvider plainAuthProvider = new
PlainTextAuthProvider(
env.getProperty("cassandra.username"),
env.getProperty("cassandra.password"));
// Probably needs to set Pooling Options etc
//
Cluster.builder().getConfiguration().getPoolingOptions()
// Cluster.builder().withReconnectionPolicy(policy)
// Cluster.builder().withRetryPolicy(policy)
cluster = Cluster.builder()
.addContactPoints(contactpoints.split(","))
.withAuthProvider(plainAuthProvider).build();
// connect() without keyspace
// Session cs = cluster.connect();
keyspace = env.getProperty("cassandra.oauthKeyspace");
//session = cluster.connect(keyspace);
// conect without specifying keyspace
session = cluster.connect();
}
}
For the node configuration in terms of connections I have the default which is
unlimited:
# The maximum number of concurrent client connections.
# The default is -1, which means unlimited.
# native_transport_max_concurrent_connections: -1
# The maximum number of concurrent client connections per source ip.
# The default is -1, which means unlimited.
# native_transport_max_concurrent_connections_per_ip: -1
I had also checked Tomcat limits and I have no limits on opened files with it:
cat /proc/30948/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 256854 256854 processes
Max open files 4096 4096 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 256854 256854 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
> Memory leak with Cassadra java driver 3.0.0-beta1 and Cassandra 3.0.1
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-10861
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10861
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: Ubuntu 14.04.3 LTS 64 bits, Java build 1.8.0_66-b17,
> tomcat 8.0.23
> Reporter: Carlos Scheidecker
> Fix For: 3.0.1
>
> Attachments: error_log_tomcat.txt
>
>
> Same dev environment with same application on Tomcat 8.0.23. However the dev
> nodes have been upgraded to 3.0.0 and later to 3.0.1. The Cassandra driver is
> version 3.0.0-beta1.
> It seems that connections crash, do not get cleared and it leads to a memory
> leak stack overflow condition.
> Attached is an error log file from tomcat.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
