[jira] [Commented] (CASSANDRA-10861) Memory leak with Cassadra java driver 3.0.0-beta1 and Cassandra 3.0.1
[
https://issues.apache.org/jira/browse/CASSANDRA-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15056758#comment-15056758
]
Michael Shuler commented on CASSANDRA-10861:
bq. It seems that connections crash
Could you provide some details on this? How can someone reproduce this?
{noformat}
14-Dec-2015 13:24:38.379 SEVERE [http-nio-8080-Acceptor-0]
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:686)
at java.lang.Thread.run(Thread.java:745)
{noformat}
The short snippet of logs you attached appears that your application is
creating a large number of connections and running out of file descriptors.
Have you set up the server(s) sysctl settings to handle the number of
connections your application requires?
What is the Cassandra server state and what do the Cassandra server logs show
when this occurs? What does the driver debug logging show? Have you gotten help
from the java driver devs or is there a java driver JIRA ticket to look at?
This ticket is a little vague on details.
> Memory leak with Cassadra java driver 3.0.0-beta1 and Cassandra 3.0.1
> -
>
> Key: CASSANDRA-10861
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10861
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: Ubuntu 14.04.3 LTS 64 bits, Java build 1.8.0_66-b17,
> tomcat 8.0.23
>Reporter: Carlos Scheidecker
> Fix For: 3.0.1
>
> Attachments: error_log_tomcat.txt
>
>
> Same dev environment with same application on Tomcat 8.0.23. However the dev
> nodes have been upgraded to 3.0.0 and later to 3.0.1. The Cassandra driver is
> version 3.0.0-beta1.
> It seems that connections crash, do not get cleared and it leads to a memory
> leak stack overflow condition.
> Attached is an error log file from tomcat.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10861) Memory leak with Cassadra java driver 3.0.0-beta1 and Cassandra 3.0.1
[
https://issues.apache.org/jira/browse/CASSANDRA-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15056787#comment-15056787
]
Carlos Scheidecker commented on CASSANDRA-10861:
Michael,
Sure, sorry if I had tried to be brief.
I cannot reproduce this. It only occurs after 1 or 2 days. But it occurs. Once
Tomcat is restarted it is all good. As the API has not changed only the server
and driver, the only errors I have are on the application driver side.
There are no errors on Cassandra side that I could find either on system.log
nor debug.log. This seems to be a driver error.
The only thing I had from the same time on Cassandra debug.log is as follows:
INFO [IndexSummaryManager:1] 2015-12-14 11:49:18,904
IndexSummaryManager.java:257 - Redistributing index summaries
DEBUG [ScheduledTasks:1] 2015-12-14 12:49:15,198 ColumnFamilyStore.java:829 -
Enqueuing flush of size_estimates: 5545698 (1%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:141] 2015-12-14 12:49:15,198 Memtable.java:363 -
Writing Memtable-size_estimates@1082810684(514.892KiB serialized bytes, 52200
ops, 1%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:141] 2015-12-14 12:49:15,226 Memtable.java:396 -
Completed flushing
/var/lib/cassandra/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/ma-486-big-Data.db
(351.674KiB) for commitlog position ReplayPosition(segmentId=1449870554433,
position=15986025)
DEBUG [ScheduledTasks:1] 2015-12-14 12:49:17,758 ColumnFamilyStore.java:829 -
Enqueuing flush of sstable_activity: 9294 (0%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:142] 2015-12-14 12:49:17,758 Memtable.java:363 -
Writing Memtable-sstable_activity@1052341053(1.162KiB serialized bytes, 168
ops, 0%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:142] 2015-12-14 12:49:17,770 Memtable.java:396 -
Completed flushing
/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-484-big-Data.db
(1.297KiB) for commitlog position ReplayPosition(segmentId=1449870554433,
position=15986025)
DEBUG [CompactionExecutor:3412] 2015-12-14 12:49:17,814 CompactionTask.java:146
- Compacting (c2e5d160-a29b-11e5-bbad-cd29e2a8cd4b)
[/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-484-big-Data.db:level=0,
/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-483-big-Data.db:level=0,
/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-482-big-Data.db:level=0,
/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-481-big-Data.db:level=0,
]
DEBUG [CompactionExecutor:3412] 2015-12-14 12:49:17,902 CompactionTask.java:217
- Compacted (c2e5d160-a29b-11e5-bbad-cd29e2a8cd4b) 4 sstables to
[/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-485-big,]
to level=0. 2,555 bytes to 603 (~23% of original) in 88ms = 0.006535MB/s. 0
total partitions merged to 14. Partition merge counts were {1:8, 4:14, }
INFO [IndexSummaryManager:1] 2015-12-14 12:49:18,914
IndexSummaryManager.java:257 - Redistributing index summaries
DEBUG [ScheduledTasks:1] 2015-12-14 13:49:15,200 ColumnFamilyStore.java:829 -
Enqueuing flush of size_estimates: 5545698 (1%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:143] 2015-12-14 13:49:15,200 Memtable.java:363 -
Writing Memtable-size_estimates@647480674(514.892KiB serialized bytes, 52200
ops, 1%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:143] 2015-12-14 13:49:15,220 Memtable.java:396 -
Completed flushing
/var/lib/cassandra/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/ma-487-big-Data.db
(351.674KiB) for commitlog position ReplayPosition(segmentId=1449870554433,
position=20068341)
DEBUG [ScheduledTasks:1] 2015-12-14 13:49:17,756 ColumnFamilyStore.java:829 -
Enqueuing flush of compaction_history: 1147 (0%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:144] 2015-12-14 13:49:17,756 Memtable.java:363 -
Writing Memtable-compaction_history@953834276(0.226KiB serialized bytes, 1 ops,
0%/0% of on/off-heap limit)
DEBUG [ScheduledTasks:1] 2015-12-14 13:49:17,760 ColumnFamilyStore.java:829 -
Enqueuing flush of sstable_activity: 9902 (0%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:143] 2015-12-14 13:49:17,760 Memtable.java:363 -
Writing Memtable-sstable_activity@481319531(1.193KiB serialized bytes, 172 ops,
0%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:144] 2015-12-14 13:49:17,772 Memtable.java:396 -
Completed flushing
/var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/ma-208-big-Data.db
(0.122KiB) for commitlog position ReplayPosition(segmentId=1449870554433,
position=20068341)
DEBUG [MemtableFlushWriter:143] 2015-12-14 13:49:17,772 Memtable.java:396 -
Completed flushing
[jira] [Commented] (CASSANDRA-10861) Memory leak with Cassadra java driver 3.0.0-beta1 and Cassandra 3.0.1
[ https://issues.apache.org/jira/browse/CASSANDRA-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15056803#comment-15056803 ] Carlos Scheidecker commented on CASSANDRA-10861: As you can see, looks like that heartbeat does not work on all nodes (192.168.1.15, 192.168.1.16 and 192.168.1.17) at first just 192.168.1.15 and then works well next. These are from Tomcat logs: 13:12:33.904 [cluster1-nio-worker-0] DEBUG com.datastax.driver.core.Connection - Connection[/192.168.1.15:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat 13:12:33.905 [cluster1-nio-worker-0] DEBUG com.datastax.driver.core.Connection - Connection[/192.168.1.15:9042-1, inFlight=0, closed=false] heartbeat query succeeded 14-Dec-2015 13:12:33.941 SEVERE [http-nio-8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:686) at java.lang.Thread.run(Thread.java:745) 14-Dec-2015 13:12:34.041 SEVERE [http-nio-8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:686) at java.lang.Thread.run(Thread.java:745) 14-Dec-2015 13:12:34.242 SEVERE [http-nio-8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:686) at java.lang.Thread.run(Thread.java:745) 14-Dec-2015 13:12:34.242 SEVERE [http-nio-8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:686) at java.lang.Thread.run(Thread.java:745) 14-Dec-2015 13:12:34.292 SEVERE [http-nio-8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:686) at java.lang.Thread.run(Thread.java:745) 14-Dec-2015 13:12:34.393 SEVERE [http-nio-8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:686) at java.lang.Thread.run(Thread.java:745) 14-Dec-2015 13:12:34.593 SEVERE [http-nio-8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:686) at java.lang.Thread.run(Thread.java:745) 14-Dec-2015 13:12:34.994 SEVERE [http-nio-8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint$Acceptor.run Socket accept failed java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at
[jira] [Commented] (CASSANDRA-10861) Memory leak with Cassadra java driver 3.0.0-beta1 and Cassandra 3.0.1
[ https://issues.apache.org/jira/browse/CASSANDRA-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15056814#comment-15056814 ] Michael Shuler commented on CASSANDRA-10861: Max open files 4096 4096 files > Memory leak with Cassadra java driver 3.0.0-beta1 and Cassandra 3.0.1 > - > > Key: CASSANDRA-10861 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10861 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Ubuntu 14.04.3 LTS 64 bits, Java build 1.8.0_66-b17, > tomcat 8.0.23 >Reporter: Carlos Scheidecker > Fix For: 3.0.1 > > Attachments: error_log_tomcat.txt > > > Same dev environment with same application on Tomcat 8.0.23. However the dev > nodes have been upgraded to 3.0.0 and later to 3.0.1. The Cassandra driver is > version 3.0.0-beta1. > It seems that connections crash, do not get cleared and it leads to a memory > leak stack overflow condition. > Attached is an error log file from tomcat. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10861) Memory leak with Cassadra java driver 3.0.0-beta1 and Cassandra 3.0.1
[ https://issues.apache.org/jira/browse/CASSANDRA-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15056883#comment-15056883 ] Carlos Scheidecker commented on CASSANDRA-10861: Michael, I had it increased on Tomcat updstart script to 5 let's see how it goes now. Limit Soft Limit Hard Limit Units Max cpu time unlimitedunlimitedseconds Max file size unlimitedunlimitedbytes Max data size unlimitedunlimitedbytes Max stack size8388608 unlimitedbytes Max core file size0unlimitedbytes Max resident set unlimitedunlimitedbytes Max processes 256854 256854 processes Max open files55files Max locked memory 6553665536bytes Max address space unlimitedunlimitedbytes Max file locksunlimitedunlimitedlocks Max pending signals 256854 256854 signals Max msgqueue size 819200 819200 bytes Max nice priority 00 Max realtime priority 00 Max realtime timeout unlimitedunlimitedus > Memory leak with Cassadra java driver 3.0.0-beta1 and Cassandra 3.0.1 > - > > Key: CASSANDRA-10861 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10861 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging > Environment: Ubuntu 14.04.3 LTS 64 bits, Java build 1.8.0_66-b17, > tomcat 8.0.23 >Reporter: Carlos Scheidecker > Fix For: 3.0.1 > > Attachments: error_log_tomcat.txt > > > Same dev environment with same application on Tomcat 8.0.23. However the dev > nodes have been upgraded to 3.0.0 and later to 3.0.1. The Cassandra driver is > version 3.0.0-beta1. > It seems that connections crash, do not get cleared and it leads to a memory > leak stack overflow condition. > Attached is an error log file from tomcat. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
