[jira] [Updated] (CASSANDRA-10393) LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref)

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-10393:
---
Labels: memory-leak sstable  (was: memory-leak proposed-cantrepro sstable)

> LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref)
> --
>
> Key: CASSANDRA-10393
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10393
> Project: Cassandra
>  Issue Type: Bug
> Environment: v 2.2.1 (from apt)
> -> lsb_release -a
> No LSB modules are available.
> Distributor ID:   Debian
> Description:  Debian GNU/Linux 7.8 (wheezy)
> Release:  7.8
> Codename: wheezy
> -> java -version
> java version "1.8.0_60"
> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
>Reporter: Christian Winther
>Priority: Major
>  Labels: memory-leak, sstable
> Fix For: 2.2.x
>
>
> When trying to repair full on a table with the following schema my nodes 
> stall 
> and end up with spamming this 
> I've recently changed the table from SizeTieredCompactionStrategy to 
> LeveledCompactionStrategy.
> Coming from 2.1.9 -> 2.2.0 -> 2.2.1 i ran upgradesstable without issue as well
> When trying to full repair post compaction change, I got "out of order" 
> errors. A few google searches later, I was told to "scrub" the keyspace - did 
> that during the night (no problems logged, and no data lost)
> Now a repair just stalls and output memory leaks all over the place 
> {code}
> CREATE KEYSPACE sessions WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': '3'}  AND durable_writes = true;
> CREATE TABLE sessions.sessions (
> id text PRIMARY KEY,
> client_ip text,
> controller text,
> controller_action text,
> created timestamp,
> data text,
> expires timestamp,
> http_host text,
> modified timestamp,
> request_agent text,
> request_agent_bot boolean,
> request_path text,
> site_id int,
> user_id int
> ) WITH bloom_filter_fp_chance = 0.01
> AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'sstable_compression': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> {code}
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@4428a373) to class 
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@184765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big
>  was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@368dd97) to class 
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@184765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big
>  was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@66fb78be) to class 
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@184765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big
>  was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@9fdd2e6) to class 
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1460906269:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104788-big
>  was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@84fcb91) to class 
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1460906269:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104788-big
>  was not released before the reference was garbage collected
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (CASSANDRA-10393) LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref)

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-10393:
---
Labels: memory-leak proposed-cantrepro sstable  (was: memory-leak sstable)

> LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref)
> --
>
> Key: CASSANDRA-10393
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10393
> Project: Cassandra
>  Issue Type: Bug
> Environment: v 2.2.1 (from apt)
> -> lsb_release -a
> No LSB modules are available.
> Distributor ID:   Debian
> Description:  Debian GNU/Linux 7.8 (wheezy)
> Release:  7.8
> Codename: wheezy
> -> java -version
> java version "1.8.0_60"
> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
>Reporter: Christian Winther
>  Labels: memory-leak, proposed-cantrepro, sstable
> Fix For: 2.2.x
>
>
> When trying to repair full on a table with the following schema my nodes 
> stall 
> and end up with spamming this 
> I've recently changed the table from SizeTieredCompactionStrategy to 
> LeveledCompactionStrategy.
> Coming from 2.1.9 -> 2.2.0 -> 2.2.1 i ran upgradesstable without issue as well
> When trying to full repair post compaction change, I got "out of order" 
> errors. A few google searches later, I was told to "scrub" the keyspace - did 
> that during the night (no problems logged, and no data lost)
> Now a repair just stalls and output memory leaks all over the place 
> {code}
> CREATE KEYSPACE sessions WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': '3'}  AND durable_writes = true;
> CREATE TABLE sessions.sessions (
> id text PRIMARY KEY,
> client_ip text,
> controller text,
> controller_action text,
> created timestamp,
> data text,
> expires timestamp,
> http_host text,
> modified timestamp,
> request_agent text,
> request_agent_bot boolean,
> request_path text,
> site_id int,
> user_id int
> ) WITH bloom_filter_fp_chance = 0.01
> AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'sstable_compression': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> {code}
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@4428a373) to class 
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@184765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big
>  was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@368dd97) to class 
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@184765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big
>  was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@66fb78be) to class 
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@184765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big
>  was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@9fdd2e6) to class 
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1460906269:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104788-big
>  was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@84fcb91) to class 
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1460906269:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104788-big
>  was not released before the reference was garbage collected
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (CASSANDRA-10393) LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref)

[Philip Thompson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Thompson updated CASSANDRA-10393:

Fix Version/s: 2.2.x

> LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref)
> --
>
> Key: CASSANDRA-10393
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10393
> Project: Cassandra
>  Issue Type: Bug
> Environment: v 2.2.1 (from apt)
> -> lsb_release -a
> No LSB modules are available.
> Distributor ID:   Debian
> Description:  Debian GNU/Linux 7.8 (wheezy)
> Release:  7.8
> Codename: wheezy
> -> java -version
> java version "1.8.0_60"
> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
>Reporter: Christian Winther
>  Labels: memory-leak, sstable
> Fix For: 2.2.x
>
>
> When trying to repair full on a table with the following schema my nodes 
> stall 
> and end up with spamming this 
> I've recently changed the table from SizeTieredCompactionStrategy to 
> LeveledCompactionStrategy.
> Coming from 2.1.9 -> 2.2.0 -> 2.2.1 i ran upgradesstable without issue as well
> When trying to full repair post compaction change, I got "out of order" 
> errors. A few google searches later, I was told to "scrub" the keyspace - did 
> that during the night (no problems logged, and no data lost)
> Now a repair just stalls and output memory leaks all over the place 
> {code}
> CREATE KEYSPACE sessions WITH replication = {'class': 'SimpleStrategy', 
> 'replication_factor': '3'}  AND durable_writes = true;
> CREATE TABLE sessions.sessions (
> id text PRIMARY KEY,
> client_ip text,
> controller text,
> controller_action text,
> created timestamp,
> data text,
> expires timestamp,
> http_host text,
> modified timestamp,
> request_agent text,
> request_agent_bot boolean,
> request_path text,
> site_id int,
> user_id int
> ) WITH bloom_filter_fp_chance = 0.01
> AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'sstable_compression': 
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> {code}
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@4428a373) to class 
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@184765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big
>  was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@368dd97) to class 
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@184765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big
>  was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@66fb78be) to class 
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@184765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big
>  was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@9fdd2e6) to class 
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1460906269:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104788-big
>  was not released before the reference was garbage collected
> ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
> DETECTED: a reference 
> (org.apache.cassandra.utils.concurrent.Ref$State@84fcb91) to class 
> org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1460906269:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104788-big
>  was not released before the reference was garbage collected
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (CASSANDRA-10393) LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref)

[Christian Winther (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christian Winther updated CASSANDRA-10393:
--
Description: 
When trying to repair full on a table with the following schema my nodes stall 
and end up with spamming this 

I've recently changed the table from SizeTieredCompactionStrategy to 
LeveledCompactionStrategy.

Coming from 2.1.9 -> 2.2.0 -> 2.2.1 i ran upgradesstable without issue as well

When trying to full repair post compaction change, I got "out of order" errors. 
A few google searches later, I was told to "scrub" the keyspace - did that 
during the night (no problems logged, and no data lost)

Now a repair just stalls and output memory leaks all over the place 

{code}
CREATE KEYSPACE sessions WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '3'}  AND durable_writes = true;

CREATE TABLE sessions.sessions (
id text PRIMARY KEY,
client_ip text,
controller text,
controller_action text,
created timestamp,
data text,
expires timestamp,
http_host text,
modified timestamp,
request_agent text,
request_agent_bot boolean,
request_path text,
site_id int,
user_id int
) WITH bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'class': 
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'sstable_compression': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
{code}

{code}
ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
DETECTED: a reference 
(org.apache.cassandra.utils.concurrent.Ref$State@4428a373) to class 
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@184765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big
 was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@368dd97) 
to class 
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@184765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big
 was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
DETECTED: a reference 
(org.apache.cassandra.utils.concurrent.Ref$State@66fb78be) to class 
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@184765:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104037-big
 was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@9fdd2e6) 
to class 
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1460906269:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104788-big
 was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2015-09-24 10:25:28,475 Ref.java:187 - LEAK 
DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@84fcb91) 
to class 
org.apache.cassandra.io.sstable.format.SSTableReader$InstanceTidier@1460906269:/data/1/cassandra/sessions/sessions-77dd22f0ab9711e49cbc410c6b6f53a6/la-104788-big
 was not released before the reference was garbage collected
{code}

  was:
When trying to repair full on a table with the following schema my nodes stall 
and end up with spamming this 

I've recently changed the table from SizeTieredCompactionStrategy to 
LeveledCompactionStrategy.

Coming from 2.1.9 -> 2.2.0 -> 2.2.1 i ran upgradesstable without issue as well

When trying to full repair post compaction change, I got "out of order" errors. 
A few google searches later, I was told to "scrub" the keyspace - did that 
during the night (no problems logged, and no data lost)

Now a repair just stalls and output memory leaks all over the place 

{code}
CREATE KEYSPACE sessions WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '3'}  AND durable_writes = true;

CREATE TABLE sessions.sessions (
id text PRIMARY KEY,
client_ip text,
controller text,
controller_action text,
created timestamp,
data text,
expires timestamp,
http_host text,
modified timestamp,
request_agent text,
request_agent_bot boolean,
request_path text,
site_id int,
user_id int
) WITH bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"NONE",