[jira] [Commented] (CASSANDRA-12662) OOM when using SASI index
[ https://issues.apache.org/jira/browse/CASSANDRA-12662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15540328#comment-15540328 ] Maxim Podkolzine commented on CASSANDRA-12662: -- Thanks Alex, we are considering to set `memtable_heap_space_in_mb` and `memtable_offheap_space_in_mb` explicitly. What can a reasonable value on a 20Gb machine? Also we've been thinking to change the garbage collector settings. Do you think G1 would be better? > OOM when using SASI index > - > > Key: CASSANDRA-12662 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12662 > Project: Cassandra > Issue Type: Bug > Environment: Linux, 4 CPU cores, 16Gb RAM, Cassandra process utilizes > ~8Gb, of which ~4Gb is Java heap >Reporter: Maxim Podkolzine >Priority: Critical > Fix For: 3.x > > Attachments: memory-dump.png > > > 2.8Gb of the heap is taken by the index data, pending for flush (see the > screenshot). As a result the node fails with OOM. > Questions: > - Why can't Cassandra keep up with the inserted data and flush it? > - What resources/configuration should be changed to improve the performance? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12662) OOM when using SASI index
[ https://issues.apache.org/jira/browse/CASSANDRA-12662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15523048#comment-15523048 ] Maxim Podkolzine commented on CASSANDRA-12662: -- OK. Here's an update. We've added SSD disks and 20Gb RAM per node. Since we run three nodes, this is practically the limit we can allocate for Cassandra. The load is the same. Cassandra fairly quickly crashes with OOM, a glance over hprof shows 4Gb of PartitionUpdates. Do you think it's possible to configure the nodes with current hardware limitations? Or should we shift to one big node with big RAM and lots of CPU? Thanks. > OOM when using SASI index > - > > Key: CASSANDRA-12662 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12662 > Project: Cassandra > Issue Type: Bug > Environment: Linux, 4 CPU cores, 16Gb RAM, Cassandra process utilizes > ~8Gb, of which ~4Gb is Java heap >Reporter: Maxim Podkolzine >Priority: Critical > Fix For: 3.x > > Attachments: memory-dump.png > > > 2.8Gb of the heap is taken by the index data, pending for flush (see the > screenshot). As a result the node fails with OOM. > Questions: > - Why can't Cassandra keep up with the inserted data and flush it? > - What resources/configuration should be changed to improve the performance? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12662) OOM when using SASI index
[ https://issues.apache.org/jira/browse/CASSANDRA-12662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15501465#comment-15501465 ] Maxim Podkolzine commented on CASSANDRA-12662: -- Got it, thanks a lot! We don't use SSD right now, I need to check what the actual storage is. I'll try to get as much CPU and RAM as possible and get back with the results. > OOM when using SASI index > - > > Key: CASSANDRA-12662 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12662 > Project: Cassandra > Issue Type: Bug > Environment: Linux, 4 CPU cores, 16Gb RAM, Cassandra process utilizes > ~8Gb, of which ~4Gb is Java heap >Reporter: Maxim Podkolzine >Priority: Critical > Fix For: 3.6 > > Attachments: memory-dump.png > > > 2.8Gb of the heap is taken by the index data, pending for flush (see the > screenshot). As a result the node fails with OOM. > Questions: > - Why can't Cassandra keep up with the inserted data and flush it? > - What resources/configuration should be changed to improve the performance? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12573) SASI index. Incorrect results for '%foo%bar%'-like search pattern.
[
https://issues.apache.org/jira/browse/CASSANDRA-12573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15501443#comment-15501443
]
Maxim Podkolzine commented on CASSANDRA-12573:
--
There are 7 rows that contain "RevisionDiff" and 2 rows that contain
"ItemImpl". There are 9 rows that contain "RevisionDiff" OR "ItemImpl".
Here they are (only the name):
- RevisionDiffType.java: it contains "RevisionDiff", hence it contains
"RevisionDiff" OR "ItemImpl"
- RevisionDiffItem.java: it contains "RevisionDiff", hence it contains
"RevisionDiff" OR "ItemImpl"
- RevisionDiffItemDTO.java: it contains "RevisionDiff", hence it contains
"RevisionDiff" OR "ItemImpl"
- GetRevisionDiff.java: it contains "RevisionDiff", hence it contains
"RevisionDiff" OR "ItemImpl"
- RevisionDiffItemDTO.java (twice): it contains "RevisionDiff", hence it
contains "RevisionDiff" OR "ItemImpl"
- RevisionDiffItemImpl.java: it contains "RevisionDiff", hence it contains
"RevisionDiff" OR "ItemImpl"
- FastTreeItemImpl.java: it contains "ItemImpl", hence it contains
"RevisionDiff" OR "ItemImpl"
- RevisionDiffItemImpl.java: it contains "ItemImpl", hence it contains
"RevisionDiff" OR "ItemImpl"
Of these 9 rows there is one row that contains both "RevisionDiff" AND
"ItemImpl": "RevisionDiffItemImpl.java".
> SASI index. Incorrect results for '%foo%bar%'-like search pattern.
> ---
>
> Key: CASSANDRA-12573
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12573
> Project: Cassandra
> Issue Type: Bug
>Reporter: Mikhail Krupitskiy
>Priority: Critical
> Labels: sasi
>
> We use Cassandra 3.7 and have faced a strange behaviour of SELECT requests
> with "LIKE '%foo%bar%'" constraints on a column with SASI index.
> Below are few experiments that show this behaviour.
> Experiment 1:
> {noformat}
> drop keyspace if exists kmv;
> create keyspace if not exists kmv WITH REPLICATION = { 'class' :
> 'SimpleStrategy', 'replication_factor':'1'} ;
> use kmv;
> CREATE TABLE if not exists kmv (id int primary key, c1 text, c2 text);
> CREATE CUSTOM INDEX ON kmv.kmv ( c2 ) USING
> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
> 'mode': 'CONTAINS'
> };
> insert into kmv (id, c1, c2) values (1, 'f21', 'qwe') ;
> insert into kmv (id, c1, c2) values (2, 'f22', 'qweasd') ;
> insert into kmv (id, c1, c2) values (3, 'f23', 'qwea1') ;
> insert into kmv (id, c1, c2) values (4, 'f24', '1qwe') ;
> insert into kmv (id, c1, c2) values (5, 'f25', 'asdqwe') ;
> select c2 from kmv.kmv where c2 like '%w%a%';
> {noformat}
> Expected result: qweasd, qwea1.
> Actual result: no rows.
> Experiment 2 (NOTE: definition of index is changed):
> {noformat}
> drop keyspace if exists kmv;
> create keyspace if not exists kmv WITH REPLICATION = { 'class' :
> 'SimpleStrategy', 'replication_factor':'1'} ;
> use kmv;
> CREATE TABLE if not exists kmv (id int primary key, c1 text, c2 text);
> CREATE CUSTOM INDEX ON kmv.kmv ( c2 ) USING
> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
> 'mode': 'CONTAINS',
> 'analyzer_class':
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
> 'analyzed': 'true'
> };
> insert into kmv (id, c1, c2) values (1, 'f21', 'qwe') ;
> insert into kmv (id, c1, c2) values (2, 'f22', 'qweasd') ;
> insert into kmv (id, c1, c2) values (3, 'f23', 'qwea1') ;
> insert into kmv (id, c1, c2) values (4, 'f24', '1qwe') ;
> insert into kmv (id, c1, c2) values (5, 'f25', 'asdqwe') ;
> select c2 from kmv.kmv where c2 like '%w%a%';
> {noformat}
> Expected result: qweasd, qwea1.
> Actual result: asdqwe, qweasd, qwea1.
> Experiment 3 (NOTE: primary key is compound now and inserted data was
> changed):
> {noformat}
> drop keyspace if exists kmv;
> create keyspace if not exists kmv WITH REPLICATION = { 'class' :
> 'SimpleStrategy', 'replication_factor':'1'} ;
> use kmv;
> CREATE TABLE if not exists kmv (id int, c1 text, c2 text, PRIMARY KEY(id,
> c1));
> CREATE CUSTOM INDEX ON kmv.kmv ( c2 ) USING
> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
> 'mode': 'CONTAINS',
> 'analyzer_class':
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
> 'analyzed': 'true'
> };
> insert into kmv (id, c1, c2) values (1, 'f21', 'qwe') ;
> insert into kmv (id, c1, c2) values (1, 'f22', 'qweasd') ;
> insert into kmv (id, c1, c2) values (1, 'f23', 'qwea1') ;
> insert into kmv (id, c1, c2) values (1, 'f24', '1qwe') ;
> insert into kmv (id, c1, c2) values (1, 'f25', 'asdqwe') ;
> select c2 from kmv.kmv where c2 like '%w%a%';
> {noformat}
> Expected result: qweasd, qwea1.
> Actual result: qwe, qweasd, qwea1, 1qwe, asdqwe.
> Experiment 4 (NOTE: search criteria is changed):
> {noformat}
> drop keyspace if exists kmv;
> create keyspace if not exists kmv WITH REPLICATION = { 'class' :
>
[jira] [Comment Edited] (CASSANDRA-12573) SASI index. Incorrect results for '%foo%bar%'-like search pattern.
[
https://issues.apache.org/jira/browse/CASSANDRA-12573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15501293#comment-15501293
]
Maxim Podkolzine edited comment on CASSANDRA-12573 at 9/18/16 5:03 PM:
---
> SASI initially support multiple predicates, something like : WHERE ((col1=xxx
> OR col2=yyy) AND (col3 LIKE '%zzz')) but it is not merged yet into the 3.x
> trunk
That's good news. When do you plan to merge it?
> Wrong, a bug is something that does not work as expected e.g that does not
> work as documented.
As a customer I have a slightly different view on this. My expectations are
based on prior experience and common sense.
I understand when certain features that are usual in other products are not
implemented by design. This is obviously not the case.
My current impression is that this feature is half-baked and not well tested.
But it's just my opinion.
I think I have a stronger argument that this is a bug. I have created a DB and
filled it with some data from my disk:
{code}
CREATE KEYSPACE Excelsior WITH REPLICATION = { 'class' : 'SimpleStrategy',
'replication_factor' : 3 };
use excelsior;
create table demo (id text primary key, name text, content text);
CREATE CUSTOM INDEX name_index ON demo (name) USING
'org.apache.cassandra.index.sasi.SASIIndex'
WITH OPTIONS = {
'mode': 'CONTAINS',
'analyzer_class': 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
'analyzed': 'true'
};
{code}
After that I run the queries with '%' inside. As you can see multi-patterns are
handled by AND:
{code}
cqlsh:excelsior> select id, name from demo where name like '%RevisionDiff%';
id | name
--+
93dce11a-cfdd-4c16-b3b3-7537c7af03ec | RevisionDiffType.java
6586058f-bd57-4fc7-ae12-e6d8ddcd2ceb | RevisionDiffItem.java
d16dff53-002b-4fe6-9a10-bb32425360e0 | RevisionDiffItemDTO.java
bb20981e-714f-4eac-802f-6191dba5a301 | GetRevisionDiff.java
1c53574b-2eea-46f8-bcbc-5e295ef9c70a | RevisionDiffItemDTO.java
7366f852-d63c-4d07-86b3-18a3bf47e79b | RevisionDiffItemDTO.java
7f18accb-9832-4303-8227-43aa89534cde | RevisionDiffItemImpl.java
(7 rows)
cqlsh:excelsior> select id, name from demo where name like '%ItemImpl%';
id | name
--+---
603c1d12-4871-4244-896a-54ddb76dbd3b | FastTreeItemImpl.java
7f18accb-9832-4303-8227-43aa89534cde | RevisionDiffItemImpl.java
(2 rows)
cqlsh:excelsior> select id, name from demo where name like
'%RevisionDiff%ItemImpl%';
id | name
--+--
7f18accb-9832-4303-8227-43aa89534cde | RevisionDiffItemImpl.java
(1 rows)
{code}
was (Author: maximp):
> SASI initially support multiple predicates, something like : WHERE ((col1=xxx
> OR col2=yyy) AND (col3 LIKE '%zzz')) but it is not merged yet into the 3.x
> trunk
That's good news. When do you plan to merge it?
> Wrong, a bug is something that does not work as expected e.g that does not
> work as documented.
As a customer I have a slightly different view on this. My expectations are
based on prior experience and common sense.
I understand when certain features that are usual in other products are not
implemented by design. This is obviously not the case.
My current impression is that this feature is half-baked and not well tested.
But it's just my opinion.
I think I have a stronger argument that this is a bug. I have created a DB and
filled it with some data from my disk:
```
CREATE KEYSPACE Excelsior WITH REPLICATION = { 'class' : 'SimpleStrategy',
'replication_factor' : 3 };
use excelsior;
create table demo (id text primary key, name text, content text);
CREATE CUSTOM INDEX name_index ON demo (name) USING
'org.apache.cassandra.index.sasi.SASIIndex'
WITH OPTIONS = {
'mode': 'CONTAINS',
'analyzer_class': 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
'analyzed': 'true'
};
```
After that I run the queries with '%' inside. As you can see multi-patterns are
handled by AND:
```
cqlsh:excelsior> select id, name from demo where name like '%RevisionDiff%';
id | name
--+
93dce11a-cfdd-4c16-b3b3-7537c7af03ec | RevisionDiffType.java
6586058f-bd57-4fc7-ae12-e6d8ddcd2ceb | RevisionDiffItem.java
d16dff53-002b-4fe6-9a10-bb32425360e0 | RevisionDiffItemDTO.java
bb20981e-714f-4eac-802f-6191dba5a301 | GetRevisionDiff.java
1c53574b-2eea-46f8-bcbc-5e295ef9c70a | RevisionDiffItemDTO.java
7366f852-d63c-4d07-86b3-18a3bf47e79b | RevisionDiffItemDTO.java
7f18accb-9832-4303-8227-43aa89534cde | RevisionDiffItemImpl.java
(7 rows)
cqlsh:excelsior> select id, name from demo where name like '%ItemImpl%';
id
[jira] [Commented] (CASSANDRA-12573) SASI index. Incorrect results for '%foo%bar%'-like search pattern.
[
https://issues.apache.org/jira/browse/CASSANDRA-12573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15501293#comment-15501293
]
Maxim Podkolzine commented on CASSANDRA-12573:
--
> SASI initially support multiple predicates, something like : WHERE ((col1=xxx
> OR col2=yyy) AND (col3 LIKE '%zzz')) but it is not merged yet into the 3.x
> trunk
That's good news. When do you plan to merge it?
> Wrong, a bug is something that does not work as expected e.g that does not
> work as documented.
As a customer I have a slightly different view on this. My expectations are
based on prior experience and common sense.
I understand when certain features that are usual in other products are not
implemented by design. This is obviously not the case.
My current impression is that this feature is half-baked and not well tested.
But it's just my opinion.
I think I have a stronger argument that this is a bug. I have created a DB and
filled it with some data from my disk:
```
CREATE KEYSPACE Excelsior WITH REPLICATION = { 'class' : 'SimpleStrategy',
'replication_factor' : 3 };
use excelsior;
create table demo (id text primary key, name text, content text);
CREATE CUSTOM INDEX name_index ON demo (name) USING
'org.apache.cassandra.index.sasi.SASIIndex'
WITH OPTIONS = {
'mode': 'CONTAINS',
'analyzer_class': 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
'analyzed': 'true'
};
```
After that I run the queries with '%' inside. As you can see multi-patterns are
handled by AND:
```
cqlsh:excelsior> select id, name from demo where name like '%RevisionDiff%';
id | name
--+
93dce11a-cfdd-4c16-b3b3-7537c7af03ec | RevisionDiffType.java
6586058f-bd57-4fc7-ae12-e6d8ddcd2ceb | RevisionDiffItem.java
d16dff53-002b-4fe6-9a10-bb32425360e0 | RevisionDiffItemDTO.java
bb20981e-714f-4eac-802f-6191dba5a301 | GetRevisionDiff.java
1c53574b-2eea-46f8-bcbc-5e295ef9c70a | RevisionDiffItemDTO.java
7366f852-d63c-4d07-86b3-18a3bf47e79b | RevisionDiffItemDTO.java
7f18accb-9832-4303-8227-43aa89534cde | RevisionDiffItemImpl.java
(7 rows)
cqlsh:excelsior> select id, name from demo where name like '%ItemImpl%';
id | name
--+---
603c1d12-4871-4244-896a-54ddb76dbd3b | FastTreeItemImpl.java
7f18accb-9832-4303-8227-43aa89534cde | RevisionDiffItemImpl.java
(2 rows)
cqlsh:excelsior> select id, name from demo where name like
'%RevisionDiff%ItemImpl%';
id | name
--+--
7f18accb-9832-4303-8227-43aa89534cde | RevisionDiffItemImpl.java
(1 rows)
```
> SASI index. Incorrect results for '%foo%bar%'-like search pattern.
> ---
>
> Key: CASSANDRA-12573
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12573
> Project: Cassandra
> Issue Type: Bug
>Reporter: Mikhail Krupitskiy
>Priority: Critical
> Labels: sasi
>
> We use Cassandra 3.7 and have faced a strange behaviour of SELECT requests
> with "LIKE '%foo%bar%'" constraints on a column with SASI index.
> Below are few experiments that show this behaviour.
> Experiment 1:
> {noformat}
> drop keyspace if exists kmv;
> create keyspace if not exists kmv WITH REPLICATION = { 'class' :
> 'SimpleStrategy', 'replication_factor':'1'} ;
> use kmv;
> CREATE TABLE if not exists kmv (id int primary key, c1 text, c2 text);
> CREATE CUSTOM INDEX ON kmv.kmv ( c2 ) USING
> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
> 'mode': 'CONTAINS'
> };
> insert into kmv (id, c1, c2) values (1, 'f21', 'qwe') ;
> insert into kmv (id, c1, c2) values (2, 'f22', 'qweasd') ;
> insert into kmv (id, c1, c2) values (3, 'f23', 'qwea1') ;
> insert into kmv (id, c1, c2) values (4, 'f24', '1qwe') ;
> insert into kmv (id, c1, c2) values (5, 'f25', 'asdqwe') ;
> select c2 from kmv.kmv where c2 like '%w%a%';
> {noformat}
> Expected result: qweasd, qwea1.
> Actual result: no rows.
> Experiment 2 (NOTE: definition of index is changed):
> {noformat}
> drop keyspace if exists kmv;
> create keyspace if not exists kmv WITH REPLICATION = { 'class' :
> 'SimpleStrategy', 'replication_factor':'1'} ;
> use kmv;
> CREATE TABLE if not exists kmv (id int primary key, c1 text, c2 text);
> CREATE CUSTOM INDEX ON kmv.kmv ( c2 ) USING
> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
> 'mode': 'CONTAINS',
> 'analyzer_class':
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
> 'analyzed': 'true'
> };
> insert into kmv (id, c1, c2) values (1, 'f21', 'qwe') ;
> insert into kmv (id, c1, c2) values (2, 'f22', 'qweasd') ;
> insert into kmv (id, c1, c2) values (3, 'f23', 'qwea1')
[jira] [Created] (CASSANDRA-12662) OOM when using SASI index
Maxim Podkolzine created CASSANDRA-12662: Summary: OOM when using SASI index Key: CASSANDRA-12662 URL: https://issues.apache.org/jira/browse/CASSANDRA-12662 Project: Cassandra Issue Type: Bug Environment: Linux, 4 CPU cores, 16Gb RAM, Cassandra process utilizes ~8Gb, of which ~4Gb is Java heap Reporter: Maxim Podkolzine Priority: Critical Fix For: 3.6 Attachments: memory-dump.png 2.8Gb of the heap is taken by the index data, pending for flush (see the screenshot). As a result the node fails with OOM. Questions: - Why can't Cassandra keep up with the inserted data and flush it? - What resources/configuration should be changed to improve the performance? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12573) SASI index. Incorrect results for '%foo%bar%'-like search pattern.
[
https://issues.apache.org/jira/browse/CASSANDRA-12573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496711#comment-15496711
]
Maxim Podkolzine commented on CASSANDRA-12573:
--
[~doanduyhai] Since CQL doesn't allow multiple LIKE constraints in one query,
it does look like a bug.
Can you suggest a workaround to search for several patterns?
> SASI index. Incorrect results for '%foo%bar%'-like search pattern.
> ---
>
> Key: CASSANDRA-12573
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12573
> Project: Cassandra
> Issue Type: Bug
>Reporter: Mikhail Krupitskiy
>Priority: Critical
> Labels: sasi
>
> We use Cassandra 3.7 and have faced a strange behaviour of SELECT requests
> with "LIKE '%foo%bar%'" constraints on a column with SASI index.
> Below are few experiments that show this behaviour.
> Experiment 1:
> {noformat}
> drop keyspace if exists kmv;
> create keyspace if not exists kmv WITH REPLICATION = { 'class' :
> 'SimpleStrategy', 'replication_factor':'1'} ;
> use kmv;
> CREATE TABLE if not exists kmv (id int primary key, c1 text, c2 text);
> CREATE CUSTOM INDEX ON kmv.kmv ( c2 ) USING
> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
> 'mode': 'CONTAINS'
> };
> insert into kmv (id, c1, c2) values (1, 'f21', 'qwe') ;
> insert into kmv (id, c1, c2) values (2, 'f22', 'qweasd') ;
> insert into kmv (id, c1, c2) values (3, 'f23', 'qwea1') ;
> insert into kmv (id, c1, c2) values (4, 'f24', '1qwe') ;
> insert into kmv (id, c1, c2) values (5, 'f25', 'asdqwe') ;
> select c2 from kmv.kmv where c2 like '%w%a%';
> {noformat}
> Expected result: qweasd, qwea1.
> Actual result: no rows.
> Experiment 2 (NOTE: definition of index is changed):
> {noformat}
> drop keyspace if exists kmv;
> create keyspace if not exists kmv WITH REPLICATION = { 'class' :
> 'SimpleStrategy', 'replication_factor':'1'} ;
> use kmv;
> CREATE TABLE if not exists kmv (id int primary key, c1 text, c2 text);
> CREATE CUSTOM INDEX ON kmv.kmv ( c2 ) USING
> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
> 'mode': 'CONTAINS',
> 'analyzer_class':
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
> 'analyzed': 'true'
> };
> insert into kmv (id, c1, c2) values (1, 'f21', 'qwe') ;
> insert into kmv (id, c1, c2) values (2, 'f22', 'qweasd') ;
> insert into kmv (id, c1, c2) values (3, 'f23', 'qwea1') ;
> insert into kmv (id, c1, c2) values (4, 'f24', '1qwe') ;
> insert into kmv (id, c1, c2) values (5, 'f25', 'asdqwe') ;
> select c2 from kmv.kmv where c2 like '%w%a%';
> {noformat}
> Expected result: qweasd, qwea1.
> Actual result: asdqwe, qweasd, qwea1.
> Experiment 3 (NOTE: primary key is compound now and inserted data was
> changed):
> {noformat}
> drop keyspace if exists kmv;
> create keyspace if not exists kmv WITH REPLICATION = { 'class' :
> 'SimpleStrategy', 'replication_factor':'1'} ;
> use kmv;
> CREATE TABLE if not exists kmv (id int, c1 text, c2 text, PRIMARY KEY(id,
> c1));
> CREATE CUSTOM INDEX ON kmv.kmv ( c2 ) USING
> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
> 'mode': 'CONTAINS',
> 'analyzer_class':
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
> 'analyzed': 'true'
> };
> insert into kmv (id, c1, c2) values (1, 'f21', 'qwe') ;
> insert into kmv (id, c1, c2) values (1, 'f22', 'qweasd') ;
> insert into kmv (id, c1, c2) values (1, 'f23', 'qwea1') ;
> insert into kmv (id, c1, c2) values (1, 'f24', '1qwe') ;
> insert into kmv (id, c1, c2) values (1, 'f25', 'asdqwe') ;
> select c2 from kmv.kmv where c2 like '%w%a%';
> {noformat}
> Expected result: qweasd, qwea1.
> Actual result: qwe, qweasd, qwea1, 1qwe, asdqwe.
> Experiment 4 (NOTE: search criteria is changed):
> {noformat}
> drop keyspace if exists kmv;
> create keyspace if not exists kmv WITH REPLICATION = { 'class' :
> 'SimpleStrategy', 'replication_factor':'1'} ;
> use kmv;
> CREATE TABLE if not exists kmv (id int, c1 text, c2 text, PRIMARY KEY(id,
> c1));
> CREATE CUSTOM INDEX ON kmv.kmv ( c2 ) USING
> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
> 'mode': 'CONTAINS',
> 'analyzer_class':
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
> 'analyzed': 'true'
> };
> insert into kmv (id, c1, c2) values (1, 'f21', 'qwe') ;
> insert into kmv (id, c1, c2) values (1, 'f22', 'qweasd') ;
> insert into kmv (id, c1, c2) values (1, 'f23', 'qwea1') ;
> insert into kmv (id, c1, c2) values (1, 'f24', '1qwe') ;
> insert into kmv (id, c1, c2) values (1, 'f25', 'asdqwe') ;
> select c2 from kmv.kmv where c2 like '%w22%a%';
> {noformat}
> Expected result: no rows.
> Actual result: qweasd, qwea1, asdqwe.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11456) support for PreparedStatement with LIKE
[
https://issues.apache.org/jira/browse/CASSANDRA-11456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15487016#comment-15487016
]
Maxim Podkolzine commented on CASSANDRA-11456:
--
[~beobal] Oh, it's so unfortunate.
We're currently using 3.5, have tried 3.7 and it turned out to be slower. So
we're evaluating 3.6 and probably will stick with that.
But in case it fails, I don't see any workaround to construct a LIKE constraint
in 3.5 with the query builder.
> support for PreparedStatement with LIKE
> ---
>
> Key: CASSANDRA-11456
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11456
> Project: Cassandra
> Issue Type: Improvement
> Components: CQL
>Reporter: Pavel Yaskevich
>Assignee: Sam Tunnicliffe
>Priority: Minor
> Labels: sasi
> Fix For: 3.6
>
>
> Using the Java driver for example:
> {code}
> PreparedStatement pst = session.prepare("select * from test.users where
> first_name LIKE ?");
> BoundStatement bs = pst.bind("Jon%");
> {code}
> The first line fails with {{SyntaxError: line 1:47 mismatched input '?'
> expecting STRING_LITERAL}} (which makes sense since it's how it's declared in
> the grammar). Other operators declare the right-hand side value as a
> {{Term.Raw}}, which can also be a bind marker.
> I think users will expect to be able to bind the argument this way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11456) support for PreparedStatement with LIKE
[
https://issues.apache.org/jira/browse/CASSANDRA-11456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15486896#comment-15486896
]
Maxim Podkolzine commented on CASSANDRA-11456:
--
[~beobal] is it possible to port the fix to 3.5 as well?
> support for PreparedStatement with LIKE
> ---
>
> Key: CASSANDRA-11456
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11456
> Project: Cassandra
> Issue Type: Improvement
> Components: CQL
>Reporter: Pavel Yaskevich
>Assignee: Sam Tunnicliffe
>Priority: Minor
> Labels: sasi
> Fix For: 3.6
>
>
> Using the Java driver for example:
> {code}
> PreparedStatement pst = session.prepare("select * from test.users where
> first_name LIKE ?");
> BoundStatement bs = pst.bind("Jon%");
> {code}
> The first line fails with {{SyntaxError: line 1:47 mismatched input '?'
> expecting STRING_LITERAL}} (which makes sense since it's how it's declared in
> the grammar). Other operators declare the right-hand side value as a
> {{Term.Raw}}, which can also be a bind marker.
> I think users will expect to be able to bind the argument this way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-12573) SASI index. Incorrect results for '%foo%bar%'-like search pattern.
[
https://issues.apache.org/jira/browse/CASSANDRA-12573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15473404#comment-15473404
]
Maxim Podkolzine commented on CASSANDRA-12573:
--
Can anyone please take a look at this?
> SASI index. Incorrect results for '%foo%bar%'-like search pattern.
> ---
>
> Key: CASSANDRA-12573
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12573
> Project: Cassandra
> Issue Type: Bug
>Reporter: Mikhail Krupitskiy
>Assignee: Arunkumar M
>Priority: Critical
>
> We use Cassandra 3.7 and have faced a strange behaviour of SELECT requests
> with "LIKE '%foo%bar%'" constraints on a column with SASI index.
> Below are few experiments that show this behaviour.
> Experiment 1:
> {noformat}
> drop keyspace if exists kmv;
> create keyspace if not exists kmv WITH REPLICATION = { 'class' :
> 'SimpleStrategy', 'replication_factor':'1'} ;
> use kmv;
> CREATE TABLE if not exists kmv (id int primary key, c1 text, c2 text);
> CREATE CUSTOM INDEX ON kmv.kmv ( c2 ) USING
> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
> 'mode': 'CONTAINS'
> };
> insert into kmv (id, c1, c2) values (1, 'f21', 'qwe') ;
> insert into kmv (id, c1, c2) values (2, 'f22', 'qweasd') ;
> insert into kmv (id, c1, c2) values (3, 'f23', 'qwea1') ;
> insert into kmv (id, c1, c2) values (4, 'f24', '1qwe') ;
> insert into kmv (id, c1, c2) values (5, 'f25', 'asdqwe') ;
> select c2 from kmv.kmv where c2 like '%w%a%';
> {noformat}
> Expected result: qweasd, qwea1.
> Actual result: no rows.
> Experiment 2 (NOTE: definition of index is changed):
> {noformat}
> drop keyspace if exists kmv;
> create keyspace if not exists kmv WITH REPLICATION = { 'class' :
> 'SimpleStrategy', 'replication_factor':'1'} ;
> use kmv;
> CREATE TABLE if not exists kmv (id int primary key, c1 text, c2 text);
> CREATE CUSTOM INDEX ON kmv.kmv ( c2 ) USING
> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
> 'mode': 'CONTAINS',
> 'analyzer_class':
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
> 'analyzed': 'true'
> };
> insert into kmv (id, c1, c2) values (1, 'f21', 'qwe') ;
> insert into kmv (id, c1, c2) values (2, 'f22', 'qweasd') ;
> insert into kmv (id, c1, c2) values (3, 'f23', 'qwea1') ;
> insert into kmv (id, c1, c2) values (4, 'f24', '1qwe') ;
> insert into kmv (id, c1, c2) values (5, 'f25', 'asdqwe') ;
> select c2 from kmv.kmv where c2 like '%w%a%';
> {noformat}
> Expected result: qweasd, qwea1.
> Actual result: asdqwe, qweasd, qwea1.
> Experiment 3 (NOTE: primary key is compound now and inserted data was
> changed):
> {noformat}
> drop keyspace if exists kmv;
> create keyspace if not exists kmv WITH REPLICATION = { 'class' :
> 'SimpleStrategy', 'replication_factor':'1'} ;
> use kmv;
> CREATE TABLE if not exists kmv (id int, c1 text, c2 text, PRIMARY KEY(id,
> c1));
> CREATE CUSTOM INDEX ON kmv.kmv ( c2 ) USING
> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
> 'mode': 'CONTAINS',
> 'analyzer_class':
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
> 'analyzed': 'true'
> };
> insert into kmv (id, c1, c2) values (1, 'f21', 'qwe') ;
> insert into kmv (id, c1, c2) values (1, 'f22', 'qweasd') ;
> insert into kmv (id, c1, c2) values (1, 'f23', 'qwea1') ;
> insert into kmv (id, c1, c2) values (1, 'f24', '1qwe') ;
> insert into kmv (id, c1, c2) values (1, 'f25', 'asdqwe') ;
> select c2 from kmv.kmv where c2 like '%w%a%';
> {noformat}
> Expected result: qweasd, qwea1.
> Actual result: qwe, qweasd, qwea1, 1qwe, asdqwe.
> Experiment 4 (NOTE: search criteria is changed):
> {noformat}
> drop keyspace if exists kmv;
> create keyspace if not exists kmv WITH REPLICATION = { 'class' :
> 'SimpleStrategy', 'replication_factor':'1'} ;
> use kmv;
> CREATE TABLE if not exists kmv (id int, c1 text, c2 text, PRIMARY KEY(id,
> c1));
> CREATE CUSTOM INDEX ON kmv.kmv ( c2 ) USING
> 'org.apache.cassandra.index.sasi.SASIIndex' WITH OPTIONS = {
> 'mode': 'CONTAINS',
> 'analyzer_class':
> 'org.apache.cassandra.index.sasi.analyzer.StandardAnalyzer',
> 'analyzed': 'true'
> };
> insert into kmv (id, c1, c2) values (1, 'f21', 'qwe') ;
> insert into kmv (id, c1, c2) values (1, 'f22', 'qweasd') ;
> insert into kmv (id, c1, c2) values (1, 'f23', 'qwea1') ;
> insert into kmv (id, c1, c2) values (1, 'f24', '1qwe') ;
> insert into kmv (id, c1, c2) values (1, 'f25', 'asdqwe') ;
> select c2 from kmv.kmv where c2 like '%w22%a%';
> {noformat}
> Expected result: no rows.
> Actual result: qweasd, qwea1, asdqwe.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11026) OOM
[ https://issues.apache.org/jira/browse/CASSANDRA-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Podkolzine updated CASSANDRA-11026: - Attachment: Screenshot.png One more occurrence. > OOM > --- > > Key: CASSANDRA-11026 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11026 > Project: Cassandra > Issue Type: Bug >Reporter: Maxim Podkolzine > Attachments: Screenshot.png, dump.png > > > Cassandra 3.0.2 fails with OOM. The heapdump shows large number of > HeapByteBuffer instances, each retaining 1Mb (see the details on the > screenshot). Overall retained size is ~2Gb. > We can provide the additional info and the whole heapdump if necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-11026) OOM
Maxim Podkolzine created CASSANDRA-11026: Summary: OOM Key: CASSANDRA-11026 URL: https://issues.apache.org/jira/browse/CASSANDRA-11026 Project: Cassandra Issue Type: Bug Reporter: Maxim Podkolzine Attachments: dump.png Cassandra 3.0.2 fails with OOM. The heapdump shows large number of HeapByteBuffer instances, each retaining 1Gb (see the details on the screenshot). Overall retained size is ~2Gb. We can provide the additional info and the whole heapdump if necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11026) OOM
[ https://issues.apache.org/jira/browse/CASSANDRA-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Podkolzine updated CASSANDRA-11026: - Description: Cassandra 3.0.2 fails with OOM. The heapdump shows large number of HeapByteBuffer instances, each retaining 1Mb (see the details on the screenshot). Overall retained size is ~2Gb. We can provide the additional info and the whole heapdump if necessary. was: Cassandra 3.0.2 fails with OOM. The heapdump shows large number of HeapByteBuffer instances, each retaining 1Gb (see the details on the screenshot). Overall retained size is ~2Gb. We can provide the additional info and the whole heapdump if necessary. > OOM > --- > > Key: CASSANDRA-11026 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11026 > Project: Cassandra > Issue Type: Bug >Reporter: Maxim Podkolzine > Attachments: dump.png > > > Cassandra 3.0.2 fails with OOM. The heapdump shows large number of > HeapByteBuffer instances, each retaining 1Mb (see the details on the > screenshot). Overall retained size is ~2Gb. > We can provide the additional info and the whole heapdump if necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10922) Inconsistent query results
[ https://issues.apache.org/jira/browse/CASSANDRA-10922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101962#comment-15101962 ] Maxim Podkolzine commented on CASSANDRA-10922: -- Unfortunately, we don't know how to reproduce. But we can run any commands and provide all info that can help you. Could you please describe how to run scrub, or turn on tracing? > Inconsistent query results > -- > > Key: CASSANDRA-10922 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10922 > Project: Cassandra > Issue Type: Bug >Reporter: Maxim Podkolzine >Priority: Critical > > I have a DB created with Cassandra 2.2.3. And currently I'm running it by > Cassandra 3.0.2. > The value of a particular cell is returned depending on the query I run (in > cqlsh): > - returned when iterate all columns, i.e. > SELECT value FROM "3xupsource".Content WHERE databaseid=0x2112 LIMIT 2 > (I can see the columns 0x and 0x0100 there, the values seem > correct) > - not returned when I specify a particular column > SELECT value FROM "3xupsource".Content WHERE databaseid=0x2112 AND > columnid=0x0100 > Other queries like SELECT value FROM "3xupsource".Content WHERE > databaseid=0x2112 AND columnid=0x work consistently. > There is nothing in Cassandra error log, so it does not look like a > corruption. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-10922) Inconsistent query results
Maxim Podkolzine created CASSANDRA-10922: Summary: Inconsistent query results Key: CASSANDRA-10922 URL: https://issues.apache.org/jira/browse/CASSANDRA-10922 Project: Cassandra Issue Type: Bug Reporter: Maxim Podkolzine Priority: Critical I have a DB created with Cassandra 2.2.3. And currently I'm running it by Cassandra 3.0.2. The value of a particular cell is returned depending on the query I run (in cqlsh): - returned when iterate all columns, i.e. SELECT value FROM "3xupsource".Content WHERE databaseid=0x2112 LIMIT 2 (I can see the columns 0x and 0x0100 there, the values seem correct) - not returned when I specify a particular column SELECT value FROM "3xupsource".Content WHERE databaseid=0x2112 AND columnid=0x0100 Other queries like SELECT value FROM "3xupsource".Content WHERE databaseid=0x2112 AND columnid=0x work consistently. There is nothing in Cassandra error log, so it does not look like a corruption. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6411) Issue with reading from sstable
[
https://issues.apache.org/jira/browse/CASSANDRA-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14976225#comment-14976225
]
Maxim Podkolzine commented on CASSANDRA-6411:
-
The status of the issue is "Awaiting feedback". If you need some some
additional information from me, please let me know.
> Issue with reading from sstable
> ---
>
> Key: CASSANDRA-6411
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6411
> Project: Cassandra
> Issue Type: Bug
> Components: API
>Reporter: Mike Konobeevskiy
>Assignee: Yuki Morishita
> Attachments: 6411-log.zip, 6411-sstables.zip
>
>
> With Cassandra 1.2.5 this happens almost every week.
> {noformat}
> java.lang.RuntimeException:
> org.apache.cassandra.io.sstable.CorruptSSTableException:
> java.io.EOFException: EOF after 5105 bytes out of 19815
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException:
> java.io.EOFException: EOF after 5105 bytes out of 19815
> at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.(SimpleSliceReader.java:91)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:68)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:44)
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:101)
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:274)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
> at org.apache.cassandra.db.Table.getRow(Table.java:347)
> at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
> at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1052)
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578)
> ... 3 more
> Caused by: java.io.EOFException: EOF after 5105 bytes out of 19815
> at
> org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:350)
> at
> org.apache.cassandra.utils.ByteBufferUtil.skipShortLength(ByteBufferUtil.java:382)
> at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.(SimpleSliceReader.java:72)
> ... 16 more
> {noformat}
> This is occurring roughly weekly with quite minimal usage.
> Recreation of CF does not help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9549) Memory leak in Ref.GlobalState due to pathological ConcurrentLinkedQueue.remove behaviour
[
https://issues.apache.org/jira/browse/CASSANDRA-9549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968882#comment-14968882
]
Maxim Podkolzine commented on CASSANDRA-9549:
-
Is this bug fixed in Cassandra 2.2.0?
> Memory leak in Ref.GlobalState due to pathological
> ConcurrentLinkedQueue.remove behaviour
> -
>
> Key: CASSANDRA-9549
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9549
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 2.1.5. 9 node cluster in EC2 (m1.large nodes,
> 2 cores 7.5G memory, 800G platter for cassandra data, root partition and
> commit log are on SSD EBS with sufficient IOPS), 3 nodes/availablity zone, 1
> replica/zone
> JVM: /usr/java/jdk1.8.0_40/jre/bin/java
> JVM Flags besides CP: -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar
> -XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42 -Xms2G -Xmx2G -Xmn200M
> -XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103
> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB -XX:CompileCommandFile=/etc/cassandra/conf/hotspot_compiler
> -XX:CMSWaitDuration=1 -XX:+CMSParallelInitialMarkEnabled
> -XX:+CMSEdenChunksRecordAlways -XX:CMSWaitDuration=1 -XX:+UseCondCardMark
> -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199
> -Dcom.sun.management.jmxremote.rmi.port=7199
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dlogback.configurationFile=logback.xml -Dcassandra.logdir=/var/log/cassandra
> -Dcassandra.storagedir= -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid
> Kernel: Linux 2.6.32-504.16.2.el6.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Ivar Thorson
>Assignee: Benedict
>Priority: Critical
> Fix For: 2.1.7
>
> Attachments: c4_system.log, c7fromboot.zip, cassandra.yaml,
> cpu-load.png, memoryuse.png, ref-java-errors.jpeg, suspect.png, two-loads.png
>
>
> We have been experiencing a severe memory leak with Cassandra 2.1.5 that,
> over the period of a couple of days, eventually consumes all of the available
> JVM heap space, putting the JVM into GC hell where it keeps trying CMS
> collection but can't free up any heap space. This pattern happens for every
> node in our cluster and is requiring rolling cassandra restarts just to keep
> the cluster running. We have upgraded the cluster per Datastax docs from the
> 2.0 branch a couple of months ago and have been using the data from this
> cluster for more than a year without problem.
> As the heap fills up with non-GC-able objects, the CPU/OS load average grows
> along with it. Heap dumps reveal an increasing number of
> java.util.concurrent.ConcurrentLinkedQueue$Node objects. We took heap dumps
> over a 2 day period, and watched the number of Node objects go from 4M, to
> 19M, to 36M, and eventually about 65M objects before the node stops
> responding. The screen capture of our heap dump is from the 19M measurement.
> Load on the cluster is minimal. We can see this effect even with only a
> handful of writes per second. (See attachments for Opscenter snapshots during
> very light loads and heavier loads). Even with only 5 reads a sec we see this
> behavior.
> Log files show repeated errors in Ref.java:181 and Ref.java:279 and "LEAK
> detected" messages:
> {code}
> ERROR [CompactionExecutor:557] 2015-06-01 18:27:36,978 Ref.java:279 - Error
> when closing class
> org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1302301946:/data1/data/ourtablegoeshere-ka-1150
> java.util.concurrent.RejectedExecutionException: Task
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@32680b31
> rejected from
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor@573464d6[Terminated,
> pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1644]
> {code}
> {code}
> ERROR [Reference-Reaper:1] 2015-06-01 18:27:37,083 Ref.java:181 - LEAK
> DETECTED: a reference
> (org.apache.cassandra.utils.concurrent.Ref$State@74b5df92) to class
> org.apache.cassandra.io.sstable.SSTableReader$DescriptorTypeTidy@2054303604:/data2/data/ourtablegoeshere-ka-1151
> was not released before the reference was garbage collected
> {code}
> This might be related to [CASSANDRA-8723]?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-10548) OOM in Ref#GlobalState
Maxim Podkolzine created CASSANDRA-10548: Summary: OOM in Ref#GlobalState Key: CASSANDRA-10548 URL: https://issues.apache.org/jira/browse/CASSANDRA-10548 Project: Cassandra Issue Type: Bug Reporter: Maxim Podkolzine Attachments: Screenshot.png Cassandra eats up all memory (at one time 24Gb) and crashes with OOM. I was able to analyze one instance of hprof and concluded that ~3Gb were referenced by GlobalState instances, mostly ConcurrentLinkedQueues (see the details in screenshot attached). I can provide the hprof if necessary. I noticed that in 2.2 the source code changed in this place. Can you confirm it's better in 2.2 in terms of memory management? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6411) Issue with reading from sstable
[
https://issues.apache.org/jira/browse/CASSANDRA-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14941013#comment-14941013
]
Maxim Podkolzine commented on CASSANDRA-6411:
-
[~yukim] Yes, it is from CompressionMetadata. If there is a dedicated issue,
I'd be happy to watch it too.
But most importantly: what is the cause of this? Anything I can do to prevent
it from happening?
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
at
org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:131)
~[apache-cassandra-2.1-879691dd.jar:2.1-879691dd]
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85)
~[apache-cassandra-2.1-879691dd.jar:2.1-879691dd]
at
org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79)
~[apache-cassandra-2.1-879691dd.jar:2.1-879691dd]
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72)
~[apache-cassandra-2.1-879691dd.jar:2.1-879691dd]
at
org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:168)
~[apache-cassandra-2.1-879691dd.jar:2.1-879691dd]
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:711)
~[apache-cassandra-2.1-879691dd.jar:2.1-879691dd]
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:666)
~[apache-cassandra-2.1-879691dd.jar:2.1-879691dd]
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:479)
~[apache-cassandra-2.1-879691dd.jar:2.1-879691dd]
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:378)
~[apache-cassandra-2.1-879691dd.jar:2.1-879691dd]
at
org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:516)
~[apache-cassandra-2.1-879691dd.jar:2.1-879691dd]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
Caused by: java.io.EOFException: null
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
~[na:1.8.0_45]
at java.io.DataInputStream.readUTF(DataInputStream.java:589)
~[na:1.8.0_45]
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
~[na:1.8.0_45]
at
org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:106)
~[apache-cassandra-2.1-879691dd.jar:2.1-879691dd]
... 14 common frames omitted
> Issue with reading from sstable
> ---
>
> Key: CASSANDRA-6411
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6411
> Project: Cassandra
> Issue Type: Bug
> Components: API
>Reporter: Mike Konobeevskiy
>Assignee: Yuki Morishita
> Attachments: 6411-log.zip, 6411-sstables.zip
>
>
> With Cassandra 1.2.5 this happens almost every week.
> {noformat}
> java.lang.RuntimeException:
> org.apache.cassandra.io.sstable.CorruptSSTableException:
> java.io.EOFException: EOF after 5105 bytes out of 19815
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException:
> java.io.EOFException: EOF after 5105 bytes out of 19815
> at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.(SimpleSliceReader.java:91)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:68)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:44)
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:101)
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:274)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
> at
>
[jira] [Commented] (CASSANDRA-6411) Issue with reading from sstable
[
https://issues.apache.org/jira/browse/CASSANDRA-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14936703#comment-14936703
]
Maxim Podkolzine commented on CASSANDRA-6411:
-
We observed this issue in 2.2.0. Is there any workaround?
> Issue with reading from sstable
> ---
>
> Key: CASSANDRA-6411
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6411
> Project: Cassandra
> Issue Type: Bug
> Components: API
>Reporter: Mike Konobeevskiy
>Assignee: Yuki Morishita
> Attachments: 6411-log.zip, 6411-sstables.zip
>
>
> With Cassandra 1.2.5 this happens almost every week.
> {noformat}
> java.lang.RuntimeException:
> org.apache.cassandra.io.sstable.CorruptSSTableException:
> java.io.EOFException: EOF after 5105 bytes out of 19815
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException:
> java.io.EOFException: EOF after 5105 bytes out of 19815
> at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.(SimpleSliceReader.java:91)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:68)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.(SSTableSliceIterator.java:44)
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:101)
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:274)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
> at org.apache.cassandra.db.Table.getRow(Table.java:347)
> at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:70)
> at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1052)
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578)
> ... 3 more
> Caused by: java.io.EOFException: EOF after 5105 bytes out of 19815
> at
> org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:350)
> at
> org.apache.cassandra.utils.ByteBufferUtil.skipShortLength(ByteBufferUtil.java:382)
> at
> org.apache.cassandra.db.columniterator.SimpleSliceReader.(SimpleSliceReader.java:72)
> ... 16 more
> {noformat}
> This is occurring roughly weekly with quite minimal usage.
> Recreation of CF does not help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8535) java.lang.RuntimeException: Failed to rename XXX to YYY
[
https://issues.apache.org/jira/browse/CASSANDRA-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14589583#comment-14589583
]
Maxim Podkolzine commented on CASSANDRA-8535:
-
I'm afraid this issue has to be reopened. I've got this in 2.1.6:
2015-06-15T17:31:44,797 [CompactionExecutor:1] ERROR
o.a.c.service.CassandraDaemon - Exception in thread
Thread[CompactionExecutor:1,1,main]
java.lang.RuntimeException: Failed to rename
E:\Upsource\Upsource_2.0.262_new_cassandra\data\cassandra\data\upsource\content-f13ce210136211e59a87137398728adc\upsource-content-tmp-ka-18-Index.db
to E:\Upsource\Upsource_2.0.262_new_cassandra\
data\cassandra\data\upsource\content-f13ce210136211e59a87137398728adc\upsource-content-ka-18-Index.db
at
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:170)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:154)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:541)
~[apache-cassandra-2.1.6.jar:2.1.6]
at
org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:533)
~[apache-cassandra-2.1.6.jar:2.1.6]
Can provide the full stacktrace if needed, but it looks the same as in the
issue:
Caused by: java.nio.file.FileSystemException:
E:\Upsource\Upsource_2.0.262_new_cassandra\data\cassandra\data\upsource\uniqueidhistory_t-eef64d70136211e59a87137398728adc\upsource-uniqueidhistory_t-tmplink-ka-23-Index.db:
The process cannot access the file because it is being used by another process.
I can add that we've been running Cassandra build on 879691dd revision and it
worked fine on Windows under same load. So it looks like one of the last
changes in 2.1 branch broke it again.
java.lang.RuntimeException: Failed to rename XXX to YYY
---
Key: CASSANDRA-8535
URL: https://issues.apache.org/jira/browse/CASSANDRA-8535
Project: Cassandra
Issue Type: Bug
Environment: Windows 2008 X64
Reporter: Leonid Shalupov
Assignee: Joshua McKenzie
Labels: Windows
Fix For: 2.1.5
Attachments: 8535_v1.txt, 8535_v2.txt, 8535_v3.txt
{code}
java.lang.RuntimeException: Failed to rename
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db
to
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-ka-5-Index.db
at
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:170)
~[main/:na]
at
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:154)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:569)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:561)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.close(SSTableWriter.java:535)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:470)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finishAndMaybeThrow(SSTableRewriter.java:349)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:324)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:304)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:200)
~[main/:na]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
~[main/:na]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:226)
~[main/:na]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.nio.file.FileSystemException:
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db
-
[jira] [Commented] (CASSANDRA-9127) Out of memory failure: ~2Gb retained
[ https://issues.apache.org/jira/browse/CASSANDRA-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485066#comment-14485066 ] Maxim Podkolzine commented on CASSANDRA-9127: - Damn, I did clean, but I really missed realclean =) Thanks! We've started to use 2.1.4 once it got out. I'll update the bug if we see anything like OOM Out of memory failure: ~2Gb retained Key: CASSANDRA-9127 URL: https://issues.apache.org/jira/browse/CASSANDRA-9127 Project: Cassandra Issue Type: Bug Reporter: Maxim Podkolzine Assignee: Benedict Fix For: 2.1.5 Attachments: snapshot.png See the snapshot analysis. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8390) The process cannot access the file because it is being used by another process
[
https://issues.apache.org/jira/browse/CASSANDRA-8390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485150#comment-14485150
]
Maxim Podkolzine commented on CASSANDRA-8390:
-
We have tested Cassandra 2.1.4 and still see The process cannot access the
file because it is being used by another process every 2-3 hours.
Though this time Cassandra does not fail, but just stops gently.
Does it mean that patches for CASSANDRA-8535, CASSANDRA-8812 and others were
not included in 2.1.4?
The process cannot access the file because it is being used by another process
--
Key: CASSANDRA-8390
URL: https://issues.apache.org/jira/browse/CASSANDRA-8390
Project: Cassandra
Issue Type: Bug
Reporter: Ilya Komolkin
Assignee: Joshua McKenzie
Fix For: 3.0
Attachments: CassandraDiedWithDiskAccessModeStandardLogs.7z, FSD.PNG,
NoHostAvailableLogs.zip, recreate-CASSANDRA-8390.tgz
{code}21:46:27.810 [NonPeriodicTasks:1] ERROR o.a.c.service.CassandraDaemon -
Exception in thread Thread[NonPeriodicTasks:1,5,main]
org.apache.cassandra.io.FSWriteError: java.nio.file.FileSystemException:
E:\Upsource_12391\data\cassandra\data\kernel\filechangehistory_t-a277b560764611e48c8e4915424c75fe\kernel-filechangehistory_t-ka-33-Index.db:
The process cannot access the file because it is being used by another
process.
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135)
~[cassandra-all-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:121)
~[cassandra-all-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:113)
~[cassandra-all-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableDeletingTask.run(SSTableDeletingTask.java:94)
~[cassandra-all-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:664)
~[cassandra-all-2.1.1.jar:2.1.1]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_71]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
~[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: java.nio.file.FileSystemException:
E:\Upsource_12391\data\cassandra\data\kernel\filechangehistory_t-a277b560764611e48c8e4915424c75fe\kernel-filechangehistory_t-ka-33-Index.db:
The process cannot access the file because it is being used by another
process.
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
~[na:1.7.0_71]
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
~[na:1.7.0_71]
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
~[na:1.7.0_71]
at
sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
~[na:1.7.0_71]
at
sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
~[na:1.7.0_71]
at java.nio.file.Files.delete(Files.java:1079) ~[na:1.7.0_71]
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131)
~[cassandra-all-2.1.1.jar:2.1.1]
... 11 common frames omitted{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9127) Out of memory failure: ~2Gb retained
[ https://issues.apache.org/jira/browse/CASSANDRA-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485057#comment-14485057 ] Maxim Podkolzine commented on CASSANDRA-9127: - The logs and hprof are available here: ftp://ftp.jetbrains.net/.upsource/2.1.3/OOM/ Out of memory failure: ~2Gb retained Key: CASSANDRA-9127 URL: https://issues.apache.org/jira/browse/CASSANDRA-9127 Project: Cassandra Issue Type: Bug Reporter: Maxim Podkolzine Assignee: Benedict Fix For: 2.1.5 Attachments: snapshot.png See the snapshot analysis. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8390) The process cannot access the file because it is being used by another process
[
https://issues.apache.org/jira/browse/CASSANDRA-8390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485150#comment-14485150
]
Maxim Podkolzine edited comment on CASSANDRA-8390 at 4/8/15 12:30 PM:
--
We have tested Cassandra 2.1.4 and still see The process cannot access the
file because it is being used by another process every 2-3 hours.
Though this time Cassandra does not fail, but just stops gently.
Does it mean that patches for CASSANDRA-8535, CASSANDRA-8812 and others were
not included in 2.1.4?
UPD: I see from commits that these patches got into 2.1 branch, but not into
2.1.4
was (Author: maximp):
We have tested Cassandra 2.1.4 and still see The process cannot access the
file because it is being used by another process every 2-3 hours.
Though this time Cassandra does not fail, but just stops gently.
Does it mean that patches for CASSANDRA-8535, CASSANDRA-8812 and others were
not included in 2.1.4?
The process cannot access the file because it is being used by another process
--
Key: CASSANDRA-8390
URL: https://issues.apache.org/jira/browse/CASSANDRA-8390
Project: Cassandra
Issue Type: Bug
Reporter: Ilya Komolkin
Assignee: Joshua McKenzie
Fix For: 3.0
Attachments: CassandraDiedWithDiskAccessModeStandardLogs.7z, FSD.PNG,
NoHostAvailableLogs.zip, recreate-CASSANDRA-8390.tgz
{code}21:46:27.810 [NonPeriodicTasks:1] ERROR o.a.c.service.CassandraDaemon -
Exception in thread Thread[NonPeriodicTasks:1,5,main]
org.apache.cassandra.io.FSWriteError: java.nio.file.FileSystemException:
E:\Upsource_12391\data\cassandra\data\kernel\filechangehistory_t-a277b560764611e48c8e4915424c75fe\kernel-filechangehistory_t-ka-33-Index.db:
The process cannot access the file because it is being used by another
process.
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135)
~[cassandra-all-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:121)
~[cassandra-all-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:113)
~[cassandra-all-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableDeletingTask.run(SSTableDeletingTask.java:94)
~[cassandra-all-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:664)
~[cassandra-all-2.1.1.jar:2.1.1]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_71]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
~[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: java.nio.file.FileSystemException:
E:\Upsource_12391\data\cassandra\data\kernel\filechangehistory_t-a277b560764611e48c8e4915424c75fe\kernel-filechangehistory_t-ka-33-Index.db:
The process cannot access the file because it is being used by another
process.
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
~[na:1.7.0_71]
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
~[na:1.7.0_71]
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
~[na:1.7.0_71]
at
sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
~[na:1.7.0_71]
at
sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
~[na:1.7.0_71]
at java.nio.file.Files.delete(Files.java:1079) ~[na:1.7.0_71]
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131)
~[cassandra-all-2.1.1.jar:2.1.1]
... 11 common frames omitted{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-9127) Out of memory failure: ~2.7Gb retained
Maxim Podkolzine created CASSANDRA-9127: --- Summary: Out of memory failure: ~2.7Gb retained Key: CASSANDRA-9127 URL: https://issues.apache.org/jira/browse/CASSANDRA-9127 Project: Cassandra Issue Type: Bug Reporter: Maxim Podkolzine Attachments: snapshot.png See the snapshot analysis. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9127) Out of memory failure: ~2Gb retained
[ https://issues.apache.org/jira/browse/CASSANDRA-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Podkolzine updated CASSANDRA-9127: Summary: Out of memory failure: ~2Gb retained (was: Out of memory failure: ~2.7Gb retained) Out of memory failure: ~2Gb retained Key: CASSANDRA-9127 URL: https://issues.apache.org/jira/browse/CASSANDRA-9127 Project: Cassandra Issue Type: Bug Reporter: Maxim Podkolzine Attachments: snapshot.png See the snapshot analysis. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9127) Out of memory failure: ~2Gb retained
[ https://issues.apache.org/jira/browse/CASSANDRA-9127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483203#comment-14483203 ] Maxim Podkolzine commented on CASSANDRA-9127: - It's 2.1-HEAD, much closer to 2.1.4. OK, we'll try to share the full dump, it's pretty big Out of memory failure: ~2Gb retained Key: CASSANDRA-9127 URL: https://issues.apache.org/jira/browse/CASSANDRA-9127 Project: Cassandra Issue Type: Bug Reporter: Maxim Podkolzine Assignee: Benedict Attachments: snapshot.png See the snapshot analysis. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9018) Dropped keyspace is not collected
[ https://issues.apache.org/jira/browse/CASSANDRA-9018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377851#comment-14377851 ] Maxim Podkolzine commented on CASSANDRA-9018: - I see, thanks for clarification. Here's I find misleading here: - The two settings are obviously conflicting. I personally find gc_grace_seconds option a better choice, because it gives a sense of control: I can decide the expiration of my data, per table on top of that. - More general: there are several notions dealing with the same problem: grace period, snapshot and backup. In such cases conflicts often arise. - Strong recommended settings (auto_snapshot: true) require an operator to monitor the disk space. It seems more convenient if Cassandra takes care of its own space. - The log doesn't give enough information about what's going on. Can advise the proper settings for our case: it's perfectly normal that the data is dropped, and it should be kept _just in case_ for some period? When needed we do a full backup ourselves. Are there any potential problems with auto_snapshot: false? Dropped keyspace is not collected - Key: CASSANDRA-9018 URL: https://issues.apache.org/jira/browse/CASSANDRA-9018 Project: Cassandra Issue Type: Bug Reporter: Maxim Podkolzine Attachments: cassandra-log.zip As far as I understand when a keyspace is dropped, the data is marked as tombstone. We expect that after the grace period (all tables are created with gc_grace_seconds=7200), this data is automatically removed during the compaction process, which means that keyspace no longer takes any space on disk. This is not happening (not after 2 or 24 hours). The log keeps saying No files to compact for user defined compaction, keyspace files remain on disk. It's not clear whether Cassandra is still waiting for certain event, or decided not to collect the data. Is there any setting that I missed? Any clues to figure out from the log, what's the current state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9018) Dropped keyspace is not collected
[ https://issues.apache.org/jira/browse/CASSANDRA-9018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377620#comment-14377620 ] Maxim Podkolzine commented on CASSANDRA-9018: - It's 2.1.1. No, I can't be sure of that, maybe the files are kept for a reason. But I didn't do anything specific using a command line, and as I said the log does not explain much. Essentially the keyspace was created along with the tables, some data was inserted, and then it was dropped. According to the yaml data is not incrementally backed up: incremental_backups: false I do see auto_snapshot: true, but that only reason for that is because the doc STRONGLY advised it. Dropped keyspace is not collected - Key: CASSANDRA-9018 URL: https://issues.apache.org/jira/browse/CASSANDRA-9018 Project: Cassandra Issue Type: Bug Reporter: Maxim Podkolzine Attachments: cassandra-log.zip As far as I understand when a keyspace is dropped, the data is marked as tombstone. We expect that after the grace period (all tables are created with gc_grace_seconds=7200), this data is automatically removed during the compaction process, which means that keyspace no longer takes any space on disk. This is not happening (not after 2 or 24 hours). The log keeps saying No files to compact for user defined compaction, keyspace files remain on disk. It's not clear whether Cassandra is still waiting for certain event, or decided not to collect the data. Is there any setting that I missed? Any clues to figure out from the log, what's the current state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9018) Dropped keyspace is not collected
Maxim Podkolzine created CASSANDRA-9018: --- Summary: Dropped keyspace is not collected Key: CASSANDRA-9018 URL: https://issues.apache.org/jira/browse/CASSANDRA-9018 Project: Cassandra Issue Type: Bug Reporter: Maxim Podkolzine Attachments: cassandra-log.zip As far as I understand when a keyspace is dropped, the data is marked as tombstone. We expect that after the grace period (all tables are created with gc_grace_seconds=7200), this data is automatically removed during the compaction process, which means that keyspace no longer takes any space on disk. This is not happening (not after 2 or 24 hours). The log keeps saying No files to compact for user defined compaction, keyspace files remain on disk. It's not clear whether Cassandra is still waiting for certain event, or decided not to collect the data. Is there any setting that I missed? Any clues to figure out from the log, what's the current state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8535) java.lang.RuntimeException: Failed to rename XXX to YYY
[
https://issues.apache.org/jira/browse/CASSANDRA-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319978#comment-14319978
]
Maxim Podkolzine commented on CASSANDRA-8535:
-
It'll be very disappointing for us, if the fix is postponed until 2.1.4.
I understand that quick fixes are undesirable, but we are so eager to see it
working on Windows, and this issue is one of the most frequent we see.
java.lang.RuntimeException: Failed to rename XXX to YYY
---
Key: CASSANDRA-8535
URL: https://issues.apache.org/jira/browse/CASSANDRA-8535
Project: Cassandra
Issue Type: Bug
Environment: Windows 2008 X64
Reporter: Leonid Shalupov
Assignee: Joshua McKenzie
Fix For: 2.1.3
Attachments: 8535_v1.txt, 8535_v2.txt
{code}
java.lang.RuntimeException: Failed to rename
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db
to
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-ka-5-Index.db
at
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:170)
~[main/:na]
at
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:154)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:569)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:561)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.close(SSTableWriter.java:535)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:470)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finishAndMaybeThrow(SSTableRewriter.java:349)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:324)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:304)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:200)
~[main/:na]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
~[main/:na]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:226)
~[main/:na]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.nio.file.FileSystemException:
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db
-
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-ka-5-Index.db:
The process cannot access the file because it is being used by another
process.
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
~[na:1.7.0_45]
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
~[na:1.7.0_45]
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
~[na:1.7.0_45]
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
~[na:1.7.0_45]
at java.nio.file.Files.move(Files.java:1345) ~[na:1.7.0_45]
at
org.apache.cassandra.io.util.FileUtils.atomicMoveWithFallback(FileUtils.java:184)
~[main/:na]
at
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:166)
~[main/:na]
... 18 common frames omitted
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-8800) Cassandra fails to start with OOM after disk space problem occurred
Maxim Podkolzine created CASSANDRA-8800: --- Summary: Cassandra fails to start with OOM after disk space problem occurred Key: CASSANDRA-8800 URL: https://issues.apache.org/jira/browse/CASSANDRA-8800 Project: Cassandra Issue Type: Bug Reporter: Maxim Podkolzine It one point the Cassandra failed with org.apache.cassandra.io.FSWriteError: java.io.IOException: No space left on device Which seems pretty normal situation. But after the disk space was increased, Cassandra refused to start up again. The memory snapshot shows a big ArrayList of org.apache.cassandra.io.sstable.SSTableScanner is allocated, holding around 2.8Gb. It looks like the DB is corrupted, but we're not entirely sure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8535) java.lang.RuntimeException: Failed to rename XXX to YYY
[
https://issues.apache.org/jira/browse/CASSANDRA-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320044#comment-14320044
]
Maxim Podkolzine commented on CASSANDRA-8535:
-
We can build and go on with our own distributive, but I don't feel like it's a
good way to go, because of any other issues that can crop up later. Besides
difficulties with reproducing and figuring out whether the patch had an effect,
I don't think these issues will get any priority. It sounds like last resort.
But I will try the second patch, no problem here.
java.lang.RuntimeException: Failed to rename XXX to YYY
---
Key: CASSANDRA-8535
URL: https://issues.apache.org/jira/browse/CASSANDRA-8535
Project: Cassandra
Issue Type: Bug
Environment: Windows 2008 X64
Reporter: Leonid Shalupov
Assignee: Joshua McKenzie
Fix For: 2.1.3
Attachments: 8535_v1.txt, 8535_v2.txt
{code}
java.lang.RuntimeException: Failed to rename
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db
to
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-ka-5-Index.db
at
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:170)
~[main/:na]
at
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:154)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:569)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:561)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.close(SSTableWriter.java:535)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:470)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finishAndMaybeThrow(SSTableRewriter.java:349)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:324)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:304)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:200)
~[main/:na]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
~[main/:na]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:226)
~[main/:na]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.nio.file.FileSystemException:
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db
-
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-ka-5-Index.db:
The process cannot access the file because it is being used by another
process.
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
~[na:1.7.0_45]
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
~[na:1.7.0_45]
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
~[na:1.7.0_45]
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
~[na:1.7.0_45]
at java.nio.file.Files.move(Files.java:1345) ~[na:1.7.0_45]
at
org.apache.cassandra.io.util.FileUtils.atomicMoveWithFallback(FileUtils.java:184)
~[main/:na]
at
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:166)
~[main/:na]
... 18 common frames omitted
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8800) Cassandra fails to start with OOM after disk space problem occurred
[ https://issues.apache.org/jira/browse/CASSANDRA-8800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320206#comment-14320206 ] Maxim Podkolzine commented on CASSANDRA-8800: - Cassandra 2.1.1. We got this once. The DB is no longer available to check it unfortunately. Cassandra fails to start with OOM after disk space problem occurred --- Key: CASSANDRA-8800 URL: https://issues.apache.org/jira/browse/CASSANDRA-8800 Project: Cassandra Issue Type: Bug Reporter: Maxim Podkolzine It one point the Cassandra failed with org.apache.cassandra.io.FSWriteError: java.io.IOException: No space left on device Which seems pretty normal situation. But after the disk space was increased, Cassandra refused to start up again. The memory snapshot shows a big ArrayList of org.apache.cassandra.io.sstable.SSTableScanner is allocated, holding around 2.8Gb. It looks like the DB is corrupted, but we're not entirely sure. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8535) java.lang.RuntimeException: Failed to rename XXX to YYY
[
https://issues.apache.org/jira/browse/CASSANDRA-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293985#comment-14293985
]
Maxim Podkolzine commented on CASSANDRA-8535:
-
Joshua, no, I haven't run a scrub. There were several more exceptions, like
that one. Let's discuss in a separate issue.
But I did not see {{FileSystemException}}.
I think the second exception is not relevant now, I have an idea, what might
have caused it. Seems it is not Cassandra fault.
java.lang.RuntimeException: Failed to rename XXX to YYY
---
Key: CASSANDRA-8535
URL: https://issues.apache.org/jira/browse/CASSANDRA-8535
Project: Cassandra
Issue Type: Bug
Environment: Windows 2008 X64
Reporter: Leonid Shalupov
Assignee: Joshua McKenzie
Fix For: 2.1.3
Attachments: 8535_v1.txt
{code}
java.lang.RuntimeException: Failed to rename
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db
to
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-ka-5-Index.db
at
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:170)
~[main/:na]
at
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:154)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:569)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:561)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.close(SSTableWriter.java:535)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:470)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finishAndMaybeThrow(SSTableRewriter.java:349)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:324)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:304)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:200)
~[main/:na]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
~[main/:na]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:226)
~[main/:na]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.nio.file.FileSystemException:
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db
-
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-ka-5-Index.db:
The process cannot access the file because it is being used by another
process.
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
~[na:1.7.0_45]
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
~[na:1.7.0_45]
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
~[na:1.7.0_45]
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
~[na:1.7.0_45]
at java.nio.file.Files.move(Files.java:1345) ~[na:1.7.0_45]
at
org.apache.cassandra.io.util.FileUtils.atomicMoveWithFallback(FileUtils.java:184)
~[main/:na]
at
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:166)
~[main/:na]
... 18 common frames omitted
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8535) java.lang.RuntimeException: Failed to rename XXX to YYY
[
https://issues.apache.org/jira/browse/CASSANDRA-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291760#comment-14291760
]
Maxim Podkolzine commented on CASSANDRA-8535:
-
Hi Joshua,
Yes, it looks better now. Before that I got the {{FileSystemException}} very
quickly, now I don't see it, at least after right from the start. I will keep
running to check it in long term.
There is a new one, that you might be interested in, since {{SSTableRewriter}}
was patched:
2015-01-15T16:23:47,760 [CompactionExecutor:6] ERROR
o.a.c.service.CassandraDaemon - Exception in thread
Thread[CompactionExecutor:6,1,main]
java.lang.AssertionError: null
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.build(IndexSummaryBuilder.java:133)
~[apache-cassandra-2.1.1.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.IndexSummaryBuilder.build(IndexSummaryBuilder.java:128)
~[apache-cassandra-2.1.1.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:476)
~[apache-cassandra-2.1.1.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:459)
~[apache-cassandra-2.1.1.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finishAndMaybeThrowWindows(SSTableRewriter.java:394)
~[apache-cassandra-2.1.1.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finishAndMaybeThrow(SSTableRewriter.java:353)
~[apache-cassandra-2.1.1.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:341)
~[apache-cassandra-2.1.1.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:321)
~[apache-cassandra-2.1.1.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:213)
~[apache-cassandra-2.1.1.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.1.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:76)
~[apache-cassandra-2.1.1.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[apache-cassandra-2.1.1.jar:2.1.2-SNAPSHOT]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:227)
~[apache-cassandra-2.1.1.jar:2.1.2-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_25]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_25]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_25]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_25]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_25]
Also I see {{DecoderException}}s, which I previously assumed to be caused by
other errors, like one in this issue. Now it looks like a separate issue:
2015-01-15T16:03:25,613 [SharedPool-Worker-3] ERROR
o.apache.cassandra.transport.Message - Unexpected exception during request;
channel = [id: 0x9a614ce6, /127.0.0.1:51881 = /127.0.0.1:10030]
io.netty.handler.codec.DecoderException:
org.apache.cassandra.transport.ProtocolException: Invalid or unsupported
protocol version: 71
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:280)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:149)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
~[netty-all-4.0.23.Final.jar:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
[jira] [Commented] (CASSANDRA-8535) java.lang.RuntimeException: Failed to rename XXX to YYY
[
https://issues.apache.org/jira/browse/CASSANDRA-8535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14278643#comment-14278643
]
Maxim Podkolzine commented on CASSANDRA-8535:
-
Joshua,
Just in case: I've managed to reproduce this exception in Windows 7 with the
latest cassandra shapshot build.
No A/V in the system, windows search does not index cassandra directory.
I can try the fix as soon as it is ready.
java.lang.RuntimeException: Failed to rename XXX to YYY
---
Key: CASSANDRA-8535
URL: https://issues.apache.org/jira/browse/CASSANDRA-8535
Project: Cassandra
Issue Type: Bug
Environment: Windows 2008 X64
Reporter: Leonid Shalupov
Assignee: Joshua McKenzie
Fix For: 2.1.3
{code}
java.lang.RuntimeException: Failed to rename
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db
to
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-ka-5-Index.db
at
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:170)
~[main/:na]
at
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:154)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:569)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.rename(SSTableWriter.java:561)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.close(SSTableWriter.java:535)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableWriter.finish(SSTableWriter.java:470)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finishAndMaybeThrow(SSTableRewriter.java:349)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:324)
~[main/:na]
at
org.apache.cassandra.io.sstable.SSTableRewriter.finish(SSTableRewriter.java:304)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:200)
~[main/:na]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
~[main/:na]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[main/:na]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:226)
~[main/:na]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: java.nio.file.FileSystemException:
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-tmp-ka-5-Index.db
-
build\test\cassandra\data;0\system\schema_keyspaces-b0f2235744583cdb9631c43e59ce3676\system-schema_keyspaces-ka-5-Index.db:
The process cannot access the file because it is being used by another
process.
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
~[na:1.7.0_45]
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
~[na:1.7.0_45]
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
~[na:1.7.0_45]
at
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
~[na:1.7.0_45]
at java.nio.file.Files.move(Files.java:1345) ~[na:1.7.0_45]
at
org.apache.cassandra.io.util.FileUtils.atomicMoveWithFallback(FileUtils.java:184)
~[main/:na]
at
org.apache.cassandra.io.util.FileUtils.renameWithConfirm(FileUtils.java:166)
~[main/:na]
... 18 common frames omitted
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
