[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations are not accounted
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13663802#comment-13663802 ] Radim Kolar commented on CASSANDRA-3741: i retested it. bug from 1.0 do not exists in 1.1 and 1.2. But its still not optimal and can lead to OOM because it do not adds enough bytes count for tombstone to live data count. If i remember some hardcoded constant was used, it needs to be raised. OOMs because delete operations are not accounted Key: CASSANDRA-3741 URL: https://issues.apache.org/jira/browse/CASSANDRA-3741 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.0 Environment: FreeBSD Reporter: Vitalii Tymchyshyn Assignee: Andriy Kolyadenko Fix For: 1.1.1 Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key. I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager. This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations are not accounted
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13461662#comment-13461662 ] Radim Kolar commented on CASSANDRA-3741: If you do not want to get fixed, then it should be documented as known bug in NEWS.TXT OOMs because delete operations are not accounted Key: CASSANDRA-3741 URL: https://issues.apache.org/jira/browse/CASSANDRA-3741 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.0 Environment: FreeBSD Reporter: Vitalii Tymchyshyn Assignee: Andriy Kolyadenko Fix For: 1.1.1 Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key. I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager. This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations are not accounted
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13458849#comment-13458849 ] Radim Kolar commented on CASSANDRA-3741: was this backported to cassandra 1.0? OOMs because delete operations are not accounted Key: CASSANDRA-3741 URL: https://issues.apache.org/jira/browse/CASSANDRA-3741 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.0 Environment: FreeBSD Reporter: Vitalii Tymchyshyn Assignee: Andriy Kolyadenko Fix For: 1.1.1 Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key. I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager. This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations are not accounted
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13458860#comment-13458860 ] Jonathan Ellis commented on CASSANDRA-3741: --- No, this is a sensitive area of the code and we don't want to risk destabilizing it. OOMs because delete operations are not accounted Key: CASSANDRA-3741 URL: https://issues.apache.org/jira/browse/CASSANDRA-3741 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.0 Environment: FreeBSD Reporter: Vitalii Tymchyshyn Assignee: Andriy Kolyadenko Fix For: 1.1.1 Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key. I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager. This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations are not accounted
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13458932#comment-13458932 ] Radim Kolar commented on CASSANDRA-3741: 1.0 is already destabilized by this bug. If you want to have minimal effect, then add just 1 byte to live bytes count for every delete. In non delete only workload, it will have minimal effect. Its important to have non zero count otherwise it will not be flushed to disk on memory pressure. OOMs because delete operations are not accounted Key: CASSANDRA-3741 URL: https://issues.apache.org/jira/browse/CASSANDRA-3741 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.0 Environment: FreeBSD Reporter: Vitalii Tymchyshyn Assignee: Andriy Kolyadenko Fix For: 1.1.1 Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key. I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager. This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations are not accounted
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267238#comment-13267238 ] Andriy Kolyadenko commented on CASSANDRA-3741: -- Another attempt to fix this: https://github.com/apache/cassandra/pull/10 Please consider my patch. It's very annoying to have Cassandra dying in such situations. OOMs because delete operations are not accounted Key: CASSANDRA-3741 URL: https://issues.apache.org/jira/browse/CASSANDRA-3741 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.0 Environment: FreeBSD Reporter: Vitalii Tymchyshyn Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key. I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager. This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations are not accounted
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267506#comment-13267506 ] Jonathan Ellis commented on CASSANDRA-3741: --- Thanks, Andriy. You (and Vitalii) are right; whole-row deletions did indeed have zero throughput because of that behavior. Committed with a change to 12 bytes (long + int from deletion info) to match what we do with Column sizes. (We measure the serialized size in Memtable.currentThroughput, then multiply by liveRatio to get a better estimate of the in-memory size. Mixing internal overhead as in CSLM would actually double count that. I've also introduced CASSANDRA-4215 to clean this up more for 1.2. OOMs because delete operations are not accounted Key: CASSANDRA-3741 URL: https://issues.apache.org/jira/browse/CASSANDRA-3741 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.0 Environment: FreeBSD Reporter: Vitalii Tymchyshyn Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key. I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager. This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations are not accounted
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267510#comment-13267510 ] Jonathan Ellis commented on CASSANDRA-3741: --- (That said, truncate is still the right solution for mass deletes.) OOMs because delete operations are not accounted Key: CASSANDRA-3741 URL: https://issues.apache.org/jira/browse/CASSANDRA-3741 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1.0 Environment: FreeBSD Reporter: Vitalii Tymchyshyn Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key. I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager. This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations are not accounted
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267781#comment-13267781 ] Jeremiah Jordan commented on CASSANDRA-3741: Procedural question, for git stuff should we still be attaching patches to JIRA? Or is a link to the github diff enough? OOMs because delete operations are not accounted Key: CASSANDRA-3741 URL: https://issues.apache.org/jira/browse/CASSANDRA-3741 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.0 Environment: FreeBSD Reporter: Vitalii Tymchyshyn Assignee: Andriy Kolyadenko Fix For: 1.1.1 Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key. I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager. This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations are not accounted
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13267783#comment-13267783 ] Jonathan Ellis commented on CASSANDRA-3741: --- Intent to contribute in public is enough. github pull requests (and patches sent to the mailing list for that matter) are fine. For the purposes of record-keeping though, we like to associate these with Jira tickets. OOMs because delete operations are not accounted Key: CASSANDRA-3741 URL: https://issues.apache.org/jira/browse/CASSANDRA-3741 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.0.0 Environment: FreeBSD Reporter: Vitalii Tymchyshyn Assignee: Andriy Kolyadenko Fix For: 1.1.1 Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key. I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager. This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations are not accounted
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13186448#comment-13186448 ] Vitalii Tymchyshyn commented on CASSANDRA-3741: --- Throughput size for deletion IS 0 for me. I am not deleting for some columns, but all columns in a key and operation has 0 columns in this case, this means 0 throughput so I did introduce an overhead for memory operation storage that looks good for me (20 bytes is ~ConcurrentMap size). To note: I was getting OOM on startup/log replay. The problem is that I have started delete-only workload from some Column Families and this won't change as all inserts go into another column families: we are moving. And for full-key delete-only throughput is 0 OOMs because delete operations are not accounted Key: CASSANDRA-3741 URL: https://issues.apache.org/jira/browse/CASSANDRA-3741 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1 Environment: FreeBSD Reporter: Vitalii Tymchyshyn Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key. I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager. This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations are not accounted
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13185742#comment-13185742 ] Jonathan Ellis commented on CASSANDRA-3741: --- Thanks, Vitalii! Unfortunately we can't use that patch as is because adding ops * 20 in there is going to throw off the size calculation for other workloads. Note that the throughput size for a deletion is NOT zero (see Column.size implementation). It sounds like you abruptly changed your workload from doing a bunch of larger inserts, then hit it with a ton of deletes all at once and OOMed before it was able to update its liveRatio estimate. So the real problem is that if you change workloads dramatically enough, Cassandra's estimates can be off. OOMs because delete operations are not accounted Key: CASSANDRA-3741 URL: https://issues.apache.org/jira/browse/CASSANDRA-3741 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1 Environment: FreeBSD Reporter: Vitalii Tymchyshyn Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key. I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager. This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3741) OOMs because delete operations are not accounted
[ https://issues.apache.org/jira/browse/CASSANDRA-3741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13185836#comment-13185836 ] Radim Kolar commented on CASSANDRA-3741: How can you OOM if you replace large inserts with small deletes? OOMs because delete operations are not accounted Key: CASSANDRA-3741 URL: https://issues.apache.org/jira/browse/CASSANDRA-3741 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.1 Environment: FreeBSD Reporter: Vitalii Tymchyshyn Currently we are moving to new data format where new format is written into new CFs and old one is deleted key-by-key. I have started getting OOMs and found out that delete operations are not accounted and so, column families are not flushed (changed == 0 with delete only operations) by storage manager. This is pull request that fixed this problem for me: https://github.com/apache/cassandra/pull/5 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
