[jira] [Created] (CASSANDRA-9339) Commit Log completed tasks incremented during getCompletedTasks
graham sanderson created CASSANDRA-9339: --- Summary: Commit Log completed tasks incremented during getCompletedTasks Key: CASSANDRA-9339 URL: https://issues.apache.org/jira/browse/CASSANDRA-9339 Project: Cassandra Issue Type: Bug Reporter: graham sanderson Fix For: 2.1.6 This is the same problem as CASSANDRA-8662 just less noticeable since completed tasks is generally quite large -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9339) Commit Log completed tasks incremented during getCompletedTasks
[ https://issues.apache.org/jira/browse/CASSANDRA-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] graham sanderson updated CASSANDRA-9339: Priority: Minor (was: Major) Commit Log completed tasks incremented during getCompletedTasks --- Key: CASSANDRA-9339 URL: https://issues.apache.org/jira/browse/CASSANDRA-9339 Project: Cassandra Issue Type: Bug Reporter: graham sanderson Priority: Minor Fix For: 2.1.6 This is the same problem as CASSANDRA-8662 just less noticeable since completed tasks is generally quite large -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9339) Commit Log completed tasks incremented during getCompletedTasks
[ https://issues.apache.org/jira/browse/CASSANDRA-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537287#comment-14537287 ] graham sanderson commented on CASSANDRA-9339: - Note I commented this on the original issue, but that didn't get much traction so opened a new one Commit Log completed tasks incremented during getCompletedTasks --- Key: CASSANDRA-9339 URL: https://issues.apache.org/jira/browse/CASSANDRA-9339 Project: Cassandra Issue Type: Bug Reporter: graham sanderson Priority: Minor Fix For: 2.1.6 This is the same problem as CASSANDRA-8662 just less noticeable since completed tasks is generally quite large -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9339) Commit Log completed tasks incremented during getCompletedTasks
[ https://issues.apache.org/jira/browse/CASSANDRA-9339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] graham sanderson updated CASSANDRA-9339: Attachment: cassandra-2.1-9339.txt Commit Log completed tasks incremented during getCompletedTasks --- Key: CASSANDRA-9339 URL: https://issues.apache.org/jira/browse/CASSANDRA-9339 Project: Cassandra Issue Type: Bug Reporter: graham sanderson Priority: Minor Fix For: 2.1.6 Attachments: cassandra-2.1-9339.txt This is the same problem as CASSANDRA-8662 just less noticeable since completed tasks is generally quite large -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522753#comment-14522753
]
graham sanderson commented on CASSANDRA-8325:
-
My suggestion would be to see if you can perf test this yourself on Linux.
Benedict is right that it might pollute the instruction cache, but then it
might not (it really depends on the mix of calls).. I think the point is that
someone who needs FreeBSD is probably going to have to do the leg work here; or
just run with my patch which may be sub-optimal.
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Fix For: 2.1.x
Attachments: hs_err_pid1856.log, system.log, unsafeCopy1.txt,
untested_8325.patch
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8061) tmplink files are not removed
[
https://issues.apache.org/jira/browse/CASSANDRA-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349402#comment-14349402
]
graham sanderson commented on CASSANDRA-8061:
-
So w.r.t. to
{code}
ERROR [CompactionExecutor:4967] 2015-01-29 18:54:10,709
CassandraDaemon.java:170 - Exception in thread
Thread[CompactionExecutor:4967,1,main]
java.lang.AssertionError: null
at org.apache.cassandra.io.util.Memory.size(Memory.java:307) ~[main/:na]
at org.apache.cassandra.utils.obs.OffHeapBitSet.capacity(OffHeapBitSet.java:61)
~[main/:na]
at org.apache.cassandra.utils.BloomFilter.indexes(BloomFilter.java:74)
~[main/:na]
{code}
We are also seeing this on 2.1.3 (nothing to do with windows) - it sounds like
this is an end result of a race condition... as the code never explicitly
appears to allocate empty bit sets.
[~benedict] is your sense that at this point data is already corrupted, or is
this assertion actually protecting data at the cost of compaction (because
there is another patch which disables assertions about 0 size off heap objects,
which would suppress this)
tmplink files are not removed
-
Key: CASSANDRA-8061
URL: https://issues.apache.org/jira/browse/CASSANDRA-8061
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Linux
Reporter: Gianluca Borello
Assignee: Joshua McKenzie
Fix For: 2.1.4
Attachments: 8061_v1.txt, 8248-thread_dump.txt
After installing 2.1.0, I'm experiencing a bunch of tmplink files that are
filling my disk. I found https://issues.apache.org/jira/browse/CASSANDRA-7803
and that is very similar, and I confirm it happens both on 2.1.0 as well as
from the latest commit on the cassandra-2.1 branch
(https://github.com/apache/cassandra/commit/aca80da38c3d86a40cc63d9a122f7d45258e4685
from the cassandra-2.1)
Even starting with a clean keyspace, after a few hours I get:
{noformat}
$ sudo find /raid0 | grep tmplink | xargs du -hs
2.7G
/raid0/cassandra/data/draios/protobuf1-ccc6dce04beb11e4abf997b38fbf920b/draios-protobuf1-tmplink-ka-4515-Data.db
13M
/raid0/cassandra/data/draios/protobuf1-ccc6dce04beb11e4abf997b38fbf920b/draios-protobuf1-tmplink-ka-4515-Index.db
1.8G
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-1788-Data.db
12M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-1788-Index.db
5.2M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-2678-Index.db
822M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-2678-Data.db
7.3M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-3283-Index.db
1.2G
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-3283-Data.db
6.7M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-3951-Index.db
1.1G
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-3951-Data.db
11M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-4799-Index.db
1.7G
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-4799-Data.db
812K
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-234-Index.db
122M
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-208-Data.db
744K
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-739-Index.db
660K
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-193-Index.db
796K
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-230-Index.db
137M
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-230-Data.db
161M
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-269-Data.db
139M
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-234-Data.db
940K
[jira] [Commented] (CASSANDRA-8061) tmplink files are not removed
[
https://issues.apache.org/jira/browse/CASSANDRA-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349879#comment-14349879
]
graham sanderson commented on CASSANDRA-8061:
-
Thanks that is good news esp. w.r.t. 2.1.4 as there are a few patches there
we'd like but are pretty much impossible to rebase onto 2.1.3
Note just FYI we are getting the assertion above on
system.compactions_in_progress, and whilst I haven't studied its use, I imagine
the inputs must thus be large, and that might be a good indicator that we are
indeed also seeing CASSANDRA-8860 occasionally (something we suspect and are
going to patch anyway since it restores 2.0.x behavior)
tmplink files are not removed
-
Key: CASSANDRA-8061
URL: https://issues.apache.org/jira/browse/CASSANDRA-8061
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Linux
Reporter: Gianluca Borello
Assignee: Joshua McKenzie
Fix For: 2.1.4
Attachments: 8061_v1.txt, 8248-thread_dump.txt
After installing 2.1.0, I'm experiencing a bunch of tmplink files that are
filling my disk. I found https://issues.apache.org/jira/browse/CASSANDRA-7803
and that is very similar, and I confirm it happens both on 2.1.0 as well as
from the latest commit on the cassandra-2.1 branch
(https://github.com/apache/cassandra/commit/aca80da38c3d86a40cc63d9a122f7d45258e4685
from the cassandra-2.1)
Even starting with a clean keyspace, after a few hours I get:
{noformat}
$ sudo find /raid0 | grep tmplink | xargs du -hs
2.7G
/raid0/cassandra/data/draios/protobuf1-ccc6dce04beb11e4abf997b38fbf920b/draios-protobuf1-tmplink-ka-4515-Data.db
13M
/raid0/cassandra/data/draios/protobuf1-ccc6dce04beb11e4abf997b38fbf920b/draios-protobuf1-tmplink-ka-4515-Index.db
1.8G
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-1788-Data.db
12M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-1788-Index.db
5.2M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-2678-Index.db
822M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-2678-Data.db
7.3M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-3283-Index.db
1.2G
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-3283-Data.db
6.7M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-3951-Index.db
1.1G
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-3951-Data.db
11M
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-4799-Index.db
1.7G
/raid0/cassandra/data/draios/protobuf_by_agent1-cd071a304beb11e4abf997b38fbf920b/draios-protobuf_by_agent1-tmplink-ka-4799-Data.db
812K
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-234-Index.db
122M
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-208-Data.db
744K
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-739-Index.db
660K
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-193-Index.db
796K
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-230-Index.db
137M
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-230-Data.db
161M
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-269-Data.db
139M
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-234-Data.db
940K
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-786-Index.db
936K
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-269-Index.db
161M
/raid0/cassandra/data/draios/mounted_fs_by_agent1-d7bf3e304beb11e4abf997b38fbf920b/draios-mounted_fs_by_agent1-tmplink-ka-786-Data.db
672K
[jira] [Comment Edited] (CASSANDRA-8860) Too many java.util.HashMap$Entry objects in heap
[ https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348034#comment-14348034 ] graham sanderson edited comment on CASSANDRA-8860 at 3/5/15 2:53 AM: - Note we are going to try a simpler patch on 2.1.3 (because we believe we are seeing this too, though our heaps are simply too big to dump reasonably it would seem). We're just reverting the cold_reads_to_omit default to 0.0, and adding logging. We could probably change the schema for all tables to do this explicitly, but I haven't looked at whether this affects LCS L0 This is just an FYI to see if it helps us - we are seeing memory issue that we did not see in testing with 2.1.3, but may well be related to a proliferation of sstables was (Author: graham sanderson): Note we are going to try a simpler patch on 2.1.3 (because we believe we are seeing this too, though our heaps are simply too big to dump reasonably it would seem). We're just reverting the cold_reads_to_omit default to 0.0, and adding logging. We could probably change the schema for all tables to do this explicitly, but I haven't looked at whether this affects LCS L0 Too many java.util.HashMap$Entry objects in heap Key: CASSANDRA-8860 URL: https://issues.apache.org/jira/browse/CASSANDRA-8860 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.1.3, jdk 1.7u51 Reporter: Phil Yang Assignee: Marcus Eriksson Fix For: 2.1.4 Attachments: 0001-remove-cold_reads_to_omit.patch, 8860-v2.txt, 8860.txt, cassandra-env.sh, cassandra.yaml, jmap.txt, jstack.txt, jstat-afterv1.txt, jstat-afterv2.txt, jstat-before.txt While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have GC issue after the node restarting successfully. Old gen grows very fast and most of the space can not be recycled after setting its status to normal immediately. The qps of both reading and writing are very low and there is no heavy compaction. Jmap result seems strange that there are too many java.util.HashMap$Entry objects in heap, where in my experience the [B is usually the No1. If I downgrade it to 2.1.1, this issue will not appear. I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if someone need it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8860) Too many java.util.HashMap$Entry objects in heap
[ https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348034#comment-14348034 ] graham sanderson commented on CASSANDRA-8860: - Note we are going to try a simpler patch on 2.1.3 (because we believe we are seeing this too, though our heaps are simply too big to dump reasonably it would seem). We're just reverting the cold_reads_to_omit default to 0.0, and adding logging. We could probably change the schema for all tables to do this explicitly, but I haven't looked at whether this affects LCS L0 Too many java.util.HashMap$Entry objects in heap Key: CASSANDRA-8860 URL: https://issues.apache.org/jira/browse/CASSANDRA-8860 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.1.3, jdk 1.7u51 Reporter: Phil Yang Assignee: Marcus Eriksson Fix For: 2.1.4 Attachments: 0001-remove-cold_reads_to_omit.patch, 8860-v2.txt, 8860.txt, cassandra-env.sh, cassandra.yaml, jmap.txt, jstack.txt, jstat-afterv1.txt, jstat-afterv2.txt, jstat-before.txt While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have GC issue after the node restarting successfully. Old gen grows very fast and most of the space can not be recycled after setting its status to normal immediately. The qps of both reading and writing are very low and there is no heavy compaction. Jmap result seems strange that there are too many java.util.HashMap$Entry objects in heap, where in my experience the [B is usually the No1. If I downgrade it to 2.1.1, this issue will not appear. I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if someone need it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8862) Commit log pending tasks incremented during getPendingTasks
[ https://issues.apache.org/jira/browse/CASSANDRA-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348055#comment-14348055 ] graham sanderson commented on CASSANDRA-8862: - Note getCompletedTasks() has the same problem, just harder to see in the noise Commit log pending tasks incremented during getPendingTasks --- Key: CASSANDRA-8862 URL: https://issues.apache.org/jira/browse/CASSANDRA-8862 Project: Cassandra Issue Type: Bug Reporter: Michael Swiecicki Assignee: Jonathan Ellis Priority: Minor Fix For: 2.1.4 After upgrading from Cassandra 2.0.8 to 2.1.2 our monitoring based on JMX metrics shows increasing number of commit log pending tasks. From what we found out, this value is now incremented every time it is fetched. It is easy to reproduce using tool like JConsole by refreshing value of PendingTasks attribute. Quick look at the code shows that this behavior was probably introduced in 2.1 with CASSANDRA-3578 where getPendingTasks in AbstractCommitLogService was changed from returning queue.size() to returning pending.incrementAndGet(). As a result this counter is incremented when getting value of Commitlog.PendingTasks from org.apache.cassandra.db (now deprecated) or CommitLog.PendingTasks from org.apache.cassandra.metrics. Could you please clarify if this is expected behavior? This was already mentioned in CASSANDRA-8531. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8795) Cassandra (possibly under load) occasionally throws an exception during CQL create table
[
https://issues.apache.org/jira/browse/CASSANDRA-8795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348179#comment-14348179
]
graham sanderson commented on CASSANDRA-8795:
-
Note just as an aside - I see frenetic MigrationStage activity on 2.1.3 - like
dozens of tasks per ms sometimes. Not sure under what conditions, and sorry for
being vague, but it seems unlikely to be correct. I think there may be a busy
retry loop in the case of failure
Cassandra (possibly under load) occasionally throws an exception during CQL
create table
Key: CASSANDRA-8795
URL: https://issues.apache.org/jira/browse/CASSANDRA-8795
Project: Cassandra
Issue Type: Bug
Reporter: Darren Warner
Fix For: 2.1.4
CQLSH will return the following:
{code}
{ name: 'ResponseError',
message: 'java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.NullPointerException',
info: 'Represents an error message from the server',
code: 0,
query: 'CREATE TABLE IF NOT EXISTS roles_by_users( userid TIMEUUID, role
INT, entityid TIMEUUID, entity_type TEXT, enabled BOOLEAN, PRIMARY KEY
(userid, role, entityid, entity_type) );' }
{code}
Cassandra system.log shows:
{code}
ERROR [MigrationStage:1] 2015-02-11 14:38:48,610 CassandraDaemon.java:153 -
Exception in thread Thread[MigrationStage:1,5,main]
java.lang.NullPointerException: null
at
org.apache.cassandra.db.DefsTables.addColumnFamily(DefsTables.java:371)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:293)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.java:194)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:166)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.service.MigrationManager$2.runMayThrow(MigrationManager.java:393)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_31]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_31]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_31]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_31]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31]
ERROR [SharedPool-Worker-2] 2015-02-11 14:38:48,620 QueryMessage.java:132 -
Unexpected error during query
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.NullPointerException
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:398)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:374)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.service.MigrationManager.announceNewColumnFamily(MigrationManager.java:249)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.cql3.statements.CreateTableStatement.announceMigration(CreateTableStatement.java:113)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute(SchemaAlteringStatement.java:80)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:226)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:248)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
~[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439)
[apache-cassandra-2.1.2.jar:2.1.2]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335)
[apache-cassandra-2.1.2.jar:2.1.2]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
[netty-all-4.0.23.Final.jar:4.0.23.Final]
at
[jira] [Commented] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265799#comment-14265799
]
graham sanderson commented on CASSANDRA-8325:
-
Well that is good news I guess; I think we need someone from DataStax to make a
call on whether this should be refactored, and what perf tests need to be
redone if this is going to be fixed on all platforms.
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log, unsafeCopy1.txt,
untested_8325.patch
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262669#comment-14262669
]
graham sanderson edited comment on CASSANDRA-8325 at 1/1/15 9:53 PM:
-
Since I'm feeling new-yearly I have made untested_8325.patch which seems (again
untested on anything least of all FreeBSD) like it should address the problems
Benedict you might want to take a quick look - this isn't going to win any
awards for prettiness or encapsulation, but it isn't immediately obvious that
it should make *anything* slower.
was (Author: graham sanderson):
Since I'm feeling new-yearly I have made untested_8325.patch which seems (again
untested on anything least of all FreeBSD) like it should address the problems
Benedict you might want to take a quick look - this isn't going to win any
awards for prettyness or encapsulation, but it isn't immediately obvious that
it should make *anything* slower.
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log, unsafeCopy1.txt,
untested_8325.patch
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
graham sanderson updated CASSANDRA-8325:
Attachment: untested_8325.patch
Since I'm feeling new-yearly I have made untested_8325.patch which seems (again
untested on anything least of all FreeBSD) like it should address the problems
Benedict you might want to take a quick look - this isn't going to win any
awards for prettyness or encapsulation, but it isn't immediately obvious that
it should make *anything* slower.
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log, unsafeCopy1.txt,
untested_8325.patch
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14262670#comment-14262670
]
graham sanderson commented on CASSANDRA-8325:
-
Note that patch supersedes the old one (unsafeCopy1.txt)
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log, unsafeCopy1.txt,
untested_8325.patch
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14261151#comment-14261151
]
graham sanderson commented on CASSANDRA-8325:
-
My first patch was kind of ugly (shortest path to probable correctness); whilst
I could do the same thing for the comparators, at that point the code will
cease to be very elegant, and probably in need of some refactoring to make it
more approachable.
[~benedict] do you want to take this issue. I was really only taking a look out
of curiosity (hadn't seen it in our playings with 2.1.x), but it just seemed
odd that there should be an issue with Sun.misc.Unsafe on FreeBSD. We don't use
FreeBSD or plan to, but as mentioned above, this is actually incorrect code
although it happens to work on most other things.
It seems like you took great pains to get the {{@Inline}} mix right, and you
have much more context than me with what you were thinking (again as per
{{MIN_COPY_THRESHOLD}} and its only partial use - which may not have been
intended).
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log, unsafeCopy1.txt
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227227#comment-14227227
]
graham sanderson edited comment on CASSANDRA-8325 at 12/29/14 4:28 PM:
---
unsafeCopy1.txt has a minimal code change that *might* based on the
documentation for Unsafe.java fix this one code path (others would remain in
the comparators).
basically this is untested (even not on FreeBSD) so YMMV. Feel free to try it
and see if it fixes the exception in the copy() method
was (Author: graham sanderson):
unsafeCopy1.txt has a minimal code change that *might* based on the
documentation for Unsafe.java fix this one code path (others would remain in
the compators).
basically this is untested (even not on FreeBSD) so YMMV. Feel free to try it
and see if it fixes the exception in the copy() method
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log, unsafeCopy1.txt
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227049#comment-14227049
]
graham sanderson commented on CASSANDRA-8325:
-
Yeah the code in library_call.cpp is quite hard to introspect, however my sense
is this probably comes down to a mixture of the case where a null (0) 32 bit
compressedOop pointer is not equivalent to a null 64 bit address (i.e. the case
where the heap does not fit in the 32 gig of address space) possibly coupled
with whether the hotspot compiler could prove at compile time whether the value
was statically null.
It seems like a slightly strange distinction to make given that you are using a
64 bit offset anyway (and that other methods do using them interchangeably) but
that doesn't change the fact that they made the distinction and apparently it
matters at least in some implementations.
The question I was trying to get a sense of, is whether this explicitly
disallows the sorts of pointer arithmetic C* is trying to achieve, or whether
C* is just doing it wrong. Maybe that is actually quite simple and there are
clearly distinctions between baseless pointers and static or otherwise field
level offsets, but someone would have to have to dive in (probably with the
ability to test) to make sure.
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227049#comment-14227049
]
graham sanderson edited comment on CASSANDRA-8325 at 11/27/14 12:30 AM:
Yeah the code in library_call.cpp is quite hard to introspect, however my sense
is this probably comes down to a mixture of the case where a null (0) 32 bit
compressedOop pointer is not equivalent to a null 64 bit address (i.e. the case
where the heap does not fit in the 32 gig of address space) possibly coupled
with whether the hotspot compiler could prove at compile time whether the value
was statically null.
It seems like a slightly strange distinction to make given that you are using a
64 bit offset anyway (and that other methods do using them interchangeably) but
that doesn't change the fact that they made the distinction and apparently it
matters at least in some implementations.
The question I was trying to get a sense of, is whether this explicitly
disallows the sorts of pointer arithmetic C* is trying to achieve, or whether
C* is just doing it wrong. Maybe that is actually quite simple and there are
clearly distinctions in the C* usage between baseless pointers and static or
otherwise field level offsets, but someone would have to have to dive in
(probably with the ability to test) to make sure.
was (Author: graham sanderson):
Yeah the code in library_call.cpp is quite hard to introspect, however my sense
is this probably comes down to a mixture of the case where a null (0) 32 bit
compressedOop pointer is not equivalent to a null 64 bit address (i.e. the case
where the heap does not fit in the 32 gig of address space) possibly coupled
with whether the hotspot compiler could prove at compile time whether the value
was statically null.
It seems like a slightly strange distinction to make given that you are using a
64 bit offset anyway (and that other methods do using them interchangeably) but
that doesn't change the fact that they made the distinction and apparently it
matters at least in some implementations.
The question I was trying to get a sense of, is whether this explicitly
disallows the sorts of pointer arithmetic C* is trying to achieve, or whether
C* is just doing it wrong. Maybe that is actually quite simple and there are
clearly distinctions between baseless pointers and static or otherwise field
level offsets, but someone would have to have to dive in (probably with the
ability to test) to make sure.
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
graham sanderson updated CASSANDRA-8325:
Attachment: unsafeCopy1.txt
unsafeCopy1.txt has a minimal code change that *might* based on the
documentation for Unsafe.java fix this one code path (others would remain in
the compators).
basically this is untested (even not on FreeBSD) so YMMV
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log, unsafeCopy1.txt
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227227#comment-14227227
]
graham sanderson edited comment on CASSANDRA-8325 at 11/27/14 3:52 AM:
---
unsafeCopy1.txt has a minimal code change that *might* based on the
documentation for Unsafe.java fix this one code path (others would remain in
the compators).
basically this is untested (even not on FreeBSD) so YMMV. Feel free to try it
and see if it fixes the exception in the copy() method
was (Author: graham sanderson):
unsafeCopy1.txt has a minimal code change that *might* based on the
documentation for Unsafe.java fix this one code path (others would remain in
the compators).
basically this is untested (even not on FreeBSD) so YMMV
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log, unsafeCopy1.txt
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227257#comment-14227257
]
graham sanderson commented on CASSANDRA-8325:
-
Note also Benedict that I noticed in passing that MIN_COPY_THRESHOLD is only
checked in some code paths - was this deliberate?
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log, unsafeCopy1.txt
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14224734#comment-14224734
]
graham sanderson commented on CASSANDRA-8325:
-
Well that was a good guess ;-)
So {{unsafe.getByte(null, address)}} and {{unsafe.getByte(address)}} are not
documented to be the same, but do seem to be on most platforms
However from getInt() docs:
{code}
* This method refers to a variable by means of two parameters, and so
* it provides (in effect) a emdouble-register/em addressing mode
* for Java variables. When the object reference is null, this method
* uses its offset as an absolute address. This is similar in operation
* to methods such as {@link #getInt(long)}, which provide (in effect) a
* emsingle-register/em addressing mode for non-Java variables.
* However, because Java variables may have a different layout in memory
* from non-Java variables, programmers should not assume that these
* two addressing modes are ever equivalent. Also, programmers should
* remember that offsets from the double-register addressing mode cannot
* be portably confused with longs used in the single-register addressing
* mode.
{code}
The why? Well you could look on FreeBSD OpenJDK mailing lists, or dig up the
source - changes to {{hotspot/src/share/vm/opto/library_call.cpp}} may help.
However this may not be a JVM bug, it might just be a consequence of address
space layout on FreeBSD. Things like -XX:+UseCompressedOps and heap size = or
= 32gig may have an effect.
Anyway, you might consider this a C* bug... sort of fuzzy there. If you're
building C* yourself, I'd start by changing (though it isn't necessarily the
only effected method)
In FastByteOperations.UnsafeOperations, I'd change
{code}
public static void copy(Object src, long srcOffset, byte[] trg, int
trgPosition, int length)
{
if (length = MIN_COPY_THRESHOLD)
{
for (int i = 0 ; i length ; i++)
trg[trgPosition + i] = theUnsafe.getByte(src, srcOffset +
i);
}
else
{
copy(src, srcOffset, trg, BYTE_ARRAY_BASE_OFFSET + trgPosition,
length);
}
}
{code}
to
{code}
public static void copy(Object src, long srcOffset, byte[] trg, int
trgPosition, int length)
{
if (length = MIN_COPY_THRESHOLD)
{
if (src == null)
{
for (int i = 0 ; i length ; i++)
trg[trgPosition + i] = theUnsafe.getByte(srcOffset + i);
}
else
{
for (int i = 0 ; i length ; i++)
trg[trgPosition + i] = theUnsafe.getByte(src, srcOffset
+ i);
}
}
else
{
copy(src, srcOffset, trg, BYTE_ARRAY_BASE_OFFSET + trgPosition,
length);
}
}
{code}
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225120#comment-14225120
]
graham sanderson commented on CASSANDRA-8325:
-
True - it would seem that probably most of {{FastByteOperations}} may be making
invalid assumptions, and indeed appears to be new in 2.1 (I thought this might
be avoided by memtable_allocation_type: heap_buffers, but it isn't)
So yes... I guess this could be classed as a bug, maybe ~benedict has some ideas
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225120#comment-14225120
]
graham sanderson edited comment on CASSANDRA-8325 at 11/25/14 7:57 PM:
---
True - it would seem that probably most of {{FastByteOperations}} may be making
invalid assumptions, and indeed appears to be new in 2.1 (I thought this might
be avoided by memtable_allocation_type: heap_buffers, but it isn't)
So yes... I guess this should be classed as a bug, maybe [~benedict] has some
ideas
was (Author: graham sanderson):
True - it would seem that probably most of {{FastByteOperations}} may be making
invalid assumptions, and indeed appears to be new in 2.1 (I thought this might
be avoided by memtable_allocation_type: heap_buffers, but it isn't)
So yes... I guess this could be classed as a bug, maybe [~benedict] has some
ideas
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225120#comment-14225120
]
graham sanderson edited comment on CASSANDRA-8325 at 11/25/14 7:57 PM:
---
True - it would seem that probably most of {{FastByteOperations}} may be making
invalid assumptions, and indeed appears to be new in 2.1 (I thought this might
be avoided by memtable_allocation_type: heap_buffers, but it isn't)
So yes... I guess this should be classed as a bug, maybe [~benedict] has some
ideas regarding the origin of this code
was (Author: graham sanderson):
True - it would seem that probably most of {{FastByteOperations}} may be making
invalid assumptions, and indeed appears to be new in 2.1 (I thought this might
be avoided by memtable_allocation_type: heap_buffers, but it isn't)
So yes... I guess this should be classed as a bug, maybe [~benedict] has some
ideas
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225120#comment-14225120
]
graham sanderson edited comment on CASSANDRA-8325 at 11/25/14 7:57 PM:
---
True - it would seem that probably most of {{FastByteOperations}} may be making
invalid assumptions, and indeed appears to be new in 2.1 (I thought this might
be avoided by memtable_allocation_type: heap_buffers, but it isn't)
So yes... I guess this could be classed as a bug, maybe [~benedict] has some
ideas
was (Author: graham sanderson):
True - it would seem that probably most of {{FastByteOperations}} may be making
invalid assumptions, and indeed appears to be new in 2.1 (I thought this might
be avoided by memtable_allocation_type: heap_buffers, but it isn't)
So yes... I guess this could be classed as a bug, maybe ~benedict has some ideas
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225137#comment-14225137
]
graham sanderson commented on CASSANDRA-8325:
-
Posting full javadoc (which is on getInt but applies below).
Note this is reasonable ammunition to get the bug fixed, however reproducing it
on something other than FreeBSD would probably help move things along, which
begs the question what is it about FreeBSD OpenJDK that is the issue.
Things that pop into my head are
- Heap either less than or greater than 4 gig in size
- within this also if heap is fully located within fist 4 gig of virtual
address space
- Heap either less than or greater than 32 gig in size
- within this also if heap is fully located within fist 32 gig of virtual
address space
- And whether you are using compressed oops (-XX:+/-UsedCompressedOops)
{code}
/**
* Fetches a value from a given Java variable.
* More specifically, fetches a field or array element within the given
* object codeo/code at the given offset, or (if codeo/code is
* null) from the memory address whose numerical value is the given
* offset.
* p
* The results are undefined unless one of the following cases is true:
* ul
* liThe offset was obtained from {@link #objectFieldOffset} on
* the {@link java.lang.reflect.Field} of some Java field and the object
* referred to by codeo/code is of a class compatible with that
* field's class.
*
* liThe offset and object reference codeo/code (either null or
* non-null) were both obtained via {@link #staticFieldOffset}
* and {@link #staticFieldBase} (respectively) from the
* reflective {@link Field} representation of some Java field.
*
* liThe object referred to by codeo/code is an array, and the offset
* is an integer of the form codeB+N*S/code, where codeN/code is
* a valid index into the array, and codeB/code and codeS/code are
* the values obtained by {@link #arrayBaseOffset} and {@link
* #arrayIndexScale} (respectively) from the array's class. The value
* referred to is the codeN/codeemth/em element of the array.
*
* /ul
* p
* If one of the above cases is true, the call references a specific Java
* variable (field or array element). However, the results are undefined
* if that variable is not in fact of the type returned by this method.
* p
* This method refers to a variable by means of two parameters, and so
* it provides (in effect) a emdouble-register/em addressing mode
* for Java variables. When the object reference is null, this method
* uses its offset as an absolute address. This is similar in operation
* to methods such as {@link #getInt(long)}, which provide (in effect) a
* emsingle-register/em addressing mode for non-Java variables.
* However, because Java variables may have a different layout in memory
* from non-Java variables, programmers should not assume that these
* two addressing modes are ever equivalent. Also, programmers should
* remember that offsets from the double-register addressing mode cannot
* be portably confused with longs used in the single-register addressing
* mode.
*
* @param o Java heap object in which the variable resides, if any, else
*null
* @param offset indication of where the variable resides in a Java heap
*object, if any, else a memory address locating the variable
*statically
* @return the value fetched from the indicated Java variable
* @throws RuntimeException No defined exceptions are thrown, not even
* {@link NullPointerException}
*/
{code}
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225147#comment-14225147
]
graham sanderson commented on CASSANDRA-8325:
-
The only other thing to note is that other methods (e.g. copyMemory) in Unsafe
use (null, longValue) and (longValue) interchangeably which adds to the
confusion
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225151#comment-14225151
]
graham sanderson commented on CASSANDRA-8325:
-
And finally, the C* code base seems to use the two interchangeably in some
cases which is might make it non trivial to disentangle
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225151#comment-14225151
]
graham sanderson edited comment on CASSANDRA-8325 at 11/25/14 8:27 PM:
---
And finally, the C* code base seems to use the two interchangeably in some
cases which is might make it non trivial to disentangle (thus my suggested fix
above might just push the problem around in so far as the caller may have
incorrectly converted (obj, offset) to (null, address_of_obj + offset))
was (Author: graham sanderson):
And finally, the C* code base seems to use the two interchangeably in some
cases which is might make it non trivial to disentangle
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14219824#comment-14219824
]
graham sanderson commented on CASSANDRA-8325:
-
Just for sanity checking's sake - otherwise my previous comments about {{peer}}
still stand, can you check:
{code}
System.err.println(byteA + theUnsafe.getByte(null, l)); // that is your long
l
System.err.println(byteB + theUnsafe.getByte(l)); // that is your long l
{code}
To make sure neither fail.
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14219824#comment-14219824
]
graham sanderson edited comment on CASSANDRA-8325 at 11/20/14 7:23 PM:
---
Just for sanity checking's sake - otherwise my previous comments about {{peer}}
still stand, can you check:
{code}
System.err.println(byteA + unsafe.getByte(null, l)); // that is your long l
System.err.println(byteB + unsafe.getByte(l)); // that is your long l
{code}
To make sure neither fail.
was (Author: graham sanderson):
Just for sanity checking's sake - otherwise my previous comments about {{peer}}
still stand, can you check:
{code}
System.err.println(byteA + theUnsafe.getByte(null, l)); // that is your long
l
System.err.println(byteB + theUnsafe.getByte(l)); // that is your long l
{code}
To make sure neither fail.
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14219829#comment-14219829
]
graham sanderson commented on CASSANDRA-8325:
-
You might also want to try putting them (without the println in a long tight
loop to get them out of the interpreter also)
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14219829#comment-14219829
]
graham sanderson edited comment on CASSANDRA-8325 at 11/20/14 10:16 PM:
You might also want to try putting them (without the println) in a long tight
loop to get them out of the interpreter
was (Author: graham sanderson):
You might also want to try putting them (without the println in a long tight
loop to get them out of the interpreter also)
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14218918#comment-14218918
]
graham sanderson commented on CASSANDRA-8325:
-
I don't have FreeBSD, but posting my comment from user thread
{bq}
Only thing I can see from looking at the exception, is that it looks like maybe
the “peer” value in the RefCountedMemory object is probably 0
Given that Unsafe.allocateMemory should not return 0 even on allocation failure
(it should throw OOM, though you should add a log statement to the Memory class
to check that) - I’d suggest logging to see if anyone is calling
SSTableReader.releaseSummary, which could set the peer to 0
{bq}
Note this particular use of sun.misc.Unsafe is new in 2.1, however I thought
there were others in 2.0. It is possible your JVM doesn't support {{public
native long Unsafe.allocateMemory(long l);}} and returns 0, though in that case
you pretty much just need to switch back to the on heap memtable allocator.
In either case, if you cannot recompile Cassandra for new logging, you can
certainly write a simple Jave program which calls {{public native long
allocateMemory(long l);}}
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14218918#comment-14218918
]
graham sanderson edited comment on CASSANDRA-8325 at 11/20/14 2:55 AM:
---
I don't have FreeBSD, but posting my comment from user thread
{quote}
Only thing I can see from looking at the exception, is that it looks like maybe
the “peer” value in the RefCountedMemory object is probably 0
Given that Unsafe.allocateMemory should not return 0 even on allocation failure
(it should throw OOM, though you should add a log statement to the Memory class
to check that) - I’d suggest logging to see if anyone is calling
SSTableReader.releaseSummary, which could set the peer to 0
{quote}
Note this particular use of sun.misc.Unsafe is new in 2.1, however I thought
there were others in 2.0. It is possible your JVM doesn't support {{public
native long Unsafe.allocateMemory(long l);}} and returns 0, though in that case
you pretty much just need to switch back to the on heap memtable allocator.
In either case, if you cannot recompile Cassandra for new logging, you can
certainly write a simple Jave program which calls {{public native long
allocateMemory(long l);}}
was (Author: graham sanderson):
I don't have FreeBSD, but posting my comment from user thread
{bq}
Only thing I can see from looking at the exception, is that it looks like maybe
the “peer” value in the RefCountedMemory object is probably 0
Given that Unsafe.allocateMemory should not return 0 even on allocation failure
(it should throw OOM, though you should add a log statement to the Memory class
to check that) - I’d suggest logging to see if anyone is calling
SSTableReader.releaseSummary, which could set the peer to 0
{bq}
Note this particular use of sun.misc.Unsafe is new in 2.1, however I thought
there were others in 2.0. It is possible your JVM doesn't support {{public
native long Unsafe.allocateMemory(long l);}} and returns 0, though in that case
you pretty much just need to switch back to the on heap memtable allocator.
In either case, if you cannot recompile Cassandra for new logging, you can
certainly write a simple Jave program which calls {{public native long
allocateMemory(long l);}}
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14218918#comment-14218918
]
graham sanderson edited comment on CASSANDRA-8325 at 11/20/14 2:57 AM:
---
I don't have FreeBSD, but posting my comment from user thread
{quote}
Only thing I can see from looking at the exception, is that it looks like maybe
the “peer” value in the RefCountedMemory object is probably 0
Given that Unsafe.allocateMemory should not return 0 even on allocation failure
(it should throw OOM, though you should add a log statement to the Memory class
to check that) - I’d suggest logging to see if anyone is calling
SSTableReader.releaseSummary, which could set the peer to 0
{quote}
Note this particular use of sun.misc.Unsafe is new in 2.1, however I thought
there were others in 2.0. It is possible your JVM doesn't support {{public
native long Unsafe.allocateMemory(long l);}} and returns 0, though in that case
you pretty much just need to switch back to the on heap memtable allocator.
Also IMHO it'd seem a bit bizarre to support sun.misc.Unsafe, but silently not
implement the methods.
In either case, if you cannot recompile Cassandra for new logging, you can
certainly write a simple Jave program which calls {{public native long
allocateMemory(long l);}}
was (Author: graham sanderson):
I don't have FreeBSD, but posting my comment from user thread
{quote}
Only thing I can see from looking at the exception, is that it looks like maybe
the “peer” value in the RefCountedMemory object is probably 0
Given that Unsafe.allocateMemory should not return 0 even on allocation failure
(it should throw OOM, though you should add a log statement to the Memory class
to check that) - I’d suggest logging to see if anyone is calling
SSTableReader.releaseSummary, which could set the peer to 0
{quote}
Note this particular use of sun.misc.Unsafe is new in 2.1, however I thought
there were others in 2.0. It is possible your JVM doesn't support {{public
native long Unsafe.allocateMemory(long l);}} and returns 0, though in that case
you pretty much just need to switch back to the on heap memtable allocator.
In either case, if you cannot recompile Cassandra for new logging, you can
certainly write a simple Jave program which calls {{public native long
allocateMemory(long l);}}
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[
https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14218918#comment-14218918
]
graham sanderson edited comment on CASSANDRA-8325 at 11/20/14 2:57 AM:
---
I don't have FreeBSD, but posting my comment from user thread
{quote}
Only thing I can see from looking at the exception, is that it looks like maybe
the “peer” value in the RefCountedMemory object is probably 0
Given that Unsafe.allocateMemory should not return 0 even on allocation failure
(it should throw OOM, though you should add a log statement to the Memory class
to check that) - I’d suggest logging to see if anyone is calling
SSTableReader.releaseSummary, which could set the peer to 0
{quote}
Note this particular use of sun.misc.Unsafe is new in 2.1, however I thought
there were others in 2.0. It is possible your JVM doesn't support {{public
native long Unsafe.allocateMemory(long l);}} and returns 0, though in that case
you pretty much just need to switch back to the on heap memtable allocator.
Also IMHO it'd seem a bit bizarre to support sun.misc.Unsafe, but silently not
implement the methods.
In any case, if you cannot recompile Cassandra for new logging, you can
certainly write a simple Jave program which calls {{public native long
allocateMemory(long l);}}
was (Author: graham sanderson):
I don't have FreeBSD, but posting my comment from user thread
{quote}
Only thing I can see from looking at the exception, is that it looks like maybe
the “peer” value in the RefCountedMemory object is probably 0
Given that Unsafe.allocateMemory should not return 0 even on allocation failure
(it should throw OOM, though you should add a log statement to the Memory class
to check that) - I’d suggest logging to see if anyone is calling
SSTableReader.releaseSummary, which could set the peer to 0
{quote}
Note this particular use of sun.misc.Unsafe is new in 2.1, however I thought
there were others in 2.0. It is possible your JVM doesn't support {{public
native long Unsafe.allocateMemory(long l);}} and returns 0, though in that case
you pretty much just need to switch back to the on heap memtable allocator.
Also IMHO it'd seem a bit bizarre to support sun.misc.Unsafe, but silently not
implement the methods.
In either case, if you cannot recompile Cassandra for new logging, you can
certainly write a simple Jave program which calls {{public native long
allocateMemory(long l);}}
Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
-
Key: CASSANDRA-8325
URL: https://issues.apache.org/jira/browse/CASSANDRA-8325
Project: Cassandra
Issue Type: Bug
Environment: FreeBSD 10.0 with openjdk version 1.7.0_71, 64-Bit
Server VM
Reporter: Leonid Shalupov
Attachments: hs_err_pid1856.log, system.log
See attached error file after JVM crash
{quote}
FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu
Jan 16 22:34:59 UTC 2014
r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
{quote}
{quote}
% java -version
openjdk version 1.7.0_71
OpenJDK Runtime Environment (build 1.7.0_71-b14)
OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-6890) Standardize on a single read path
[ https://issues.apache.org/jira/browse/CASSANDRA-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14179429#comment-14179429 ] graham sanderson edited comment on CASSANDRA-6890 at 10/22/14 12:55 AM: I haven't followed this entire thread - I just happened across it - so this may be moot at this point, but I thought I'd throw it out there. bq. mmap is better for the number of reasons but mostly because we have to do less syscalls and less copies (especially cross mode boundary). So I would, as well, vote to leave it and remove buffered I/O instead. We just observed this in the wild on similar high end machines to that on which we run C* (40 cores, lots of high perf disks, RAM etc). With a large number of reader threads (in this cased doing random pread64 I/O against a single fd) we were seeing a *significant* cliff where performance dropped off alarmingly quickly (very high user and kernel CPU, with little to no actual disk I/O). In this case the data files fit in the page cache which obviously makes the problem more visible. Anyway. was (Author: graham sanderson): I haven't followed this entire thread - I just happened across it - so this may be moot at this point, but I thought I'd throw it out there. bq. mmap is better for the number of reasons but mostly because we have to do less syscalls and less copies (especially cross mode boundary). So I would, as well, vote to leave it and remove buffered I/O instead. We just observed this in the wild on similar high end machines to that on which we run C* (40 cores, lots of high perf disks, RAM etc). With a large number of reader threads (in this cased doing random pread64 I/O against a single fd) we were seeing a *significant* cliff where performance dropped off alarmingly quickly (very high user and kernel CPU, with little to no actual disk I/O). In this case the data files fit in the page cache which obviously makes the problem more visible. Anyway. Standardize on a single read path - Key: CASSANDRA-6890 URL: https://issues.apache.org/jira/browse/CASSANDRA-6890 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Joshua McKenzie Assignee: Joshua McKenzie Labels: performance Fix For: 3.0 Attachments: 6890_v1.txt, mmap_gc.jpg, mmap_jstat.txt, mmap_perf.txt, nommap_gc.jpg, nommap_jstat.txt Since we actively unmap unreferenced SSTR's and also copy data out of those readers on the read path, the current memory mapped i/o is a lot of complexity for very little payoff. Clean out the mmapp'ed i/o on the read path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6890) Standardize on a single read path
[ https://issues.apache.org/jira/browse/CASSANDRA-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14179429#comment-14179429 ] graham sanderson commented on CASSANDRA-6890: - I haven't followed this entire thread - I just happened across it - so this may be moot at this point, but I thought I'd throw it out there. bq. mmap is better for the number of reasons but mostly because we have to do less syscalls and less copies (especially cross mode boundary). So I would, as well, vote to leave it and remove buffered I/O instead. We just observed this in the wild on similar high end machines to that on which we run C* (40 cores, lots of high perf disks, RAM etc). With a large number of reader threads (in this cased doing random pread64 I/O against a single fd) we were seeing a *significant* cliff where performance dropped off alarmingly quickly (very high user and kernel CPU, with little to no actual disk I/O). In this case the data files fit in the page cache which obviously makes the problem more visible. Anyway. Standardize on a single read path - Key: CASSANDRA-6890 URL: https://issues.apache.org/jira/browse/CASSANDRA-6890 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Joshua McKenzie Assignee: Joshua McKenzie Labels: performance Fix For: 3.0 Attachments: 6890_v1.txt, mmap_gc.jpg, mmap_jstat.txt, mmap_perf.txt, nommap_gc.jpg, nommap_jstat.txt Since we actively unmap unreferenced SSTR's and also copy data out of those readers on the read path, the current memory mapped i/o is a lot of complexity for very little payoff. Clean out the mmapp'ed i/o on the read path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6890) Standardize on a single read path
[
https://issues.apache.org/jira/browse/CASSANDRA-6890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14179432#comment-14179432
]
graham sanderson commented on CASSANDRA-6890:
-
I guess I should point out that the pread64s were coming from NIO(1)
{{FileChannel.read(ByteBuffer dst, long position)}}
Standardize on a single read path
-
Key: CASSANDRA-6890
URL: https://issues.apache.org/jira/browse/CASSANDRA-6890
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Joshua McKenzie
Assignee: Joshua McKenzie
Labels: performance
Fix For: 3.0
Attachments: 6890_v1.txt, mmap_gc.jpg, mmap_jstat.txt, mmap_perf.txt,
nommap_gc.jpg, nommap_jstat.txt
Since we actively unmap unreferenced SSTR's and also copy data out of those
readers on the read path, the current memory mapped i/o is a lot of
complexity for very little payoff. Clean out the mmapp'ed i/o on the read
path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14175425#comment-14175425 ] graham sanderson commented on CASSANDRA-7546: - Thanks [~yukim] ... note I just noticed that in CHANGES.txt this is recorded in the merge from 2.0: section AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, cassandra-2.1-7546-v2.txt, cassandra-2.1-7546-v3.txt, cassandra-2.1-7546.txt, graph2_7546.png, graph3_7546.png, graph4_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] graham sanderson updated CASSANDRA-7546: Attachment: cassandra-2.1-7546-v3.txt Oops - my bad - Locks.java wasn't already committed (I though Benedict might have added it as part of something else), but it turns out I just had it in my working copy not commit... uploaded cassandra-2.1-7546-v3.txt which adds it back AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, cassandra-2.1-7546-v2.txt, cassandra-2.1-7546-v3.txt, cassandra-2.1-7546.txt, graph2_7546.png, graph3_7546.png, graph4_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14168841#comment-14168841 ] graham sanderson commented on CASSANDRA-7546: - Actually this is the first time I've looked at the Locks.java code in detail myself - it should probably not throw an AssertionError on failure (it should log) since it is optional - and maybe the methods should be renamed to indicate that it may be a noop AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, cassandra-2.1-7546-v2.txt, cassandra-2.1-7546-v3.txt, cassandra-2.1-7546.txt, graph2_7546.png, graph3_7546.png, graph4_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14168489#comment-14168489 ] graham sanderson commented on CASSANDRA-7546: - For what it's worth, I happened to be poking around the JVM source today debugging something, and so stopped to take a look - the monitorEnter does indeed just revoke any bios and inflate the lock... so seems perfectly fine for our purposes (since we expect lock contention anyway) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, cassandra-2.1-7546-v2.txt, cassandra-2.1-7546.txt, graph2_7546.png, graph3_7546.png, graph4_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14167613#comment-14167613 ] graham sanderson commented on CASSANDRA-7546: - Sorry [~yukim], I somehow missed your update - I'm about to attach the test results here... note they show much higher GC issues in native_obj than heap_buffers without the fix, I'm guessing because the spinning is much faster with native_obj As for monitorEnter/monitorExit Benedict and I had a discussion about that above (I originally had it with either multiple copies of the code, or nested functions), but it complicated stuff, and I was unable to prove any issues with monitorEnter or monitorExit (or indeed reference any, other than some vague suspicions I had that maybe this excludes biased locking or anything else which assumes these are neatly paired in a stack frame). In any case we don't really care because if we are using them we've already proved we're contended, and the monitor would be inflated anyway. The other issue was the use of Unsafe, but Benedict seemed fine with that also, since without Unsafe (which most people have) you just get the old behavior So, I say go ahead and promote the fix as is (yes current 2.1 trunk seemed to have Locks.java already added - I didn't diff them, but I peeked briefly and it looked about the same) It is possible someone will find a usage scenario that this makes slower, in which case we can look at that, but I suspect as mentioned before, in all of these cases where we degrade performance it is probably because the original performance is just on a lucky knife edge between under utilization, and a complete mess! Finally, I'll summarize what Benedict said up above, that whilst we could add a switch for this, this is really an internal implementation fix, the goal of which is eventually that there should be no bottleneck even when mutation the same partition (something he planned to address in version =3.0 with lazy updates, and repair on read) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, cassandra-2.1-7546-v2.txt, cassandra-2.1-7546.txt, graph2_7546.png, graph3_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] graham sanderson updated CASSANDRA-7546: Attachment: graph4_7546.png See previous comment for discussion (note I also realized I didn't change the labels correctly on graph3, but that was of heap_buffers and hinting) !graph4_7546.png! AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, cassandra-2.1-7546-v2.txt, cassandra-2.1-7546.txt, graph2_7546.png, graph3_7546.png, graph4_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14167613#comment-14167613 ] graham sanderson edited comment on CASSANDRA-7546 at 10/10/14 10:02 PM: Sorry [~yukim], I somehow missed your update - I'm about to attach the test results here... note they show much higher GC issues in native_obj than heap_buffers without the fix, I'm guessing because the spinning is much faster with native_obj As for monitorEnter/monitorExit Benedict and I had a discussion about that above (I originally had it with either multiple copies of the code, or nested functions), but it complicated stuff, and I was unable to prove any issues with monitorEnter or monitorExit (or indeed reference any, other than some vague suspicions I had that maybe this excludes biased locking or anything else which assumes these are neatly paired in a stack frame). In any case we don't really care because if we are using them we've already proved we're contended, and the monitor would be inflated anyway. The other issue was the use of Unsafe, but Benedict seemed fine with that also, since without Unsafe (which most people have) you just get the old behavior So, I say go ahead and promote the fix as is (yes current 2.1 trunk seemed to have Locks.java already added - I didn't diff them, but I peeked briefly and it looked about the same) It is possible someone will find a usage scenario that this makes slower, in which case we can look at that, but I suspect as mentioned before, in all of these cases where we degrade performance it is probably because the original performance is just on a lucky knife edge between under utilization, and a complete mess! Finally, I'll summarize what Benedict said up above, that whilst we could add a switch for this, this is really an internal implementation fix, the goal of which is eventually that there should be no bottleneck even when mutating the same partition (something he planned to address in version =3.0 with lazy updates, and repair on read) was (Author: graham sanderson): Sorry [~yukim], I somehow missed your update - I'm about to attach the test results here... note they show much higher GC issues in native_obj than heap_buffers without the fix, I'm guessing because the spinning is much faster with native_obj As for monitorEnter/monitorExit Benedict and I had a discussion about that above (I originally had it with either multiple copies of the code, or nested functions), but it complicated stuff, and I was unable to prove any issues with monitorEnter or monitorExit (or indeed reference any, other than some vague suspicions I had that maybe this excludes biased locking or anything else which assumes these are neatly paired in a stack frame). In any case we don't really care because if we are using them we've already proved we're contended, and the monitor would be inflated anyway. The other issue was the use of Unsafe, but Benedict seemed fine with that also, since without Unsafe (which most people have) you just get the old behavior So, I say go ahead and promote the fix as is (yes current 2.1 trunk seemed to have Locks.java already added - I didn't diff them, but I peeked briefly and it looked about the same) It is possible someone will find a usage scenario that this makes slower, in which case we can look at that, but I suspect as mentioned before, in all of these cases where we degrade performance it is probably because the original performance is just on a lucky knife edge between under utilization, and a complete mess! Finally, I'll summarize what Benedict said up above, that whilst we could add a switch for this, this is really an internal implementation fix, the goal of which is eventually that there should be no bottleneck even when mutation the same partition (something he planned to address in version =3.0 with lazy updates, and repair on read) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, cassandra-2.1-7546-v2.txt, cassandra-2.1-7546.txt, graph2_7546.png, graph3_7546.png, graph4_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the
[jira] [Comment Edited] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14167613#comment-14167613 ] graham sanderson edited comment on CASSANDRA-7546 at 10/10/14 10:03 PM: Sorry [~yukim], I somehow missed your update - I'm about to attach the test results here... note they show much higher GC issues in native_obj than heap_buffers without the fix, I'm guessing because the spinning is much faster with native_obj. Note as with heap_buffers and hints, we are getting a little bit of noise in the results (likely from compaction) As for monitorEnter/monitorExit Benedict and I had a discussion about that above (I originally had it with either multiple copies of the code, or nested functions), but it complicated stuff, and I was unable to prove any issues with monitorEnter or monitorExit (or indeed reference any, other than some vague suspicions I had that maybe this excludes biased locking or anything else which assumes these are neatly paired in a stack frame). In any case we don't really care because if we are using them we've already proved we're contended, and the monitor would be inflated anyway. The other issue was the use of Unsafe, but Benedict seemed fine with that also, since without Unsafe (which most people have) you just get the old behavior So, I say go ahead and promote the fix as is (yes current 2.1 trunk seemed to have Locks.java already added - I didn't diff them, but I peeked briefly and it looked about the same) It is possible someone will find a usage scenario that this makes slower, in which case we can look at that, but I suspect as mentioned before, in all of these cases where we degrade performance it is probably because the original performance is just on a lucky knife edge between under utilization, and a complete mess! Finally, I'll summarize what Benedict said up above, that whilst we could add a switch for this, this is really an internal implementation fix, the goal of which is eventually that there should be no bottleneck even when mutating the same partition (something he planned to address in version =3.0 with lazy updates, and repair on read) was (Author: graham sanderson): Sorry [~yukim], I somehow missed your update - I'm about to attach the test results here... note they show much higher GC issues in native_obj than heap_buffers without the fix, I'm guessing because the spinning is much faster with native_obj As for monitorEnter/monitorExit Benedict and I had a discussion about that above (I originally had it with either multiple copies of the code, or nested functions), but it complicated stuff, and I was unable to prove any issues with monitorEnter or monitorExit (or indeed reference any, other than some vague suspicions I had that maybe this excludes biased locking or anything else which assumes these are neatly paired in a stack frame). In any case we don't really care because if we are using them we've already proved we're contended, and the monitor would be inflated anyway. The other issue was the use of Unsafe, but Benedict seemed fine with that also, since without Unsafe (which most people have) you just get the old behavior So, I say go ahead and promote the fix as is (yes current 2.1 trunk seemed to have Locks.java already added - I didn't diff them, but I peeked briefly and it looked about the same) It is possible someone will find a usage scenario that this makes slower, in which case we can look at that, but I suspect as mentioned before, in all of these cases where we degrade performance it is probably because the original performance is just on a lucky knife edge between under utilization, and a complete mess! Finally, I'll summarize what Benedict said up above, that whilst we could add a switch for this, this is really an internal implementation fix, the goal of which is eventually that there should be no bottleneck even when mutating the same partition (something he planned to address in version =3.0 with lazy updates, and repair on read) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, cassandra-2.1-7546-v2.txt, cassandra-2.1-7546.txt, graph2_7546.png, graph3_7546.png, graph4_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt,
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14167633#comment-14167633 ] graham sanderson commented on CASSANDRA-7546: - Just to be clear from the graphs - that is 70gig of GC during the 913 thread count run! AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, cassandra-2.1-7546-v2.txt, cassandra-2.1-7546.txt, graph2_7546.png, graph3_7546.png, graph4_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163922#comment-14163922 ] graham sanderson edited comment on CASSANDRA-7546 at 10/8/14 6:30 PM: -- Attached graph3_7546.png, which is heap buffer and one node down (hinting)... things look good - there is some noise in .999 !graph3_7456.png! was (Author: graham sanderson): Attached graph3_7546.png, which is heap buffer and one node down (hinting)... things look good - there is some noise in .999 AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, graph2_7546.png, graph3_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] graham sanderson updated CASSANDRA-7546: Attachment: graph3_7546.png Attached graph3_7546.png, which is heap buffer and one node down (hinting)... things look good - there is some noise in .999 AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, graph2_7546.png, graph3_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163922#comment-14163922 ] graham sanderson edited comment on CASSANDRA-7546 at 10/8/14 6:31 PM: -- Attached graph3_7546.png, which is heap buffer and one node down (hinting)... things look good - there is some noise in .999 !graph3_7546.png! was (Author: graham sanderson): Attached graph3_7546.png, which is heap buffer and one node down (hinting)... things look good - there is some noise in .999 !graph3_7456.png! AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, graph2_7546.png, graph3_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] graham sanderson updated CASSANDRA-7546: Attachment: cassandra-2.1-7546.txt Added better named patch (cassandra-2.1-7546.txt) rebased against current 2.1 head and with log statement switched from WARN to DEBUG I will do the native_objects test with hinting on, but as far as I'm concerned this patch is ready to go (For me hints and OpsCenter.pdps are the only tables which it affects (the latter because it is highly bursty, dumping lots of (concurrent) low cardinality partioned writes periodically, and as such is neither here nor there when it comes to evaluating this fix)) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, cassandra-2.1-7546.txt, graph2_7546.png, graph3_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] graham sanderson updated CASSANDRA-7546: Attachment: cassandra-2.1-7546-v2.txt Made minor change to patch (uploaded as cassandra-2.1-7546-v2.txt) - Having switch to DEBUG logging of heavily contended partitions, I changed Memtable flush code to only count the contended partitions when DEBUG logging is enabled (since it involves a volatile read per partition which might have a -cost - though likely not a big deal - on non intel h/w) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, cassandra-2.1-7546-v2.txt, cassandra-2.1-7546.txt, graph2_7546.png, graph3_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14163964#comment-14163964 ] graham sanderson edited comment on CASSANDRA-7546 at 10/8/14 6:48 PM: -- Made minor change to patch (uploaded as cassandra-2.1-7546-v2.txt) - Having switch to DEBUG logging of heavily contended partitions, I changed Memtable flush code to only count the contended partitions when DEBUG logging is enabled (since it involves a volatile read per partition which might have a cost - though likely not a big deal - on non intel h/w) was (Author: graham sanderson): Made minor change to patch (uploaded as cassandra-2.1-7546-v2.txt) - Having switch to DEBUG logging of heavily contended partitions, I changed Memtable flush code to only count the contended partitions when DEBUG logging is enabled (since it involves a volatile read per partition which might have a -cost - though likely not a big deal - on non intel h/w) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, cassandra-2.1-7546-v2.txt, cassandra-2.1-7546.txt, graph2_7546.png, graph3_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14153780#comment-14153780 ] graham sanderson commented on CASSANDRA-7546: - Just a little update: I have numbers for one node down hinting with heap_buffers, I just need to re-run a few tests since there were a couple of spurious points (might have be due to not using a totally clean cluster every time - this is not a cluster I can easily re-create) that I want to verify before I post them. Generally this patch thus far seems to be good, and while there is a non-sweet spot where it can be mildly harmful, this is basically on the knife edge of where you are almost overcommitting your hardware, which is probably not where people are hoping to be running. The other point to note is that while the excess GC allocation here does not cause huge issues, in a busy cluster which had a huge number of resident slabs to start off with, this can cause major knock on GC - head-aches (with slabs spilling into old gen with other garbage etc)... The GC issue isn't as much of a problem with the native allocators in 2.1 (though they do seem to become a bottleneck under high allocation rates), the fact that it is still generally faster with this patch suggests we should keep it on for those too. AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, graph2_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7734) Schema pushes (seemingly) randomly not happening
[
https://issues.apache.org/jira/browse/CASSANDRA-7734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14152041#comment-14152041
]
graham sanderson commented on CASSANDRA-7734:
-
Giving this a little nudge - I am happy to take a look, but I don't really
understand what the intention is with the original code (i.e. lifecycle of
{{IncomingTcpConnection}})
Schema pushes (seemingly) randomly not happening
Key: CASSANDRA-7734
URL: https://issues.apache.org/jira/browse/CASSANDRA-7734
Project: Cassandra
Issue Type: Bug
Reporter: graham sanderson
Assignee: Aleksey Yeschenko
We have been seeing problems since upgrade to 2.0.9 from 2.0.5.
Basically after a while, new schema changes (we periodically add tables)
start propagating very slowly to some nodes and fast to others. It looks from
the logs and trace that in this case the push of the schema never happens
(note a node has decided not to push to another node, it doesn't seem to
start again) from the originating node to some of the other nodes. In this
case though, we do see the other node end up pulling the schema some time
later when it notices its schema is out of date.
Here is code from 2.0.9 MigrationManager.announce
{code}
for (InetAddress endpoint : Gossiper.instance.getLiveMembers())
{
// only push schema to nodes with known and equal versions
if (!endpoint.equals(FBUtilities.getBroadcastAddress())
MessagingService.instance().knowsVersion(endpoint)
MessagingService.instance().getRawVersion(endpoint) ==
MessagingService.current_version)
pushSchemaMutation(endpoint, schema);
}
{code}
and from 2.0.5
{code}
for (InetAddress endpoint : Gossiper.instance.getLiveMembers())
{
if (endpoint.equals(FBUtilities.getBroadcastAddress()))
continue; // we've dealt with localhost already
// don't send schema to the nodes with the versions older than
current major
if (MessagingService.instance().getVersion(endpoint)
MessagingService.current_version)
continue;
pushSchemaMutation(endpoint, schema);
}
{code}
the old getVersion() call would return MessagingService.current_version if
the version was unknown, so the push would occur in this case. I don't have
logging to prove this, but have strong suspicion that the version may end up
null in some cases (which would have allowed schema propagation in 2.0.5, but
not by somewhere after that and = 2.0.9)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] graham sanderson updated CASSANDRA-7546: Attachment: graph2_7546.png AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, graph2_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14150409#comment-14150409 ] graham sanderson commented on CASSANDRA-7546: - Busy week - I did the native_objects graphs. It actually really helps out here too - seems like native allocation starts taking a hit with too much concurrency. I was about to do the hinting graphs, but cassandra-stress seems to be pulling the server names from the server (so I can't start it with one node down) - or maybe I can, and I should just ignore the errors (I just tried giving it 4/5 nodes on the command line) What would you like me to do for n= ... I do have the full raw output for all these runs !graphs2_7546.png! AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, graph2_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14150409#comment-14150409 ] graham sanderson edited comment on CASSANDRA-7546 at 9/27/14 4:55 AM: -- Busy week - I I just did native_objects runs. It actually really helps out here a lot too - seems like native allocation starts taking a hit with too much concurrency. I was about to do the hinting graphs, but cassandra-stress seems to be pulling the server names from the server (so I can't start it with one node down) - or maybe I can, and I should just ignore the errors (I just tried giving it 4/5 nodes on the command line) What would you like me to do for n= ... I do have the full raw output for all these runs !graphs2_7546.png! was (Author: graham sanderson): Busy week - I did the native_objects graphs. It actually really helps out here too - seems like native allocation starts taking a hit with too much concurrency. I was about to do the hinting graphs, but cassandra-stress seems to be pulling the server names from the server (so I can't start it with one node down) - or maybe I can, and I should just ignore the errors (I just tried giving it 4/5 nodes on the command line) What would you like me to do for n= ... I do have the full raw output for all these runs !graphs2_7546.png! AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, graph2_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14150409#comment-14150409 ] graham sanderson edited comment on CASSANDRA-7546 at 9/27/14 4:56 AM: -- Busy week - I I just did native_objects runs. It actually really helps out here a lot too - seems like native allocation starts taking a hit with too much concurrency. I was about to do the hinting graphs, but cassandra-stress seems to be pulling the server names from the server (so I can't start it with one node down) - or maybe I can, and I should just ignore the errors (I just tried giving it 4/5 nodes on the command line) What would you like me to do for n= ... I do have the full raw output for all these runs !graph2_7546.png! was (Author: graham sanderson): Busy week - I I just did native_objects runs. It actually really helps out here a lot too - seems like native allocation starts taking a hit with too much concurrency. I was about to do the hinting graphs, but cassandra-stress seems to be pulling the server names from the server (so I can't start it with one node down) - or maybe I can, and I should just ignore the errors (I just tried giving it 4/5 nodes on the command line) What would you like me to do for n= ... I do have the full raw output for all these runs !graphs2_7546.png! AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, graph2_7546.png, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] graham sanderson updated CASSANDRA-7546: Attachment: graphs1.png AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14141812#comment-14141812 ] graham sanderson commented on CASSANDRA-7546: - Make of this what you will (these are the 1-1024 partitions with and without the patch as mentioned above)... You can clearly see the higher mem usage without the patch. Beyond that there looks to be some noise from compaction. As expected, the patch helps under high contention... dosen't seem to hurt at the low end (some of the low thread count stuff looks like it might be cassandra-stress related), and I'm not sure yet if the small differences in the middle thread counts are just noise. !graphs1.png! AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, graphs1.png, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14141764#comment-14141764 ] graham sanderson commented on CASSANDRA-7546: - Thanks - I updated, and have run 1/16/256/1024 partitions against both my baseline 2.1.1, and patched (with 7546.21_v1.txt) 2.1.1 using heap_buffers and all 5 nodes up. Things look promising so far, I need to run with a node down (I assume I take it out of the seeds list), and also with native_objects/native_buffers... this is something I can do in parallel with other work, but will still take some time. Random cassandra-stress question: Generally it seems that the threadCount where it stops seems to be the one after it has started overloading the system. Maybe this is what is wanted for the final results, but generally it seems that the latency of this final run is not representative of the previous one or two thread counts which were doing about the same number of ops/second (hence why it stopped). Not sure what the thinking is on that, I'm sure it has come up before. AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14141766#comment-14141766 ] graham sanderson commented on CASSANDRA-7546: - I'll try and make a graph of the data I have so far at some point over the weekend anyway. AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7849) Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful
[
https://issues.apache.org/jira/browse/CASSANDRA-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
graham sanderson updated CASSANDRA-7849:
Attachment: cassandra-1.2-7849_v3.txt
Updated patch v3 that
# Adds channel info to logged error from call sites in {{Message.java}}
# Keeps everything at ERROR level exception from code path
{{Dispatcher.exceptionCaught}}, which logs IOException at INFO or DEBUG for 3
specific messages (Connection reset by peer, Broken pipe, Connection timed
out)
Server logged error messages (in binary protocol) for unexpected exceptions
could be more helpful
-
Key: CASSANDRA-7849
URL: https://issues.apache.org/jira/browse/CASSANDRA-7849
Project: Cassandra
Issue Type: Improvement
Reporter: graham sanderson
Assignee: graham sanderson
Fix For: 1.2.19, 2.0.11
Attachments: cassandra-1.2-7849.txt, cassandra-1.2-7849_v2.txt,
cassandra-1.2-7849_v3.txt
From time to time (actually quite frequently) we get error messages in the
server logs like this
{code}
ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118
ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
These particular cases are almost certainly problems with the client driver,
client machine, client process, however after the fact this particular
exception is practically impossible to debug because there is no indication
in the underlying JVM/netty exception of who the peer was. I should note we
have lots of different types of applications running against the cluster so
it is very hard to correlate these to anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7849) Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful
[
https://issues.apache.org/jira/browse/CASSANDRA-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139144#comment-14139144
]
graham sanderson edited comment on CASSANDRA-7849 at 9/18/14 4:42 PM:
--
Updated patch v3 that
# Adds channel info to logged error from call sites in {{Message.java}}
# Keeps everything at ERROR level exception from code path
{{Dispatcher.exceptionCaught}}, which logs IOException at INFO except for 3
specific message strings at DEBUG (Connection reset by peer, Broken pipe,
Connection timed out) - corresponding to likely client disconnects
Note that since {{Throwable#getLocalizedMessage}} exists, and the Windows JVM
code path seems to map windows error codes to the *nix error messages, I think
these message strings are actually more robust than I thought across platforms
and/or locales
was (Author: graham sanderson):
Updated patch v3 that
# Adds channel info to logged error from call sites in {{Message.java}}
# Keeps everything at ERROR level exception from code path
{{Dispatcher.exceptionCaught}}, which logs IOException at INFO or DEBUG for 3
specific messages (Connection reset by peer, Broken pipe, Connection timed
out)
Server logged error messages (in binary protocol) for unexpected exceptions
could be more helpful
-
Key: CASSANDRA-7849
URL: https://issues.apache.org/jira/browse/CASSANDRA-7849
Project: Cassandra
Issue Type: Improvement
Reporter: graham sanderson
Assignee: graham sanderson
Fix For: 1.2.19, 2.0.11
Attachments: cassandra-1.2-7849.txt, cassandra-1.2-7849_v2.txt,
cassandra-1.2-7849_v3.txt
From time to time (actually quite frequently) we get error messages in the
server logs like this
{code}
ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118
ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
These particular cases are almost certainly problems with the client driver,
client machine, client process, however after the fact this particular
exception is practically impossible to debug because there is no indication
in the underlying JVM/netty exception of who the peer was. I should note we
have lots of different types of applications running against the cluster so
it is very hard to correlate these to anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[
https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139982#comment-14139982
]
graham sanderson commented on CASSANDRA-7546:
-
Ok, cool thanks - I've upgraded my 2.1.0 to 2.1.1... {{7cfd3ed}} for what it's
worth.
I merged {{7964+7926}} into that and updated my load machine with that.
I switched to 40x40x40x40 clustering keys as suggested and changed the 10M
entries in the command line args to 256 accordingly (it now runs
successfully)
The output is below
Note I ended up with 1275 partitions (note during the warmup I ended up with
1025 so there may be a 1-off bug there also either in stress or my config!)...
still not sure this is what we expect - each node has only seen about 3M
mutations total (and I've run the stress test twice - once without the GC stuff
working)
Anyway, let me know what you think - I won't be running more tests until
tomorrow US time.
Another question - what do you usually do to get comparable results; right now
I have been blowing away the stresscql keyspace every time to at least keep
compaction out of the equation. Given the length of the cassandra-stress run,
I'm not sure there is much to be gained by bouncing the cluster in between
runs, but you probably know better having used it before.
{code}
Results:
op rate : 10595
partition rate: 10595
row rate : 10595
latency mean : 85.8
latency median: 49.9
latency 95th percentile : 360.0
latency 99th percentile : 417.9
latency 99.9th percentile : 491.9
latency max : 552.2
total gc count: 3
total gc mb : 19471
total gc time (s) : 0
avg gc time(ms) : 67
stdev gc time(ms) : 5
Total operation time : 00:00:40
Improvement over 609 threadCount: -1%
id, total ops , adj row/s,op/s,pk/s, row/s,mean,
med, .95, .99,.999, max, time, stderr, gc: #, max ms,
sum ms, sdv ms, mb
4 threadCount, 6939 ,-0, 226, 226, 226,17.6,
16.3,40.3,49.4,51.1, 131.8, 30.6, 0.01464, 0, 0,
0, 0, 0
8 threadCount, 11827 , 385, 385, 385, 385,20.7,
15.1,47.5,51.3,82.1, 111.7, 30.7, 0.02511, 0, 0,
0, 0, 0
16 threadCount, 19068 ,-0, 612, 612, 612,26.1,
28.8,49.9,60.6,89.7, 172.1, 31.2, 0.01924, 0, 0,
0, 0, 0
24 threadCount, 24441 ,-0, 775, 775, 775,30.9,
32.6,52.1,80.3,88.3, 150.4, 31.5, 0.01508, 0, 0,
0, 0, 0
36 threadCount, 36641 ,-0,1155,1155,1155,31.1,
30.2,59.0,78.1,89.7, 172.1, 31.7, 0.01127, 0, 0,
0, 0, 0
54 threadCount, 55220 ,-0,1730,1730,1730,31.1,
29.1,54.5,74.3,84.3, 164.4, 31.9, 0.00883, 0, 0,
0, 0, 0
81 threadCount, 83460 ,-0,2609,2609,2609,31.0,
28.9,51.2,71.0,79.2, 175.4, 32.0, 0.01678, 0, 0,
0, 0, 0
121 threadCount, 140705,-0,4402,4402,4402,27.4,
25.8,49.7,53.2,70.3, 302.8, 32.0, 0.01438, 2, 462,
462, 11, 12889
181 threadCount, 226213,-0,7116,7116,7116,25.4,
24.2,48.8,51.8,60.1, 279.0, 31.8, 0.01335, 1, 230,
230, 0,6401
271 threadCount, 320658,-0, 10089, 10089, 10089,26.8,
25.0,48.3,50.1,57.4, 297.0, 31.8, 0.01256, 2, 425,
425, 14, 12786
406 threadCount, 342451,-0, 10609, 10609, 10609,38.2,
40.3,59.0,77.5,81.7, 142.4, 32.3, 0.00920, 0, 0,
0, 0, 0
609 threadCount, 381058,-0, 10651, 10651, 10651,57.0,
48.6, 171.5, 224.4, 248.4, 342.0, 35.8, 0.01234, 1, 66,
66, 0,6520
913 threadCount, 432518,-0, 10595, 10595, 10595,85.8,
49.9, 360.0, 417.9, 491.9, 552.2, 40.8, 0.01471, 3, 200,
200, 5, 19471
END
{code}
AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
-
Key: CASSANDRA-7546
URL: https://issues.apache.org/jira/browse/CASSANDRA-7546
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: graham sanderson
Assignee: graham sanderson
Fix For:
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14139986#comment-14139986 ] graham sanderson commented on CASSANDRA-7546: - FYI in case I didn't mention it, this is a 5 node cluster, and we're running LOCAL_QUORUM and repl factor 3 AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[
https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14140007#comment-14140007
]
graham sanderson commented on CASSANDRA-7546:
-
Didn't want to deep dive, but out of curiosity I did do one run configured for
a single partition
{code}
Results:
op rate : 5760
partition rate: 5760
row rate : 5760
latency mean : 158.7
latency median: 151.2
latency 95th percentile : 221.5
latency 99th percentile : 262.3
latency 99.9th percentile : 282.4
latency max : 396.0
total gc count: 3
total gc mb : 18779
total gc time (s) : 0
avg gc time(ms) : 67
stdev gc time(ms) : 26
Total operation time : 00:00:35
Improvement over 609 threadCount: 4%
id, total ops , adj row/s,op/s,pk/s, row/s,mean,
med, .95, .99,.999, max, time, stderr, gc: #, max ms,
sum ms, sdv ms, mb
4 threadCount, 6782 ,-0, 120, 120, 120,33.3,
43.8,50.6,63.0,83.9,85.7, 56.6, 0.01940, 0, 0,
0, 0, 0
8 threadCount, 6629 ,-0, 212, 212, 212,37.7,
39.1,57.0,75.0, 127.2, 138.2, 31.3, 0.00868, 0, 0,
0, 0, 0
16 threadCount, 27730 ,-0, 566, 566, 566,28.2,
26.2,50.6,75.7, 125.5, 170.4, 49.0, 0.01963, 0, 0,
0, 0, 0
24 threadCount, 51763 , 798, 796, 796, 796,30.1,
29.5,51.0,76.9,90.8, 144.4, 65.0, 0.01977, 2, 203,
203, 10, 12877
36 threadCount, 74953 ,-0,1253,1253,1253,28.7,
27.8,50.7,60.5,79.6, 308.0, 59.8, 0.01938, 0, 0,
0, 0, 0
54 threadCount, 56948 ,-0,1807,1807,1807,29.8,
27.6,52.6,63.1,78.1, 121.1, 31.5, 0.01170, 3, 176,
176, 12, 19816
81 threadCount, 74856 ,-0,2369,2369,2369,34.1,
33.2,57.2,67.6,76.6, 108.6, 31.6, 0.00946, 0, 0,
0, 0, 0
121 threadCount, 100526,-0,3158,3158,3158,38.2,
37.8,63.4,78.9,89.1, 446.6, 31.8, 0.01805, 2, 93,
93, 1, 13063
181 threadCount, 277875,-0,4491,4491,4491,40.2,
40.2,63.1,79.1,94.0, 679.7, 61.9, 0.01985, 5, 286,
286, 28, 32541
271 threadCount, 169870,-0,5205,5205,5205,52.0,
49.2,84.9, 110.5, 140.5, 843.9, 32.6, 0.01320, 3, 157,
157, 11, 19408
406 threadCount, 187985, 5648,,,,73.0,
64.2, 122.1, 156.0, 285.3, 848.6, 33.8, 0.01421, 3, 173,
173, 12, 19570
609 threadCount, 201184, 5540,5534,5534,5534, 110.1,
101.1, 160.5, 230.1, 378.9, 555.6, 36.4, 0.01917, 3, 163,
163, 17, 19709
913 threadCount, 205466, 5787,5760,5760,5760, 158.7,
151.2, 221.5, 262.3, 282.4, 396.0, 35.7, 0.01335, 3, 200,
200, 26, 18779
{code}
Obviously I don't know if the slowdown is on the load end or the server end
(though there is some GC increase here - we'll see what the patch for this
issue does). Note that if this is a synchronization problem with the load
generator still, we do know for a fact that hinting is a good way of turning a
large partition domain into a small partition domain (so I'll obviously be
testing that too, though that isn't apples to apples either).
AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
-
Key: CASSANDRA-7546
URL: https://issues.apache.org/jira/browse/CASSANDRA-7546
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: graham sanderson
Assignee: graham sanderson
Fix For: 2.1.1
Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt,
7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt,
7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, hint_spikes.png,
suggestion1.txt, suggestion1_21.txt, young_gen_gc.png
In order to preserve atomicity, this code attempts to read, clone/update,
then CAS the state of the partition.
Under heavy contention for updating a single partition this can cause some
fairly staggering memory growth (the more cores on your machine the worst it
gets).
Whilst many usage patterns don't
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14140011#comment-14140011 ] graham sanderson commented on CASSANDRA-7546: - Oh I should mention the warmup ended up generating 20 partitions, and during the cause of the whole test, it got bumped to 21... maybe that'll give you an ah ha moment. AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[
https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1413#comment-1413
]
graham sanderson commented on CASSANDRA-7546:
-
OK, so I'm running latest stress.jar on my load machine - given the number of
changes to stress in 2.1.1 (and the addition by the looks of things of remote
GC logging via cassandra-stress which would be useful in this case), I guess
I'll upgrade the cluster as well.
Here is my current config (minus the comments) and the launch command... note
there were some typos in our conversation above
{code}
keyspace: stresscql
keyspace_definition: |
CREATE KEYSPACE stresscql WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 3};
table: testtable
table_definition: |
CREATE TABLE testtable (
p text,
c1 int, c2 int, c3 int,
v blob,
PRIMARY KEY(p, c1, c2, c3)
) WITH COMPACT STORAGE
AND compaction = { 'class':'LeveledCompactionStrategy' }
AND comment='TestTable'
columnspec:
- name: p
size: fixed(16)
- name: c1
cluster: fixed(100)
- name: c2
cluster: fixed(100)
- name: c3
cluster: fixed(1000) # note I made it slightly bigger since 10M is better
than 1M for a max - 1M happens pretty quickly
- name: v
size: gaussian(50..250)
queries:
simple1:
cql: select * from testtable where k = ? and v = ? LIMIT 10
fields: samerow
{code}
{code}
./cassandra-stress user profile=~/cqlstress-7546.yaml ops\(insert=1\)
cl=LOCAL_QUORUM -node $NODES -mode native prepared cql3 -pop seq=1..10M -insert
visits=fixed\(10M\) revisit=uniform\(1..1024\) | tee
results/results-2.1.0-p1024-a.txt
{code}
As of right now, we're still (8 minutes later) at:
{code}
INFO 19:11:51 Using data-center name 'Austin' for DCAwareRoundRobinPolicy (if
this is incorrect, please provide the correct datacenter name with
DCAwareRoundRobinPolicy constructor)
Connected to cluster: Austin Multi-Tenant Cassandra 1
INFO 19:11:51 New Cassandra host cassandra4.aus.vast.com/172.17.26.14:9042
added
Datatacenter: Austin; Host: cassandra4.aus.vast.com/172.17.26.14; Rack: 98.9
Datatacenter: Austin; Host: /172.17.26.15; Rack: 98.9
Datatacenter: Austin; Host: /172.17.26.13; Rack: 98.9
Datatacenter: Austin; Host: /172.17.26.12; Rack: 98.9
Datatacenter: Austin; Host: /172.17.26.11; Rack: 98.9
INFO 19:11:51 New Cassandra host /172.17.26.12:9042 added
INFO 19:11:51 New Cassandra host /172.17.26.11:9042 added
INFO 19:11:51 New Cassandra host /172.17.26.13:9042 added
INFO 19:11:51 New Cassandra host /172.17.26.15:9042 added
Created schema. Sleeping 5s for propagation.
Warming up insert with 25 iterations...
Failed to connect over JMX; not collecting these stats
Generating batches with [1..1] partitions and [1..1] rows (of
[1000..1000] total rows in the partitions)
{code}
Number of distinct partitions is currently 2365 and growing.
Is this what we expect? doesn't seem like 250,000 should have exhausted any
partitions?
AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
-
Key: CASSANDRA-7546
URL: https://issues.apache.org/jira/browse/CASSANDRA-7546
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: graham sanderson
Assignee: graham sanderson
Fix For: 2.1.1
Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt,
7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt,
7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, hint_spikes.png,
suggestion1.txt, suggestion1_21.txt, young_gen_gc.png
In order to preserve atomicity, this code attempts to read, clone/update,
then CAS the state of the partition.
Under heavy contention for updating a single partition this can cause some
fairly staggering memory growth (the more cores on your machine the worst it
gets).
Whilst many usage patterns don't do highly concurrent updates to the same
partition, hinting today, does, and in this case wild (order(s) of magnitude
more than expected) memory allocation rates can be seen (especially when the
updates being hinted are small updates to different partitions which can
happen very fast on their own) - see CASSANDRA-7545
It would be best to eliminate/reduce/limit the spinning memory allocation
whilst not slowing down the very common un-contended case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7849) Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful
[
https://issues.apache.org/jira/browse/CASSANDRA-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14137857#comment-14137857
]
graham sanderson commented on CASSANDRA-7849:
-
Yeah the error message string comes from deep in native bowels of native JVM
code, looks like it hard coded in JVM on Windows (probably to match *nix) and
comes from the OS on *nix, so is possibly platform dependent and/or localized,
which is why I was hesitant to touch it originally.
That said, if our intention is to ONLY put at DEBUG things we absolutely know
to be spurious, then we could test the error message string, and just risk some
OS or locale getting a different message at INFO - they could always open a new
issue.
Server logged error messages (in binary protocol) for unexpected exceptions
could be more helpful
-
Key: CASSANDRA-7849
URL: https://issues.apache.org/jira/browse/CASSANDRA-7849
Project: Cassandra
Issue Type: Improvement
Reporter: graham sanderson
Assignee: graham sanderson
Fix For: 1.2.19, 2.0.11
Attachments: cassandra-1.2-7849.txt, cassandra-1.2-7849_v2.txt
From time to time (actually quite frequently) we get error messages in the
server logs like this
{code}
ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118
ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
These particular cases are almost certainly problems with the client driver,
client machine, client process, however after the fact this particular
exception is practically impossible to debug because there is no indication
in the underlying JVM/netty exception of who the peer was. I should note we
have lots of different types of applications running against the cluster so
it is very hard to correlate these to anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14137872#comment-14137872 ] graham sanderson commented on CASSANDRA-7546: - Cool, thanks, I'll wait on your patch (I have plenty of other things to do ;-) ). that said, am I relatively safe to upgrade the actual nodes to current head of 2.1 branch (and thusly pick up your latest GC monitoring stuff?) if I have a spare moment before then? Ideally I'd upgrade to the last commit in 2.1 needed to be in place on test nodes for correct latest cassandra-stress operation. AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14137887#comment-14137887 ] graham sanderson commented on CASSANDRA-7546: - Hmm - i'll definitely have to try again - it didn't respond to SIGHUP or non -F jstack, and isn't responding to ctrl+C, so maybe close to OOM AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14137888#comment-14137888 ] graham sanderson commented on CASSANDRA-7546: - Yeah, this is only a cluster for my testing this... I just don't want a massive breakage that stops it working completely! I'll install head AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14135637#comment-14135637 ] graham sanderson commented on CASSANDRA-7546: - Ok, thanks Sylvain, yes I was a bit confused (also because Benedict's changes included in the incorrect tag had CHANGES.txt with his new stress change as part of listed 2.1.0 changes - which of course now makes sense); anyways... this is good news for me, I'll leave the test cluster on what I deployed (2.1.0-tentative == the real 2.1.0 as expected according to how the vote was looking at the time), and update stress.jar on my load machine to come from 2.1 head. AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7849) Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful
[
https://issues.apache.org/jira/browse/CASSANDRA-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14136390#comment-14136390
]
graham sanderson commented on CASSANDRA-7849:
-
Well our options certainly include calling code path (as it happens the only
errors I have seen, were all client related, and all {{IOException}} and all
came from the Message.exceptionCaught path)
I am happy to do whatever, I'd suggest
1) Leave in the new the extra information in the message (this IS useful) -
i.e. IP addresses of each end of the channel
2) Just use DEBUG level for Message.exceptionCaught code path
3) Possibly make the decision at that code path for ERROR vs DEBUG based on
{{instanceof IOException}}?
The crux of the issue (and hence the stuff above) is that the IOException in
particular does not help you distinguish the cause (except by examining the
message which is obviously bad) from being noise or an actual issue... I had
leaned towards INFO at some point
Note that this error message is not logged by netty, but originates there , but
I think 2) pretty much covers that (since it is netty specific exception
handling)
Thoughts? I'll go ahead and update the patch if we're in agreement (currently
with 1,2,3 and no INFO level)
Server logged error messages (in binary protocol) for unexpected exceptions
could be more helpful
-
Key: CASSANDRA-7849
URL: https://issues.apache.org/jira/browse/CASSANDRA-7849
Project: Cassandra
Issue Type: Improvement
Reporter: graham sanderson
Assignee: graham sanderson
Fix For: 1.2.19, 2.0.11
Attachments: cassandra-1.2-7849.txt, cassandra-1.2-7849_v2.txt
From time to time (actually quite frequently) we get error messages in the
server logs like this
{code}
ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118
ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
These particular cases are almost certainly problems with the client driver,
client machine, client process, however after the fact this particular
exception is practically impossible to debug because there is no indication
in the underlying JVM/netty exception of who the peer was. I should note we
have lots of different types of applications running against the cluster so
it is very hard to correlate these to anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[
https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14134635#comment-14134635
]
graham sanderson commented on CASSANDRA-7546:
-
Finally getting back to this, been doing other things (this slightly lower
priority as we have it in production already)... I just realized that the
version c6a2c65a75ade being voted on for 2.1.0 that I deployed is not the same
as 2.1.0 released. I am now upgrading, since cassandra-stress changes snuck in.
Note, than I plan to stress using 1024, 256, 16, 1 partitions, with all 5 nodes
up, and then with 4 nodes up and one down to test effect of hinting, (note repl
factor of 3 and cl=LOCAL_QUORUM)
I want to do one cell insert per batch... I'm upgrading in part because of the
new visit/revisit stuff - I'm not 100% sure how to use them correctly, I'll
keep playing but you may answer before I have finished upgrading and tried with
this. My first attempt on the original 2.1.0 revision, ended up with only one
clustering key value per partition which is not what I wanted (because it'll
make trees small)
Sample YAML for 1024 partitions
{code}
#
# This is an example YAML profile for cassandra-stress
#
# insert data
# cassandra-stress user profile=/home/jake/stress1.yaml ops(insert=1)
#
# read, using query simple1:
# cassandra-stress profile=/home/jake/stress1.yaml ops(simple1=1)
#
# mixed workload (90/10)
# cassandra-stress user profile=/home/jake/stress1.yaml ops(insert=1,simple1=9)
#
# Keyspace info
#
keyspace: stresscql
#
# The CQL for creating a keyspace (optional if it already exists)
#
keyspace_definition: |
CREATE KEYSPACE stresscql WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 3};
#
# Table info
#
table: testtable
#
# The CQL for creating a table you wish to stress (optional if it already
exists)
#
table_definition: |
CREATE TABLE testtable (
p text,
c text,
v blob,
PRIMARY KEY(p, c)
) WITH COMPACT STORAGE
AND compaction = { 'class':'LeveledCompactionStrategy' }
AND comment='TestTable'
#
# Optional meta information on the generated columns in the above table
# The min and max only apply to text and blob types
# The distribution field represents the total unique population
# distribution of that column across rows. Supported types are
#
# EXP(min..max)An exponential distribution over
the range [min..max]
# EXTREME(min..max,shape) An extreme value (Weibull)
distribution over the range [min..max]
# GAUSSIAN(min..max,stdvrng) A gaussian/normal distribution,
where mean=(min+max)/2, and stdev is (mean-min)/stdvrng
# GAUSSIAN(min..max,mean,stdev)A gaussian/normal distribution,
with explicitly defined mean and stdev
# UNIFORM(min..max)A uniform distribution over the
range [min, max]
# FIXED(val) A fixed distribution, always
returning the same value
# Aliases: extr, gauss, normal, norm, weibull
#
# If preceded by ~, the distribution is inverted
#
# Defaults for all columns are size: uniform(4..8), population:
uniform(1..100B), cluster: fixed(1)
#
columnspec:
- name: p
size: fixed(16)
population: uniform(1..1024) # the range of unique values to select for
the field (default is 100Billion)
- name: c
size: fixed(26)
#cluster: uniform(1..100B)
- name: v
size: gaussian(50..250)
insert:
partitions: fixed(1)# number of unique partitions to update in a
single operation
# if batchcount 1, multiple batches will be
used but all partitions will
# occur in all batches (unless they finish
early); only the row counts will vary
batchtype: LOGGED # type of batch to use
visits: fixed(10M)# not sure about this
queries:
simple1: select * from testtable where k = ? and v = ? LIMIT 10
{code}
Command-line
{code}
./cassandra-stress user profile=~/cqlstress-1024.yaml ops\(insert=1\)
cl=LOCAL_QUORUM -node $NODES -mode native prepared cql3 | tee
results/results-2.1.0-p1024-a.txt
{code}
AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
-
Key: CASSANDRA-7546
URL: https://issues.apache.org/jira/browse/CASSANDRA-7546
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: graham sanderson
Assignee: graham sanderson
Fix For: 2.1.1
Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt,
7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt,
7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, hint_spikes.png,
suggestion1.txt, suggestion1_21.txt,
[jira] [Comment Edited] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[
https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14134635#comment-14134635
]
graham sanderson edited comment on CASSANDRA-7546 at 9/15/14 10:50 PM:
---
Finally getting back to this, been doing other things (this slightly lower
priority as we have it in production already) as well as keeping breaking
myself physically, requiring orthopedic visits! I just realized that the
version c6a2c65a75ade being voted on for 2.1.0 that I deployed is not the same
as 2.1.0 released. I am now upgrading, since cassandra-stress changes snuck in.
Note, than I plan to stress using 1024, 256, 16, 1 partitions, with all 5 nodes
up, and then with 4 nodes up and one down to test effect of hinting, (note repl
factor of 3 and cl=LOCAL_QUORUM), as well as with at least
memtable_allocation_type = heap_buffers off_heap_buffers
I want to do one cell insert per batch... I'm upgrading in part because of the
new visit/revisit stuff - I'm not 100% sure how to use them correctly, I'll
keep playing but you may answer before I have finished upgrading and tried with
this. My first attempt on the original 2.1.0 revision, ended up with only one
clustering key value per partition which is not what I wanted (because it'll
make trees small)
Sample YAML for 1024 partitions
{code}
#
# This is an example YAML profile for cassandra-stress
#
# insert data
# cassandra-stress user profile=/home/jake/stress1.yaml ops(insert=1)
#
# read, using query simple1:
# cassandra-stress profile=/home/jake/stress1.yaml ops(simple1=1)
#
# mixed workload (90/10)
# cassandra-stress user profile=/home/jake/stress1.yaml ops(insert=1,simple1=9)
#
# Keyspace info
#
keyspace: stresscql
#
# The CQL for creating a keyspace (optional if it already exists)
#
keyspace_definition: |
CREATE KEYSPACE stresscql WITH replication = {'class': 'SimpleStrategy',
'replication_factor': 3};
#
# Table info
#
table: testtable
#
# The CQL for creating a table you wish to stress (optional if it already
exists)
#
table_definition: |
CREATE TABLE testtable (
p text,
c text,
v blob,
PRIMARY KEY(p, c)
) WITH COMPACT STORAGE
AND compaction = { 'class':'LeveledCompactionStrategy' }
AND comment='TestTable'
#
# Optional meta information on the generated columns in the above table
# The min and max only apply to text and blob types
# The distribution field represents the total unique population
# distribution of that column across rows. Supported types are
#
# EXP(min..max)An exponential distribution over
the range [min..max]
# EXTREME(min..max,shape) An extreme value (Weibull)
distribution over the range [min..max]
# GAUSSIAN(min..max,stdvrng) A gaussian/normal distribution,
where mean=(min+max)/2, and stdev is (mean-min)/stdvrng
# GAUSSIAN(min..max,mean,stdev)A gaussian/normal distribution,
with explicitly defined mean and stdev
# UNIFORM(min..max)A uniform distribution over the
range [min, max]
# FIXED(val) A fixed distribution, always
returning the same value
# Aliases: extr, gauss, normal, norm, weibull
#
# If preceded by ~, the distribution is inverted
#
# Defaults for all columns are size: uniform(4..8), population:
uniform(1..100B), cluster: fixed(1)
#
columnspec:
- name: p
size: fixed(16)
population: uniform(1..1024) # the range of unique values to select for
the field (default is 100Billion)
- name: c
size: fixed(26)
#cluster: uniform(1..100B)
- name: v
size: gaussian(50..250)
insert:
partitions: fixed(1)# number of unique partitions to update in a
single operation
# if batchcount 1, multiple batches will be
used but all partitions will
# occur in all batches (unless they finish
early); only the row counts will vary
batchtype: LOGGED # type of batch to use
visits: fixed(10M)# not sure about this
queries:
simple1: select * from testtable where k = ? and v = ? LIMIT 10
{code}
Command-line
{code}
./cassandra-stress user profile=~/cqlstress-1024.yaml ops\(insert=1\)
cl=LOCAL_QUORUM -node $NODES -mode native prepared cql3 | tee
results/results-2.1.0-p1024-a.txt
{code}
was (Author: graham sanderson):
Finally getting back to this, been doing other things (this slightly lower
priority as we have it in production already)... I just realized that the
version c6a2c65a75ade being voted on for 2.1.0 that I deployed is not the same
as 2.1.0 released. I am now upgrading, since cassandra-stress changes snuck in.
Note, than I plan to stress using 1024, 256, 16, 1 partitions, with all 5 nodes
up, and then with 4 nodes up and one down to test effect of hinting, (note repl
[jira] [Comment Edited] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14134695#comment-14134695 ] graham sanderson edited comment on CASSANDRA-7546 at 9/15/14 11:35 PM: --- # https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=log;h=f099e086f3f002789e24bd6c58e52b7553cd5381 is what was released according to the 2.1.0 tag in git vs what [~slebresne] said in the email thread regarding no changes after c6a2c65a75adea9a62896269da98dd036c8e57f3 which was 2.1.0-tentative # ok, I'll try offheap_objects instead (or as well) # I'm still a bit confused about visit/revisit (which are in the 2.1.0 tagged release)... I want to evenly spread the load across all my partitions (genernally using a new clustering key every time, though I want to put a practical limit on the size of the partitions, so I was hoping to let it wrap at 10M or so unique clustering key values)... so it ounds like i want a visits=fixed(1) and revisits=not quite sure was (Author: graham sanderson): # https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=log;h=f099e086f3f002789e24bd6c58e52b7553cd5381 is what was released according to the 2.1.0 tag in git vs despite what [~slebresne] said in the email thread regarding no changes after c6a2c65a75adea9a62896269da98dd036c8e57f3 which was 2.1.0-tentative # ok, I'll try offheap_objects instead (or as well) # I'm still a bit confused about visit/revisit (which are in the 2.1.0 tagged release)... I want to evenly spread the load across all my partitions (genernally using a new clustering key every time, though I want to put a practical limit on the size of the partitions, so I was hoping to let it wrap at 10M or so unique clustering key values)... so it ounds like i want a visits=fixed(1) and revisits=not quite sure AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14134695#comment-14134695 ] graham sanderson commented on CASSANDRA-7546: - # https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=log;h=f099e086f3f002789e24bd6c58e52b7553cd5381 is what was released according to the 2.1.0 tag in git vs despite what [~slebresne] said in the email thread regarding no changes after c6a2c65a75adea9a62896269da98dd036c8e57f3 which was 2.1.0-tentative # ok, I'll try offheap_objects instead (or as well) # I'm still a bit confused about visit/revisit (which are in the 2.1.0 tagged release)... I want to evenly spread the load across all my partitions (genernally using a new clustering key every time, though I want to put a practical limit on the size of the partitions, so I was hoping to let it wrap at 10M or so unique clustering key values)... so it ounds like i want a visits=fixed(1) and revisits=not quite sure AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7849) Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful
[
https://issues.apache.org/jira/browse/CASSANDRA-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
graham sanderson updated CASSANDRA-7849:
Attachment: cassandra-1.2-7849_v2.txt
Attached patch which just logs everything as debug ... can someone take it from
here to maybe
# rename the issue (or at least give it a better commit message)
# maybe update NEWS.txt to say that these errors have been set to DEBUG since
they are almost always client problems
Server logged error messages (in binary protocol) for unexpected exceptions
could be more helpful
-
Key: CASSANDRA-7849
URL: https://issues.apache.org/jira/browse/CASSANDRA-7849
Project: Cassandra
Issue Type: Improvement
Reporter: graham sanderson
Fix For: 1.2.19, 2.0.11
Attachments: cassandra-1.2-7849.txt, cassandra-1.2-7849_v2.txt
From time to time (actually quite frequently) we get error messages in the
server logs like this
{code}
ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118
ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
These particular cases are almost certainly problems with the client driver,
client machine, client process, however after the fact this particular
exception is practically impossible to debug because there is no indication
in the underlying JVM/netty exception of who the peer was. I should note we
have lots of different types of applications running against the cluster so
it is very hard to correlate these to anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7849) Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful
[
https://issues.apache.org/jira/browse/CASSANDRA-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125698#comment-14125698
]
graham sanderson commented on CASSANDRA-7849:
-
# So the vote is to change all logging from this class to debug?
# We should probably add something to NEWS.txt don't you think, about turning
on debugging for the class to debug network issues or binary client driver
problems
#* Do I do that as part of the patch
Server logged error messages (in binary protocol) for unexpected exceptions
could be more helpful
-
Key: CASSANDRA-7849
URL: https://issues.apache.org/jira/browse/CASSANDRA-7849
Project: Cassandra
Issue Type: Improvement
Reporter: graham sanderson
Fix For: 1.2.19, 2.0.11
Attachments: cassandra-1.2-7849.txt
From time to time (actually quite frequently) we get error messages in the
server logs like this
{code}
ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118
ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
These particular cases are almost certainly problems with the client driver,
client machine, client process, however after the fact this particular
exception is practically impossible to debug because there is no indication
in the underlying JVM/netty exception of who the peer was. I should note we
have lots of different types of applications running against the cluster so
it is very hard to correlate these to anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7849) Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful
[
https://issues.apache.org/jira/browse/CASSANDRA-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14125698#comment-14125698
]
graham sanderson edited comment on CASSANDRA-7849 at 9/8/14 4:05 PM:
-
# So the vote is to change all logging from this class to debug?
# We should probably add something to NEWS.txt don't you think, about turning
on debugging for the class to debug network issues or binary client driver
problems
#* Do I do that as part of the patch?
was (Author: graham sanderson):
# So the vote is to change all logging from this class to debug?
# We should probably add something to NEWS.txt don't you think, about turning
on debugging for the class to debug network issues or binary client driver
problems
#* Do I do that as part of the patch
Server logged error messages (in binary protocol) for unexpected exceptions
could be more helpful
-
Key: CASSANDRA-7849
URL: https://issues.apache.org/jira/browse/CASSANDRA-7849
Project: Cassandra
Issue Type: Improvement
Reporter: graham sanderson
Fix For: 1.2.19, 2.0.11
Attachments: cassandra-1.2-7849.txt
From time to time (actually quite frequently) we get error messages in the
server logs like this
{code}
ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118
ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
These particular cases are almost certainly problems with the client driver,
client machine, client process, however after the fact this particular
exception is practically impossible to debug because there is no indication
in the underlying JVM/netty exception of who the peer was. I should note we
have lots of different types of applications running against the cluster so
it is very hard to correlate these to anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7849) Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful
[
https://issues.apache.org/jira/browse/CASSANDRA-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
graham sanderson updated CASSANDRA-7849:
Fix Version/s: 2.1.0
2.0.11
1.2.19
Server logged error messages (in binary protocol) for unexpected exceptions
could be more helpful
-
Key: CASSANDRA-7849
URL: https://issues.apache.org/jira/browse/CASSANDRA-7849
Project: Cassandra
Issue Type: Improvement
Reporter: graham sanderson
Fix For: 1.2.19, 2.0.11, 2.1.0
From time to time (actually quite frequently) we get error messages in the
server logs like this
{code}
ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118
ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
These particular cases are almost certainly problems with the client driver,
client machine, client process, however after the fact this particular
exception is practically impossible to debug because there is no indication
in the underlying JVM/netty exception of who the peer was. I should note we
have lots of different types of applications running against the cluster so
it is very hard to correlate these to anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7849) Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful
[
https://issues.apache.org/jira/browse/CASSANDRA-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
graham sanderson updated CASSANDRA-7849:
Attachment: cassandra-1.2-7849.txt
We're using 2.0.9, but I'm submitting patch against 1.2 since it will be useful
from there on
Server logged error messages (in binary protocol) for unexpected exceptions
could be more helpful
-
Key: CASSANDRA-7849
URL: https://issues.apache.org/jira/browse/CASSANDRA-7849
Project: Cassandra
Issue Type: Improvement
Reporter: graham sanderson
Fix For: 1.2.19, 2.0.11, 2.1.0
Attachments: cassandra-1.2-7849.txt
From time to time (actually quite frequently) we get error messages in the
server logs like this
{code}
ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118
ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
These particular cases are almost certainly problems with the client driver,
client machine, client process, however after the fact this particular
exception is practically impossible to debug because there is no indication
in the underlying JVM/netty exception of who the peer was. I should note we
have lots of different types of applications running against the cluster so
it is very hard to correlate these to anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7849) Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful
[
https://issues.apache.org/jira/browse/CASSANDRA-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123124#comment-14123124
]
graham sanderson commented on CASSANDRA-7849:
-
Example with patch in place
{code}
ERROR [Native-Transport-Requests:445039] 2014-09-05 18:18:21,809
ErrorMessage.java (line 238) Unexpected exception during request. channel =
[id: 0xe91ab64e, /172.17.70.12:32874 = /172.16.26.11:9042]
java.io.IOException: Connection timed out
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
Server logged error messages (in binary protocol) for unexpected exceptions
could be more helpful
-
Key: CASSANDRA-7849
URL: https://issues.apache.org/jira/browse/CASSANDRA-7849
Project: Cassandra
Issue Type: Improvement
Reporter: graham sanderson
Fix For: 1.2.19, 2.0.11, 2.1.0
Attachments: cassandra-1.2-7849.txt
From time to time (actually quite frequently) we get error messages in the
server logs like this
{code}
ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118
ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
These particular cases are almost certainly problems with the client driver,
client machine, client process, however after the fact this particular
exception is practically impossible to debug because there is no indication
in the underlying JVM/netty exception of who the peer was. I should note we
have lots of different types of applications running against the cluster so
it is very hard to correlate these to anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7849) Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful
[
https://issues.apache.org/jira/browse/CASSANDRA-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123425#comment-14123425
]
graham sanderson commented on CASSANDRA-7849:
-
Certainly a connection reset by peer or a connection reset don't count as
server errors, and there was a similar discussion on CASSANDRA-6424 about not
really being able to tell what is a _real_ error vs something inconsequential.
I will say that these server logs are useful, for detecting bugs in client
drivers (we use several), application bugs (unclean shutdowns, incorrect
pooling etc), as well as more subtle things (we discovered today, that through
a chain of distributed configs, some service in beta was pulling data from
production cassandra).
Therefore my vote would be to change this to INFO, and add the additional
logging as per the patch.
Server logged error messages (in binary protocol) for unexpected exceptions
could be more helpful
-
Key: CASSANDRA-7849
URL: https://issues.apache.org/jira/browse/CASSANDRA-7849
Project: Cassandra
Issue Type: Improvement
Reporter: graham sanderson
Fix For: 1.2.19, 2.0.11, 2.1.0
Attachments: cassandra-1.2-7849.txt
From time to time (actually quite frequently) we get error messages in the
server logs like this
{code}
ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118
ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
These particular cases are almost certainly problems with the client driver,
client machine, client process, however after the fact this particular
exception is practically impossible to debug because there is no indication
in the underlying JVM/netty exception of who the peer was. I should note we
have lots of different types of applications running against the cluster so
it is very hard to correlate these to anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7849) Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful
[
https://issues.apache.org/jira/browse/CASSANDRA-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123425#comment-14123425
]
graham sanderson edited comment on CASSANDRA-7849 at 9/5/14 7:23 PM:
-
Certainly a connection reset by peer or a connection timed out don't count
as server errors, and there was a similar discussion on CASSANDRA-6424 about
not really being able to tell what is a _real_ error vs something
inconsequential.
I will say that these server logs are useful, for detecting bugs in client
drivers (we use several), application bugs (unclean shutdowns, incorrect
pooling etc), as well as more subtle things (we discovered today, that through
a chain of distributed configs, some service in beta was pulling data from
production cassandra).
Therefore my vote would be to change this to INFO, and add the additional
logging as per the patch.
was (Author: graham sanderson):
Certainly a connection reset by peer or a connection reset don't count as
server errors, and there was a similar discussion on CASSANDRA-6424 about not
really being able to tell what is a _real_ error vs something inconsequential.
I will say that these server logs are useful, for detecting bugs in client
drivers (we use several), application bugs (unclean shutdowns, incorrect
pooling etc), as well as more subtle things (we discovered today, that through
a chain of distributed configs, some service in beta was pulling data from
production cassandra).
Therefore my vote would be to change this to INFO, and add the additional
logging as per the patch.
Server logged error messages (in binary protocol) for unexpected exceptions
could be more helpful
-
Key: CASSANDRA-7849
URL: https://issues.apache.org/jira/browse/CASSANDRA-7849
Project: Cassandra
Issue Type: Improvement
Reporter: graham sanderson
Fix For: 1.2.19, 2.0.11, 2.1.0
Attachments: cassandra-1.2-7849.txt
From time to time (actually quite frequently) we get error messages in the
server logs like this
{code}
ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118
ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
These particular cases are almost certainly problems with the client driver,
client machine, client process, however after the fact this particular
exception is practically impossible to debug because there is no indication
in the underlying JVM/netty exception of who the peer was. I should note we
have lots of different types of applications running against the cluster so
it is very hard to correlate these to anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7849) Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful
[
https://issues.apache.org/jira/browse/CASSANDRA-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123908#comment-14123908
]
graham sanderson commented on CASSANDRA-7849:
-
Those are the only log statements in the class, so you are right it is easy to
turn from DEBUG to something higher.
Just to confirm, you are cool with hiding at DEBUG level any unexpected
exception coming thru this code path, or just the channel related ones I
modified (there are still users of the original raw {{public static
ErrorMessage fromException(Throwable e)}} method). As I say, there isn't a nice
way to detect specifically the silly exceptions we've mostly been talking about
(connection reset, and timeout)
Server logged error messages (in binary protocol) for unexpected exceptions
could be more helpful
-
Key: CASSANDRA-7849
URL: https://issues.apache.org/jira/browse/CASSANDRA-7849
Project: Cassandra
Issue Type: Improvement
Reporter: graham sanderson
Fix For: 1.2.19, 2.0.11, 2.1.0
Attachments: cassandra-1.2-7849.txt
From time to time (actually quite frequently) we get error messages in the
server logs like this
{code}
ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118
ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
These particular cases are almost certainly problems with the client driver,
client machine, client process, however after the fact this particular
exception is practically impossible to debug because there is no indication
in the underlying JVM/netty exception of who the peer was. I should note we
have lots of different types of applications running against the cluster so
it is very hard to correlate these to anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7849) Server logged error messages (in binary protocol) for unexpected exceptions could be more helpful
[
https://issues.apache.org/jira/browse/CASSANDRA-7849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123908#comment-14123908
]
graham sanderson edited comment on CASSANDRA-7849 at 9/6/14 12:52 AM:
--
Those are the only log statements in the class, so you are right it is easy to
selectively turn on this DEBUG logging
Just to confirm, you are cool with hiding at DEBUG level any unexpected
exception coming thru this code path, or just the channel related ones I
modified (there are still users of the original raw {{public static
ErrorMessage fromException(Throwable e)}} method). As I say, there isn't a nice
way to detect specifically the silly exceptions we've mostly been talking about
(connection reset, and timeout)
was (Author: graham sanderson):
Those are the only log statements in the class, so you are right it is easy to
turn from DEBUG to something higher.
Just to confirm, you are cool with hiding at DEBUG level any unexpected
exception coming thru this code path, or just the channel related ones I
modified (there are still users of the original raw {{public static
ErrorMessage fromException(Throwable e)}} method). As I say, there isn't a nice
way to detect specifically the silly exceptions we've mostly been talking about
(connection reset, and timeout)
Server logged error messages (in binary protocol) for unexpected exceptions
could be more helpful
-
Key: CASSANDRA-7849
URL: https://issues.apache.org/jira/browse/CASSANDRA-7849
Project: Cassandra
Issue Type: Improvement
Reporter: graham sanderson
Fix For: 1.2.19, 2.0.11, 2.1.0
Attachments: cassandra-1.2-7849.txt
From time to time (actually quite frequently) we get error messages in the
server logs like this
{code}
ERROR [Native-Transport-Requests:288] 2014-08-29 04:48:07,118
ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
These particular cases are almost certainly problems with the client driver,
client machine, client process, however after the fact this particular
exception is practically impossible to debug because there is no indication
in the underlying JVM/netty exception of who the peer was. I should note we
have lots of different types of applications running against the cluster so
it is very hard to correlate these to anything
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123920#comment-14123920 ] graham sanderson commented on CASSANDRA-7546: - Yes, I'm actually waiting on one of our main Cassandra Ops guys to come back from vacation on Monday to upgrade one of our clusters to 2.1 before I can run the stress tests, but we do have the patch running in production on 2.0.x. It detects hints, and it would also seem (which makes sense) fast hint playback of things with low cardinality keys I will certainly change the log level to INFO or DEBUG though... as this shouldn't really be a WARNING. AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] graham sanderson updated CASSANDRA-7546: Attachment: 7546.21_v1.txt Well, I made a patch for 2.1 anyway in case anyone wants to take a look... I'll try it out as soon as I can (7546.21_v1.txt) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, 7546.21_v1.txt, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7734) Schema pushes (seemingly) randomly not happening
[
https://issues.apache.org/jira/browse/CASSANDRA-7734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117613#comment-14117613
]
graham sanderson commented on CASSANDRA-7734:
-
Additional data point - while investigating 2 errors in production, I noticed
that in each case the exception is next to a transition of the version to null.
Note it is always the same thread and time, so it seems like it must be the
same request
e.g.
{code}
INFO [Thread-46312] 2014-09-01 07:05:26,055 MessagingService.java (line 782)
Reseting version 7 - null for /172.16.26.16
ERROR [Thread-46312] 2014-09-01 07:05:26,055 CassandraDaemon.java (line 199)
Exception in thread Thread[Thread-46312,5,main]
java.lang.NullPointerException
at
org.apache.cassandra.db.SliceByNamesReadCommandSerializer.deserialize(SliceByNamesReadCommand.java:150)
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:175)
at
org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:134)
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
at
org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:153)
at
org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(IncomingTcpConnection.java:130)
{code}
Schema pushes (seemingly) randomly not happening
Key: CASSANDRA-7734
URL: https://issues.apache.org/jira/browse/CASSANDRA-7734
Project: Cassandra
Issue Type: Bug
Reporter: graham sanderson
Assignee: Aleksey Yeschenko
We have been seeing problems since upgrade to 2.0.9 from 2.0.5.
Basically after a while, new schema changes (we periodically add tables)
start propagating very slowly to some nodes and fast to others. It looks from
the logs and trace that in this case the push of the schema never happens
(note a node has decided not to push to another node, it doesn't seem to
start again) from the originating node to some of the other nodes. In this
case though, we do see the other node end up pulling the schema some time
later when it notices its schema is out of date.
Here is code from 2.0.9 MigrationManager.announce
{code}
for (InetAddress endpoint : Gossiper.instance.getLiveMembers())
{
// only push schema to nodes with known and equal versions
if (!endpoint.equals(FBUtilities.getBroadcastAddress())
MessagingService.instance().knowsVersion(endpoint)
MessagingService.instance().getRawVersion(endpoint) ==
MessagingService.current_version)
pushSchemaMutation(endpoint, schema);
}
{code}
and from 2.0.5
{code}
for (InetAddress endpoint : Gossiper.instance.getLiveMembers())
{
if (endpoint.equals(FBUtilities.getBroadcastAddress()))
continue; // we've dealt with localhost already
// don't send schema to the nodes with the versions older than
current major
if (MessagingService.instance().getVersion(endpoint)
MessagingService.current_version)
continue;
pushSchemaMutation(endpoint, schema);
}
{code}
the old getVersion() call would return MessagingService.current_version if
the version was unknown, so the push would occur in this case. I don't have
logging to prove this, but have strong suspicion that the version may end up
null in some cases (which would have allowed schema propagation in 2.0.5, but
not by somewhere after that and = 2.0.9)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117845#comment-14117845 ] graham sanderson commented on CASSANDRA-7546: - Ok, NP, we can do our own custom builds with it in 2.0.x... I'll make and attach a 2.1.x patch for this sensible (sensitive?) part of the code soon. AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117896#comment-14117896 ] graham sanderson commented on CASSANDRA-7546: - Hi Benedict, I hope you are OK and get well soon... it will likely be a week or two before we can prove in production that this is fixing the problem. I also have been on vacation and then sick, so I have a lot of other catching up to do. Once I have some time, I will play with the new stress testing stuff in 2.1 along with this and try and get some firm evidence there. All I ask is that it doesn't get pushed to 3.0.x ;-) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] graham sanderson updated CASSANDRA-7546: Attachment: 7546.20_async.txt While I wait for the ability to test in production, I've made an (untested) version of the fix which uses your preferred asynchronous monitor enter/exit. !7546.20_async.txt! AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Fix For: 2.1.1 Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_3.txt, 7546.20_4.txt, 7546.20_5.txt, 7546.20_6.txt, 7546.20_7.txt, 7546.20_7b.txt, 7546.20_alt.txt, 7546.20_async.txt, hint_spikes.png, suggestion1.txt, suggestion1_21.txt, young_gen_gc.png In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.2#6252)
