[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin lock that allocates memory
[
https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061757#comment-14061757
]
graham sanderson commented on CASSANDRA-7546:
-
I tried on 2.1 with pretty similar results (note I used a fresh HeapPool for
each iteration, since it got stuck with the default Native stuff)
{code}
[junit] --
[junit] 1 THREAD; ELEMENT SIZE 64
[junit]
[junit] Threads = 1 elements = 10 (of size 64) partitions = 1
[junit] original code:
[junit] Duration = 562ms maxConcurrency = 1
[junit] GC for PS Scavenge: 11 ms for 1 collections
[junit] Approx allocation = 238MB vs 8MB; ratio to raw data size =
29.79389619047619
[junit] loopRatio (closest to 1 best) 1.0 raw 10/10 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 480ms maxConcurrency = 1
[junit] GC for PS Scavenge: 10 ms for 1 collections
[junit] Approx allocation = 239MB vs 8MB; ratio to raw data size =
29.95105523809524
[junit] loopRatio (closest to 1 best) 1.0 raw 10/10 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 10 (of size 64) partitions = 16
[junit] original code:
[junit] Duration = 376ms maxConcurrency = 1
[junit] GC for PS Scavenge: 11 ms for 1 collections
[junit] Approx allocation = 219MB vs 8MB; ratio to raw data size =
27.37058095238095
[junit] loopRatio (closest to 1 best) 1.0 raw 10/10 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 349ms maxConcurrency = 1
[junit] GC for PS Scavenge: 12 ms for 1 collections
[junit] Approx allocation = 219MB vs 8MB; ratio to raw data size =
27.40619619047619
[junit] loopRatio (closest to 1 best) 1.0 raw 10/10 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 10 (of size 64) partitions = 256
[junit] original code:
[junit] Duration = 268ms maxConcurrency = 1
[junit] Approx allocation = 198MB vs 8MB; ratio to raw data size =
24.8206867
[junit] loopRatio (closest to 1 best) 1.0 raw 10/10 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 277ms maxConcurrency = 1
[junit] Approx allocation = 198MB vs 8MB; ratio to raw data size =
24.82146285714286
[junit] loopRatio (closest to 1 best) 1.0 raw 10/10 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 10 (of size 64) partitions = 1024
[junit] original code:
[junit] Duration = 258ms maxConcurrency = 1
[junit] Approx allocation = 190MB vs 8MB; ratio to raw data size =
23.770030476190477
[junit] loopRatio (closest to 1 best) 1.0 raw 10/10 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 234ms maxConcurrency = 1
[junit] Approx allocation = 190MB vs 8MB; ratio to raw data size =
23.769555238095236
[junit] loopRatio (closest to 1 best) 1.0 raw 10/10 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] --
[junit] 100 THREADS; ELEMENT SIZE 64
[junit]
[junit] Threads = 100 elements = 10 (of size 64) partitions = 1
[junit] original code:
[junit] Duration = 3908ms maxConcurrency = 100
[junit] GC for PS Scavenge: 36 ms for 10 collections
[junit] Approx allocation = 2481MB vs 8MB; ratio to raw data size =
309.730900952381
[junit] loopRatio (closest to 1 best) 16.04825 raw 10/1604825 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 697ms maxConcurrency = 100
[junit] GC for PS Scavenge: 12 ms for 1 collections
[junit] Approx allocation = 241MB vs 8MB; ratio to raw data size =
30.141012380952382
[junit] loopRatio (closest to 1 best) 1.4 raw 287/289 counted 2/2
sync 99712/99713 up 1 down 1
[junit]
[junit]
[junit] Threads = 100 elements = 10 (of size 64) partitions = 16
[junit] original code:
[junit] Duration = 867ms maxConcurrency = 99
[junit] GC for PS Scavenge: 10 ms for 1 collections
[junit] Approx allocation = 378MB vs 8MB; ratio to raw data size =
47.22428
[junit] loopRatio (closest to 1 best) 2.03201 raw 10/203201 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 797ms maxConcurrency = 99
[junit] GC for PS Scavenge: 13 ms for 1 collections
[junit] Approx allocation = 252MB vs 8MB; ratio to raw data size =
31.476938095238093
[junit] loopRatio (closest to 1 best) 1.17123 raw 22254/30788 counted
[jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin lock that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061758#comment-14061758 ] graham sanderson commented on CASSANDRA-7546: - Note that given we are fine with allocating an object (i.e. ColumnUpdater in 2.1) there is no reason we can't encapsulate the common loop logic inside of an instance, to avoid code duplication - leaving the 3 loops may still be clearer if we decide to stick with this route AtomicSortedColumns.addAllWithSizeDelta has a spin lock that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Attachments: suggestion1.txt In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin lock that allocates memory
[
https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061757#comment-14061757
]
graham sanderson edited comment on CASSANDRA-7546 at 7/15/14 6:30 AM:
--
I tried on 2.1 with pretty similar results (note I used a fresh HeapPool for
each iteration, since it got stuck with the default Native stuff) - although
the 2.1 BTree is much cheaper than the SnapTreeMap
{code}
[junit] --
[junit] 1 THREAD; ELEMENT SIZE 64
[junit]
[junit] Threads = 1 elements = 10 (of size 64) partitions = 1
[junit] original code:
[junit] Duration = 562ms maxConcurrency = 1
[junit] GC for PS Scavenge: 11 ms for 1 collections
[junit] Approx allocation = 238MB vs 8MB; ratio to raw data size =
29.79389619047619
[junit] loopRatio (closest to 1 best) 1.0 raw 10/10 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 480ms maxConcurrency = 1
[junit] GC for PS Scavenge: 10 ms for 1 collections
[junit] Approx allocation = 239MB vs 8MB; ratio to raw data size =
29.95105523809524
[junit] loopRatio (closest to 1 best) 1.0 raw 10/10 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 10 (of size 64) partitions = 16
[junit] original code:
[junit] Duration = 376ms maxConcurrency = 1
[junit] GC for PS Scavenge: 11 ms for 1 collections
[junit] Approx allocation = 219MB vs 8MB; ratio to raw data size =
27.37058095238095
[junit] loopRatio (closest to 1 best) 1.0 raw 10/10 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 349ms maxConcurrency = 1
[junit] GC for PS Scavenge: 12 ms for 1 collections
[junit] Approx allocation = 219MB vs 8MB; ratio to raw data size =
27.40619619047619
[junit] loopRatio (closest to 1 best) 1.0 raw 10/10 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 10 (of size 64) partitions = 256
[junit] original code:
[junit] Duration = 268ms maxConcurrency = 1
[junit] Approx allocation = 198MB vs 8MB; ratio to raw data size =
24.8206867
[junit] loopRatio (closest to 1 best) 1.0 raw 10/10 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 277ms maxConcurrency = 1
[junit] Approx allocation = 198MB vs 8MB; ratio to raw data size =
24.82146285714286
[junit] loopRatio (closest to 1 best) 1.0 raw 10/10 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] Threads = 1 elements = 10 (of size 64) partitions = 1024
[junit] original code:
[junit] Duration = 258ms maxConcurrency = 1
[junit] Approx allocation = 190MB vs 8MB; ratio to raw data size =
23.770030476190477
[junit] loopRatio (closest to 1 best) 1.0 raw 10/10 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 234ms maxConcurrency = 1
[junit] Approx allocation = 190MB vs 8MB; ratio to raw data size =
23.769555238095236
[junit] loopRatio (closest to 1 best) 1.0 raw 10/10 counted 0/0
sync 0/0 up 0 down 0
[junit]
[junit]
[junit] --
[junit] 100 THREADS; ELEMENT SIZE 64
[junit]
[junit] Threads = 100 elements = 10 (of size 64) partitions = 1
[junit] original code:
[junit] Duration = 3908ms maxConcurrency = 100
[junit] GC for PS Scavenge: 36 ms for 10 collections
[junit] Approx allocation = 2481MB vs 8MB; ratio to raw data size =
309.730900952381
[junit] loopRatio (closest to 1 best) 16.04825 raw 10/1604825 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 697ms maxConcurrency = 100
[junit] GC for PS Scavenge: 12 ms for 1 collections
[junit] Approx allocation = 241MB vs 8MB; ratio to raw data size =
30.141012380952382
[junit] loopRatio (closest to 1 best) 1.4 raw 287/289 counted 2/2
sync 99712/99713 up 1 down 1
[junit]
[junit]
[junit] Threads = 100 elements = 10 (of size 64) partitions = 16
[junit] original code:
[junit] Duration = 867ms maxConcurrency = 99
[junit] GC for PS Scavenge: 10 ms for 1 collections
[junit] Approx allocation = 378MB vs 8MB; ratio to raw data size =
47.22428
[junit] loopRatio (closest to 1 best) 2.03201 raw 10/203201 counted
0/0 sync 0/0 up 0 down 0
[junit]
[junit] modified code:
[junit] Duration = 797ms maxConcurrency = 99
[junit] GC for PS Scavenge: 13 ms for 1 collections
[junit] Approx allocation = 252MB vs 8MB; ratio to
[jira] [Updated] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin lock that allocates memory
[ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] graham sanderson updated CASSANDRA-7546: Attachment: suggestion1_21.txt Added version of suggestion1 for 2.1 branch AtomicSortedColumns.addAllWithSizeDelta has a spin lock that allocates memory - Key: CASSANDRA-7546 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546 Project: Cassandra Issue Type: Bug Components: Core Reporter: graham sanderson Assignee: graham sanderson Attachments: suggestion1.txt, suggestion1_21.txt In order to preserve atomicity, this code attempts to read, clone/update, then CAS the state of the partition. Under heavy contention for updating a single partition this can cause some fairly staggering memory growth (the more cores on your machine the worst it gets). Whilst many usage patterns don't do highly concurrent updates to the same partition, hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory allocation rates can be seen (especially when the updates being hinted are small updates to different partitions which can happen very fast on their own) - see CASSANDRA-7545 It would be best to eliminate/reduce/limit the spinning memory allocation whilst not slowing down the very common un-contended case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7541) Windows: IOException when repairing a range of tokens
[ https://issues.apache.org/jira/browse/CASSANDRA-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061807#comment-14061807 ] Chienyu Chen commented on CASSANDRA-7541: - Thanks, I've re-run the repair to test. It has fixed the issue for me. Windows: IOException when repairing a range of tokens - Key: CASSANDRA-7541 URL: https://issues.apache.org/jira/browse/CASSANDRA-7541 Project: Cassandra Issue Type: Bug Reporter: Joshua McKenzie Assignee: Joshua McKenzie Priority: Minor Labels: Windows Fix For: 2.0.10 Attachments: 7541_2.1_v1.txt, 7541_v1.txt Looks like we missed this in CASSANDRA-6907 - range-based repair still default to snapshot-based on Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (CASSANDRA-7547) EC2 seed provider using Datastax Reflector
Pekka Enberg created CASSANDRA-7547: --- Summary: EC2 seed provider using Datastax Reflector Key: CASSANDRA-7547 URL: https://issues.apache.org/jira/browse/CASSANDRA-7547 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Pekka Enberg Priority: Minor This is a request for comments. I am using this to build our EC2 AMIs but I thought I'd ask if this makes sense as a generic feature for Cassandra. Cassandra cluster auto-configuration on EC2 uses the Datastax reflector service for discovering seed nodes. Instead of relying on external scripts, this patch implements EC2 seed provider that uses the Datastax reflector service. This is particularly useful for EC2 AMIs that don't include a complete userspace (such as those built with OSv) where we ideally want to push as much configuration to the application itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-7547) EC2 seed provider using Datastax Reflector
[ https://issues.apache.org/jira/browse/CASSANDRA-7547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pekka Enberg updated CASSANDRA-7547: Attachment: 0001-EC2-seed-provider.patch EC2 seed provider using Datastax Reflector -- Key: CASSANDRA-7547 URL: https://issues.apache.org/jira/browse/CASSANDRA-7547 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Pekka Enberg Priority: Minor Attachments: 0001-EC2-seed-provider.patch This is a request for comments. I am using this to build our EC2 AMIs but I thought I'd ask if this makes sense as a generic feature for Cassandra. Cassandra cluster auto-configuration on EC2 uses the Datastax reflector service for discovering seed nodes. Instead of relying on external scripts, this patch implements EC2 seed provider that uses the Datastax reflector service. This is particularly useful for EC2 AMIs that don't include a complete userspace (such as those built with OSv) where we ideally want to push as much configuration to the application itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7477) JSON to SSTable import failing
[
https://issues.apache.org/jira/browse/CASSANDRA-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061825#comment-14061825
]
Mikhail Stepura commented on CASSANDRA-7477:
[~kishkaru] is it possible to share {{json_tools_test.py}}? Or is there any
other way to reproduce the problem?
JSON to SSTable import failing
--
Key: CASSANDRA-7477
URL: https://issues.apache.org/jira/browse/CASSANDRA-7477
Project: Cassandra
Issue Type: Bug
Environment: Linux Mint 17 64-bit | 16GiB | C* 2.1
Reporter: Kishan Karunaratne
Assignee: Mikhail Stepura
Fix For: 2.1.0
Attachments: log2.log
Issue affects C* version = 2.1. Commit found by using git bisect. The
previous commit to this one also fails, but due to other reasons (CCM server
won't start). This commit is the one that give the same error as 2.1 HEAD:
{noformat}
02d1e7497a9930120fac367ce82a3b22940acafb is the first bad commit
commit 02d1e7497a9930120fac367ce82a3b22940acafb
Author: Brandon Williams brandonwilli...@apache.org
Date: Mon Apr 21 14:42:29 2014 -0500
Default flush dir to data dir.
Patch by brandonwilliams, reviewed by yukim for CASSANDRA-7064
:04 04 c50a123f305b73583ccbfa9c455efc4e4cee228f
507a90290dccb8a929afadf1f833d926049c46ad Mconf
{noformat}
{noformat}
$ PRINT_DEBUG=true nosetests -x -s -v json_tools_test.py
json_tools_test (json_tools_test.TestJson) ... cluster ccm directory:
/tmp/dtest-8WVBq9
Starting cluster...
Version: 2.1.0
Getting CQLSH...
Inserting data...
Flushing and stopping cluster...
Exporting to JSON file...
-- test-users-ka-1-Data.db -
Deleting cluster and creating new...
Inserting data...
Importing JSON file...
Counting keys to import, please wait... (NOTE: to skip this use -n num_keys)
Importing 2 keys...
java.lang.ClassCastException:
org.apache.cassandra.db.composites.Composites$EmptyComposite cannot be cast
to org.apache.cassandra.db.composites.CellName
at
org.apache.cassandra.db.composites.AbstractCellNameType.cellFromByteBuffer(AbstractCellNameType.java:168)
at
org.apache.cassandra.tools.SSTableImport$JsonColumn.init(SSTableImport.java:165)
at
org.apache.cassandra.tools.SSTableImport.addColumnsToCF(SSTableImport.java:242)
at
org.apache.cassandra.tools.SSTableImport.addToStandardCF(SSTableImport.java:225)
at
org.apache.cassandra.tools.SSTableImport.importSorted(SSTableImport.java:464)
at
org.apache.cassandra.tools.SSTableImport.importJson(SSTableImport.java:351)
at org.apache.cassandra.tools.SSTableImport.main(SSTableImport.java:575)
ERROR: org.apache.cassandra.db.composites.Composites$EmptyComposite cannot be
cast to org.apache.cassandra.db.composites.CellName
Verifying import...
data: [[u'gandalf', 1955, u'male', u'p@$$', u'WA']]
FAIL
removing ccm cluster test at: /tmp/dtest-8WVBq9
ERROR
==
ERROR: json_tools_test (json_tools_test.TestJson)
--
Traceback (most recent call last):
File /home/kishan/git/cstar/cassandra-dtest/dtest.py, line 214, in
tearDown
raise AssertionError('Unexpected error in %s node log: %s' % (node.name,
errors))
AssertionError: Unexpected error in node1 node log: ['ERROR
[SSTableBatchOpen:1] 2014-06-30 13:56:01,032 CassandraDaemon.java:166 -
Exception in thread Thread[SSTableBatchOpen:1,5,main]\n']
begin captured logging
dtest: DEBUG: cluster ccm directory: /tmp/dtest-8WVBq9
dtest: DEBUG: Starting cluster...
dtest: DEBUG: Version: 2.1.0
dtest: DEBUG: Getting CQLSH...
dtest: DEBUG: Inserting data...
dtest: DEBUG: Flushing and stopping cluster...
dtest: DEBUG: Exporting to JSON file...
dtest: DEBUG: Deleting cluster and creating new...
dtest: DEBUG: Inserting data...
dtest: DEBUG: Importing JSON file...
dtest: DEBUG: Verifying import...
dtest: DEBUG: data: [[u'gandalf', 1955, u'male', u'p@$$', u'WA']]
dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-8WVBq9
- end captured logging -
==
FAIL: json_tools_test (json_tools_test.TestJson)
--
Traceback (most recent call last):
File /home/kishan/git/cstar/cassandra-dtest/json_tools_test.py, line 91,
in json_tools_test
[u'gandalf', 1955, u'male', u'p@$$', u'WA'] ] )
AssertionError: Element counts were not equal:
First has 0, Second has 1: [u'frodo', 1985, u'male', u'pass@', u'CA']
First has 0, Second has 1: [u'sam', 1980, u'male', u'@pass', u'NY']
begin captured logging
[jira] [Commented] (CASSANDRA-7477) JSON to SSTable import failing
[
https://issues.apache.org/jira/browse/CASSANDRA-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061844#comment-14061844
]
Mikhail Stepura commented on CASSANDRA-7477:
[~kishkaru] I've found that at
https://github.com/riptano/cassandra-dtest/tree/json_tools. But it gives me a
different error (I'm using the latest {ccm} from git, and latest changes from
2.1.0 branch
{code}
mstepura-mac:cassandra-dtest mikhail$ PRINT_DEBUG=true nosetests -x -s -v
json_tools_test.py
json_tools_test (json_tools_test.TestJson) ... cluster ccm directory:
/var/folders/xl/w12639217y3041zwnjstr3lcgn/T/dtest-yrqnIG
Starting cluster...
[node1 ERROR] objc[52456]: Class JavaLaunchHelper is implemented in both
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/bin/java and
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
Version: 2.1.0
Getting CQLSH...
Inserting data...
Flushing and stopping cluster...
objc[52474]: Class JavaLaunchHelper is implemented in both
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/bin/java and
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
Exporting to JSON file...
-- test-users-ka-1-Data.db -
objc[52476]: Class JavaLaunchHelper is implemented in both
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/bin/java and
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
Deleting cluster and creating new...
Inserting data...
objc[52510]: Class JavaLaunchHelper is implemented in both
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/bin/java and
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
Importing JSON file...
objc[52512]: Class JavaLaunchHelper is implemented in both
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/bin/java and
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
WARN 08:35:00 JNA link failure, one or more native method will be unavailable.
Counting keys to import, please wait... (NOTE: to skip this use -n num_keys)
org.codehaus.jackson.JsonParseException: Unexpected character ('W' (code 87)):
expected a valid value (number, String, array, object, 'true', 'false' or
'null')
at [Source:
/var/folders/xl/w12639217y3041zwnjstr3lcgn/T/tmppUlYbe.schema.json; line:
1, column: 2]
at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1432)
at
org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
at
org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
at
org.codehaus.jackson.impl.Utf8StreamParser._handleUnexpectedValue(Utf8StreamParser.java:2084)
at
org.codehaus.jackson.impl.Utf8StreamParser._nextTokenNotInObject(Utf8StreamParser.java:600)
at
org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:486)
at
org.apache.cassandra.tools.SSTableImport.importSorted(SSTableImport.java:360)
at
org.apache.cassandra.tools.SSTableImport.importJson(SSTableImport.java:282)
at org.apache.cassandra.tools.SSTableImport.main(SSTableImport.java:494)
ERROR: Unexpected character ('W' (code 87)): expected a valid value (number,
String, array, object, 'true', 'false' or 'null')
at [Source:
/var/folders/xl/w12639217y3041zwnjstr3lcgn/T/tmppUlYbe.schema.json; line:
1, column: 2]
Verifying import...
data: [[u'gandalf', 1955, u'male', u'p@$$', u'WA']]
FAIL
removing ccm cluster test at:
/var/folders/xl/w12639217y3041zwnjstr3lcgn/T/dtest-yrqnIG
==
FAIL: json_tools_test (json_tools_test.TestJson)
--
Traceback (most recent call last):
File /Users/mikhail/Documents/workspace/cassandra-dtest/json_tools_test.py,
line 91, in json_tools_test
[u'gandalf', 1955, u'male', u'p@$$', u'WA'] ] )
AssertionError: Element counts were not equal:
First has 0, Second has 1: [u'frodo', 1985, u'male', u'pass@', u'CA']
First has 0, Second has 1: [u'sam', 1980, u'male', u'@pass', u'NY']
begin captured logging
dtest: DEBUG: cluster ccm directory:
/var/folders/xl/w12639217y3041zwnjstr3lcgn/T/dtest-yrqnIG
dtest: DEBUG: Starting cluster...
dtest: DEBUG: Version: 2.1.0
dtest: DEBUG: Getting CQLSH...
dtest: DEBUG: Inserting data...
dtest: DEBUG: Flushing and stopping cluster...
dtest:
[jira] [Comment Edited] (CASSANDRA-7477) JSON to SSTable import failing
[
https://issues.apache.org/jira/browse/CASSANDRA-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061844#comment-14061844
]
Mikhail Stepura edited comment on CASSANDRA-7477 at 7/15/14 8:46 AM:
-
[~kishkaru] I've found that at
https://github.com/riptano/cassandra-dtest/tree/json_tools. But it gives me a
different error (I'm using the latest {{ccm}} from git, and latest changes from
2.1.0 branch
{code}
mstepura-mac:cassandra-dtest mikhail$ PRINT_DEBUG=true nosetests -x -s -v
json_tools_test.py
json_tools_test (json_tools_test.TestJson) ... cluster ccm directory:
/var/folders/xl/w12639217y3041zwnjstr3lcgn/T/dtest-yrqnIG
Starting cluster...
[node1 ERROR] objc[52456]: Class JavaLaunchHelper is implemented in both
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/bin/java and
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
Version: 2.1.0
Getting CQLSH...
Inserting data...
Flushing and stopping cluster...
objc[52474]: Class JavaLaunchHelper is implemented in both
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/bin/java and
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
Exporting to JSON file...
-- test-users-ka-1-Data.db -
objc[52476]: Class JavaLaunchHelper is implemented in both
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/bin/java and
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
Deleting cluster and creating new...
Inserting data...
objc[52510]: Class JavaLaunchHelper is implemented in both
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/bin/java and
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
Importing JSON file...
objc[52512]: Class JavaLaunchHelper is implemented in both
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/bin/java and
/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home/jre/lib/libinstrument.dylib.
One of the two will be used. Which one is undefined.
WARN 08:35:00 JNA link failure, one or more native method will be unavailable.
Counting keys to import, please wait... (NOTE: to skip this use -n num_keys)
org.codehaus.jackson.JsonParseException: Unexpected character ('W' (code 87)):
expected a valid value (number, String, array, object, 'true', 'false' or
'null')
at [Source:
/var/folders/xl/w12639217y3041zwnjstr3lcgn/T/tmppUlYbe.schema.json; line:
1, column: 2]
at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1432)
at
org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:385)
at
org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:306)
at
org.codehaus.jackson.impl.Utf8StreamParser._handleUnexpectedValue(Utf8StreamParser.java:2084)
at
org.codehaus.jackson.impl.Utf8StreamParser._nextTokenNotInObject(Utf8StreamParser.java:600)
at
org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:486)
at
org.apache.cassandra.tools.SSTableImport.importSorted(SSTableImport.java:360)
at
org.apache.cassandra.tools.SSTableImport.importJson(SSTableImport.java:282)
at org.apache.cassandra.tools.SSTableImport.main(SSTableImport.java:494)
ERROR: Unexpected character ('W' (code 87)): expected a valid value (number,
String, array, object, 'true', 'false' or 'null')
at [Source:
/var/folders/xl/w12639217y3041zwnjstr3lcgn/T/tmppUlYbe.schema.json; line:
1, column: 2]
Verifying import...
data: [[u'gandalf', 1955, u'male', u'p@$$', u'WA']]
FAIL
removing ccm cluster test at:
/var/folders/xl/w12639217y3041zwnjstr3lcgn/T/dtest-yrqnIG
==
FAIL: json_tools_test (json_tools_test.TestJson)
--
Traceback (most recent call last):
File /Users/mikhail/Documents/workspace/cassandra-dtest/json_tools_test.py,
line 91, in json_tools_test
[u'gandalf', 1955, u'male', u'p@$$', u'WA'] ] )
AssertionError: Element counts were not equal:
First has 0, Second has 1: [u'frodo', 1985, u'male', u'pass@', u'CA']
First has 0, Second has 1: [u'sam', 1980, u'male', u'@pass', u'NY']
begin captured logging
dtest: DEBUG: cluster ccm directory:
/var/folders/xl/w12639217y3041zwnjstr3lcgn/T/dtest-yrqnIG
dtest: DEBUG: Starting cluster...
dtest: DEBUG: Version: 2.1.0
dtest: DEBUG: Getting CQLSH...
dtest: DEBUG: Inserting data...
[jira] [Created] (CASSANDRA-7548) Disk I/O priority control for Compaction and Flusher
Hanson created CASSANDRA-7548: - Summary: Disk I/O priority control for Compaction and Flusher Key: CASSANDRA-7548 URL: https://issues.apache.org/jira/browse/CASSANDRA-7548 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Hanson Disk I/O priority: Memtables Flusher shall have higher priority than Compaction. This is to avoid DB Insert/Update hung (spikes) during SSTables Compaction. The Compaction shall be able to detect the in-progress of Memtables flushing, and slow down itself for disk I/O. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (CASSANDRA-7549) Heavy Disk Read I/O
Hanson created CASSANDRA-7549: - Summary: Heavy Disk Read I/O Key: CASSANDRA-7549 URL: https://issues.apache.org/jira/browse/CASSANDRA-7549 Project: Cassandra Issue Type: Improvement Environment: Cassandra 2.0.6 Reporter: Hanson We observed heavy disk Read I/O, sometimes almost ~100% disk I/O %util. The block size for Disk Read seems too small per “iostat”: - DB Query: ~40KB per read - SSTables Compaction : ~120KB per read Could it use larger block size for Disk Read? (from Cassandra or OS disk driver tuning) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (CASSANDRA-7550) Reduce tombstones
Hanson created CASSANDRA-7550: - Summary: Reduce tombstones Key: CASSANDRA-7550 URL: https://issues.apache.org/jira/browse/CASSANDRA-7550 Project: Cassandra Issue Type: Improvement Reporter: Hanson If a record is still in Memtable, the Delete to that record shall be done in memory (Make Compaction in memory), instead of placing a tombstone. This way it can save disk I/O and space a lot for records with short lifecycle. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (CASSANDRA-7550) Reduce tombstones
[ https://issues.apache.org/jira/browse/CASSANDRA-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko resolved CASSANDRA-7550. -- Resolution: Invalid Unfortunately, it's not that easy. You can only do it if you are sure that there is nothing covered by the tombstone in any of the sstables on this, or any other node in the cluster. Reduce tombstones - Key: CASSANDRA-7550 URL: https://issues.apache.org/jira/browse/CASSANDRA-7550 Project: Cassandra Issue Type: Improvement Reporter: Hanson If a record is still in Memtable, the Delete to that record shall be done in memory (Make Compaction in memory), instead of placing a tombstone. This way it can save disk I/O and space a lot for records with short lifecycle. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-7547) EC2 seed provider using DataStax Reflector
[ https://issues.apache.org/jira/browse/CASSANDRA-7547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pekka Enberg updated CASSANDRA-7547: Summary: EC2 seed provider using DataStax Reflector (was: EC2 seed provider using Datastax Reflector) EC2 seed provider using DataStax Reflector -- Key: CASSANDRA-7547 URL: https://issues.apache.org/jira/browse/CASSANDRA-7547 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Pekka Enberg Priority: Minor Attachments: 0001-EC2-seed-provider.patch This is a request for comments. I am using this to build our EC2 AMIs but I thought I'd ask if this makes sense as a generic feature for Cassandra. Cassandra cluster auto-configuration on EC2 uses the Datastax reflector service for discovering seed nodes. Instead of relying on external scripts, this patch implements EC2 seed provider that uses the Datastax reflector service. This is particularly useful for EC2 AMIs that don't include a complete userspace (such as those built with OSv) where we ideally want to push as much configuration to the application itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7232) Enable live replay of commit logs
[
https://issues.apache.org/jira/browse/CASSANDRA-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061934#comment-14061934
]
Lyuben Todorov commented on CASSANDRA-7232:
---
So we want the nodetool command to accept a list of commit logs and the
point-in-time to recover to? Something along the lines of: nodetool
replaycommitlogs ks cf point-in-time list-of-commitlogs
eg:
{{nodetool replaycommitogs demodb usertable -point-in-time 1404777106690
C*_DIR/data/commitlogs/CommitLog-4-1404777106690.log
C*_DIR/data/commitlogs/CommitLog-4-1404777106696.log
C*_DIR/data/commitlogs/CommitLog-4-1404777106698.log}}
Enable live replay of commit logs
-
Key: CASSANDRA-7232
URL: https://issues.apache.org/jira/browse/CASSANDRA-7232
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Fix For: 2.0.10
Attachments:
0001-Expose-CommitLog-recover-to-JMX-add-nodetool-cmd-for.patch,
0001-TRUNK-JMX-and-nodetool-cmd-for-commitlog-replay.patch
Replaying commit logs takes a restart but restoring sstables can be an online
operation with refresh. In order to restore a point-in-time without a
restart, the node needs to live replay the commit logs from JMX and a
nodetool command.
nodetool refreshcommitlogs keyspace table
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7232) Enable live replay of commit logs
[ https://issues.apache.org/jira/browse/CASSANDRA-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061965#comment-14061965 ] Benedict commented on CASSANDRA-7232: - That looks a bit error prone: it should at least take a list of directories. However I'm a little uncomfortable passing node-local paths remotely over nodetool in general, especially when we have very little to help ensure we got it right (we should at least print a list of restored log files when done). In general, I think it would be safer to specify the commit log restore/archive location at startup, but not automatically restore them if specified (have a separate option); then the nodetool command always uses that path, and only accepts a point-in-time option. This is ripe for some bikeshedding, though. [~jbellis] [~thobbs] [~pmcfadin]? Enable live replay of commit logs - Key: CASSANDRA-7232 URL: https://issues.apache.org/jira/browse/CASSANDRA-7232 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Patrick McFadin Assignee: Lyuben Todorov Priority: Minor Fix For: 2.0.10 Attachments: 0001-Expose-CommitLog-recover-to-JMX-add-nodetool-cmd-for.patch, 0001-TRUNK-JMX-and-nodetool-cmd-for-commitlog-replay.patch Replaying commit logs takes a restart but restoring sstables can be an online operation with refresh. In order to restore a point-in-time without a restart, the node needs to live replay the commit logs from JMX and a nodetool command. nodetool refreshcommitlogs keyspace table -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7550) Reduce tombstones
[ https://issues.apache.org/jira/browse/CASSANDRA-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061974#comment-14061974 ] Hanson commented on CASSANDRA-7550: --- I mean that record is newly inserted and has not been flushed into SSTable yet, only existed in memory at the moment to be deleted. So it shall be safety to do the Delete in memory without putting a tombstone. In our use case, the records only stay in DB for hours (random duration) and will then be deleted. Reduce tombstones - Key: CASSANDRA-7550 URL: https://issues.apache.org/jira/browse/CASSANDRA-7550 Project: Cassandra Issue Type: Improvement Reporter: Hanson If a record is still in Memtable, the Delete to that record shall be done in memory (Make Compaction in memory), instead of placing a tombstone. This way it can save disk I/O and space a lot for records with short lifecycle. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7550) Reduce tombstones
[ https://issues.apache.org/jira/browse/CASSANDRA-7550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061979#comment-14061979 ] Aleksey Yeschenko commented on CASSANDRA-7550: -- You can't (generally) know for a fact that a partition with the same key/a cell with the same name doesn't exist elsewhere on disk or on another node. Reduce tombstones - Key: CASSANDRA-7550 URL: https://issues.apache.org/jira/browse/CASSANDRA-7550 Project: Cassandra Issue Type: Improvement Reporter: Hanson If a record is still in Memtable, the Delete to that record shall be done in memory (Make Compaction in memory), instead of placing a tombstone. This way it can save disk I/O and space a lot for records with short lifecycle. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7549) Heavy Disk Read I/O
[ https://issues.apache.org/jira/browse/CASSANDRA-7549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061991#comment-14061991 ] Benedict commented on CASSANDRA-7549: - Can you clarify what this ticket is hoping to achieve? Too small compared to what? The default Cassandra block size is 64Kb, and since currently reading data in a block requires a sequential scan of data preceding the target record, reading ~40Kb is about correct for reading a very small row with an 8Kb read ahead, and zero page cache utilisation (i.e. completely random reads for a substantially larger than memory dataset). Increasing block size is likely to reduce performance for this kind of workload, not improve it. Heavy Disk Read I/O --- Key: CASSANDRA-7549 URL: https://issues.apache.org/jira/browse/CASSANDRA-7549 Project: Cassandra Issue Type: Improvement Environment: Cassandra 2.0.6 Reporter: Hanson We observed heavy disk Read I/O, sometimes almost ~100% disk I/O %util. The block size for Disk Read seems too small per “iostat”: - DB Query: ~40KB per read - SSTables Compaction : ~120KB per read Could it use larger block size for Disk Read? (from Cassandra or OS disk driver tuning) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (CASSANDRA-7542) Reduce CAS contention
[ https://issues.apache.org/jira/browse/CASSANDRA-7542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams reassigned CASSANDRA-7542: --- Assignee: Benedict I think Benedict had some ideas here. Reduce CAS contention - Key: CASSANDRA-7542 URL: https://issues.apache.org/jira/browse/CASSANDRA-7542 Project: Cassandra Issue Type: Improvement Reporter: sankalp kohli Assignee: Benedict CAS updates on same CQL partition can lead to heavy contention inside C*. I am looking for simple ways(no algorithmic changes) to reduce contention as the penalty of it is high in terms of latency, specially for reads. We can put some sort of synchronization on CQL partition at StorageProxy level. This will reduce contention at least for all requests landing on one box for same partition. Here is an example of why it will help: 1) Say 1 write and 2 read CAS requests for the same partition key is send to C* in parallel. 2) Since client is token-aware, it sends these 3 request to the same C* instance A. (Lets assume that all 3 requests goto same instance A) 3) In this C* instance A, all 3 CAS requests will contend with each other in Paxos. (This is bad) To improve contention in 3), what I am proposing is to add a lock on partition key similar to what we do in PaxosState.java to serialize these 3 requests. This will remove the contention and improve performance as these 3 requests will not collide with each other. Another improvement we can do in client is to pick a deterministic live replica for a given partition doing CAS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-7544) Allow storage port to be configurable per node
[ https://issues.apache.org/jira/browse/CASSANDRA-7544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-7544: Reviewer: Brandon Williams Fix Version/s: 3.0 Allow storage port to be configurable per node -- Key: CASSANDRA-7544 URL: https://issues.apache.org/jira/browse/CASSANDRA-7544 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Sam Overton Assignee: Sam Overton Fix For: 3.0 Currently storage_port must be configured identically on all nodes in a cluster and it is assumed that this is the case when connecting to a remote node. This prevents running in any environment that requires multiple nodes to be able to bind to the same network interface, such as with many automatic provisioning/deployment frameworks. The current solutions seems to be * use a separate network interface for each node deployed to the same box. This puts a big requirement on IP allocation at large scale. * allow multiple clusters to be provisioned from the same resource pool, but restrict allocation to a maximum of one node per host from each cluster, assuming each cluster is running on a different storage port. It would make operations much simpler in these kind of environments if the environment provisioning the resources could assign the ports to be used when bringing up a new node on shared hardware. The changes required would be at least the following: 1. configure seeds as IP:port instead of just IP 2. gossip the storage port as part of a node's ApplicationState 3. refer internally to nodes by hostID instead of IP, since there will be multiple nodes with the same IP (1) (2) are mostly trivial and I already have a patch for these. The bulk of the work to enable this is (3), and I would structure this as a separate pre-requisite patch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (CASSANDRA-7543) Assertion error when compacting large row with map//list field or range tombstone
[
https://issues.apache.org/jira/browse/CASSANDRA-7543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams reassigned CASSANDRA-7543:
---
Assignee: Yuki Morishita
Assertion error when compacting large row with map//list field or range
tombstone
-
Key: CASSANDRA-7543
URL: https://issues.apache.org/jira/browse/CASSANDRA-7543
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: linux
Reporter: Matt Byrd
Assignee: Yuki Morishita
Labels: compaction, map
Hi,
So in a couple of clusters we're hitting this problem when compacting large
rows with a schema which contains the map data-type.
Here is an example of the error:
{code}
java.lang.AssertionError: incorrect row data size 87776427 written to
/cassandra/X/Y/X-Y-tmp-ic-2381-Data.db; correct is 87845952
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163)
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:208)
{code}
I have a python script which reproduces the problem, by just writing lots of
data to a single partition key with a schema that contains the map data-type.
I added some debug logging and found that the difference in bytes seen in the
reproduction (255) was due to the following pieces of data being written:
{code}
DEBUG [CompactionExecutor:3] 2014-07-13 00:38:42,891 ColumnIndex.java (line
168) DATASIZE writeOpenedMarker columnIndex:
org.apache.cassandra.db.ColumnIndex$Builder@6678a9d0 firstColumn:
[java.nio.HeapByteBuffer[pos=0 lim=34 cap=34], java.nio.HeapByteBuffer[pos=0
lim=34 cap=34]](deletedAt=1405237116014999, localDeletion=1405237116)
startPosition: 262476 endPosition: 262561 diff: 85
DEBUG [CompactionExecutor:3] 2014-07-13 00:38:43,007 ColumnIndex.java (line
168) DATASIZE writeOpenedMarker columnIndex:
org.apache.cassandra.db.ColumnIndex$Builder@6678a9d0 firstColumn:
org.apache.cassandra.db.Column@3e5b5939 startPosition: 328157 endPosition:
328242 diff: 85
DEBUG [CompactionExecutor:3] 2014-07-13 00:38:44,159 ColumnIndex.java (line
168) DATASIZE writeOpenedMarker columnIndex:
org.apache.cassandra.db.ColumnIndex$Builder@6678a9d0 firstColumn:
org.apache.cassandra.db.Column@fc3299b startPosition: 984105 endPosition:
984190 diff: 85
{code}
So looking at the code you can see that there are extra range tombstones
written on the column index border (in ColumnIndex where
tombstoneTracker.writeOpenedMarker is called) which aren't accounted for in
LazilyCompactedRow.columnSerializedSize.
This is where the difference comes from in the assertion error, so the
solution is just to account for this data.
I have a patch which does just this, by keeping track of the extra data
written out via tombstoneTracker.writeOpenedMarker in ColumnIndex and adding
it back to the dataSize in LazilyCompactedRow.java, where it serialises out
the row size.
After applying the patch the reproduction stops producing the AssertionError.
I know this is not a problem in 2.0 + because of singe pass compaction,
however there are lots of 1.2 clusters out there still which might run into
this.
Please let me know if you've any questions.
Thanks,
Matt
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (CASSANDRA-7547) EC2 seed provider using DataStax Reflector
[ https://issues.apache.org/jira/browse/CASSANDRA-7547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-7547: -- Reviewer: Brandon Williams [~brandon.williams] to review EC2 seed provider using DataStax Reflector -- Key: CASSANDRA-7547 URL: https://issues.apache.org/jira/browse/CASSANDRA-7547 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Pekka Enberg Priority: Minor Attachments: 0001-EC2-seed-provider.patch This is a request for comments. I am using this to build our EC2 AMIs but I thought I'd ask if this makes sense as a generic feature for Cassandra. Cassandra cluster auto-configuration on EC2 uses the Datastax reflector service for discovering seed nodes. Instead of relying on external scripts, this patch implements EC2 seed provider that uses the Datastax reflector service. This is particularly useful for EC2 AMIs that don't include a complete userspace (such as those built with OSv) where we ideally want to push as much configuration to the application itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7547) EC2 seed provider using DataStax Reflector
[ https://issues.apache.org/jira/browse/CASSANDRA-7547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062161#comment-14062161 ] Brandon Williams commented on CASSANDRA-7547: - I have to say I'm pretty -1 on something that connects to a datastax.com host and relies on php. EC2 seed provider using DataStax Reflector -- Key: CASSANDRA-7547 URL: https://issues.apache.org/jira/browse/CASSANDRA-7547 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Pekka Enberg Priority: Minor Attachments: 0001-EC2-seed-provider.patch This is a request for comments. I am using this to build our EC2 AMIs but I thought I'd ask if this makes sense as a generic feature for Cassandra. Cassandra cluster auto-configuration on EC2 uses the Datastax reflector service for discovering seed nodes. Instead of relying on external scripts, this patch implements EC2 seed provider that uses the Datastax reflector service. This is particularly useful for EC2 AMIs that don't include a complete userspace (such as those built with OSv) where we ideally want to push as much configuration to the application itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[06/10] git commit: merge from 2.0
merge from 2.0
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/04017616
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/04017616
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/04017616
Branch: refs/heads/cassandra-2.1.0
Commit: 04017616e8cfb7952fcc909d69aa644f6cb981f3
Parents: 9460503 1630ebf
Author: Jonathan Ellis jbel...@apache.org
Authored: Tue Jul 15 09:57:21 2014 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Tue Jul 15 09:57:21 2014 -0500
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/service/StorageService.java | 5 +
2 files changed, 6 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/04017616/CHANGES.txt
--
diff --cc CHANGES.txt
index d3390f9,d339309..193cff8
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,32 -1,7 +1,33 @@@
-2.0.10
+2.1.0-final
+ * (Windows) fix startup when WMI memory query fails (CASSANDRA-7505)
+ * Anti-compaction proceeds if any part of the repair failed (CASANDRA-7521)
+Merged from 2.0:
+ * (Windows) force range-based repair to non-sequential mode (CASSANDRA-7541)
* Fix range merging when DES scores are zero (CASSANDRA-7535)
* Warn when SSL certificates have expired (CASSANDRA-7528)
+ * Fix error when doing reversed queries with static columns (CASSANDRA-7490)
+Merged from 1.2:
+ * Set correct stream ID on responses when non-Exception Throwables
+ are thrown while handling native protocol messages (CASSANDRA-7470)
+
+
+2.1.0-rc3
+ * Consider expiry when reconciling otherwise equal cells (CASSANDRA-7403)
+ * Introduce CQL support for stress tool (CASSANDRA-6146)
+ * Fix ClassCastException processing expired messages (CASSANDRA-7496)
+ * Fix prepared marker for collections inside UDT (CASSANDRA-7472)
+ * Remove left-over populate_io_cache_on_flush and replicate_on_write
+ uses (CASSANDRA-7493)
+ * (Windows) handle spaces in path names (CASSANDRA-7451)
+ * Ensure writes have completed after dropping a table, before recycling
+ commit log segments (CASSANDRA-7437)
+ * Remove left-over rows_per_partition_to_cache (CASSANDRA-7493)
+ * Fix error when CONTAINS is used with a bind marker (CASSANDRA-7502)
+ * Properly reject unknown UDT field (CASSANDRA-7484)
+Merged from 2.0:
+ * Fix CC#collectTimeOrderedData() tombstone optimisations (CASSANDRA-7394)
+ * Support DISTINCT for static columns and fix behaviour when DISTINC is
+ not use (CASSANDRA-7305).
* Workaround JVM NPE on JMX bind failure (CASSANDRA-7254)
* Fix race in FileCacheService RemovalListener (CASSANDRA-7278)
* Fix inconsistent use of consistencyForCommit that allowed LOCAL_QUORUM
http://git-wip-us.apache.org/repos/asf/cassandra/blob/04017616/src/java/org/apache/cassandra/service/StorageService.java
--
diff --cc src/java/org/apache/cassandra/service/StorageService.java
index 3fd3781,c6f4ec9..8ace958
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@@ -2570,10 -2504,10 +2570,15 @@@ public class StorageService extends Not
if (Keyspace.SYSTEM_KS.equals(keyspace) || ranges.isEmpty())
return 0;
-final int cmd = nextRepairCommand.incrementAndGet();
+int cmd = nextRepairCommand.incrementAndGet();
if (ranges.size() 0)
{
-new Thread(createRepairTask(cmd, keyspace, ranges, isSequential,
dataCenters, hosts, columnFamilies)).start();
++if (!FBUtilities.isUnix() isSequential)
++{
++logger.warn(Snapshot-based repair is not yet supported on
Windows. Reverting to parallel repair.);
++isSequential = false;
++}
+new Thread(createRepairTask(cmd, keyspace, ranges, isSequential,
dataCenters, hosts, fullRepair, columnFamilies)).start();
}
return cmd;
}
[02/10] git commit: (Windows) force range-based repair to non-sequential mode patch by Josh McKenzie; reviewed by jbellis for CASSANDRA-7541
(Windows) force range-based repair to non-sequential mode
patch by Josh McKenzie; reviewed by jbellis for CASSANDRA-7541
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1630ebf6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1630ebf6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1630ebf6
Branch: refs/heads/cassandra-2.1.0
Commit: 1630ebf6afa6c3ceb4751dfd9bb3e444b99211e2
Parents: a454461
Author: Jonathan Ellis jbel...@apache.org
Authored: Mon Jul 14 11:11:07 2014 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Mon Jul 14 11:11:07 2014 -0500
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/service/StorageService.java | 6 ++
2 files changed, 7 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1630ebf6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e1c8922..d339309 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.10
+ * (Windows) force range-based repair to non-sequential mode (CASSANDRA-7541)
* Fix range merging when DES scores are zero (CASSANDRA-7535)
* Warn when SSL certificates have expired (CASSANDRA-7528)
* Workaround JVM NPE on JMX bind failure (CASSANDRA-7254)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1630ebf6/src/java/org/apache/cassandra/service/StorageService.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageService.java
b/src/java/org/apache/cassandra/service/StorageService.java
index 77ad4f0..c6f4ec9 100644
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@ -2544,6 +2544,12 @@ public class StorageService extends
NotificationBroadcasterSupport implements IE
logger.info(starting user-requested repair of range {} for keyspace
{} and column families {},
repairingRange, keyspaceName, columnFamilies);
+
+if (!FBUtilities.isUnix() isSequential)
+{
+logger.warn(Snapshot-based repair is not yet supported on
Windows. Reverting to parallel repair.);
+isSequential = false;
+}
return forceRepairAsync(keyspaceName, isSequential, dataCenters,
hosts, repairingRange, columnFamilies);
}
[10/10] git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3966935b Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3966935b Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3966935b Branch: refs/heads/trunk Commit: 3966935b43d4007123869e18d93bab4808e0efbe Parents: 8243b6a 52c6dfb Author: Jonathan Ellis jbel...@apache.org Authored: Tue Jul 15 09:57:34 2014 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Tue Jul 15 09:57:34 2014 -0500 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/service/StorageService.java | 5 + 2 files changed, 6 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/3966935b/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/3966935b/src/java/org/apache/cassandra/service/StorageService.java --
[04/10] git commit: (Windows) force range-based repair to non-sequential mode patch by Josh McKenzie; reviewed by jbellis for CASSANDRA-7541
(Windows) force range-based repair to non-sequential mode
patch by Josh McKenzie; reviewed by jbellis for CASSANDRA-7541
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1630ebf6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1630ebf6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1630ebf6
Branch: refs/heads/trunk
Commit: 1630ebf6afa6c3ceb4751dfd9bb3e444b99211e2
Parents: a454461
Author: Jonathan Ellis jbel...@apache.org
Authored: Mon Jul 14 11:11:07 2014 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Mon Jul 14 11:11:07 2014 -0500
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/service/StorageService.java | 6 ++
2 files changed, 7 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1630ebf6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e1c8922..d339309 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.10
+ * (Windows) force range-based repair to non-sequential mode (CASSANDRA-7541)
* Fix range merging when DES scores are zero (CASSANDRA-7535)
* Warn when SSL certificates have expired (CASSANDRA-7528)
* Workaround JVM NPE on JMX bind failure (CASSANDRA-7254)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1630ebf6/src/java/org/apache/cassandra/service/StorageService.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageService.java
b/src/java/org/apache/cassandra/service/StorageService.java
index 77ad4f0..c6f4ec9 100644
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@ -2544,6 +2544,12 @@ public class StorageService extends
NotificationBroadcasterSupport implements IE
logger.info(starting user-requested repair of range {} for keyspace
{} and column families {},
repairingRange, keyspaceName, columnFamilies);
+
+if (!FBUtilities.isUnix() isSequential)
+{
+logger.warn(Snapshot-based repair is not yet supported on
Windows. Reverting to parallel repair.);
+isSequential = false;
+}
return forceRepairAsync(keyspaceName, isSequential, dataCenters,
hosts, repairingRange, columnFamilies);
}
[08/10] git commit: Merge branch 'cassandra-2.1.0' into cassandra-2.1
Merge branch 'cassandra-2.1.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/52c6dfb8 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/52c6dfb8 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/52c6dfb8 Branch: refs/heads/trunk Commit: 52c6dfb8f56b7bbf43fd1f8eac0052a1c8f24d12 Parents: f45a4e3 0401761 Author: Jonathan Ellis jbel...@apache.org Authored: Tue Jul 15 09:57:27 2014 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Tue Jul 15 09:57:27 2014 -0500 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/service/StorageService.java | 5 + 2 files changed, 6 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/52c6dfb8/CHANGES.txt --
[jira] [Commented] (CASSANDRA-7541) Windows: IOException when repairing a range of tokens
[ https://issues.apache.org/jira/browse/CASSANDRA-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062174#comment-14062174 ] Jonathan Ellis commented on CASSANDRA-7541: --- committed Windows: IOException when repairing a range of tokens - Key: CASSANDRA-7541 URL: https://issues.apache.org/jira/browse/CASSANDRA-7541 Project: Cassandra Issue Type: Bug Reporter: Joshua McKenzie Assignee: Joshua McKenzie Priority: Minor Labels: Windows Fix For: 2.0.10, 2.1.0 Attachments: 7541_2.1_v1.txt, 7541_v1.txt Looks like we missed this in CASSANDRA-6907 - range-based repair still default to snapshot-based on Windows. -- This message was sent by Atlassian JIRA (v6.2#6252)
[07/10] git commit: merge from 2.0
merge from 2.0
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/04017616
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/04017616
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/04017616
Branch: refs/heads/trunk
Commit: 04017616e8cfb7952fcc909d69aa644f6cb981f3
Parents: 9460503 1630ebf
Author: Jonathan Ellis jbel...@apache.org
Authored: Tue Jul 15 09:57:21 2014 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Tue Jul 15 09:57:21 2014 -0500
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/service/StorageService.java | 5 +
2 files changed, 6 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/04017616/CHANGES.txt
--
diff --cc CHANGES.txt
index d3390f9,d339309..193cff8
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,32 -1,7 +1,33 @@@
-2.0.10
+2.1.0-final
+ * (Windows) fix startup when WMI memory query fails (CASSANDRA-7505)
+ * Anti-compaction proceeds if any part of the repair failed (CASANDRA-7521)
+Merged from 2.0:
+ * (Windows) force range-based repair to non-sequential mode (CASSANDRA-7541)
* Fix range merging when DES scores are zero (CASSANDRA-7535)
* Warn when SSL certificates have expired (CASSANDRA-7528)
+ * Fix error when doing reversed queries with static columns (CASSANDRA-7490)
+Merged from 1.2:
+ * Set correct stream ID on responses when non-Exception Throwables
+ are thrown while handling native protocol messages (CASSANDRA-7470)
+
+
+2.1.0-rc3
+ * Consider expiry when reconciling otherwise equal cells (CASSANDRA-7403)
+ * Introduce CQL support for stress tool (CASSANDRA-6146)
+ * Fix ClassCastException processing expired messages (CASSANDRA-7496)
+ * Fix prepared marker for collections inside UDT (CASSANDRA-7472)
+ * Remove left-over populate_io_cache_on_flush and replicate_on_write
+ uses (CASSANDRA-7493)
+ * (Windows) handle spaces in path names (CASSANDRA-7451)
+ * Ensure writes have completed after dropping a table, before recycling
+ commit log segments (CASSANDRA-7437)
+ * Remove left-over rows_per_partition_to_cache (CASSANDRA-7493)
+ * Fix error when CONTAINS is used with a bind marker (CASSANDRA-7502)
+ * Properly reject unknown UDT field (CASSANDRA-7484)
+Merged from 2.0:
+ * Fix CC#collectTimeOrderedData() tombstone optimisations (CASSANDRA-7394)
+ * Support DISTINCT for static columns and fix behaviour when DISTINC is
+ not use (CASSANDRA-7305).
* Workaround JVM NPE on JMX bind failure (CASSANDRA-7254)
* Fix race in FileCacheService RemovalListener (CASSANDRA-7278)
* Fix inconsistent use of consistencyForCommit that allowed LOCAL_QUORUM
http://git-wip-us.apache.org/repos/asf/cassandra/blob/04017616/src/java/org/apache/cassandra/service/StorageService.java
--
diff --cc src/java/org/apache/cassandra/service/StorageService.java
index 3fd3781,c6f4ec9..8ace958
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@@ -2570,10 -2504,10 +2570,15 @@@ public class StorageService extends Not
if (Keyspace.SYSTEM_KS.equals(keyspace) || ranges.isEmpty())
return 0;
-final int cmd = nextRepairCommand.incrementAndGet();
+int cmd = nextRepairCommand.incrementAndGet();
if (ranges.size() 0)
{
-new Thread(createRepairTask(cmd, keyspace, ranges, isSequential,
dataCenters, hosts, columnFamilies)).start();
++if (!FBUtilities.isUnix() isSequential)
++{
++logger.warn(Snapshot-based repair is not yet supported on
Windows. Reverting to parallel repair.);
++isSequential = false;
++}
+new Thread(createRepairTask(cmd, keyspace, ranges, isSequential,
dataCenters, hosts, fullRepair, columnFamilies)).start();
}
return cmd;
}
[03/10] git commit: (Windows) force range-based repair to non-sequential mode patch by Josh McKenzie; reviewed by jbellis for CASSANDRA-7541
(Windows) force range-based repair to non-sequential mode
patch by Josh McKenzie; reviewed by jbellis for CASSANDRA-7541
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1630ebf6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1630ebf6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1630ebf6
Branch: refs/heads/cassandra-2.1
Commit: 1630ebf6afa6c3ceb4751dfd9bb3e444b99211e2
Parents: a454461
Author: Jonathan Ellis jbel...@apache.org
Authored: Mon Jul 14 11:11:07 2014 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Mon Jul 14 11:11:07 2014 -0500
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/service/StorageService.java | 6 ++
2 files changed, 7 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1630ebf6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e1c8922..d339309 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.10
+ * (Windows) force range-based repair to non-sequential mode (CASSANDRA-7541)
* Fix range merging when DES scores are zero (CASSANDRA-7535)
* Warn when SSL certificates have expired (CASSANDRA-7528)
* Workaround JVM NPE on JMX bind failure (CASSANDRA-7254)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1630ebf6/src/java/org/apache/cassandra/service/StorageService.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageService.java
b/src/java/org/apache/cassandra/service/StorageService.java
index 77ad4f0..c6f4ec9 100644
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@ -2544,6 +2544,12 @@ public class StorageService extends
NotificationBroadcasterSupport implements IE
logger.info(starting user-requested repair of range {} for keyspace
{} and column families {},
repairingRange, keyspaceName, columnFamilies);
+
+if (!FBUtilities.isUnix() isSequential)
+{
+logger.warn(Snapshot-based repair is not yet supported on
Windows. Reverting to parallel repair.);
+isSequential = false;
+}
return forceRepairAsync(keyspaceName, isSequential, dataCenters,
hosts, repairingRange, columnFamilies);
}
[01/10] git commit: (Windows) force range-based repair to non-sequential mode patch by Josh McKenzie; reviewed by jbellis for CASSANDRA-7541
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.0 a4544615d - 1630ebf6a
refs/heads/cassandra-2.1 f45a4e3a1 - 52c6dfb8f
refs/heads/cassandra-2.1.0 946050371 - 04017616e
refs/heads/trunk 8243b6ad0 - 3966935b4
(Windows) force range-based repair to non-sequential mode
patch by Josh McKenzie; reviewed by jbellis for CASSANDRA-7541
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1630ebf6
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1630ebf6
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1630ebf6
Branch: refs/heads/cassandra-2.0
Commit: 1630ebf6afa6c3ceb4751dfd9bb3e444b99211e2
Parents: a454461
Author: Jonathan Ellis jbel...@apache.org
Authored: Mon Jul 14 11:11:07 2014 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Mon Jul 14 11:11:07 2014 -0500
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/service/StorageService.java | 6 ++
2 files changed, 7 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1630ebf6/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index e1c8922..d339309 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.10
+ * (Windows) force range-based repair to non-sequential mode (CASSANDRA-7541)
* Fix range merging when DES scores are zero (CASSANDRA-7535)
* Warn when SSL certificates have expired (CASSANDRA-7528)
* Workaround JVM NPE on JMX bind failure (CASSANDRA-7254)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1630ebf6/src/java/org/apache/cassandra/service/StorageService.java
--
diff --git a/src/java/org/apache/cassandra/service/StorageService.java
b/src/java/org/apache/cassandra/service/StorageService.java
index 77ad4f0..c6f4ec9 100644
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@ -2544,6 +2544,12 @@ public class StorageService extends
NotificationBroadcasterSupport implements IE
logger.info(starting user-requested repair of range {} for keyspace
{} and column families {},
repairingRange, keyspaceName, columnFamilies);
+
+if (!FBUtilities.isUnix() isSequential)
+{
+logger.warn(Snapshot-based repair is not yet supported on
Windows. Reverting to parallel repair.);
+isSequential = false;
+}
return forceRepairAsync(keyspaceName, isSequential, dataCenters,
hosts, repairingRange, columnFamilies);
}
[09/10] git commit: Merge branch 'cassandra-2.1.0' into cassandra-2.1
Merge branch 'cassandra-2.1.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/52c6dfb8 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/52c6dfb8 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/52c6dfb8 Branch: refs/heads/cassandra-2.1 Commit: 52c6dfb8f56b7bbf43fd1f8eac0052a1c8f24d12 Parents: f45a4e3 0401761 Author: Jonathan Ellis jbel...@apache.org Authored: Tue Jul 15 09:57:27 2014 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Tue Jul 15 09:57:27 2014 -0500 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/service/StorageService.java | 5 + 2 files changed, 6 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/52c6dfb8/CHANGES.txt --
[05/10] git commit: merge from 2.0
merge from 2.0
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/04017616
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/04017616
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/04017616
Branch: refs/heads/cassandra-2.1
Commit: 04017616e8cfb7952fcc909d69aa644f6cb981f3
Parents: 9460503 1630ebf
Author: Jonathan Ellis jbel...@apache.org
Authored: Tue Jul 15 09:57:21 2014 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Tue Jul 15 09:57:21 2014 -0500
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/service/StorageService.java | 5 +
2 files changed, 6 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/04017616/CHANGES.txt
--
diff --cc CHANGES.txt
index d3390f9,d339309..193cff8
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,32 -1,7 +1,33 @@@
-2.0.10
+2.1.0-final
+ * (Windows) fix startup when WMI memory query fails (CASSANDRA-7505)
+ * Anti-compaction proceeds if any part of the repair failed (CASANDRA-7521)
+Merged from 2.0:
+ * (Windows) force range-based repair to non-sequential mode (CASSANDRA-7541)
* Fix range merging when DES scores are zero (CASSANDRA-7535)
* Warn when SSL certificates have expired (CASSANDRA-7528)
+ * Fix error when doing reversed queries with static columns (CASSANDRA-7490)
+Merged from 1.2:
+ * Set correct stream ID on responses when non-Exception Throwables
+ are thrown while handling native protocol messages (CASSANDRA-7470)
+
+
+2.1.0-rc3
+ * Consider expiry when reconciling otherwise equal cells (CASSANDRA-7403)
+ * Introduce CQL support for stress tool (CASSANDRA-6146)
+ * Fix ClassCastException processing expired messages (CASSANDRA-7496)
+ * Fix prepared marker for collections inside UDT (CASSANDRA-7472)
+ * Remove left-over populate_io_cache_on_flush and replicate_on_write
+ uses (CASSANDRA-7493)
+ * (Windows) handle spaces in path names (CASSANDRA-7451)
+ * Ensure writes have completed after dropping a table, before recycling
+ commit log segments (CASSANDRA-7437)
+ * Remove left-over rows_per_partition_to_cache (CASSANDRA-7493)
+ * Fix error when CONTAINS is used with a bind marker (CASSANDRA-7502)
+ * Properly reject unknown UDT field (CASSANDRA-7484)
+Merged from 2.0:
+ * Fix CC#collectTimeOrderedData() tombstone optimisations (CASSANDRA-7394)
+ * Support DISTINCT for static columns and fix behaviour when DISTINC is
+ not use (CASSANDRA-7305).
* Workaround JVM NPE on JMX bind failure (CASSANDRA-7254)
* Fix race in FileCacheService RemovalListener (CASSANDRA-7278)
* Fix inconsistent use of consistencyForCommit that allowed LOCAL_QUORUM
http://git-wip-us.apache.org/repos/asf/cassandra/blob/04017616/src/java/org/apache/cassandra/service/StorageService.java
--
diff --cc src/java/org/apache/cassandra/service/StorageService.java
index 3fd3781,c6f4ec9..8ace958
--- a/src/java/org/apache/cassandra/service/StorageService.java
+++ b/src/java/org/apache/cassandra/service/StorageService.java
@@@ -2570,10 -2504,10 +2570,15 @@@ public class StorageService extends Not
if (Keyspace.SYSTEM_KS.equals(keyspace) || ranges.isEmpty())
return 0;
-final int cmd = nextRepairCommand.incrementAndGet();
+int cmd = nextRepairCommand.incrementAndGet();
if (ranges.size() 0)
{
-new Thread(createRepairTask(cmd, keyspace, ranges, isSequential,
dataCenters, hosts, columnFamilies)).start();
++if (!FBUtilities.isUnix() isSequential)
++{
++logger.warn(Snapshot-based repair is not yet supported on
Windows. Reverting to parallel repair.);
++isSequential = false;
++}
+new Thread(createRepairTask(cmd, keyspace, ranges, isSequential,
dataCenters, hosts, fullRepair, columnFamilies)).start();
}
return cmd;
}
[jira] [Commented] (CASSANDRA-7547) EC2 seed provider using DataStax Reflector
[ https://issues.apache.org/jira/browse/CASSANDRA-7547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062183#comment-14062183 ] Pekka Enberg commented on CASSANDRA-7547: - Would the patch be more palatable without the default? The infrastructure is open source so you can set up your own Reflector service as well: https://github.com/riptano/ComboAMI EC2 seed provider using DataStax Reflector -- Key: CASSANDRA-7547 URL: https://issues.apache.org/jira/browse/CASSANDRA-7547 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Pekka Enberg Priority: Minor Attachments: 0001-EC2-seed-provider.patch This is a request for comments. I am using this to build our EC2 AMIs but I thought I'd ask if this makes sense as a generic feature for Cassandra. Cassandra cluster auto-configuration on EC2 uses the Datastax reflector service for discovering seed nodes. Instead of relying on external scripts, this patch implements EC2 seed provider that uses the Datastax reflector service. This is particularly useful for EC2 AMIs that don't include a complete userspace (such as those built with OSv) where we ideally want to push as much configuration to the application itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-7551) improve 2.1 flush defaults
[ https://issues.apache.org/jira/browse/CASSANDRA-7551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-7551: -- Attachment: 7551.txt Patch to adapt to multiple disks, same logic as concurrent_compactors. improve 2.1 flush defaults -- Key: CASSANDRA-7551 URL: https://issues.apache.org/jira/browse/CASSANDRA-7551 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Jonathan Ellis Assignee: Jonathan Ellis Fix For: 2.1.0 Attachments: 7551.txt Default of mfw=2 and mct=0.4 does a poor job saturating multi-disk systems. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (CASSANDRA-7551) improve 2.1 flush defaults
Jonathan Ellis created CASSANDRA-7551: - Summary: improve 2.1 flush defaults Key: CASSANDRA-7551 URL: https://issues.apache.org/jira/browse/CASSANDRA-7551 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Jonathan Ellis Assignee: Jonathan Ellis Fix For: 2.1.0 Default of mfw=2 and mct=0.4 does a poor job saturating multi-disk systems. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7547) EC2 seed provider using DataStax Reflector
[ https://issues.apache.org/jira/browse/CASSANDRA-7547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062210#comment-14062210 ] Pekka Enberg commented on CASSANDRA-7547: - OK, any suggestions how to package that in a way that makes it possible for Cassandra to pick it up? EC2 seed provider using DataStax Reflector -- Key: CASSANDRA-7547 URL: https://issues.apache.org/jira/browse/CASSANDRA-7547 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Pekka Enberg Priority: Minor Attachments: 0001-EC2-seed-provider.patch This is a request for comments. I am using this to build our EC2 AMIs but I thought I'd ask if this makes sense as a generic feature for Cassandra. Cassandra cluster auto-configuration on EC2 uses the Datastax reflector service for discovering seed nodes. Instead of relying on external scripts, this patch implements EC2 seed provider that uses the Datastax reflector service. This is particularly useful for EC2 AMIs that don't include a complete userspace (such as those built with OSv) where we ideally want to push as much configuration to the application itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
buildbot success in ASF Buildbot on cassandra-2.0
The Buildbot has detected a restored build on builder cassandra-2.0 while building cassandra. Full details are available at: http://ci.apache.org/builders/cassandra-2.0/builds/117 Buildbot URL: http://ci.apache.org/ Buildslave for this Build: portunus_ubuntu Build Reason: scheduler Build Source Stamp: [branch cassandra-2.0] 1630ebf6afa6c3ceb4751dfd9bb3e444b99211e2 Blamelist: Jonathan Ellis jbel...@apache.org Build succeeded! sincerely, -The Buildbot
[jira] [Commented] (CASSANDRA-4718) More-efficient ExecutorService for improved throughput
[ https://issues.apache.org/jira/browse/CASSANDRA-4718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062217#comment-14062217 ] Jonathan Ellis commented on CASSANDRA-4718: --- To summarize for those browsing, the primary result here was the introduction of SharedExecutorPool: https://github.com/apache/cassandra/blob/cassandra-2.1.0/src/java/org/apache/cassandra/concurrent/SharedExecutorPool.java More-efficient ExecutorService for improved throughput -- Key: CASSANDRA-4718 URL: https://issues.apache.org/jira/browse/CASSANDRA-4718 Project: Cassandra Issue Type: Improvement Reporter: Jonathan Ellis Assignee: Benedict Priority: Minor Labels: performance Fix For: 2.1 rc1 Attachments: 4718-v1.patch, E100M_summary_key_s.svg, E10M_summary_key_s.svg, E600M_summary_key_s.svg, PerThreadQueue.java, austin_diskbound_read.svg, aws.svg, aws_read.svg, backpressure-stress.out.txt, baq vs trunk.png, belliotsmith_branches-stress.out.txt, jason_read.svg, jason_read_latency.svg, jason_run1.svg, jason_run2.svg, jason_run3.svg, jason_write.svg, op costs of various queues.ods, stress op rate with various queues.ods, stress_2014May15.txt, stress_2014May16.txt, v1-stress.out Currently all our execution stages dequeue tasks one at a time. This can result in contention between producers and consumers (although we do our best to minimize this by using LinkedBlockingQueue). One approach to mitigating this would be to make consumer threads do more work in bulk instead of just one task per dequeue. (Producer threads tend to be single-task oriented by nature, so I don't see an equivalent opportunity there.) BlockingQueue has a drainTo(collection, int) method that would be perfect for this. However, no ExecutorService in the jdk supports using drainTo, nor could I google one. What I would like to do here is create just such a beast and wire it into (at least) the write and read stages. (Other possible candidates for such an optimization, such as the CommitLog and OutboundTCPConnection, are not ExecutorService-based and will need to be one-offs.) AbstractExecutorService may be useful. The implementations of ICommitLogExecutorService may also be useful. (Despite the name these are not actual ExecutorServices, although they share the most important properties of one.) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7547) EC2 seed provider using DataStax Reflector
[ https://issues.apache.org/jira/browse/CASSANDRA-7547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062237#comment-14062237 ] Joaquin Casares commented on CASSANDRA-7547: Before we go any further with the using the DataStax reflector, you should consider its initial purpose was to find a simple way for individual nodes to cluster together on startup without initially knowing anything about each other. The current reflector is built with a short-term memory of about 10 minutes to get the list of seeds. If a node is slow to boot and comes in on the 11th minute, it will never know of its peers. If a pre-chosen seed node is slow to boot, the nodes may never properly cluster together. This is important because the seed provider is pinged multiple times during the lifetime of the cluster, mainly during periods of topological changes: removal, bootstrap, replace, etc. If these happen outside of a window of 10 minutes for all nodes, you'll get an empty or incomplete list of seeds. Taking the concept of using a reflector may be worth doing, but keep these things in mind: * ensure you use a private service with long-term memory, * you should rely on a reflector for assistance in configuring the seed list, not the seed-provider directly, * always assume the service can and will go down so write to disk appropriately, perhaps conf/seed-list.txt, * you must account for topological changes that will occur in long running clusters, * and all seed lists on each node should be identical. The last point is probably the hardest. I'm not sure if this infrastructure fits best inside of Cassandra or as external tools. I hope this helps you build what you have in mind. Cheers! EC2 seed provider using DataStax Reflector -- Key: CASSANDRA-7547 URL: https://issues.apache.org/jira/browse/CASSANDRA-7547 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Pekka Enberg Priority: Minor Attachments: 0001-EC2-seed-provider.patch This is a request for comments. I am using this to build our EC2 AMIs but I thought I'd ask if this makes sense as a generic feature for Cassandra. Cassandra cluster auto-configuration on EC2 uses the Datastax reflector service for discovering seed nodes. Instead of relying on external scripts, this patch implements EC2 seed provider that uses the Datastax reflector service. This is particularly useful for EC2 AMIs that don't include a complete userspace (such as those built with OSv) where we ideally want to push as much configuration to the application itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-7542) Reduce CAS contention
[ https://issues.apache.org/jira/browse/CASSANDRA-7542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sankalp kohli updated CASSANDRA-7542: - Fix Version/s: 2.0.10 Reduce CAS contention - Key: CASSANDRA-7542 URL: https://issues.apache.org/jira/browse/CASSANDRA-7542 Project: Cassandra Issue Type: Improvement Reporter: sankalp kohli Assignee: Benedict Fix For: 2.0.10 CAS updates on same CQL partition can lead to heavy contention inside C*. I am looking for simple ways(no algorithmic changes) to reduce contention as the penalty of it is high in terms of latency, specially for reads. We can put some sort of synchronization on CQL partition at StorageProxy level. This will reduce contention at least for all requests landing on one box for same partition. Here is an example of why it will help: 1) Say 1 write and 2 read CAS requests for the same partition key is send to C* in parallel. 2) Since client is token-aware, it sends these 3 request to the same C* instance A. (Lets assume that all 3 requests goto same instance A) 3) In this C* instance A, all 3 CAS requests will contend with each other in Paxos. (This is bad) To improve contention in 3), what I am proposing is to add a lock on partition key similar to what we do in PaxosState.java to serialize these 3 requests. This will remove the contention and improve performance as these 3 requests will not collide with each other. Another improvement we can do in client is to pick a deterministic live replica for a given partition doing CAS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7551) improve 2.1 flush defaults
[ https://issues.apache.org/jira/browse/CASSANDRA-7551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062252#comment-14062252 ] Benedict commented on CASSANDRA-7551: - LGTM. improve 2.1 flush defaults -- Key: CASSANDRA-7551 URL: https://issues.apache.org/jira/browse/CASSANDRA-7551 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Jonathan Ellis Assignee: Jonathan Ellis Fix For: 2.1.0 Attachments: 7551.txt Default of mfw=2 and mct=0.4 does a poor job saturating multi-disk systems. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (CASSANDRA-7547) EC2 seed provider using DataStax Reflector
[ https://issues.apache.org/jira/browse/CASSANDRA-7547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062237#comment-14062237 ] Joaquin Casares edited comment on CASSANDRA-7547 at 7/15/14 4:21 PM: - Before we go any further with the using the DataStax reflector, you should consider its initial purpose was to find a simple way for individual nodes to cluster together on startup without initially knowing anything about each other. The current reflector is built with a short-term memory of about 10 minutes to get the list of seeds. If a node is slow to boot and comes in on the 11th minute, it will never know of its peers. If a pre-chosen seed node is slow to boot, the nodes may never properly cluster together. This is important because the seed provider is pinged multiple times during the lifetime of the cluster, mainly during periods of topological changes: removal, bootstrap, replace, etc. If these happen outside of a window of 10 minutes for all nodes, you'll get an empty or incomplete list of seeds. Taking the concept of using a reflector may be worth doing, but keep these things in mind: * ensure you use a private service with long-term memory, * you should rely on a reflector for assistance in configuring the seed list, not the seed-provider directly, * always assume the service can and will go down so write to disk appropriately, perhaps conf/seed-list.txt, * you must account for topological changes that will occur in long running clusters, * and all seed lists on each node should be identical. The last point is probably the hardest. I'm not sure if this infrastructure fits best inside of Cassandra or as external tools. However, in order to have more control of when seed lists get updated, instead of waiting for Cassandra services to kick in, external tools will probably be your best option. I hope this helps you build what you have in mind. Cheers! was (Author: j.casares): Before we go any further with the using the DataStax reflector, you should consider its initial purpose was to find a simple way for individual nodes to cluster together on startup without initially knowing anything about each other. The current reflector is built with a short-term memory of about 10 minutes to get the list of seeds. If a node is slow to boot and comes in on the 11th minute, it will never know of its peers. If a pre-chosen seed node is slow to boot, the nodes may never properly cluster together. This is important because the seed provider is pinged multiple times during the lifetime of the cluster, mainly during periods of topological changes: removal, bootstrap, replace, etc. If these happen outside of a window of 10 minutes for all nodes, you'll get an empty or incomplete list of seeds. Taking the concept of using a reflector may be worth doing, but keep these things in mind: * ensure you use a private service with long-term memory, * you should rely on a reflector for assistance in configuring the seed list, not the seed-provider directly, * always assume the service can and will go down so write to disk appropriately, perhaps conf/seed-list.txt, * you must account for topological changes that will occur in long running clusters, * and all seed lists on each node should be identical. The last point is probably the hardest. I'm not sure if this infrastructure fits best inside of Cassandra or as external tools. I hope this helps you build what you have in mind. Cheers! EC2 seed provider using DataStax Reflector -- Key: CASSANDRA-7547 URL: https://issues.apache.org/jira/browse/CASSANDRA-7547 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Pekka Enberg Priority: Minor Attachments: 0001-EC2-seed-provider.patch This is a request for comments. I am using this to build our EC2 AMIs but I thought I'd ask if this makes sense as a generic feature for Cassandra. Cassandra cluster auto-configuration on EC2 uses the Datastax reflector service for discovering seed nodes. Instead of relying on external scripts, this patch implements EC2 seed provider that uses the Datastax reflector service. This is particularly useful for EC2 AMIs that don't include a complete userspace (such as those built with OSv) where we ideally want to push as much configuration to the application itself. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7542) Reduce CAS contention
[ https://issues.apache.org/jira/browse/CASSANDRA-7542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062247#comment-14062247 ] sankalp kohli commented on CASSANDRA-7542: -- Thanks Brandon. I have updated the fix version as 2.0.10 for this. Reduce CAS contention - Key: CASSANDRA-7542 URL: https://issues.apache.org/jira/browse/CASSANDRA-7542 Project: Cassandra Issue Type: Improvement Reporter: sankalp kohli Assignee: Benedict Fix For: 2.0.10 CAS updates on same CQL partition can lead to heavy contention inside C*. I am looking for simple ways(no algorithmic changes) to reduce contention as the penalty of it is high in terms of latency, specially for reads. We can put some sort of synchronization on CQL partition at StorageProxy level. This will reduce contention at least for all requests landing on one box for same partition. Here is an example of why it will help: 1) Say 1 write and 2 read CAS requests for the same partition key is send to C* in parallel. 2) Since client is token-aware, it sends these 3 request to the same C* instance A. (Lets assume that all 3 requests goto same instance A) 3) In this C* instance A, all 3 CAS requests will contend with each other in Paxos. (This is bad) To improve contention in 3), what I am proposing is to add a lock on partition key similar to what we do in PaxosState.java to serialize these 3 requests. This will remove the contention and improve performance as these 3 requests will not collide with each other. Another improvement we can do in client is to pick a deterministic live replica for a given partition doing CAS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7551) improve 2.1 flush defaults
[ https://issues.apache.org/jira/browse/CASSANDRA-7551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062279#comment-14062279 ] Benedict commented on CASSANDRA-7551: - Just to confirm this should only be applied to 2.1, not trunk, as trunk is expecting other changes that would make this a negative rather than positive change improve 2.1 flush defaults -- Key: CASSANDRA-7551 URL: https://issues.apache.org/jira/browse/CASSANDRA-7551 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Jonathan Ellis Assignee: Jonathan Ellis Fix For: 2.1.0 Attachments: 7551.txt Default of mfw=2 and mct=0.4 does a poor job saturating multi-disk systems. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7509) conditional updates don't work from cqlsh
[
https://issues.apache.org/jira/browse/CASSANDRA-7509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062300#comment-14062300
]
Tyler Hobbs commented on CASSANDRA-7509:
Version 2.0.2 of the python driver doesn't handle tuple types. Last week I
released version 2.1.0b1 of the driver, which adds native support for UDTs and
tuples. Upgrading to that would also allow us to remove the cqlsh patches for
CASSANDRA-7399, CASSANDRA-7267, CASSANDRA-7261, and
[PYTHON-90|https://datastax-oss.atlassian.net/browse/PYTHON-90], and would
include the [PYTHON-84|https://datastax-oss.atlassian.net/browse/PYTHON-84] fix
for {{dclocal_read_repair_chance}}.
conditional updates don't work from cqlsh
-
Key: CASSANDRA-7509
URL: https://issues.apache.org/jira/browse/CASSANDRA-7509
Project: Cassandra
Issue Type: Bug
Components: Tools
Reporter: Jeremiah Jordan
Assignee: Mikhail Stepura
Priority: Blocker
Fix For: 2.1.0
Attachments: CASSANDRA-2.1.0-7509.patch
{noformat}
apache-cassandra-2.1.0-rc2/bin $ ./cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.0-rc2 | CQL spec 3.1.7 | Native protocol v3]
Use HELP for help.
cqlsh use ks1;
cqlsh:ks1 create table users(
... userID uuid,
... fname text,
... lname text,
... email text,
... address text,
... zip int,
... state text,
... PRIMARY KEY ((userID, fname), state)
... );
cqlsh:ks1 insert into users (userID, fname, state, email) VALUES
(cfd66ccc-d857-4e90-b1e5-df98a3d40cd6, 'laura', 'ca', 'lau...@gmail.com') if
not exists;
code=2200 [Invalid query] message=Conditional updates are not supported by
the protocol version in use. You need to upgrade to a driver using the native
protocol v2.
cqlsh:ks1
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
git commit: Include snippet of query in SyntaxError messages
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 52c6dfb8f - 8a2d8a138
Include snippet of query in SyntaxError messages
Patch by Benjamin Lerer; reviewed by Tyler Hobbs for CASSANDRA-7111
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8a2d8a13
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8a2d8a13
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8a2d8a13
Branch: refs/heads/cassandra-2.1
Commit: 8a2d8a138d5ac1fc38b24bc7fd7513f6dbe86b24
Parents: 52c6dfb
Author: blerer b_le...@hotmail.com
Authored: Tue Jul 15 12:05:47 2014 -0500
Committer: Tyler Hobbs ty...@datastax.com
Committed: Tue Jul 15 12:05:47 2014 -0500
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/cql3/Cql.g| 41 ++--
.../apache/cassandra/cql3/ErrorCollector.java | 242 +++
.../apache/cassandra/cql3/ErrorListener.java| 44
.../apache/cassandra/cql3/QueryProcessor.java | 11 +-
.../apache/cassandra/cql3/CqlParserTest.java| 90 +++
.../cassandra/cql3/ErrorCollectorTest.java | 218 +
7 files changed, 621 insertions(+), 26 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/8a2d8a13/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1781ef3..89f9b7c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -6,6 +6,7 @@
* Adjust MT depth based on # of partition validating (CASSANDRA-5263)
* Optimise NativeCell comparisons (CASSANDRA-6755)
* Configurable client timeout for cqlsh (CASSANDRA-7516)
+ * Include snippet of CQL query near syntax error in messages (CASSANDRA-7111)
2.1.0-final
http://git-wip-us.apache.org/repos/asf/cassandra/blob/8a2d8a13/src/java/org/apache/cassandra/cql3/Cql.g
--
diff --git a/src/java/org/apache/cassandra/cql3/Cql.g
b/src/java/org/apache/cassandra/cql3/Cql.g
index 948cd5e..9dcc268 100644
--- a/src/java/org/apache/cassandra/cql3/Cql.g
+++ b/src/java/org/apache/cassandra/cql3/Cql.g
@@ -51,7 +51,7 @@ options {
}
@members {
-private final ListString recognitionErrors = new ArrayListString();
+private final ListErrorListener listeners = new
ArrayListErrorListener();
private final ListColumnIdentifier bindVariables = new
ArrayListColumnIdentifier();
public static final SetString reservedTypeNames = new HashSetString()
@@ -94,27 +94,26 @@ options {
return marker;
}
-public void displayRecognitionError(String[] tokenNames,
RecognitionException e)
+public void addErrorListener(ErrorListener listener)
{
-String hdr = getErrorHeader(e);
-String msg = getErrorMessage(e, tokenNames);
-recognitionErrors.add(hdr + + msg);
+this.listeners.add(listener);
}
-public void addRecognitionError(String msg)
+public void removeErrorListener(ErrorListener listener)
{
-recognitionErrors.add(msg);
+this.listeners.remove(listener);
}
-public ListString getRecognitionErrors()
+public void displayRecognitionError(String[] tokenNames,
RecognitionException e)
{
-return recognitionErrors;
+for (int i = 0, m = listeners.size(); i m; i++)
+listeners.get(i).syntaxError(this, tokenNames, e);
}
-public void throwLastRecognitionError() throws SyntaxException
+private void addRecognitionError(String msg)
{
-if (recognitionErrors.size() 0)
-throw new
SyntaxException(recognitionErrors.get((recognitionErrors.size()-1)));
+for (int i = 0, m = listeners.size(); i m; i++)
+listeners.get(i).syntaxError(this, msg);
}
public MapString, String convertPropertyMap(Maps.Literal map)
@@ -189,24 +188,22 @@ options {
return tokens.remove(0);
}
-private ListString recognitionErrors = new ArrayListString();
+private final ListErrorListener listeners = new
ArrayListErrorListener();
-public void displayRecognitionError(String[] tokenNames,
RecognitionException e)
+public void addErrorListener(ErrorListener listener)
{
-String hdr = getErrorHeader(e);
-String msg = getErrorMessage(e, tokenNames);
-recognitionErrors.add(hdr + + msg);
+this.listeners.add(listener);
}
-public ListString getRecognitionErrors()
+public void removeErrorListener(ErrorListener listener)
{
-return recognitionErrors;
+this.listeners.remove(listener);
}
-public void throwLastRecognitionError() throws SyntaxException
+public void displayRecognitionError(String[] tokenNames,
RecognitionException e)
{
-
[jira] [Updated] (CASSANDRA-6276) CQL: Map can not be created with the same name as a previously dropped list
[
https://issues.apache.org/jira/browse/CASSANDRA-6276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-6276:
-
Fix Version/s: (was: 2.1.0)
2.1.1
CQL: Map can not be created with the same name as a previously dropped list
---
Key: CASSANDRA-6276
URL: https://issues.apache.org/jira/browse/CASSANDRA-6276
Project: Cassandra
Issue Type: Bug
Environment: Cassandra 2.0.2 | CQL spec 3.1.0
centos 64 bit
Java(TM) SE Runtime Environment (build 1.7.0-b147)
Reporter: Oli Schacher
Assignee: Benjamin Lerer
Priority: Minor
Labels: cql
Fix For: 2.1.1
Attachments: CASSANDRA-6276.txt
If create a list, drop it and create a map with the same name, i get Bad
Request: comparators do not match or are not compatible.
{quote}
cqlsh:os_test1 create table thetable(id timeuuid primary key, somevalue
text);
cqlsh:os_test1 alter table thetable add mycollection listtext;
cqlsh:os_test1 alter table thetable drop mycollection;
cqlsh:os_test1 alter table thetable add mycollection maptext,text;
Bad Request: comparators do not match or are not compatible.
{quote}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[2/2] git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8a5a82b8 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8a5a82b8 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8a5a82b8 Branch: refs/heads/trunk Commit: 8a5a82b8a87314d1e740d94cd40a190bd2681a42 Parents: 3966935 8a2d8a1 Author: Tyler Hobbs ty...@datastax.com Authored: Tue Jul 15 12:06:51 2014 -0500 Committer: Tyler Hobbs ty...@datastax.com Committed: Tue Jul 15 12:06:51 2014 -0500 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/cql3/Cql.g| 41 ++-- .../apache/cassandra/cql3/ErrorCollector.java | 242 +++ .../apache/cassandra/cql3/ErrorListener.java| 44 .../apache/cassandra/cql3/QueryProcessor.java | 11 +- .../apache/cassandra/cql3/CqlParserTest.java| 90 +++ .../cassandra/cql3/ErrorCollectorTest.java | 218 + 7 files changed, 621 insertions(+), 26 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/8a5a82b8/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/8a5a82b8/src/java/org/apache/cassandra/cql3/Cql.g -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/8a5a82b8/src/java/org/apache/cassandra/cql3/QueryProcessor.java --
[1/2] git commit: Include snippet of query in SyntaxError messages
Repository: cassandra
Updated Branches:
refs/heads/trunk 3966935b4 - 8a5a82b8a
Include snippet of query in SyntaxError messages
Patch by Benjamin Lerer; reviewed by Tyler Hobbs for CASSANDRA-7111
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8a2d8a13
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8a2d8a13
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8a2d8a13
Branch: refs/heads/trunk
Commit: 8a2d8a138d5ac1fc38b24bc7fd7513f6dbe86b24
Parents: 52c6dfb
Author: blerer b_le...@hotmail.com
Authored: Tue Jul 15 12:05:47 2014 -0500
Committer: Tyler Hobbs ty...@datastax.com
Committed: Tue Jul 15 12:05:47 2014 -0500
--
CHANGES.txt | 1 +
src/java/org/apache/cassandra/cql3/Cql.g| 41 ++--
.../apache/cassandra/cql3/ErrorCollector.java | 242 +++
.../apache/cassandra/cql3/ErrorListener.java| 44
.../apache/cassandra/cql3/QueryProcessor.java | 11 +-
.../apache/cassandra/cql3/CqlParserTest.java| 90 +++
.../cassandra/cql3/ErrorCollectorTest.java | 218 +
7 files changed, 621 insertions(+), 26 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/8a2d8a13/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1781ef3..89f9b7c 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -6,6 +6,7 @@
* Adjust MT depth based on # of partition validating (CASSANDRA-5263)
* Optimise NativeCell comparisons (CASSANDRA-6755)
* Configurable client timeout for cqlsh (CASSANDRA-7516)
+ * Include snippet of CQL query near syntax error in messages (CASSANDRA-7111)
2.1.0-final
http://git-wip-us.apache.org/repos/asf/cassandra/blob/8a2d8a13/src/java/org/apache/cassandra/cql3/Cql.g
--
diff --git a/src/java/org/apache/cassandra/cql3/Cql.g
b/src/java/org/apache/cassandra/cql3/Cql.g
index 948cd5e..9dcc268 100644
--- a/src/java/org/apache/cassandra/cql3/Cql.g
+++ b/src/java/org/apache/cassandra/cql3/Cql.g
@@ -51,7 +51,7 @@ options {
}
@members {
-private final ListString recognitionErrors = new ArrayListString();
+private final ListErrorListener listeners = new
ArrayListErrorListener();
private final ListColumnIdentifier bindVariables = new
ArrayListColumnIdentifier();
public static final SetString reservedTypeNames = new HashSetString()
@@ -94,27 +94,26 @@ options {
return marker;
}
-public void displayRecognitionError(String[] tokenNames,
RecognitionException e)
+public void addErrorListener(ErrorListener listener)
{
-String hdr = getErrorHeader(e);
-String msg = getErrorMessage(e, tokenNames);
-recognitionErrors.add(hdr + + msg);
+this.listeners.add(listener);
}
-public void addRecognitionError(String msg)
+public void removeErrorListener(ErrorListener listener)
{
-recognitionErrors.add(msg);
+this.listeners.remove(listener);
}
-public ListString getRecognitionErrors()
+public void displayRecognitionError(String[] tokenNames,
RecognitionException e)
{
-return recognitionErrors;
+for (int i = 0, m = listeners.size(); i m; i++)
+listeners.get(i).syntaxError(this, tokenNames, e);
}
-public void throwLastRecognitionError() throws SyntaxException
+private void addRecognitionError(String msg)
{
-if (recognitionErrors.size() 0)
-throw new
SyntaxException(recognitionErrors.get((recognitionErrors.size()-1)));
+for (int i = 0, m = listeners.size(); i m; i++)
+listeners.get(i).syntaxError(this, msg);
}
public MapString, String convertPropertyMap(Maps.Literal map)
@@ -189,24 +188,22 @@ options {
return tokens.remove(0);
}
-private ListString recognitionErrors = new ArrayListString();
+private final ListErrorListener listeners = new
ArrayListErrorListener();
-public void displayRecognitionError(String[] tokenNames,
RecognitionException e)
+public void addErrorListener(ErrorListener listener)
{
-String hdr = getErrorHeader(e);
-String msg = getErrorMessage(e, tokenNames);
-recognitionErrors.add(hdr + + msg);
+this.listeners.add(listener);
}
-public ListString getRecognitionErrors()
+public void removeErrorListener(ErrorListener listener)
{
-return recognitionErrors;
+this.listeners.remove(listener);
}
-public void throwLastRecognitionError() throws SyntaxException
+public void displayRecognitionError(String[] tokenNames,
RecognitionException e)
{
-if
[3/3] git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1c4768a8 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1c4768a8 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1c4768a8 Branch: refs/heads/trunk Commit: 1c4768a8889122b40f57be10f8af8e2522a09ec8 Parents: 8a5a82b e66896b Author: Benedict Elliott Smith bened...@apache.org Authored: Tue Jul 15 18:21:10 2014 +0100 Committer: Benedict Elliott Smith bened...@apache.org Committed: Tue Jul 15 18:21:10 2014 +0100 -- .../cassandra/concurrent/SharedExecutorPool.java | 19 --- 1 file changed, 12 insertions(+), 7 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/1c4768a8/src/java/org/apache/cassandra/concurrent/SharedExecutorPool.java --
git commit: Add missing table name to DROP INDEX responses
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1.0 04017616e - bd0eafd58
Add missing table name to DROP INDEX responses
Patch and review by Tyler Hobbs and Aleksey Yeschenko for CASSANDRA-7539
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bd0eafd5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bd0eafd5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bd0eafd5
Branch: refs/heads/cassandra-2.1.0
Commit: bd0eafd5853a446b0db79547231a0a9653a74293
Parents: 0401761
Author: Tyler Hobbs ty...@datastax.com
Authored: Tue Jul 15 12:23:06 2014 -0500
Committer: Tyler Hobbs ty...@datastax.com
Committed: Tue Jul 15 12:23:06 2014 -0500
--
CHANGES.txt | 1 +
.../cassandra/cql3/statements/DropIndexStatement.java| 11 +++
src/java/org/apache/cassandra/transport/Event.java | 2 ++
3 files changed, 14 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd0eafd5/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 193cff8..e6ded2a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,6 +1,7 @@
2.1.0-final
* (Windows) fix startup when WMI memory query fails (CASSANDRA-7505)
* Anti-compaction proceeds if any part of the repair failed (CASANDRA-7521)
+ * Add missing table name to DROP INDEX responses and notifications
(CASSANDRA-7539)
Merged from 2.0:
* (Windows) force range-based repair to non-sequential mode (CASSANDRA-7541)
* Fix range merging when DES scores are zero (CASSANDRA-7535)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd0eafd5/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
--
diff --git
a/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
b/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
index b8a4c9a..0562cc6 100644
--- a/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
@@ -33,6 +33,9 @@ public class DropIndexStatement extends
SchemaAlteringStatement
public final String indexName;
public final boolean ifExists;
+// initialized in announceMigration()
+private String indexedCF;
+
public DropIndexStatement(IndexName indexName, boolean ifExists)
{
super(indexName.getCfName());
@@ -67,6 +70,7 @@ public class DropIndexStatement extends
SchemaAlteringStatement
return;
CFMetaData updatedCfm = updateCFMetadata(cfm);
+indexedCF = updatedCfm.cfName;
MigrationManager.announceColumnFamilyUpdate(updatedCfm, false,
isLocalOnly);
}
@@ -106,4 +110,11 @@ public class DropIndexStatement extends
SchemaAlteringStatement
}
return null;
}
+
+@Override
+public String columnFamily()
+{
+assert indexedCF != null;
+return indexedCF;
+}
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd0eafd5/src/java/org/apache/cassandra/transport/Event.java
--
diff --git a/src/java/org/apache/cassandra/transport/Event.java
b/src/java/org/apache/cassandra/transport/Event.java
index 3fa2733..1b7e487 100644
--- a/src/java/org/apache/cassandra/transport/Event.java
+++ b/src/java/org/apache/cassandra/transport/Event.java
@@ -218,6 +218,8 @@ public abstract class Event
this.target = target;
this.keyspace = keyspace;
this.tableOrType = tableOrType;
+if (target != Target.KEYSPACE)
+assert this.tableOrType != null : Table or type should be set
for non-keyspace schema change events;
}
public SchemaChange(Change change, String keyspace)
[1/2] git commit: Add missing table name to DROP INDEX responses
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 8a2d8a138 - 0b7377fed
Add missing table name to DROP INDEX responses
Patch and review by Tyler Hobbs and Aleksey Yeschenko for CASSANDRA-7539
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bd0eafd5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bd0eafd5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bd0eafd5
Branch: refs/heads/cassandra-2.1
Commit: bd0eafd5853a446b0db79547231a0a9653a74293
Parents: 0401761
Author: Tyler Hobbs ty...@datastax.com
Authored: Tue Jul 15 12:23:06 2014 -0500
Committer: Tyler Hobbs ty...@datastax.com
Committed: Tue Jul 15 12:23:06 2014 -0500
--
CHANGES.txt | 1 +
.../cassandra/cql3/statements/DropIndexStatement.java| 11 +++
src/java/org/apache/cassandra/transport/Event.java | 2 ++
3 files changed, 14 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd0eafd5/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 193cff8..e6ded2a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,6 +1,7 @@
2.1.0-final
* (Windows) fix startup when WMI memory query fails (CASSANDRA-7505)
* Anti-compaction proceeds if any part of the repair failed (CASANDRA-7521)
+ * Add missing table name to DROP INDEX responses and notifications
(CASSANDRA-7539)
Merged from 2.0:
* (Windows) force range-based repair to non-sequential mode (CASSANDRA-7541)
* Fix range merging when DES scores are zero (CASSANDRA-7535)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd0eafd5/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
--
diff --git
a/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
b/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
index b8a4c9a..0562cc6 100644
--- a/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
@@ -33,6 +33,9 @@ public class DropIndexStatement extends
SchemaAlteringStatement
public final String indexName;
public final boolean ifExists;
+// initialized in announceMigration()
+private String indexedCF;
+
public DropIndexStatement(IndexName indexName, boolean ifExists)
{
super(indexName.getCfName());
@@ -67,6 +70,7 @@ public class DropIndexStatement extends
SchemaAlteringStatement
return;
CFMetaData updatedCfm = updateCFMetadata(cfm);
+indexedCF = updatedCfm.cfName;
MigrationManager.announceColumnFamilyUpdate(updatedCfm, false,
isLocalOnly);
}
@@ -106,4 +110,11 @@ public class DropIndexStatement extends
SchemaAlteringStatement
}
return null;
}
+
+@Override
+public String columnFamily()
+{
+assert indexedCF != null;
+return indexedCF;
+}
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd0eafd5/src/java/org/apache/cassandra/transport/Event.java
--
diff --git a/src/java/org/apache/cassandra/transport/Event.java
b/src/java/org/apache/cassandra/transport/Event.java
index 3fa2733..1b7e487 100644
--- a/src/java/org/apache/cassandra/transport/Event.java
+++ b/src/java/org/apache/cassandra/transport/Event.java
@@ -218,6 +218,8 @@ public abstract class Event
this.target = target;
this.keyspace = keyspace;
this.tableOrType = tableOrType;
+if (target != Target.KEYSPACE)
+assert this.tableOrType != null : Table or type should be set
for non-keyspace schema change events;
}
public SchemaChange(Change change, String keyspace)
[2/2] git commit: Merge branch 'cassandra-2.1.0' into cassandra-2.1
Merge branch 'cassandra-2.1.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0b7377fe Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0b7377fe Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0b7377fe Branch: refs/heads/cassandra-2.1 Commit: 0b7377fed597eb41bc23224c80747130561165cf Parents: 8a2d8a1 bd0eafd Author: Tyler Hobbs ty...@datastax.com Authored: Tue Jul 15 12:24:12 2014 -0500 Committer: Tyler Hobbs ty...@datastax.com Committed: Tue Jul 15 12:24:12 2014 -0500 -- CHANGES.txt | 1 + .../cassandra/cql3/statements/DropIndexStatement.java| 11 +++ src/java/org/apache/cassandra/transport/Event.java | 2 ++ 3 files changed, 14 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/0b7377fe/CHANGES.txt --
git commit: Add missing table name to DROP INDEX responses
Repository: cassandra
Updated Branches:
refs/heads/trunk 1c4768a88 - ae25812e3
Add missing table name to DROP INDEX responses
Patch and review by Tyler Hobbs and Aleksey Yeschenko for CASSANDRA-7539
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ae25812e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ae25812e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ae25812e
Branch: refs/heads/trunk
Commit: ae25812e39cb29c6ad0d0625eec9268e55c2fb38
Parents: 1c4768a
Author: Tyler Hobbs ty...@datastax.com
Authored: Tue Jul 15 12:23:06 2014 -0500
Committer: Tyler Hobbs ty...@datastax.com
Committed: Tue Jul 15 12:25:05 2014 -0500
--
CHANGES.txt | 1 +
.../cassandra/cql3/statements/DropIndexStatement.java| 11 +++
src/java/org/apache/cassandra/transport/Event.java | 2 ++
3 files changed, 14 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/ae25812e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 27aa624..8895bf4 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -28,6 +28,7 @@
2.1.0-final
* (Windows) fix startup when WMI memory query fails (CASSANDRA-7505)
* Anti-compaction proceeds if any part of the repair failed (CASANDRA-7521)
+ * Add missing table name to DROP INDEX responses and notifications
(CASSANDRA-7539)
Merged from 2.0:
* (Windows) force range-based repair to non-sequential mode (CASSANDRA-7541)
* Fix range merging when DES scores are zero (CASSANDRA-7535)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/ae25812e/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
--
diff --git
a/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
b/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
index b8a4c9a..0562cc6 100644
--- a/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
@@ -33,6 +33,9 @@ public class DropIndexStatement extends
SchemaAlteringStatement
public final String indexName;
public final boolean ifExists;
+// initialized in announceMigration()
+private String indexedCF;
+
public DropIndexStatement(IndexName indexName, boolean ifExists)
{
super(indexName.getCfName());
@@ -67,6 +70,7 @@ public class DropIndexStatement extends
SchemaAlteringStatement
return;
CFMetaData updatedCfm = updateCFMetadata(cfm);
+indexedCF = updatedCfm.cfName;
MigrationManager.announceColumnFamilyUpdate(updatedCfm, false,
isLocalOnly);
}
@@ -106,4 +110,11 @@ public class DropIndexStatement extends
SchemaAlteringStatement
}
return null;
}
+
+@Override
+public String columnFamily()
+{
+assert indexedCF != null;
+return indexedCF;
+}
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/ae25812e/src/java/org/apache/cassandra/transport/Event.java
--
diff --git a/src/java/org/apache/cassandra/transport/Event.java
b/src/java/org/apache/cassandra/transport/Event.java
index 3fa2733..1b7e487 100644
--- a/src/java/org/apache/cassandra/transport/Event.java
+++ b/src/java/org/apache/cassandra/transport/Event.java
@@ -218,6 +218,8 @@ public abstract class Event
this.target = target;
this.keyspace = keyspace;
this.tableOrType = tableOrType;
+if (target != Target.KEYSPACE)
+assert this.tableOrType != null : Table or type should be set
for non-keyspace schema change events;
}
public SchemaChange(Change change, String keyspace)
buildbot failure in ASF Buildbot on cassandra-trunk
The Buildbot has detected a new failure on builder cassandra-trunk while building cassandra. Full details are available at: http://ci.apache.org/builders/cassandra-trunk/builds/454 Buildbot URL: http://ci.apache.org/ Buildslave for this Build: portunus_ubuntu Build Reason: scheduler Build Source Stamp: [branch trunk] 8a5a82b8a87314d1e740d94cd40a190bd2681a42 Blamelist: Tyler Hobbs ty...@datastax.com,blerer b_le...@hotmail.com BUILD FAILED: failed shell sincerely, -The Buildbot
[1/3] git commit: ninja fix comments in SharedExecutorPool
Repository: cassandra
Updated Branches:
refs/heads/trunk 8a5a82b8a - 1c4768a88
ninja fix comments in SharedExecutorPool
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ca431888
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ca431888
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ca431888
Branch: refs/heads/trunk
Commit: ca431888efd91bb8a7900c0219bd821744153686
Parents: 0401761
Author: Benedict Elliott Smith bened...@apache.org
Authored: Tue Jul 15 18:20:48 2014 +0100
Committer: Benedict Elliott Smith bened...@apache.org
Committed: Tue Jul 15 18:20:48 2014 +0100
--
.../cassandra/concurrent/SharedExecutorPool.java | 19 ---
1 file changed, 12 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/ca431888/src/java/org/apache/cassandra/concurrent/SharedExecutorPool.java
--
diff --git a/src/java/org/apache/cassandra/concurrent/SharedExecutorPool.java
b/src/java/org/apache/cassandra/concurrent/SharedExecutorPool.java
index e03ec57..161b326 100644
--- a/src/java/org/apache/cassandra/concurrent/SharedExecutorPool.java
+++ b/src/java/org/apache/cassandra/concurrent/SharedExecutorPool.java
@@ -35,17 +35,22 @@ import static
org.apache.cassandra.concurrent.SEPWorker.Work;
* To keep producers from incurring unnecessary delays, once an executor is
spun up (i.e. is processing tasks at a steady
* rate), adding tasks to the executor often involves only placing the task on
the work queue and updating the
* task permits (which imposes our max queue length constraints). Only when it
cannot be guaranteed the task will be serviced
- * promptly does the producer have to signal a thread itself to perform the
work.
+ * promptly, and the maximum concurrency has not been reached, does the
producer have to schedule a thread itself to perform
+ * the work ('promptly' in this context means we already have a worker
spinning for work, as described next).
*
- * We do this by scheduling only if
- *
- * The worker threads schedule themselves as far as possible: when they are
assigned a task, they will attempt to spawn
+ * Otherwise the worker threads schedule themselves: when they are assigned a
task, they will attempt to spawn
* a partner worker to service any other work outstanding on the queue (if
any); once they have finished the task they
* will either take another (if any remaining) and repeat this, or they will
attempt to assign themselves to another executor
* that does have tasks remaining. If both fail, it will enter a
non-busy-spinning phase, where it will sleep for a short
* random interval (based upon the number of threads in this mode, so that the
total amount of non-sleeping time remains
- * approximately fixed regardless of the number of spinning threads), and upon
waking up will again try to assign themselves
- * an executor with outstanding tasks to perform.
+ * approximately fixed regardless of the number of spinning threads), and upon
waking will again try to assign itself to
+ * an executor with outstanding tasks to perform. As a result of always
scheduling a partner before committing to performing
+ * any work, with a steady state of task arrival we should generally have
either one spinning worker ready to promptly respond
+ * to incoming work, or all possible workers actively committed to tasks.
+ *
+ * In order to prevent this executor pool acting like a noisy neighbour to
other processes on the system, workers also deschedule
+ * themselves when it is detected that there are too many for the current rate
of operation arrival. This is decided as a function
+ * of the total time spent spinning by all workers in an interval; as more
workers spin, workers are descheduled more rapidly.
*/
public class SharedExecutorPool
{
@@ -96,4 +101,4 @@ public class SharedExecutorPool
if (current == 0 spinningCount.compareAndSet(0, 1))
schedule(Work.SPINNING);
}
-}
\ No newline at end of file
+}
[jira] [Commented] (CASSANDRA-7111) Include snippet of CQL query near error in SyntaxError messages
[
https://issues.apache.org/jira/browse/CASSANDRA-7111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062344#comment-14062344
]
Tyler Hobbs commented on CASSANDRA-7111:
Thanks, committed to cassandra-2.1 and trunk.
However, I noticed that a couple of the new unit tests are failing in trunk:
{noformat}
[junit] Testsuite: org.apache.cassandra.cql3.CqlParserTest
[junit] Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed:
0.161 sec
[junit]
[junit] Testcase:
testRemoveErrorListener(org.apache.cassandra.cql3.CqlParserTest): FAILED
[junit] expected:1 but was:2
[junit] junit.framework.AssertionFailedError: expected:1 but was:2
[junit] at
org.apache.cassandra.cql3.CqlParserTest.testRemoveErrorListener(CqlParserTest.java:70)
[junit]
[junit]
[junit] Testcase:
testAddErrorListener(org.apache.cassandra.cql3.CqlParserTest):FAILED
[junit] expected:1 but was:2
[junit] junit.framework.AssertionFailedError: expected:1 but was:2
[junit] at
org.apache.cassandra.cql3.CqlParserTest.testAddErrorListener(CqlParserTest.java:49)
[junit]
[junit]
[junit] Test org.apache.cassandra.cql3.CqlParserTest FAILED
{noformat}
Can you post a second patch to fix the tests on trunk?
Include snippet of CQL query near error in SyntaxError messages
---
Key: CASSANDRA-7111
URL: https://issues.apache.org/jira/browse/CASSANDRA-7111
Project: Cassandra
Issue Type: Improvement
Components: API, Tools
Reporter: Tyler Hobbs
Assignee: Benjamin Lerer
Labels: cql
Fix For: 2.1.1
Attachments: CASSANDRA-7111-V2.txt, CASSANDRA-7111-V3.txt,
CASSANDRA-7111.txt
When a SyntaxError is returned, including a snippet of the query close to the
error would make a lot of error messages easier to understand. For example,
if you did this with the python driver:
{code}
session.execute(SELECT * FROM users WHERE username='%s', ['Joe Smith'])
{code}
you would wind up with an extra set of single quotes (the driver
automatically escapes and quotes input). If a snippet like {{...WHERE
username=''Joe Smith''}} were included in the error message, this would be
pretty easy to spot.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[2/3] git commit: Merge branch 'cassandra-2.1.0' into cassandra-2.1
Merge branch 'cassandra-2.1.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e66896b1 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e66896b1 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e66896b1 Branch: refs/heads/trunk Commit: e66896b1d326c49ee8b7b20ef343490370ba968f Parents: 8a2d8a1 ca43188 Author: Benedict Elliott Smith bened...@apache.org Authored: Tue Jul 15 18:21:01 2014 +0100 Committer: Benedict Elliott Smith bened...@apache.org Committed: Tue Jul 15 18:21:01 2014 +0100 -- .../cassandra/concurrent/SharedExecutorPool.java | 19 --- 1 file changed, 12 insertions(+), 7 deletions(-) --
[jira] [Updated] (CASSANDRA-7527) Bump CQL version and update doc for 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-7527:
---
Attachment: 7527-v2.txt
v2 patch adds CASSANDRA-7314 to the 3.2.0 changes and the {{DROP INDEX}}
statement itself.
Bump CQL version and update doc for 2.1
---
Key: CASSANDRA-7527
URL: https://issues.apache.org/jira/browse/CASSANDRA-7527
Project: Cassandra
Issue Type: Bug
Reporter: Sylvain Lebresne
Assignee: Tyler Hobbs
Fix For: 2.1.0
Attachments: 7527-v2.txt, 7527.txt
It appears we forgot to bump the CQL version for new 2.1 features (UDT, tuple
type, collection indexing), nor did we update the textile doc
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (CASSANDRA-7508) nodetool prints error if dirs don't exist.
[
https://issues.apache.org/jira/browse/CASSANDRA-7508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-7508:
---
Attachment: 7508.txt
7508.txt (and [branch|https://github.com/thobbs/cassandra/tree/CASSANDRA-7508])
uses the actual token/endpoint map to determine if vnodes are enabled instead
of using the conf, which nodetool shouldn't have been doing anyway.
nodetool prints error if dirs don't exist.
--
Key: CASSANDRA-7508
URL: https://issues.apache.org/jira/browse/CASSANDRA-7508
Project: Cassandra
Issue Type: Bug
Components: Tools
Reporter: Jeremiah Jordan
Assignee: Tyler Hobbs
Priority: Minor
Fix For: 2.1.0
Attachments: 7508.txt
The new directory stuff blows up nodetool if its not there. Like if I run
nodetool from an different untar of the rc2 tar.gz.
{noformat}
apache-cassandra-2.1.0-rc2/bin $ ./nodetool ring
Note: Ownership information does not include topology; for complete
information, specify a keyspace
Datacenter: Analytics
==
AddressRackStatus State LoadOwns
Token
127.0.0.1 rack1 Up Normal 57.88 KB100.00%
3647214374146654083
ERROR 21:59:30 Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: commitlog_directory
is missing and -Dcassandra.storagedir is not set
at
org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:461)
~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at
org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:129)
~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at org.apache.cassandra.tools.NodeTool$Ring.execute(NodeTool.java:464)
[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:245)
[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:159)
[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
commitlog_directory is missing and -Dcassandra.storagedir is not set
Fatal configuration error; unable to start. See log for stack trace.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7232) Enable live replay of commit logs
[ https://issues.apache.org/jira/browse/CASSANDRA-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062580#comment-14062580 ] Jonathan Ellis commented on CASSANDRA-7232: --- I think [~nickmbailey] was asking for this. Thoughts? Enable live replay of commit logs - Key: CASSANDRA-7232 URL: https://issues.apache.org/jira/browse/CASSANDRA-7232 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Patrick McFadin Assignee: Lyuben Todorov Priority: Minor Fix For: 2.0.10 Attachments: 0001-Expose-CommitLog-recover-to-JMX-add-nodetool-cmd-for.patch, 0001-TRUNK-JMX-and-nodetool-cmd-for-commitlog-replay.patch Replaying commit logs takes a restart but restoring sstables can be an online operation with refresh. In order to restore a point-in-time without a restart, the node needs to live replay the commit logs from JMX and a nodetool command. nodetool refreshcommitlogs keyspace table -- This message was sent by Atlassian JIRA (v6.2#6252)
[8/8] git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/da650f3c Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/da650f3c Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/da650f3c Branch: refs/heads/trunk Commit: da650f3c62bad24a3761e90b939cf98fcd1e04e5 Parents: ae25812 9a4cc22 Author: Jonathan Ellis jbel...@apache.org Authored: Tue Jul 15 15:15:16 2014 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Tue Jul 15 15:15:16 2014 -0500 -- CHANGES.txt | 2 ++ conf/cassandra.yaml | 23 +++- .../org/apache/cassandra/config/Config.java | 6 ++--- .../cassandra/config/DatabaseDescriptor.java| 6 + 4 files changed, 28 insertions(+), 9 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/da650f3c/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/da650f3c/src/java/org/apache/cassandra/config/DatabaseDescriptor.java --
[4/8] git commit: memtable_cleanup_threshold and memtable_flush_writers defaults patch by jbellis; reviewed by bes for CASSANDRA-7551
memtable_cleanup_threshold and memtable_flush_writers defaults patch by jbellis; reviewed by bes for CASSANDRA-7551 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3adf5948 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3adf5948 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3adf5948 Branch: refs/heads/trunk Commit: 3adf59486a4e55f409fcc62a4c942a070b98dd8f Parents: bd0eafd Author: Jonathan Ellis jbel...@apache.org Authored: Tue Jul 15 15:14:50 2014 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Tue Jul 15 15:14:56 2014 -0500 -- CHANGES.txt | 2 ++ conf/cassandra.yaml | 23 +++- .../org/apache/cassandra/config/Config.java | 6 ++--- .../cassandra/config/DatabaseDescriptor.java| 6 + 4 files changed, 28 insertions(+), 9 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/3adf5948/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index e6ded2a..b20c465 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,6 @@ 2.1.0-final + * Updated memtable_cleanup_threshold and memtable_flush_writers defaults + (CASSANDRA-7551) * (Windows) fix startup when WMI memory query fails (CASSANDRA-7505) * Anti-compaction proceeds if any part of the repair failed (CASANDRA-7521) * Add missing table name to DROP INDEX responses and notifications (CASSANDRA-7539) http://git-wip-us.apache.org/repos/asf/cassandra/blob/3adf5948/conf/cassandra.yaml -- diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml index f1e5576..4ce69fe 100644 --- a/conf/cassandra.yaml +++ b/conf/cassandra.yaml @@ -282,8 +282,13 @@ concurrent_counter_writes: 32 # memtable_total_space_in_mb: 2048 # Ratio of occupied non-flushing memtable size to total permitted size -# that will trigger a flush of the largest memtable. -memtable_cleanup_threshold: 0.4 +# that will trigger a flush of the largest memtable. Lager mct will +# mean larger flushes and hence less compaction, but also less concurrent +# flush activity which can make it difficult to keep your disks fed +# under heavy write load. +# +# memtable_cleanup_threshold defaults to 1 / (memtable_flush_writers + 1) +#memtable_cleanup_threshold: 0.11 # Specify the way Cassandra allocates and manages memtable memory. # Options are: @@ -304,9 +309,14 @@ memtable_allocation_type: heap_buffers # This sets the amount of memtable flush writer threads. These will # be blocked by disk io, and each one will hold a memtable in memory -# while blocked. If your flush directory is backed by SSD, you may -# want to increase this. -memtable_flush_writers: 2 +# while blocked. +# +# memtable_flush_writers defaults to the smaller of (number of disks, +# number of cores), with a minimum of 2 and a maximum of 8. +# +# If your data directories are backed by SSD, you should increase this +# to the number of cores. +#memtable_flush_writers: 8 # A fixed memory pool size in MB for for SSTable index summaries. If left # empty, this will default to 5% of the heap size. If the memory usage of @@ -503,6 +513,9 @@ batch_size_warn_threshold_in_kb: 5 # # concurrent_compactors defaults to the smaller of (number of disks, # number of cores), with a minimum of 2 and a maximum of 8. +# +# If your data directories are backed by SSD, you should increase this +# to the number of cores. #concurrent_compactors: 1 # Throttles compaction to the given total throughput across the entire http://git-wip-us.apache.org/repos/asf/cassandra/blob/3adf5948/src/java/org/apache/cassandra/config/Config.java -- diff --git a/src/java/org/apache/cassandra/config/Config.java b/src/java/org/apache/cassandra/config/Config.java index 4d4a95b..2438e09 100644 --- a/src/java/org/apache/cassandra/config/Config.java +++ b/src/java/org/apache/cassandra/config/Config.java @@ -90,12 +90,10 @@ public class Config @Deprecated public Integer concurrent_replicates = null; -// we don't want a lot of contention, but we also don't want to starve all other tables -// if a big one flushes. OS buffering should be able to minimize contention with 2 threads. -public int memtable_flush_writers = 2; +public Integer memtable_flush_writers = null; public Integer memtable_heap_space_in_mb; public Integer memtable_offheap_space_in_mb; -public float memtable_cleanup_threshold = 0.4f; +public Double memtable_cleanup_threshold = null; public Integer storage_port = 7000; public Integer ssl_storage_port = 7001;
[3/8] git commit: memtable_cleanup_threshold and memtable_flush_writers defaults patch by jbellis; reviewed by bes for CASSANDRA-7551
memtable_cleanup_threshold and memtable_flush_writers defaults patch by jbellis; reviewed by bes for CASSANDRA-7551 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3adf5948 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3adf5948 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3adf5948 Branch: refs/heads/cassandra-2.1 Commit: 3adf59486a4e55f409fcc62a4c942a070b98dd8f Parents: bd0eafd Author: Jonathan Ellis jbel...@apache.org Authored: Tue Jul 15 15:14:50 2014 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Tue Jul 15 15:14:56 2014 -0500 -- CHANGES.txt | 2 ++ conf/cassandra.yaml | 23 +++- .../org/apache/cassandra/config/Config.java | 6 ++--- .../cassandra/config/DatabaseDescriptor.java| 6 + 4 files changed, 28 insertions(+), 9 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/3adf5948/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index e6ded2a..b20c465 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,6 @@ 2.1.0-final + * Updated memtable_cleanup_threshold and memtable_flush_writers defaults + (CASSANDRA-7551) * (Windows) fix startup when WMI memory query fails (CASSANDRA-7505) * Anti-compaction proceeds if any part of the repair failed (CASANDRA-7521) * Add missing table name to DROP INDEX responses and notifications (CASSANDRA-7539) http://git-wip-us.apache.org/repos/asf/cassandra/blob/3adf5948/conf/cassandra.yaml -- diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml index f1e5576..4ce69fe 100644 --- a/conf/cassandra.yaml +++ b/conf/cassandra.yaml @@ -282,8 +282,13 @@ concurrent_counter_writes: 32 # memtable_total_space_in_mb: 2048 # Ratio of occupied non-flushing memtable size to total permitted size -# that will trigger a flush of the largest memtable. -memtable_cleanup_threshold: 0.4 +# that will trigger a flush of the largest memtable. Lager mct will +# mean larger flushes and hence less compaction, but also less concurrent +# flush activity which can make it difficult to keep your disks fed +# under heavy write load. +# +# memtable_cleanup_threshold defaults to 1 / (memtable_flush_writers + 1) +#memtable_cleanup_threshold: 0.11 # Specify the way Cassandra allocates and manages memtable memory. # Options are: @@ -304,9 +309,14 @@ memtable_allocation_type: heap_buffers # This sets the amount of memtable flush writer threads. These will # be blocked by disk io, and each one will hold a memtable in memory -# while blocked. If your flush directory is backed by SSD, you may -# want to increase this. -memtable_flush_writers: 2 +# while blocked. +# +# memtable_flush_writers defaults to the smaller of (number of disks, +# number of cores), with a minimum of 2 and a maximum of 8. +# +# If your data directories are backed by SSD, you should increase this +# to the number of cores. +#memtable_flush_writers: 8 # A fixed memory pool size in MB for for SSTable index summaries. If left # empty, this will default to 5% of the heap size. If the memory usage of @@ -503,6 +513,9 @@ batch_size_warn_threshold_in_kb: 5 # # concurrent_compactors defaults to the smaller of (number of disks, # number of cores), with a minimum of 2 and a maximum of 8. +# +# If your data directories are backed by SSD, you should increase this +# to the number of cores. #concurrent_compactors: 1 # Throttles compaction to the given total throughput across the entire http://git-wip-us.apache.org/repos/asf/cassandra/blob/3adf5948/src/java/org/apache/cassandra/config/Config.java -- diff --git a/src/java/org/apache/cassandra/config/Config.java b/src/java/org/apache/cassandra/config/Config.java index 4d4a95b..2438e09 100644 --- a/src/java/org/apache/cassandra/config/Config.java +++ b/src/java/org/apache/cassandra/config/Config.java @@ -90,12 +90,10 @@ public class Config @Deprecated public Integer concurrent_replicates = null; -// we don't want a lot of contention, but we also don't want to starve all other tables -// if a big one flushes. OS buffering should be able to minimize contention with 2 threads. -public int memtable_flush_writers = 2; +public Integer memtable_flush_writers = null; public Integer memtable_heap_space_in_mb; public Integer memtable_offheap_space_in_mb; -public float memtable_cleanup_threshold = 0.4f; +public Double memtable_cleanup_threshold = null; public Integer storage_port = 7000; public Integer ssl_storage_port = 7001;
[6/8] git commit: Merge branch 'cassandra-2.1.0' into cassandra-2.1
Merge branch 'cassandra-2.1.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9a4cc22a Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9a4cc22a Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9a4cc22a Branch: refs/heads/trunk Commit: 9a4cc22a3ebf54a52bcb7e361837ed9f51fbc58b Parents: 0b7377f 3adf594 Author: Jonathan Ellis jbel...@apache.org Authored: Tue Jul 15 15:15:05 2014 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Tue Jul 15 15:15:05 2014 -0500 -- CHANGES.txt | 2 ++ conf/cassandra.yaml | 23 +++- .../org/apache/cassandra/config/Config.java | 6 ++--- .../cassandra/config/DatabaseDescriptor.java| 6 + 4 files changed, 28 insertions(+), 9 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a4cc22a/CHANGES.txt -- diff --cc CHANGES.txt index fc6d4eb,b20c465..65763dc --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,15 -1,6 +1,17 @@@ +2.1.1 + * Add listen_interface and rpc_interface options (CASSANDRA-7417) + * Fail to start if commit log replay detects a problem (CASSANDRA-7125) + * Improve schema merge performance (CASSANDRA-7444) + * Fix NPE when unknown prepared statement ID is used (CASSANDRA-7454) + * Adjust MT depth based on # of partition validating (CASSANDRA-5263) + * Optimise NativeCell comparisons (CASSANDRA-6755) + * Configurable client timeout for cqlsh (CASSANDRA-7516) + * Include snippet of CQL query near syntax error in messages (CASSANDRA-7111) + + 2.1.0-final + * Updated memtable_cleanup_threshold and memtable_flush_writers defaults +(CASSANDRA-7551) * (Windows) fix startup when WMI memory query fails (CASSANDRA-7505) * Anti-compaction proceeds if any part of the repair failed (CASANDRA-7521) * Add missing table name to DROP INDEX responses and notifications (CASSANDRA-7539) http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a4cc22a/conf/cassandra.yaml -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a4cc22a/src/java/org/apache/cassandra/config/Config.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a4cc22a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java --
[5/8] git commit: memtable_cleanup_threshold and memtable_flush_writers defaults patch by jbellis; reviewed by bes for CASSANDRA-7551
memtable_cleanup_threshold and memtable_flush_writers defaults patch by jbellis; reviewed by bes for CASSANDRA-7551 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/3adf5948 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/3adf5948 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/3adf5948 Branch: refs/heads/cassandra-2.1.0 Commit: 3adf59486a4e55f409fcc62a4c942a070b98dd8f Parents: bd0eafd Author: Jonathan Ellis jbel...@apache.org Authored: Tue Jul 15 15:14:50 2014 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Tue Jul 15 15:14:56 2014 -0500 -- CHANGES.txt | 2 ++ conf/cassandra.yaml | 23 +++- .../org/apache/cassandra/config/Config.java | 6 ++--- .../cassandra/config/DatabaseDescriptor.java| 6 + 4 files changed, 28 insertions(+), 9 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/3adf5948/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index e6ded2a..b20c465 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,6 @@ 2.1.0-final + * Updated memtable_cleanup_threshold and memtable_flush_writers defaults + (CASSANDRA-7551) * (Windows) fix startup when WMI memory query fails (CASSANDRA-7505) * Anti-compaction proceeds if any part of the repair failed (CASANDRA-7521) * Add missing table name to DROP INDEX responses and notifications (CASSANDRA-7539) http://git-wip-us.apache.org/repos/asf/cassandra/blob/3adf5948/conf/cassandra.yaml -- diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml index f1e5576..4ce69fe 100644 --- a/conf/cassandra.yaml +++ b/conf/cassandra.yaml @@ -282,8 +282,13 @@ concurrent_counter_writes: 32 # memtable_total_space_in_mb: 2048 # Ratio of occupied non-flushing memtable size to total permitted size -# that will trigger a flush of the largest memtable. -memtable_cleanup_threshold: 0.4 +# that will trigger a flush of the largest memtable. Lager mct will +# mean larger flushes and hence less compaction, but also less concurrent +# flush activity which can make it difficult to keep your disks fed +# under heavy write load. +# +# memtable_cleanup_threshold defaults to 1 / (memtable_flush_writers + 1) +#memtable_cleanup_threshold: 0.11 # Specify the way Cassandra allocates and manages memtable memory. # Options are: @@ -304,9 +309,14 @@ memtable_allocation_type: heap_buffers # This sets the amount of memtable flush writer threads. These will # be blocked by disk io, and each one will hold a memtable in memory -# while blocked. If your flush directory is backed by SSD, you may -# want to increase this. -memtable_flush_writers: 2 +# while blocked. +# +# memtable_flush_writers defaults to the smaller of (number of disks, +# number of cores), with a minimum of 2 and a maximum of 8. +# +# If your data directories are backed by SSD, you should increase this +# to the number of cores. +#memtable_flush_writers: 8 # A fixed memory pool size in MB for for SSTable index summaries. If left # empty, this will default to 5% of the heap size. If the memory usage of @@ -503,6 +513,9 @@ batch_size_warn_threshold_in_kb: 5 # # concurrent_compactors defaults to the smaller of (number of disks, # number of cores), with a minimum of 2 and a maximum of 8. +# +# If your data directories are backed by SSD, you should increase this +# to the number of cores. #concurrent_compactors: 1 # Throttles compaction to the given total throughput across the entire http://git-wip-us.apache.org/repos/asf/cassandra/blob/3adf5948/src/java/org/apache/cassandra/config/Config.java -- diff --git a/src/java/org/apache/cassandra/config/Config.java b/src/java/org/apache/cassandra/config/Config.java index 4d4a95b..2438e09 100644 --- a/src/java/org/apache/cassandra/config/Config.java +++ b/src/java/org/apache/cassandra/config/Config.java @@ -90,12 +90,10 @@ public class Config @Deprecated public Integer concurrent_replicates = null; -// we don't want a lot of contention, but we also don't want to starve all other tables -// if a big one flushes. OS buffering should be able to minimize contention with 2 threads. -public int memtable_flush_writers = 2; +public Integer memtable_flush_writers = null; public Integer memtable_heap_space_in_mb; public Integer memtable_offheap_space_in_mb; -public float memtable_cleanup_threshold = 0.4f; +public Double memtable_cleanup_threshold = null; public Integer storage_port = 7000; public Integer ssl_storage_port = 7001;
[jira] [Commented] (CASSANDRA-7527) Bump CQL version and update doc for 2.1
[ https://issues.apache.org/jira/browse/CASSANDRA-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062586#comment-14062586 ] Aleksey Yeschenko commented on CASSANDRA-7527: -- v2 LGTM, +1 Bump CQL version and update doc for 2.1 --- Key: CASSANDRA-7527 URL: https://issues.apache.org/jira/browse/CASSANDRA-7527 Project: Cassandra Issue Type: Bug Reporter: Sylvain Lebresne Assignee: Tyler Hobbs Fix For: 2.1.0 Attachments: 7527-v2.txt, 7527.txt It appears we forgot to bump the CQL version for new 2.1 features (UDT, tuple type, collection indexing), nor did we update the textile doc -- This message was sent by Atlassian JIRA (v6.2#6252)
[1/8] git commit: Add missing table name to DROP INDEX responses
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 0b7377fed - 9a4cc22a3
refs/heads/cassandra-2.1.0 bd0eafd58 - 3adf59486
refs/heads/trunk ae25812e3 - da650f3c6
Add missing table name to DROP INDEX responses
Patch and review by Tyler Hobbs and Aleksey Yeschenko for CASSANDRA-7539
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/bd0eafd5
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/bd0eafd5
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/bd0eafd5
Branch: refs/heads/trunk
Commit: bd0eafd5853a446b0db79547231a0a9653a74293
Parents: 0401761
Author: Tyler Hobbs ty...@datastax.com
Authored: Tue Jul 15 12:23:06 2014 -0500
Committer: Tyler Hobbs ty...@datastax.com
Committed: Tue Jul 15 12:23:06 2014 -0500
--
CHANGES.txt | 1 +
.../cassandra/cql3/statements/DropIndexStatement.java| 11 +++
src/java/org/apache/cassandra/transport/Event.java | 2 ++
3 files changed, 14 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd0eafd5/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 193cff8..e6ded2a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,6 +1,7 @@
2.1.0-final
* (Windows) fix startup when WMI memory query fails (CASSANDRA-7505)
* Anti-compaction proceeds if any part of the repair failed (CASANDRA-7521)
+ * Add missing table name to DROP INDEX responses and notifications
(CASSANDRA-7539)
Merged from 2.0:
* (Windows) force range-based repair to non-sequential mode (CASSANDRA-7541)
* Fix range merging when DES scores are zero (CASSANDRA-7535)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd0eafd5/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
--
diff --git
a/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
b/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
index b8a4c9a..0562cc6 100644
--- a/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
+++ b/src/java/org/apache/cassandra/cql3/statements/DropIndexStatement.java
@@ -33,6 +33,9 @@ public class DropIndexStatement extends
SchemaAlteringStatement
public final String indexName;
public final boolean ifExists;
+// initialized in announceMigration()
+private String indexedCF;
+
public DropIndexStatement(IndexName indexName, boolean ifExists)
{
super(indexName.getCfName());
@@ -67,6 +70,7 @@ public class DropIndexStatement extends
SchemaAlteringStatement
return;
CFMetaData updatedCfm = updateCFMetadata(cfm);
+indexedCF = updatedCfm.cfName;
MigrationManager.announceColumnFamilyUpdate(updatedCfm, false,
isLocalOnly);
}
@@ -106,4 +110,11 @@ public class DropIndexStatement extends
SchemaAlteringStatement
}
return null;
}
+
+@Override
+public String columnFamily()
+{
+assert indexedCF != null;
+return indexedCF;
+}
}
http://git-wip-us.apache.org/repos/asf/cassandra/blob/bd0eafd5/src/java/org/apache/cassandra/transport/Event.java
--
diff --git a/src/java/org/apache/cassandra/transport/Event.java
b/src/java/org/apache/cassandra/transport/Event.java
index 3fa2733..1b7e487 100644
--- a/src/java/org/apache/cassandra/transport/Event.java
+++ b/src/java/org/apache/cassandra/transport/Event.java
@@ -218,6 +218,8 @@ public abstract class Event
this.target = target;
this.keyspace = keyspace;
this.tableOrType = tableOrType;
+if (target != Target.KEYSPACE)
+assert this.tableOrType != null : Table or type should be set
for non-keyspace schema change events;
}
public SchemaChange(Change change, String keyspace)
[2/8] git commit: Merge branch 'cassandra-2.1.0' into cassandra-2.1
Merge branch 'cassandra-2.1.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0b7377fe Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0b7377fe Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0b7377fe Branch: refs/heads/trunk Commit: 0b7377fed597eb41bc23224c80747130561165cf Parents: 8a2d8a1 bd0eafd Author: Tyler Hobbs ty...@datastax.com Authored: Tue Jul 15 12:24:12 2014 -0500 Committer: Tyler Hobbs ty...@datastax.com Committed: Tue Jul 15 12:24:12 2014 -0500 -- CHANGES.txt | 1 + .../cassandra/cql3/statements/DropIndexStatement.java| 11 +++ src/java/org/apache/cassandra/transport/Event.java | 2 ++ 3 files changed, 14 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/0b7377fe/CHANGES.txt --
[7/8] git commit: Merge branch 'cassandra-2.1.0' into cassandra-2.1
Merge branch 'cassandra-2.1.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/9a4cc22a Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/9a4cc22a Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/9a4cc22a Branch: refs/heads/cassandra-2.1 Commit: 9a4cc22a3ebf54a52bcb7e361837ed9f51fbc58b Parents: 0b7377f 3adf594 Author: Jonathan Ellis jbel...@apache.org Authored: Tue Jul 15 15:15:05 2014 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Tue Jul 15 15:15:05 2014 -0500 -- CHANGES.txt | 2 ++ conf/cassandra.yaml | 23 +++- .../org/apache/cassandra/config/Config.java | 6 ++--- .../cassandra/config/DatabaseDescriptor.java| 6 + 4 files changed, 28 insertions(+), 9 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a4cc22a/CHANGES.txt -- diff --cc CHANGES.txt index fc6d4eb,b20c465..65763dc --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,15 -1,6 +1,17 @@@ +2.1.1 + * Add listen_interface and rpc_interface options (CASSANDRA-7417) + * Fail to start if commit log replay detects a problem (CASSANDRA-7125) + * Improve schema merge performance (CASSANDRA-7444) + * Fix NPE when unknown prepared statement ID is used (CASSANDRA-7454) + * Adjust MT depth based on # of partition validating (CASSANDRA-5263) + * Optimise NativeCell comparisons (CASSANDRA-6755) + * Configurable client timeout for cqlsh (CASSANDRA-7516) + * Include snippet of CQL query near syntax error in messages (CASSANDRA-7111) + + 2.1.0-final + * Updated memtable_cleanup_threshold and memtable_flush_writers defaults +(CASSANDRA-7551) * (Windows) fix startup when WMI memory query fails (CASSANDRA-7505) * Anti-compaction proceeds if any part of the repair failed (CASANDRA-7521) * Add missing table name to DROP INDEX responses and notifications (CASSANDRA-7539) http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a4cc22a/conf/cassandra.yaml -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a4cc22a/src/java/org/apache/cassandra/config/Config.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/9a4cc22a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java --
[jira] [Assigned] (CASSANDRA-7525) Querying by multiple secondary indexes gives java.lang.IllegalArgumentException on some cases
[
https://issues.apache.org/jira/browse/CASSANDRA-7525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis reassigned CASSANDRA-7525:
-
Assignee: Sam Tunnicliffe (was: Tyler Hobbs)
Querying by multiple secondary indexes gives
java.lang.IllegalArgumentException on some cases
-
Key: CASSANDRA-7525
URL: https://issues.apache.org/jira/browse/CASSANDRA-7525
Project: Cassandra
Issue Type: Bug
Reporter: Tuukka Mustonen
Assignee: Sam Tunnicliffe
Fix For: 2.1.0
Querying by:
* non-primary key column with secondary index + collection with secondary
index
* primary key column with secondary index + non-primary key column with
secondary index
gives {{ErrorMessage code= [Server error]
message=java.lang.IllegalArgumentException: expected one element but was:
org.apache.cassandra.db.index.composites.CompositesSearcher@156b5576,
org.apache.cassandra.db.index.composites.CompositesSearcher@39b7c0f6}}
Steps to reproduce:
{code}
cqlsh:cs CREATE TABLE test (
... id1 text,
... id2 text,
... column1 text,
... column2 text,
... collection settext,
... PRIMARY KEY (id1, id2)
... );
cqlsh:cs
{code}
Simple query by primary key works as should:
{code}
cqlsh:cs SELECT * FROM test WHERE id1 = 'foo';
(0 rows)
cqlsh:cs SELECT * FROM test WHERE id1 = 'foo' AND id2 = 'bar';
(0 rows)
{code}
Query by secondary index + non-indexed column:
{code}
cqlsh:cs CREATE INDEX test_column1s ON test(column1);
cqlsh:cs SELECT * FROM test WHERE column1 = 'foo';
(0 rows)
cqlsh:cs SELECT * FROM test WHERE column1 = 'foo' AND column2 = 'bar' ALLOW
FILTERING;
(0 rows)
{code}
Add secondary index also for the second column:
{code}
cqlsh:cs CREATE INDEX test_column2s ON test(column2);
cqlsh:cs SELECT * FROM test WHERE column1 = 'foo' AND column2 = 'bar' ALLOW
FILTERING;
(0 rows)
{code}
Doesn't matter if we bring primary key or the collection there as well:
{code}
cqlsh:cs SELECT * FROM test WHERE id1 = 'blah' AND column1 = 'foo' AND
column2 = 'bar' ALLOW FILTERING;
(0 rows)
cqlsh:cs SELECT * FROM test WHERE column1 = 'foo' AND collection CONTAINS
'bar' ALLOW FILTERING;
(0 rows)
{code}
Let's add index for the collection:
{code}
cqlsh:cs CREATE INDEX test_collections ON test(collection);
cqlsh:cs SELECT * FROM test WHERE collection CONTAINS 'bar';
(0 rows)
{code}
But then combine secondary index column and collection with secondary index:
{code}
cqlsh:cs SELECT * FROM test WHERE column1 = 'foo' AND collection CONTAINS
'bar' ALLOW FILTERING;
ErrorMessage code= [Server error]
message=java.lang.IllegalArgumentException: expected one element but was:
org.apache.cassandra.db.index.composites.CompositesSearcher@68e37722,
org.apache.cassandra.db.index.composites.CompositesSearcher@2da28efa
{code}
Furthermore, query by primary key field with secondary index + non-primary
key column with secondary index:
{code}
cqlsh:cs CREATE INDEX test_id2s ON test(id2);
cqlsh:cs SELECT * FROM test WHERE id2 = 'foo' AND column1 = 'bar' ALLOW
FILTERING;
ErrorMessage code= [Server error]
message=java.lang.IllegalArgumentException: expected one element but was:
org.apache.cassandra.db.index.composites.CompositesSearcher@3ca899f1,
org.apache.cassandra.db.index.composites.CompositesSearcher@6112bc53
{code}
I'm a cassandra noob so maybe I'm trying to do things the db is not meant to
do?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
buildbot failure in ASF Buildbot on cassandra-2.1
The Buildbot has detected a new failure on builder cassandra-2.1 while building cassandra. Full details are available at: http://ci.apache.org/builders/cassandra-2.1/builds/240 Buildbot URL: http://ci.apache.org/ Buildslave for this Build: portunus_ubuntu Build Reason: scheduler Build Source Stamp: [branch cassandra-2.1] 9a4cc22a3ebf54a52bcb7e361837ed9f51fbc58b Blamelist: Jonathan Ellis jbel...@apache.org BUILD FAILED: failed compile sincerely, -The Buildbot
[1/6] git commit: add cast to float
Repository: cassandra Updated Branches: refs/heads/cassandra-2.1 9a4cc22a3 - 1366c0ad5 refs/heads/cassandra-2.1.0 3adf59486 - 233bd3c03 refs/heads/trunk da650f3c6 - 2d0cae4f0 add cast to float Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/233bd3c0 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/233bd3c0 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/233bd3c0 Branch: refs/heads/cassandra-2.1 Commit: 233bd3c0340d6bd19fe0c25072bd3b4931be0e1f Parents: 3adf594 Author: Jonathan Ellis jbel...@apache.org Authored: Tue Jul 15 15:21:41 2014 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Tue Jul 15 15:21:41 2014 -0500 -- src/java/org/apache/cassandra/config/Config.java | 2 +- src/java/org/apache/cassandra/config/DatabaseDescriptor.java | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/233bd3c0/src/java/org/apache/cassandra/config/Config.java -- diff --git a/src/java/org/apache/cassandra/config/Config.java b/src/java/org/apache/cassandra/config/Config.java index 2438e09..12c07bd 100644 --- a/src/java/org/apache/cassandra/config/Config.java +++ b/src/java/org/apache/cassandra/config/Config.java @@ -93,7 +93,7 @@ public class Config public Integer memtable_flush_writers = null; public Integer memtable_heap_space_in_mb; public Integer memtable_offheap_space_in_mb; -public Double memtable_cleanup_threshold = null; +public Float memtable_cleanup_threshold = null; public Integer storage_port = 7000; public Integer ssl_storage_port = 7001; http://git-wip-us.apache.org/repos/asf/cassandra/blob/233bd3c0/src/java/org/apache/cassandra/config/DatabaseDescriptor.java -- diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java index 6730cb5..d56bfe3 100644 --- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java +++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java @@ -487,7 +487,7 @@ public class DatabaseDescriptor conf.memtable_flush_writers = Math.min(8, Math.max(2, Math.min(FBUtilities.getAvailableProcessors(), conf.data_file_directories.length))); if (conf.memtable_cleanup_threshold == null) -conf.memtable_cleanup_threshold = 1.0 / (1 + conf.memtable_flush_writers); +conf.memtable_cleanup_threshold = (float) (1.0 / (1 + conf.memtable_flush_writers)); if (conf.concurrent_compactors == null) conf.concurrent_compactors = Math.min(8, Math.max(2, Math.min(FBUtilities.getAvailableProcessors(), conf.data_file_directories.length)));
[4/6] git commit: Merge branch 'cassandra-2.1.0' into cassandra-2.1
Merge branch 'cassandra-2.1.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1366c0ad Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1366c0ad Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1366c0ad Branch: refs/heads/trunk Commit: 1366c0ad51645928f35aeafb59e0e64898b19f0c Parents: 9a4cc22 233bd3c Author: Jonathan Ellis jbel...@apache.org Authored: Tue Jul 15 15:22:01 2014 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Tue Jul 15 15:22:01 2014 -0500 -- src/java/org/apache/cassandra/config/Config.java | 2 +- src/java/org/apache/cassandra/config/DatabaseDescriptor.java | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/1366c0ad/src/java/org/apache/cassandra/config/Config.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/1366c0ad/src/java/org/apache/cassandra/config/DatabaseDescriptor.java --
[6/6] git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2d0cae4f Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2d0cae4f Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2d0cae4f Branch: refs/heads/trunk Commit: 2d0cae4f0b7d5cc176c88f959dda8570c6992fb9 Parents: da650f3 1366c0a Author: Jonathan Ellis jbel...@apache.org Authored: Tue Jul 15 15:22:07 2014 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Tue Jul 15 15:22:07 2014 -0500 -- src/java/org/apache/cassandra/config/Config.java | 2 +- src/java/org/apache/cassandra/config/DatabaseDescriptor.java | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/2d0cae4f/src/java/org/apache/cassandra/config/DatabaseDescriptor.java --
[2/6] git commit: add cast to float
add cast to float Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/233bd3c0 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/233bd3c0 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/233bd3c0 Branch: refs/heads/cassandra-2.1.0 Commit: 233bd3c0340d6bd19fe0c25072bd3b4931be0e1f Parents: 3adf594 Author: Jonathan Ellis jbel...@apache.org Authored: Tue Jul 15 15:21:41 2014 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Tue Jul 15 15:21:41 2014 -0500 -- src/java/org/apache/cassandra/config/Config.java | 2 +- src/java/org/apache/cassandra/config/DatabaseDescriptor.java | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/233bd3c0/src/java/org/apache/cassandra/config/Config.java -- diff --git a/src/java/org/apache/cassandra/config/Config.java b/src/java/org/apache/cassandra/config/Config.java index 2438e09..12c07bd 100644 --- a/src/java/org/apache/cassandra/config/Config.java +++ b/src/java/org/apache/cassandra/config/Config.java @@ -93,7 +93,7 @@ public class Config public Integer memtable_flush_writers = null; public Integer memtable_heap_space_in_mb; public Integer memtable_offheap_space_in_mb; -public Double memtable_cleanup_threshold = null; +public Float memtable_cleanup_threshold = null; public Integer storage_port = 7000; public Integer ssl_storage_port = 7001; http://git-wip-us.apache.org/repos/asf/cassandra/blob/233bd3c0/src/java/org/apache/cassandra/config/DatabaseDescriptor.java -- diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java index 6730cb5..d56bfe3 100644 --- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java +++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java @@ -487,7 +487,7 @@ public class DatabaseDescriptor conf.memtable_flush_writers = Math.min(8, Math.max(2, Math.min(FBUtilities.getAvailableProcessors(), conf.data_file_directories.length))); if (conf.memtable_cleanup_threshold == null) -conf.memtable_cleanup_threshold = 1.0 / (1 + conf.memtable_flush_writers); +conf.memtable_cleanup_threshold = (float) (1.0 / (1 + conf.memtable_flush_writers)); if (conf.concurrent_compactors == null) conf.concurrent_compactors = Math.min(8, Math.max(2, Math.min(FBUtilities.getAvailableProcessors(), conf.data_file_directories.length)));
[3/6] git commit: add cast to float
add cast to float Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/233bd3c0 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/233bd3c0 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/233bd3c0 Branch: refs/heads/trunk Commit: 233bd3c0340d6bd19fe0c25072bd3b4931be0e1f Parents: 3adf594 Author: Jonathan Ellis jbel...@apache.org Authored: Tue Jul 15 15:21:41 2014 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Tue Jul 15 15:21:41 2014 -0500 -- src/java/org/apache/cassandra/config/Config.java | 2 +- src/java/org/apache/cassandra/config/DatabaseDescriptor.java | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/233bd3c0/src/java/org/apache/cassandra/config/Config.java -- diff --git a/src/java/org/apache/cassandra/config/Config.java b/src/java/org/apache/cassandra/config/Config.java index 2438e09..12c07bd 100644 --- a/src/java/org/apache/cassandra/config/Config.java +++ b/src/java/org/apache/cassandra/config/Config.java @@ -93,7 +93,7 @@ public class Config public Integer memtable_flush_writers = null; public Integer memtable_heap_space_in_mb; public Integer memtable_offheap_space_in_mb; -public Double memtable_cleanup_threshold = null; +public Float memtable_cleanup_threshold = null; public Integer storage_port = 7000; public Integer ssl_storage_port = 7001; http://git-wip-us.apache.org/repos/asf/cassandra/blob/233bd3c0/src/java/org/apache/cassandra/config/DatabaseDescriptor.java -- diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java index 6730cb5..d56bfe3 100644 --- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java +++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java @@ -487,7 +487,7 @@ public class DatabaseDescriptor conf.memtable_flush_writers = Math.min(8, Math.max(2, Math.min(FBUtilities.getAvailableProcessors(), conf.data_file_directories.length))); if (conf.memtable_cleanup_threshold == null) -conf.memtable_cleanup_threshold = 1.0 / (1 + conf.memtable_flush_writers); +conf.memtable_cleanup_threshold = (float) (1.0 / (1 + conf.memtable_flush_writers)); if (conf.concurrent_compactors == null) conf.concurrent_compactors = Math.min(8, Math.max(2, Math.min(FBUtilities.getAvailableProcessors(), conf.data_file_directories.length)));
[jira] [Commented] (CASSANDRA-7477) JSON to SSTable import failing
[
https://issues.apache.org/jira/browse/CASSANDRA-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062598#comment-14062598
]
Kishan Karunaratne commented on CASSANDRA-7477:
---
Here is the updated status, tested on various HEAD:
*Test fails on 1.2 branch: Error: Could not find or load main class
org.apache.cassandra.service.CassandraDaemon
*Test passes on 2.0 branch
*Test fails on 2.1 branch with the above original error
*Test fails on 3.0/trunk branch with an unknown error (assertion fails)
I wasn't able to replicate the error you got on any of the branches.
JSON to SSTable import failing
--
Key: CASSANDRA-7477
URL: https://issues.apache.org/jira/browse/CASSANDRA-7477
Project: Cassandra
Issue Type: Bug
Environment: Linux Mint 17 64-bit | 16GiB | C* 2.1
Reporter: Kishan Karunaratne
Assignee: Mikhail Stepura
Fix For: 2.1.0
Attachments: log2.log
Issue affects C* version = 2.1. Commit found by using git bisect. The
previous commit to this one also fails, but due to other reasons (CCM server
won't start). This commit is the one that give the same error as 2.1 HEAD:
{noformat}
02d1e7497a9930120fac367ce82a3b22940acafb is the first bad commit
commit 02d1e7497a9930120fac367ce82a3b22940acafb
Author: Brandon Williams brandonwilli...@apache.org
Date: Mon Apr 21 14:42:29 2014 -0500
Default flush dir to data dir.
Patch by brandonwilliams, reviewed by yukim for CASSANDRA-7064
:04 04 c50a123f305b73583ccbfa9c455efc4e4cee228f
507a90290dccb8a929afadf1f833d926049c46ad Mconf
{noformat}
{noformat}
$ PRINT_DEBUG=true nosetests -x -s -v json_tools_test.py
json_tools_test (json_tools_test.TestJson) ... cluster ccm directory:
/tmp/dtest-8WVBq9
Starting cluster...
Version: 2.1.0
Getting CQLSH...
Inserting data...
Flushing and stopping cluster...
Exporting to JSON file...
-- test-users-ka-1-Data.db -
Deleting cluster and creating new...
Inserting data...
Importing JSON file...
Counting keys to import, please wait... (NOTE: to skip this use -n num_keys)
Importing 2 keys...
java.lang.ClassCastException:
org.apache.cassandra.db.composites.Composites$EmptyComposite cannot be cast
to org.apache.cassandra.db.composites.CellName
at
org.apache.cassandra.db.composites.AbstractCellNameType.cellFromByteBuffer(AbstractCellNameType.java:168)
at
org.apache.cassandra.tools.SSTableImport$JsonColumn.init(SSTableImport.java:165)
at
org.apache.cassandra.tools.SSTableImport.addColumnsToCF(SSTableImport.java:242)
at
org.apache.cassandra.tools.SSTableImport.addToStandardCF(SSTableImport.java:225)
at
org.apache.cassandra.tools.SSTableImport.importSorted(SSTableImport.java:464)
at
org.apache.cassandra.tools.SSTableImport.importJson(SSTableImport.java:351)
at org.apache.cassandra.tools.SSTableImport.main(SSTableImport.java:575)
ERROR: org.apache.cassandra.db.composites.Composites$EmptyComposite cannot be
cast to org.apache.cassandra.db.composites.CellName
Verifying import...
data: [[u'gandalf', 1955, u'male', u'p@$$', u'WA']]
FAIL
removing ccm cluster test at: /tmp/dtest-8WVBq9
ERROR
==
ERROR: json_tools_test (json_tools_test.TestJson)
--
Traceback (most recent call last):
File /home/kishan/git/cstar/cassandra-dtest/dtest.py, line 214, in
tearDown
raise AssertionError('Unexpected error in %s node log: %s' % (node.name,
errors))
AssertionError: Unexpected error in node1 node log: ['ERROR
[SSTableBatchOpen:1] 2014-06-30 13:56:01,032 CassandraDaemon.java:166 -
Exception in thread Thread[SSTableBatchOpen:1,5,main]\n']
begin captured logging
dtest: DEBUG: cluster ccm directory: /tmp/dtest-8WVBq9
dtest: DEBUG: Starting cluster...
dtest: DEBUG: Version: 2.1.0
dtest: DEBUG: Getting CQLSH...
dtest: DEBUG: Inserting data...
dtest: DEBUG: Flushing and stopping cluster...
dtest: DEBUG: Exporting to JSON file...
dtest: DEBUG: Deleting cluster and creating new...
dtest: DEBUG: Inserting data...
dtest: DEBUG: Importing JSON file...
dtest: DEBUG: Verifying import...
dtest: DEBUG: data: [[u'gandalf', 1955, u'male', u'p@$$', u'WA']]
dtest: DEBUG: removing ccm cluster test at: /tmp/dtest-8WVBq9
- end captured logging -
==
FAIL: json_tools_test (json_tools_test.TestJson)
--
Traceback (most recent call last):
File /home/kishan/git/cstar/cassandra-dtest/json_tools_test.py, line 91,
in json_tools_test
[jira] [Comment Edited] (CASSANDRA-7477) JSON to SSTable import failing
[
https://issues.apache.org/jira/browse/CASSANDRA-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062598#comment-14062598
]
Kishan Karunaratne edited comment on CASSANDRA-7477 at 7/15/14 8:33 PM:
Here is the updated status, tested on various HEAD:
*Test fails on 1.2 branch: Error: Could not find or load main class
org.apache.cassandra.service.CassandraDaemon
*Test passes on 2.0 branch
*Test fails on 2.1 branch with the above original error. Also fails on 2.1.0-rc3
*Test fails on 3.0/trunk branch with an unknown error (assertion fails)
I wasn't able to replicate the error you got on any of the branches.
was (Author: kishkaru):
Here is the updated status, tested on various HEAD:
*Test fails on 1.2 branch: Error: Could not find or load main class
org.apache.cassandra.service.CassandraDaemon
*Test passes on 2.0 branch
*Test fails on 2.1 branch with the above original error
*Test fails on 3.0/trunk branch with an unknown error (assertion fails)
I wasn't able to replicate the error you got on any of the branches.
JSON to SSTable import failing
--
Key: CASSANDRA-7477
URL: https://issues.apache.org/jira/browse/CASSANDRA-7477
Project: Cassandra
Issue Type: Bug
Environment: Linux Mint 17 64-bit | 16GiB | C* 2.1
Reporter: Kishan Karunaratne
Assignee: Mikhail Stepura
Fix For: 2.1.0
Attachments: log2.log
Issue affects C* version = 2.1. Commit found by using git bisect. The
previous commit to this one also fails, but due to other reasons (CCM server
won't start). This commit is the one that give the same error as 2.1 HEAD:
{noformat}
02d1e7497a9930120fac367ce82a3b22940acafb is the first bad commit
commit 02d1e7497a9930120fac367ce82a3b22940acafb
Author: Brandon Williams brandonwilli...@apache.org
Date: Mon Apr 21 14:42:29 2014 -0500
Default flush dir to data dir.
Patch by brandonwilliams, reviewed by yukim for CASSANDRA-7064
:04 04 c50a123f305b73583ccbfa9c455efc4e4cee228f
507a90290dccb8a929afadf1f833d926049c46ad Mconf
{noformat}
{noformat}
$ PRINT_DEBUG=true nosetests -x -s -v json_tools_test.py
json_tools_test (json_tools_test.TestJson) ... cluster ccm directory:
/tmp/dtest-8WVBq9
Starting cluster...
Version: 2.1.0
Getting CQLSH...
Inserting data...
Flushing and stopping cluster...
Exporting to JSON file...
-- test-users-ka-1-Data.db -
Deleting cluster and creating new...
Inserting data...
Importing JSON file...
Counting keys to import, please wait... (NOTE: to skip this use -n num_keys)
Importing 2 keys...
java.lang.ClassCastException:
org.apache.cassandra.db.composites.Composites$EmptyComposite cannot be cast
to org.apache.cassandra.db.composites.CellName
at
org.apache.cassandra.db.composites.AbstractCellNameType.cellFromByteBuffer(AbstractCellNameType.java:168)
at
org.apache.cassandra.tools.SSTableImport$JsonColumn.init(SSTableImport.java:165)
at
org.apache.cassandra.tools.SSTableImport.addColumnsToCF(SSTableImport.java:242)
at
org.apache.cassandra.tools.SSTableImport.addToStandardCF(SSTableImport.java:225)
at
org.apache.cassandra.tools.SSTableImport.importSorted(SSTableImport.java:464)
at
org.apache.cassandra.tools.SSTableImport.importJson(SSTableImport.java:351)
at org.apache.cassandra.tools.SSTableImport.main(SSTableImport.java:575)
ERROR: org.apache.cassandra.db.composites.Composites$EmptyComposite cannot be
cast to org.apache.cassandra.db.composites.CellName
Verifying import...
data: [[u'gandalf', 1955, u'male', u'p@$$', u'WA']]
FAIL
removing ccm cluster test at: /tmp/dtest-8WVBq9
ERROR
==
ERROR: json_tools_test (json_tools_test.TestJson)
--
Traceback (most recent call last):
File /home/kishan/git/cstar/cassandra-dtest/dtest.py, line 214, in
tearDown
raise AssertionError('Unexpected error in %s node log: %s' % (node.name,
errors))
AssertionError: Unexpected error in node1 node log: ['ERROR
[SSTableBatchOpen:1] 2014-06-30 13:56:01,032 CassandraDaemon.java:166 -
Exception in thread Thread[SSTableBatchOpen:1,5,main]\n']
begin captured logging
dtest: DEBUG: cluster ccm directory: /tmp/dtest-8WVBq9
dtest: DEBUG: Starting cluster...
dtest: DEBUG: Version: 2.1.0
dtest: DEBUG: Getting CQLSH...
dtest: DEBUG: Inserting data...
dtest: DEBUG: Flushing and stopping cluster...
dtest: DEBUG: Exporting to JSON file...
dtest: DEBUG: Deleting cluster and creating new...
dtest: DEBUG: Inserting data...
dtest: DEBUG: Importing JSON file...
dtest: DEBUG: Verifying import...
dtest: DEBUG: data: [[u'gandalf', 1955,
[jira] [Commented] (CASSANDRA-7508) nodetool prints error if dirs don't exist.
[
https://issues.apache.org/jira/browse/CASSANDRA-7508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062620#comment-14062620
]
Jeremiah Jordan commented on CASSANDRA-7508:
LGTM no more error.
nodetool prints error if dirs don't exist.
--
Key: CASSANDRA-7508
URL: https://issues.apache.org/jira/browse/CASSANDRA-7508
Project: Cassandra
Issue Type: Bug
Components: Tools
Reporter: Jeremiah Jordan
Assignee: Tyler Hobbs
Priority: Minor
Fix For: 2.1.0
Attachments: 7508.txt
The new directory stuff blows up nodetool if its not there. Like if I run
nodetool from an different untar of the rc2 tar.gz.
{noformat}
apache-cassandra-2.1.0-rc2/bin $ ./nodetool ring
Note: Ownership information does not include topology; for complete
information, specify a keyspace
Datacenter: Analytics
==
AddressRackStatus State LoadOwns
Token
127.0.0.1 rack1 Up Normal 57.88 KB100.00%
3647214374146654083
ERROR 21:59:30 Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: commitlog_directory
is missing and -Dcassandra.storagedir is not set
at
org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:461)
~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at
org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:129)
~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at org.apache.cassandra.tools.NodeTool$Ring.execute(NodeTool.java:464)
[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:245)
[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:159)
[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
commitlog_directory is missing and -Dcassandra.storagedir is not set
Fatal configuration error; unable to start. See log for stack trace.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (CASSANDRA-7552) Compactions Pending build up when using LCS
Darla Baker created CASSANDRA-7552: -- Summary: Compactions Pending build up when using LCS Key: CASSANDRA-7552 URL: https://issues.apache.org/jira/browse/CASSANDRA-7552 Project: Cassandra Issue Type: Bug Components: Core Reporter: Darla Baker We seem to be hitting an issue with LeveledCompactionStrategy while running performance tests on a 4 node cassandra installation. We are currently using Cassandra 2.0.7. In summary, we run a tests consisting of approximatively, 8000 inserts/sec, 16,000 gets/sec, and 8,000 deletes/sec. We have a grace period of 12 hours on our column families. At this rate, we observe a stable pending compaction tasks for about 22 to 26 hours. After that period, something happens and the pending compaction tasks starts to increase rapidly, sometimes on one or two servers, but sometimes on all four of them. This goes on until the uncompacted SStables start consuming all the disk space, after which the cassandra cluster generally fails. When this occurs, the Compaction completed tasks rate is usually reducing over time, which seems to indicate that it takes more and more time to run the existing compaction tasks. At different occasions, I can reproduce a similar issue in less than 12 hours. While the traffic rate remains constant, we seem to be hitting this at various intervals. Yesterday I could reproduce in less than 6 hours. We have two different deployments on which we have tested this issue: 1. 4x IBM HS22, using RAMDISK as cassandra data directory (thus eliminating disk I/O) 2. 8x IBM HS23, with SSD disks, deployed in two geo-redundant data centers of 4 nodes each, and a latency of 50ms between the data centers. I can reproduce the compaction tasks falling behind on both these setup, although they could be occurring for different reasons. Because of #1, I do not believe we are hitting an I/O bottleneck just yet. As an additional interesting node, if I artificially pause the traffic when I see the pending compaction task issue occurring, then: 1. The pending compaction tasks obviously stops to increase, but stay at the same number for 15 minutes (as if nothing is running). 2. The completed compaction tasks falls to 0 for 15 minutes 3. After 15 to 20 minutes, out of the blue, all compaction completes in less than 2 minutes. If I restart the traffic after that, the system is stable for a few hours, but the issue always comes back. We have written a small test tool that reproduce our application's Cassandra interaction. We have not successfully run a test for more than 30 hours under load, and every failure after that time would follow a similar pattern. -- This message was sent by Atlassian JIRA (v6.2#6252)
[3/6] git commit: reorder to avoid NPE
reorder to avoid NPE
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fad5ad5a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fad5ad5a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fad5ad5a
Branch: refs/heads/trunk
Commit: fad5ad5a6af5a9ed2954a6395149f0ca95d34603
Parents: 233bd3c
Author: Jonathan Ellis jbel...@apache.org
Authored: Tue Jul 15 16:19:25 2014 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Tue Jul 15 16:19:25 2014 -0500
--
.../cassandra/config/DatabaseDescriptor.java | 19 ++-
1 file changed, 10 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/fad5ad5a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index d56bfe3..d89967d 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -288,15 +288,6 @@ public class DatabaseDescriptor
logger.info(Global memtable off-heap threshold is disabled,
HeapAllocator will be used instead);
else
logger.info(Global memtable off-heap threshold is enabled at
{}MB, conf.memtable_offheap_space_in_mb);
-if (conf.memtable_cleanup_threshold 0.01f)
-throw new ConfigurationException(memtable_cleanup_threshold must
be = 0.01);
-if (conf.memtable_cleanup_threshold 0.99f)
-throw new ConfigurationException(memtable_cleanup_threshold must
be = 0.99);
-if (conf.memtable_cleanup_threshold 0.1f)
-logger.warn(memtable_cleanup_threshold is set very low, which may
cause performance degradation);
-
-if (conf.memtable_flush_writers 1)
-throw new ConfigurationException(memtable_flush_writers must be
at least 1);
/* Local IP or hostname to bind services to */
if (conf.listen_address != null)
@@ -486,9 +477,19 @@ public class DatabaseDescriptor
if (conf.memtable_flush_writers == null)
conf.memtable_flush_writers = Math.min(8, Math.max(2,
Math.min(FBUtilities.getAvailableProcessors(),
conf.data_file_directories.length)));
+if (conf.memtable_flush_writers 1)
+throw new ConfigurationException(memtable_flush_writers must be
at least 1);
+
if (conf.memtable_cleanup_threshold == null)
conf.memtable_cleanup_threshold = (float) (1.0 / (1 +
conf.memtable_flush_writers));
+if (conf.memtable_cleanup_threshold 0.01f)
+throw new ConfigurationException(memtable_cleanup_threshold must
be = 0.01);
+if (conf.memtable_cleanup_threshold 0.99f)
+throw new ConfigurationException(memtable_cleanup_threshold must
be = 0.99);
+if (conf.memtable_cleanup_threshold 0.1f)
+logger.warn(memtable_cleanup_threshold is set very low, which may
cause performance degradation);
+
if (conf.concurrent_compactors == null)
conf.concurrent_compactors = Math.min(8, Math.max(2,
Math.min(FBUtilities.getAvailableProcessors(),
conf.data_file_directories.length)));
[5/6] git commit: Merge branch 'cassandra-2.1.0' into cassandra-2.1
Merge branch 'cassandra-2.1.0' into cassandra-2.1
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f9dbc10d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f9dbc10d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f9dbc10d
Branch: refs/heads/cassandra-2.1
Commit: f9dbc10db3046a783c601833ceb8f2ad0115e95e
Parents: 1366c0a fad5ad5
Author: Jonathan Ellis jbel...@apache.org
Authored: Tue Jul 15 16:20:53 2014 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Tue Jul 15 16:20:53 2014 -0500
--
.../cassandra/config/DatabaseDescriptor.java | 19 ++-
1 file changed, 10 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f9dbc10d/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --cc src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index 9a54581,d89967d..eeea33a
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@@ -290,22 -288,9 +290,13 @@@ public class DatabaseDescripto
logger.info(Global memtable off-heap threshold is disabled,
HeapAllocator will be used instead);
else
logger.info(Global memtable off-heap threshold is enabled at
{}MB, conf.memtable_offheap_space_in_mb);
- if (conf.memtable_cleanup_threshold 0.01f)
- throw new ConfigurationException(memtable_cleanup_threshold must
be = 0.01);
- if (conf.memtable_cleanup_threshold 0.99f)
- throw new ConfigurationException(memtable_cleanup_threshold must
be = 0.99);
- if (conf.memtable_cleanup_threshold 0.1f)
- logger.warn(memtable_cleanup_threshold is set very low, which
may cause performance degradation);
-
- if (conf.memtable_flush_writers 1)
- throw new ConfigurationException(memtable_flush_writers must be
at least 1);
-/* Local IP or hostname to bind services to */
-if (conf.listen_address != null)
+/* Local IP, hostname or interface to bind services to */
+if (conf.listen_address != null conf.listen_interface != null)
+{
+throw new ConfigurationException(Set listen_address OR
listen_interface, not both);
+}
+else if (conf.listen_address != null)
{
if (conf.listen_address.equals(0.0.0.0))
throw new ConfigurationException(listen_address cannot be
0.0.0.0!);
[6/6] git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e240e515 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e240e515 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e240e515 Branch: refs/heads/trunk Commit: e240e515fd1b79512904e66a6f3bfe225eca8b92 Parents: 2d0cae4 f9dbc10 Author: Jonathan Ellis jbel...@apache.org Authored: Tue Jul 15 16:21:00 2014 -0500 Committer: Jonathan Ellis jbel...@apache.org Committed: Tue Jul 15 16:21:00 2014 -0500 -- .../cassandra/config/DatabaseDescriptor.java | 19 ++- 1 file changed, 10 insertions(+), 9 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/e240e515/src/java/org/apache/cassandra/config/DatabaseDescriptor.java --
[4/6] git commit: Merge branch 'cassandra-2.1.0' into cassandra-2.1
Merge branch 'cassandra-2.1.0' into cassandra-2.1
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f9dbc10d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f9dbc10d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f9dbc10d
Branch: refs/heads/trunk
Commit: f9dbc10db3046a783c601833ceb8f2ad0115e95e
Parents: 1366c0a fad5ad5
Author: Jonathan Ellis jbel...@apache.org
Authored: Tue Jul 15 16:20:53 2014 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Tue Jul 15 16:20:53 2014 -0500
--
.../cassandra/config/DatabaseDescriptor.java | 19 ++-
1 file changed, 10 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/f9dbc10d/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --cc src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index 9a54581,d89967d..eeea33a
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@@ -290,22 -288,9 +290,13 @@@ public class DatabaseDescripto
logger.info(Global memtable off-heap threshold is disabled,
HeapAllocator will be used instead);
else
logger.info(Global memtable off-heap threshold is enabled at
{}MB, conf.memtable_offheap_space_in_mb);
- if (conf.memtable_cleanup_threshold 0.01f)
- throw new ConfigurationException(memtable_cleanup_threshold must
be = 0.01);
- if (conf.memtable_cleanup_threshold 0.99f)
- throw new ConfigurationException(memtable_cleanup_threshold must
be = 0.99);
- if (conf.memtable_cleanup_threshold 0.1f)
- logger.warn(memtable_cleanup_threshold is set very low, which
may cause performance degradation);
-
- if (conf.memtable_flush_writers 1)
- throw new ConfigurationException(memtable_flush_writers must be
at least 1);
-/* Local IP or hostname to bind services to */
-if (conf.listen_address != null)
+/* Local IP, hostname or interface to bind services to */
+if (conf.listen_address != null conf.listen_interface != null)
+{
+throw new ConfigurationException(Set listen_address OR
listen_interface, not both);
+}
+else if (conf.listen_address != null)
{
if (conf.listen_address.equals(0.0.0.0))
throw new ConfigurationException(listen_address cannot be
0.0.0.0!);
[2/6] git commit: reorder to avoid NPE
reorder to avoid NPE
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fad5ad5a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fad5ad5a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fad5ad5a
Branch: refs/heads/cassandra-2.1.0
Commit: fad5ad5a6af5a9ed2954a6395149f0ca95d34603
Parents: 233bd3c
Author: Jonathan Ellis jbel...@apache.org
Authored: Tue Jul 15 16:19:25 2014 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Tue Jul 15 16:19:25 2014 -0500
--
.../cassandra/config/DatabaseDescriptor.java | 19 ++-
1 file changed, 10 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/fad5ad5a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index d56bfe3..d89967d 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -288,15 +288,6 @@ public class DatabaseDescriptor
logger.info(Global memtable off-heap threshold is disabled,
HeapAllocator will be used instead);
else
logger.info(Global memtable off-heap threshold is enabled at
{}MB, conf.memtable_offheap_space_in_mb);
-if (conf.memtable_cleanup_threshold 0.01f)
-throw new ConfigurationException(memtable_cleanup_threshold must
be = 0.01);
-if (conf.memtable_cleanup_threshold 0.99f)
-throw new ConfigurationException(memtable_cleanup_threshold must
be = 0.99);
-if (conf.memtable_cleanup_threshold 0.1f)
-logger.warn(memtable_cleanup_threshold is set very low, which may
cause performance degradation);
-
-if (conf.memtable_flush_writers 1)
-throw new ConfigurationException(memtable_flush_writers must be
at least 1);
/* Local IP or hostname to bind services to */
if (conf.listen_address != null)
@@ -486,9 +477,19 @@ public class DatabaseDescriptor
if (conf.memtable_flush_writers == null)
conf.memtable_flush_writers = Math.min(8, Math.max(2,
Math.min(FBUtilities.getAvailableProcessors(),
conf.data_file_directories.length)));
+if (conf.memtable_flush_writers 1)
+throw new ConfigurationException(memtable_flush_writers must be
at least 1);
+
if (conf.memtable_cleanup_threshold == null)
conf.memtable_cleanup_threshold = (float) (1.0 / (1 +
conf.memtable_flush_writers));
+if (conf.memtable_cleanup_threshold 0.01f)
+throw new ConfigurationException(memtable_cleanup_threshold must
be = 0.01);
+if (conf.memtable_cleanup_threshold 0.99f)
+throw new ConfigurationException(memtable_cleanup_threshold must
be = 0.99);
+if (conf.memtable_cleanup_threshold 0.1f)
+logger.warn(memtable_cleanup_threshold is set very low, which may
cause performance degradation);
+
if (conf.concurrent_compactors == null)
conf.concurrent_compactors = Math.min(8, Math.max(2,
Math.min(FBUtilities.getAvailableProcessors(),
conf.data_file_directories.length)));
[1/6] git commit: reorder to avoid NPE
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 1366c0ad5 - f9dbc10db
refs/heads/cassandra-2.1.0 233bd3c03 - fad5ad5a6
refs/heads/trunk 2d0cae4f0 - e240e515f
reorder to avoid NPE
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/fad5ad5a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/fad5ad5a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/fad5ad5a
Branch: refs/heads/cassandra-2.1
Commit: fad5ad5a6af5a9ed2954a6395149f0ca95d34603
Parents: 233bd3c
Author: Jonathan Ellis jbel...@apache.org
Authored: Tue Jul 15 16:19:25 2014 -0500
Committer: Jonathan Ellis jbel...@apache.org
Committed: Tue Jul 15 16:19:25 2014 -0500
--
.../cassandra/config/DatabaseDescriptor.java | 19 ++-
1 file changed, 10 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/fad5ad5a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
--
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index d56bfe3..d89967d 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -288,15 +288,6 @@ public class DatabaseDescriptor
logger.info(Global memtable off-heap threshold is disabled,
HeapAllocator will be used instead);
else
logger.info(Global memtable off-heap threshold is enabled at
{}MB, conf.memtable_offheap_space_in_mb);
-if (conf.memtable_cleanup_threshold 0.01f)
-throw new ConfigurationException(memtable_cleanup_threshold must
be = 0.01);
-if (conf.memtable_cleanup_threshold 0.99f)
-throw new ConfigurationException(memtable_cleanup_threshold must
be = 0.99);
-if (conf.memtable_cleanup_threshold 0.1f)
-logger.warn(memtable_cleanup_threshold is set very low, which may
cause performance degradation);
-
-if (conf.memtable_flush_writers 1)
-throw new ConfigurationException(memtable_flush_writers must be
at least 1);
/* Local IP or hostname to bind services to */
if (conf.listen_address != null)
@@ -486,9 +477,19 @@ public class DatabaseDescriptor
if (conf.memtable_flush_writers == null)
conf.memtable_flush_writers = Math.min(8, Math.max(2,
Math.min(FBUtilities.getAvailableProcessors(),
conf.data_file_directories.length)));
+if (conf.memtable_flush_writers 1)
+throw new ConfigurationException(memtable_flush_writers must be
at least 1);
+
if (conf.memtable_cleanup_threshold == null)
conf.memtable_cleanup_threshold = (float) (1.0 / (1 +
conf.memtable_flush_writers));
+if (conf.memtable_cleanup_threshold 0.01f)
+throw new ConfigurationException(memtable_cleanup_threshold must
be = 0.01);
+if (conf.memtable_cleanup_threshold 0.99f)
+throw new ConfigurationException(memtable_cleanup_threshold must
be = 0.99);
+if (conf.memtable_cleanup_threshold 0.1f)
+logger.warn(memtable_cleanup_threshold is set very low, which may
cause performance degradation);
+
if (conf.concurrent_compactors == null)
conf.concurrent_compactors = Math.min(8, Math.max(2,
Math.min(FBUtilities.getAvailableProcessors(),
conf.data_file_directories.length)));
[jira] [Assigned] (CASSANDRA-7552) Compactions Pending build up when using LCS
[ https://issues.apache.org/jira/browse/CASSANDRA-7552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis reassigned CASSANDRA-7552: - Assignee: Yuki Morishita Compactions Pending build up when using LCS --- Key: CASSANDRA-7552 URL: https://issues.apache.org/jira/browse/CASSANDRA-7552 Project: Cassandra Issue Type: Bug Components: Core Reporter: Darla Baker Assignee: Yuki Morishita We seem to be hitting an issue with LeveledCompactionStrategy while running performance tests on a 4 node cassandra installation. We are currently using Cassandra 2.0.7. In summary, we run a tests consisting of approximatively, 8000 inserts/sec, 16,000 gets/sec, and 8,000 deletes/sec. We have a grace period of 12 hours on our column families. At this rate, we observe a stable pending compaction tasks for about 22 to 26 hours. After that period, something happens and the pending compaction tasks starts to increase rapidly, sometimes on one or two servers, but sometimes on all four of them. This goes on until the uncompacted SStables start consuming all the disk space, after which the cassandra cluster generally fails. When this occurs, the Compaction completed tasks rate is usually reducing over time, which seems to indicate that it takes more and more time to run the existing compaction tasks. At different occasions, I can reproduce a similar issue in less than 12 hours. While the traffic rate remains constant, we seem to be hitting this at various intervals. Yesterday I could reproduce in less than 6 hours. We have two different deployments on which we have tested this issue: 1. 4x IBM HS22, using RAMDISK as cassandra data directory (thus eliminating disk I/O) 2. 8x IBM HS23, with SSD disks, deployed in two geo-redundant data centers of 4 nodes each, and a latency of 50ms between the data centers. I can reproduce the compaction tasks falling behind on both these setup, although they could be occurring for different reasons. Because of #1, I do not believe we are hitting an I/O bottleneck just yet. As an additional interesting node, if I artificially pause the traffic when I see the pending compaction task issue occurring, then: 1. The pending compaction tasks obviously stops to increase, but stay at the same number for 15 minutes (as if nothing is running). 2. The completed compaction tasks falls to 0 for 15 minutes 3. After 15 to 20 minutes, out of the blue, all compaction completes in less than 2 minutes. If I restart the traffic after that, the system is stable for a few hours, but the issue always comes back. We have written a small test tool that reproduce our application's Cassandra interaction. We have not successfully run a test for more than 30 hours under load, and every failure after that time would follow a similar pattern. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-7395) Support for pure user-defined functions (UDF)
[ https://issues.apache.org/jira/browse/CASSANDRA-7395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Stupp updated CASSANDRA-7395: Attachment: (was: 7395.txt) Support for pure user-defined functions (UDF) - Key: CASSANDRA-7395 URL: https://issues.apache.org/jira/browse/CASSANDRA-7395 Project: Cassandra Issue Type: New Feature Components: API, Core Reporter: Jonathan Ellis Labels: cql Fix For: 3.0 We have some tickets for various aspects of UDF (CASSANDRA-4914, CASSANDRA-5970, CASSANDRA-4998) but they all suffer from various degrees of ocean-boiling. Let's start with something simple: allowing pure user-defined functions in the SELECT clause of a CQL query. That's it. By pure I mean, must depend only on the input parameters. No side effects. No exposure to C* internals. Column values in, result out. http://en.wikipedia.org/wiki/Pure_function -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-7395) Support for pure user-defined functions (UDF)
[
https://issues.apache.org/jira/browse/CASSANDRA-7395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Stupp updated CASSANDRA-7395:
Attachment: 7395.txt
new patch with support for UDFs as Java source using javassist
Syntax:
{noformat}
CREATE OR REPLACE FUNCTION double foobar(val double) AS JAVA
BODY
// parameter val is of type java.lang.Double
/* return type is of type java.lang.Double */
if (val == null) {
return null;
}
double v = Math.sin( val.doubleValue() );
return Double.valueOf(v);
END BODY;
{noformat}
But does not work with cqlsh (cqlsh needs to be changed to support ';' inside
statements / CREATE BUNDLE/FUNCTION syntax not yet in cqlsh).
Next steps:
* code cleanup (validate UDFs upon statement execution before writing to
system KS table)
* unit tests
Support for pure user-defined functions (UDF)
-
Key: CASSANDRA-7395
URL: https://issues.apache.org/jira/browse/CASSANDRA-7395
Project: Cassandra
Issue Type: New Feature
Components: API, Core
Reporter: Jonathan Ellis
Labels: cql
Fix For: 3.0
Attachments: 7395.txt
We have some tickets for various aspects of UDF (CASSANDRA-4914,
CASSANDRA-5970, CASSANDRA-4998) but they all suffer from various degrees of
ocean-boiling.
Let's start with something simple: allowing pure user-defined functions in
the SELECT clause of a CQL query. That's it.
By pure I mean, must depend only on the input parameters. No side effects.
No exposure to C* internals. Column values in, result out.
http://en.wikipedia.org/wiki/Pure_function
--
This message was sent by Atlassian JIRA
(v6.2#6252)
buildbot success in ASF Buildbot on cassandra-2.1
The Buildbot has detected a restored build on builder cassandra-2.1 while building cassandra. Full details are available at: http://ci.apache.org/builders/cassandra-2.1/builds/242 Buildbot URL: http://ci.apache.org/ Buildslave for this Build: portunus_ubuntu Build Reason: scheduler Build Source Stamp: [branch cassandra-2.1] f9dbc10db3046a783c601833ceb8f2ad0115e95e Blamelist: Jonathan Ellis jbel...@apache.org Build succeeded! sincerely, -The Buildbot
[jira] [Commented] (CASSANDRA-7552) Compactions Pending build up when using LCS
[ https://issues.apache.org/jira/browse/CASSANDRA-7552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062790#comment-14062790 ] Yuki Morishita commented on CASSANDRA-7552: --- Number of compaction pending tasks in LCS is estimated from total bytes of SSTable files in each level so that lower levels get more 'pending tasks' than higher level to show compaction is behind(https://github.com/apache/cassandra/blob/cassandra-2.0.7/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java#L526). I guess you have more writes that pile many SSTables on L0 than node with LCS can handle. Check your SSTable level distibution using nodetool cfstats. Compactions Pending build up when using LCS --- Key: CASSANDRA-7552 URL: https://issues.apache.org/jira/browse/CASSANDRA-7552 Project: Cassandra Issue Type: Bug Components: Core Reporter: Darla Baker Assignee: Yuki Morishita We seem to be hitting an issue with LeveledCompactionStrategy while running performance tests on a 4 node cassandra installation. We are currently using Cassandra 2.0.7. In summary, we run a tests consisting of approximatively, 8000 inserts/sec, 16,000 gets/sec, and 8,000 deletes/sec. We have a grace period of 12 hours on our column families. At this rate, we observe a stable pending compaction tasks for about 22 to 26 hours. After that period, something happens and the pending compaction tasks starts to increase rapidly, sometimes on one or two servers, but sometimes on all four of them. This goes on until the uncompacted SStables start consuming all the disk space, after which the cassandra cluster generally fails. When this occurs, the Compaction completed tasks rate is usually reducing over time, which seems to indicate that it takes more and more time to run the existing compaction tasks. At different occasions, I can reproduce a similar issue in less than 12 hours. While the traffic rate remains constant, we seem to be hitting this at various intervals. Yesterday I could reproduce in less than 6 hours. We have two different deployments on which we have tested this issue: 1. 4x IBM HS22, using RAMDISK as cassandra data directory (thus eliminating disk I/O) 2. 8x IBM HS23, with SSD disks, deployed in two geo-redundant data centers of 4 nodes each, and a latency of 50ms between the data centers. I can reproduce the compaction tasks falling behind on both these setup, although they could be occurring for different reasons. Because of #1, I do not believe we are hitting an I/O bottleneck just yet. As an additional interesting node, if I artificially pause the traffic when I see the pending compaction task issue occurring, then: 1. The pending compaction tasks obviously stops to increase, but stay at the same number for 15 minutes (as if nothing is running). 2. The completed compaction tasks falls to 0 for 15 minutes 3. After 15 to 20 minutes, out of the blue, all compaction completes in less than 2 minutes. If I restart the traffic after that, the system is stable for a few hours, but the issue always comes back. We have written a small test tool that reproduce our application's Cassandra interaction. We have not successfully run a test for more than 30 hours under load, and every failure after that time would follow a similar pattern. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-7509) conditional updates don't work from cqlsh
[
https://issues.apache.org/jira/browse/CASSANDRA-7509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs updated CASSANDRA-7509:
---
Attachment: 7509-v2.txt
7509-v2.txt (and [branch|https://github.com/datastax/python-driver]) upgrades
to the latest python driver and removes the cqlsh patches that I mentioned.
conditional updates don't work from cqlsh
-
Key: CASSANDRA-7509
URL: https://issues.apache.org/jira/browse/CASSANDRA-7509
Project: Cassandra
Issue Type: Bug
Components: Tools
Reporter: Jeremiah Jordan
Assignee: Mikhail Stepura
Priority: Blocker
Fix For: 2.1.0
Attachments: 7509-v2.txt, CASSANDRA-2.1.0-7509.patch
{noformat}
apache-cassandra-2.1.0-rc2/bin $ ./cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.0-rc2 | CQL spec 3.1.7 | Native protocol v3]
Use HELP for help.
cqlsh use ks1;
cqlsh:ks1 create table users(
... userID uuid,
... fname text,
... lname text,
... email text,
... address text,
... zip int,
... state text,
... PRIMARY KEY ((userID, fname), state)
... );
cqlsh:ks1 insert into users (userID, fname, state, email) VALUES
(cfd66ccc-d857-4e90-b1e5-df98a3d40cd6, 'laura', 'ca', 'lau...@gmail.com') if
not exists;
code=2200 [Invalid query] message=Conditional updates are not supported by
the protocol version in use. You need to upgrade to a driver using the native
protocol v2.
cqlsh:ks1
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (CASSANDRA-7509) conditional updates don't work from cqlsh
[
https://issues.apache.org/jira/browse/CASSANDRA-7509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-7509:
-
Reviewer: Aleksey Yeschenko (was: Tyler Hobbs)
conditional updates don't work from cqlsh
-
Key: CASSANDRA-7509
URL: https://issues.apache.org/jira/browse/CASSANDRA-7509
Project: Cassandra
Issue Type: Bug
Components: Tools
Reporter: Jeremiah Jordan
Assignee: Mikhail Stepura
Priority: Blocker
Fix For: 2.1.0
Attachments: 7509-v2.txt, CASSANDRA-2.1.0-7509.patch
{noformat}
apache-cassandra-2.1.0-rc2/bin $ ./cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.0-rc2 | CQL spec 3.1.7 | Native protocol v3]
Use HELP for help.
cqlsh use ks1;
cqlsh:ks1 create table users(
... userID uuid,
... fname text,
... lname text,
... email text,
... address text,
... zip int,
... state text,
... PRIMARY KEY ((userID, fname), state)
... );
cqlsh:ks1 insert into users (userID, fname, state, email) VALUES
(cfd66ccc-d857-4e90-b1e5-df98a3d40cd6, 'laura', 'ca', 'lau...@gmail.com') if
not exists;
code=2200 [Invalid query] message=Conditional updates are not supported by
the protocol version in use. You need to upgrade to a driver using the native
protocol v2.
cqlsh:ks1
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
git commit: Bump CQL version to 3.2.0 and update CQL docs
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1.0 fad5ad5a6 - e60b4e6d1
Bump CQL version to 3.2.0 and update CQL docs
Patch by Tyler Hobbs; reviewed by Aleksey Yeschenko for CASSANDRA-7527
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e60b4e6d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e60b4e6d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e60b4e6d
Branch: refs/heads/cassandra-2.1.0
Commit: e60b4e6d1d8fb94586ec3e277e77a6d2aef0be2e
Parents: fad5ad5
Author: Tyler Hobbs ty...@datastax.com
Authored: Tue Jul 15 18:18:15 2014 -0500
Committer: Tyler Hobbs ty...@datastax.com
Committed: Tue Jul 15 18:18:15 2014 -0500
--
CHANGES.txt | 1 +
bin/cqlsh | 2 +-
doc/cql3/CQL.textile| 137 +--
.../apache/cassandra/cql3/QueryProcessor.java | 2 +-
4 files changed, 127 insertions(+), 15 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/e60b4e6d/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b20c465..db66bf9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -4,6 +4,7 @@
* (Windows) fix startup when WMI memory query fails (CASSANDRA-7505)
* Anti-compaction proceeds if any part of the repair failed (CASANDRA-7521)
* Add missing table name to DROP INDEX responses and notifications
(CASSANDRA-7539)
+ * Bump CQL version to 3.2.0 and update CQL documentation (CASSANDRA-7527)
Merged from 2.0:
* (Windows) force range-based repair to non-sequential mode (CASSANDRA-7541)
* Fix range merging when DES scores are zero (CASSANDRA-7535)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/e60b4e6d/bin/cqlsh
--
diff --git a/bin/cqlsh b/bin/cqlsh
index c3a4ea6..53f0c7c 100755
--- a/bin/cqlsh
+++ b/bin/cqlsh
@@ -139,7 +139,7 @@ if os.path.exists(OLD_HISTORY):
DEFAULT_HOST = '127.0.0.1'
DEFAULT_PORT = 9042
-DEFAULT_CQLVER = '3.1.7'
+DEFAULT_CQLVER = '3.2.0'
DEFAULT_TIME_FORMAT = '%Y-%m-%d %H:%M:%S%z'
DEFAULT_FLOAT_PRECISION = 5
http://git-wip-us.apache.org/repos/asf/cassandra/blob/e60b4e6d/doc/cql3/CQL.textile
--
diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile
index 0c1d2b9..2d45cf1 100644
--- a/doc/cql3/CQL.textile
+++ b/doc/cql3/CQL.textile
@@ -1,6 +1,6 @@
link rel=StyleSheet href=CQL.css type=text/css media=screen
-h1. Cassandra Query Language (CQL) v3.1.7
+h1. Cassandra Query Language (CQL) v3.2.0
span id=tableOfContents
@@ -216,7 +216,7 @@ __Syntax:__
bc(syntax)..
create-table-stmt ::= CREATE ( TABLE | COLUMNFAMILY ) ( IF NOT EXISTS )?
tablename
- '(' definition ( ',' definition )* ')'
+ '(' column-definition ( ',' column-definition )*
')'
( WITH option ( AND option)* )?
column-definition ::= identifier type ( STATIC )? ( PRIMARY KEY )?
@@ -255,7 +255,7 @@ Attempting to create an already existing table will return
an error unless the @
h4(#createTableName). @tablename@
-Valid table names are the same than valid keyspace names:#createKeyspaceStmt
(up to 32 characters long alphanumerical identifiers). If the table name is
provided alone, the table is created within the current keyspace (see a
href=#useStmtttUSE/tt/a), but if it is prefixed by an existing
keyspace name (see @tablename@:#statements grammar), it is created in the
specified keyspace (but does *not* change the current keyspace).
+Valid table names are the same as valid keyspace names:#createKeyspaceStmt
(up to 32 characters long alphanumerical identifiers). If the table name is
provided alone, the table is created within the current keyspace (see a
href=#useStmtttUSE/tt/a), but if it is prefixed by an existing
keyspace name (see @tablename@:#statements grammar), it is created in the
specified keyspace (but does *not* change the current keyspace).
h4(#createTableColumn). @column-definition@
@@ -385,7 +385,7 @@ ALTER TABLE addamsFamily
WITH comment = 'A most excellent and useful column family'
AND read_repair_chance = 0.2;
p.
-The @ALTER@ statement is used to manipulate table definitions. It allows to
add new columns, drop existing ones, change the type of existing columns, or
update the table options. As for table creation, @ALTER COLUMNFAMILY@ is
allowed as an alias for @ALTER TABLE@.
+The @ALTER@ statement is used to manipulate table definitions. It allows for
adding new columns, dropping existing ones, changing the type of existing
columns, or updating the table options. As with table creation,
[1/2] git commit: Bump CQL version to 3.2.0 and update CQL docs
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 f9dbc10db - fbe9630d0
Bump CQL version to 3.2.0 and update CQL docs
Patch by Tyler Hobbs; reviewed by Aleksey Yeschenko for CASSANDRA-7527
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e60b4e6d
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e60b4e6d
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e60b4e6d
Branch: refs/heads/cassandra-2.1
Commit: e60b4e6d1d8fb94586ec3e277e77a6d2aef0be2e
Parents: fad5ad5
Author: Tyler Hobbs ty...@datastax.com
Authored: Tue Jul 15 18:18:15 2014 -0500
Committer: Tyler Hobbs ty...@datastax.com
Committed: Tue Jul 15 18:18:15 2014 -0500
--
CHANGES.txt | 1 +
bin/cqlsh | 2 +-
doc/cql3/CQL.textile| 137 +--
.../apache/cassandra/cql3/QueryProcessor.java | 2 +-
4 files changed, 127 insertions(+), 15 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/e60b4e6d/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index b20c465..db66bf9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -4,6 +4,7 @@
* (Windows) fix startup when WMI memory query fails (CASSANDRA-7505)
* Anti-compaction proceeds if any part of the repair failed (CASANDRA-7521)
* Add missing table name to DROP INDEX responses and notifications
(CASSANDRA-7539)
+ * Bump CQL version to 3.2.0 and update CQL documentation (CASSANDRA-7527)
Merged from 2.0:
* (Windows) force range-based repair to non-sequential mode (CASSANDRA-7541)
* Fix range merging when DES scores are zero (CASSANDRA-7535)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/e60b4e6d/bin/cqlsh
--
diff --git a/bin/cqlsh b/bin/cqlsh
index c3a4ea6..53f0c7c 100755
--- a/bin/cqlsh
+++ b/bin/cqlsh
@@ -139,7 +139,7 @@ if os.path.exists(OLD_HISTORY):
DEFAULT_HOST = '127.0.0.1'
DEFAULT_PORT = 9042
-DEFAULT_CQLVER = '3.1.7'
+DEFAULT_CQLVER = '3.2.0'
DEFAULT_TIME_FORMAT = '%Y-%m-%d %H:%M:%S%z'
DEFAULT_FLOAT_PRECISION = 5
http://git-wip-us.apache.org/repos/asf/cassandra/blob/e60b4e6d/doc/cql3/CQL.textile
--
diff --git a/doc/cql3/CQL.textile b/doc/cql3/CQL.textile
index 0c1d2b9..2d45cf1 100644
--- a/doc/cql3/CQL.textile
+++ b/doc/cql3/CQL.textile
@@ -1,6 +1,6 @@
link rel=StyleSheet href=CQL.css type=text/css media=screen
-h1. Cassandra Query Language (CQL) v3.1.7
+h1. Cassandra Query Language (CQL) v3.2.0
span id=tableOfContents
@@ -216,7 +216,7 @@ __Syntax:__
bc(syntax)..
create-table-stmt ::= CREATE ( TABLE | COLUMNFAMILY ) ( IF NOT EXISTS )?
tablename
- '(' definition ( ',' definition )* ')'
+ '(' column-definition ( ',' column-definition )*
')'
( WITH option ( AND option)* )?
column-definition ::= identifier type ( STATIC )? ( PRIMARY KEY )?
@@ -255,7 +255,7 @@ Attempting to create an already existing table will return
an error unless the @
h4(#createTableName). @tablename@
-Valid table names are the same than valid keyspace names:#createKeyspaceStmt
(up to 32 characters long alphanumerical identifiers). If the table name is
provided alone, the table is created within the current keyspace (see a
href=#useStmtttUSE/tt/a), but if it is prefixed by an existing
keyspace name (see @tablename@:#statements grammar), it is created in the
specified keyspace (but does *not* change the current keyspace).
+Valid table names are the same as valid keyspace names:#createKeyspaceStmt
(up to 32 characters long alphanumerical identifiers). If the table name is
provided alone, the table is created within the current keyspace (see a
href=#useStmtttUSE/tt/a), but if it is prefixed by an existing
keyspace name (see @tablename@:#statements grammar), it is created in the
specified keyspace (but does *not* change the current keyspace).
h4(#createTableColumn). @column-definition@
@@ -385,7 +385,7 @@ ALTER TABLE addamsFamily
WITH comment = 'A most excellent and useful column family'
AND read_repair_chance = 0.2;
p.
-The @ALTER@ statement is used to manipulate table definitions. It allows to
add new columns, drop existing ones, change the type of existing columns, or
update the table options. As for table creation, @ALTER COLUMNFAMILY@ is
allowed as an alias for @ALTER TABLE@.
+The @ALTER@ statement is used to manipulate table definitions. It allows for
adding new columns, dropping existing ones, changing the type of existing
columns, or updating the table options. As with table creation,
