[jira] [Updated] (CASSANDRA-3642) Failed to delete any Index.db on Windows
[ https://issues.apache.org/jira/browse/CASSANDRA-3642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-3642: Attachment: 3642.patch I don't have a windows readily available, but a reasonable theory would be that this is due to problem unmapping an index file (but not the data file, which at least for mmap_keys_only without JNA could be a common situation). Attaching a patch to catch that and issue a more precise message instead. Note that the patch will basically left the index file around, file that would be only deleted on the next restart. This is clearly not ideal, but if my theory is correct, this problem only happens without JNA. But my theory may not be correct. Note for the record that getting this exception is rather harmless except for some hopefully small files kept until the next restart. Failed to delete any Index.db on Windows Key: CASSANDRA-3642 URL: https://issues.apache.org/jira/browse/CASSANDRA-3642 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.0 Environment: Windows Server 2008 R2 64bit Java SE 7u1 64bit Reporter: Viktor Jevdokimov Assignee: Sylvain Lebresne Fix For: 1.0.7 Attachments: 3642.patch After upgrade Cassandra 0.8.7 to 1.0.6 under Windows Server 2008 R2 64bit with disk acces mode mmap_index_only, Cassandra failing to delete any *-Index.db files after compaction or scrub: ERROR 13:43:17,490 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main] java.lang.RuntimeException: java.io.IOException: Failed to delete D:\cassandra\data\data\system\LocationInfo-g-29-Index.db at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:689) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.IOException: Failed to delete D:\cassandra\data\data\system\LocationInfo-g-29-Index.db at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44) at org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:141) at org.apache.cassandra.io.sstable.SSTableDeletingTask.runMayThrow(SSTableDeletingTask.java:81) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 8 more ERROR 17:20:09,701 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main] java.lang.RuntimeException: java.io.IOException: Failed to delete D:\cassandra\data\data\Keyspace1\ColumnFamily1-hc-840-Index.db at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:689) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.IOException: Failed to delete D:\cassandra\data\data\ Keyspace1\ColumnFamily1-hc-840-Index.db at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44) at org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:141) at org.apache.cassandra.io.sstable.SSTableDeletingTask.runMayThrow(SSTableDeletingTask.java:81) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 8 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators:
[jira] [Updated] (CASSANDRA-3494) Streaming is mono-threaded (the bulk loader too by extension)
[
https://issues.apache.org/jira/browse/CASSANDRA-3494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sylvain Lebresne updated CASSANDRA-3494:
Attachment: 0001-cleanup-DTPE.patch
Attaching patch implementing the change described above.
Streaming is mono-threaded (the bulk loader too by extension)
-
Key: CASSANDRA-3494
URL: https://issues.apache.org/jira/browse/CASSANDRA-3494
Project: Cassandra
Issue Type: Improvement
Components: Core
Affects Versions: 0.8.0
Reporter: Sylvain Lebresne
Assignee: Peter Schuller
Priority: Minor
Fix For: 1.1
Attachments: 0001-cleanup-DTPE.patch, CASSANDRA-3494-0.8-prelim.txt,
CASSANDRA-3494-1.0.txt
The streamExecutor is define as:
{noformat}
streamExecutor_ = new DebuggableThreadPoolExecutor(Streaming,
Thread.MIN_PRIORITY);
{noformat}
In the meantime, in DebuggableThreadPoolExecutor.java:
{noformat}
public DebuggableThreadPoolExecutor(String threadPoolName, int priority)
{
this(1, Integer.MAX_VALUE, TimeUnit.SECONDS, new
LinkedBlockingQueueRunnable(), new NamedThreadFactory(threadPoolName,
priority));
}
{noformat}
In other word, since the core pool size is 1 and the queue unbounded, tasks
will always queued and the executor is essentially mono-threaded.
This is clearly not necessary since we already have stream throttling
nowadays. And it could be a limiting factor in the case of the bulk loader.
Besides, I would venture that this maybe was not the intention, because
putting the max core size to MAX_VALUE would suggest that the intention was
to spawn threads on demand.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3642) Failed to delete any Index.db on Windows
[ https://issues.apache.org/jira/browse/CASSANDRA-3642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13178698#comment-13178698 ] Viktor Jevdokimov commented on CASSANDRA-3642: -- We're now using standard disk access and sometimes Cassandra tends not to delete Data file, promising to do this after restart. Failed to delete any Index.db on Windows Key: CASSANDRA-3642 URL: https://issues.apache.org/jira/browse/CASSANDRA-3642 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.0 Environment: Windows Server 2008 R2 64bit Java SE 7u1 64bit Reporter: Viktor Jevdokimov Assignee: Sylvain Lebresne Priority: Minor Fix For: 1.0.7 Attachments: 3642.patch After upgrade Cassandra 0.8.7 to 1.0.6 under Windows Server 2008 R2 64bit with disk acces mode mmap_index_only, Cassandra failing to delete any *-Index.db files after compaction or scrub: ERROR 13:43:17,490 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main] java.lang.RuntimeException: java.io.IOException: Failed to delete D:\cassandra\data\data\system\LocationInfo-g-29-Index.db at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:689) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.IOException: Failed to delete D:\cassandra\data\data\system\LocationInfo-g-29-Index.db at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44) at org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:141) at org.apache.cassandra.io.sstable.SSTableDeletingTask.runMayThrow(SSTableDeletingTask.java:81) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 8 more ERROR 17:20:09,701 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main] java.lang.RuntimeException: java.io.IOException: Failed to delete D:\cassandra\data\data\Keyspace1\ColumnFamily1-hc-840-Index.db at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:689) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.IOException: Failed to delete D:\cassandra\data\data\ Keyspace1\ColumnFamily1-hc-840-Index.db at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44) at org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:141) at org.apache.cassandra.io.sstable.SSTableDeletingTask.runMayThrow(SSTableDeletingTask.java:81) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 8 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3486) Node Tool command to stop repair
[ https://issues.apache.org/jira/browse/CASSANDRA-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13178705#comment-13178705 ] Sylvain Lebresne commented on CASSANDRA-3486: - I don't think we should close that ticket because I don't think that we have a satisfying way to stop repair. A satisfying way to run repair would be to be able to run 'nodetool repair stop some_repair_session_id' on the host the repair was started on, on have it cleaning stop everything related to that repair (including any validation or streaming going on) and this on every participating nodes. That require a bit more work however and let me note that I don't see this ticket as a priority at all. Node Tool command to stop repair Key: CASSANDRA-3486 URL: https://issues.apache.org/jira/browse/CASSANDRA-3486 Project: Cassandra Issue Type: Improvement Components: Tools Environment: JVM Reporter: Vijay Assignee: Vijay Priority: Minor Labels: nodetool, repair, stop Fix For: 1.1 Attachments: 0001-stop-repair-3583.patch After CASSANDRA-1740, If the validation compaction is stopped then the repair will hang. This ticket will allow users to kill the original repair. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3531) Fix crack-smoking in ConsistencyLevelTest
[ https://issues.apache.org/jira/browse/CASSANDRA-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13178746#comment-13178746 ] Sylvain Lebresne commented on CASSANDRA-3531: - The deeper change I have in mind consists roughly in removing that test. It's trying to tests the result of WriteHandler.assureSufficientLiveNodes() but that method depends on the result of the FailureDetector. The problem is that I don't think we really have a good way to create real multi-nodes cluster in the unit test. Maybe we can fake live nodes but I'm not sure how and in the end it makes me wonder what the tests is really testing if we're starting to fake too much stuff. It seems to me that the distributed tests are probably a better place to do that kind of thing. In any case, It's really annoying to have unit tests failure, especially in the 1.0 branch. And as said in the description, that test never really worked anyway so any opposition to at least commenting it for now? Fix crack-smoking in ConsistencyLevelTest -- Key: CASSANDRA-3531 URL: https://issues.apache.org/jira/browse/CASSANDRA-3531 Project: Cassandra Issue Type: Bug Components: Tests Affects Versions: 1.0.4 Reporter: Sylvain Lebresne Priority: Minor Fix For: 1.0.7 First, let's note that this test fails in current 1.0 branch. It was broken (emphasis on the quotes) by CASSANDRA-3529. But it's not CASSANDRA-3529 fault, it's only that the use of NonBlockingHashMap changed the order of the tables returned by Schema.instance.getNonSystemTables(). *And*, it turns out that ConsistencyLevelTest bails out as soon as it has found one keyspace with rf = 2 due to a misplaced return. So it use to be that ConsistencyLevelTest was only ran for Keyspace5 (whose RF is 2) for which the test work. But for any RF 2, the test fails. The reason of this failing is that the test creates a 3 node cluster for whom only 1 node is alive as far as the failure detector is concerned. So for RF=3 and CL=QUORUM, the writes are unavailable (the failure detector is queried), while for reads we pretend two nodes are alive so we end up with a case where isWriteUnavailable != isReadUnavailable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3688) [patch] avoid map lookups in loops by using entrysets
[ https://issues.apache.org/jira/browse/CASSANDRA-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13178751#comment-13178751 ] Sylvain Lebresne commented on CASSANDRA-3688: - I have nothing against that per se, but it's really in non-performance sensitive code and so doesn't matter at all. And the replacement is slightly more verbose. Overall I don't really care one way or the other but it does look like change for the sake of change, which we probably should avoid. [patch] avoid map lookups in loops by using entrysets - Key: CASSANDRA-3688 URL: https://issues.apache.org/jira/browse/CASSANDRA-3688 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 1.0.6 Reporter: Dave Brosius Priority: Trivial Fix For: 1.0.7 Attachments: use_entrysets.diff code loops over the keySet and does gets for the value, just use entrySet() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3666) Changing compaction strategy from Leveled to SizeTiered logs millions of messages about nothing to compact
[ https://issues.apache.org/jira/browse/CASSANDRA-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13178755#comment-13178755 ] Sylvain Lebresne commented on CASSANDRA-3666: - +1 on the patch (but we can wait on Viktor to confirm it does fix it for him). (I actually think that it's the settings of the threshold in LCS that is the ugly part, but let's not bother with that) Changing compaction strategy from Leveled to SizeTiered logs millions of messages about nothing to compact -- Key: CASSANDRA-3666 URL: https://issues.apache.org/jira/browse/CASSANDRA-3666 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.0 Environment: Windows Server 2008 R2 64bit Reporter: Viktor Jevdokimov Assignee: Jonathan Ellis Labels: compaction Fix For: 1.0.7 Attachments: 3666.txt When column family compaction strategy is changed from Leveled to SizeTiered and there're Leveled compaction tasks pending, Cassandra starting to flood in logs with thousands per sec messages: Nothing to compact in ColumnFamily1. Use forceUserDefinedCompaction if you wish to force compaction of single sstables (e.g. for tombstone collection) As a result, log disk is full and system is down. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2749) fine-grained control over data directories
[ https://issues.apache.org/jira/browse/CASSANDRA-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-2749: Attachment: 0003-Fixes.patch 0002-fix-unit-tests.patch 0001-2749.patch Rebased patches attached. fine-grained control over data directories -- Key: CASSANDRA-2749 URL: https://issues.apache.org/jira/browse/CASSANDRA-2749 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Priority: Minor Fix For: 1.1 Attachments: 0001-2749.patch, 0001-Make-it-possible-to-put-column-families-in-subdirect.patch, 0001-non-backwards-compatible-patch-for-2749-putting-cfs-.patch.gz, 0002-fix-unit-tests.patch, 0003-Fixes.patch, 2749.tar.gz, 2749_backwards_compatible_v1.patch, 2749_backwards_compatible_v2.patch, 2749_backwards_compatible_v3.patch, 2749_backwards_compatible_v4.patch, 2749_backwards_compatible_v4_rebase1.patch, 2749_not_backwards.tar.gz, 2749_proper.tar.gz Currently Cassandra supports multiple data directories but no way to control what sstables are placed where. Particularly for systems with mixed SSDs and rotational disks, it would be nice to pin frequently accessed columnfamilies to the SSDs. Postgresql does this with tablespaces (http://www.postgresql.org/docs/9.0/static/manage-ag-tablespaces.html) but we should probably avoid using that name because of confusing similarity to keyspaces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2749) fine-grained control over data directories
[ https://issues.apache.org/jira/browse/CASSANDRA-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-2749: Attachment: (was: 0002-fix-unit-tests.patch) fine-grained control over data directories -- Key: CASSANDRA-2749 URL: https://issues.apache.org/jira/browse/CASSANDRA-2749 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Priority: Minor Fix For: 1.1 Attachments: 0001-2749.patch, 0001-Make-it-possible-to-put-column-families-in-subdirect.patch, 0001-non-backwards-compatible-patch-for-2749-putting-cfs-.patch.gz, 0002-fix-unit-tests.patch, 0003-Fixes.patch, 2749.tar.gz, 2749_backwards_compatible_v1.patch, 2749_backwards_compatible_v2.patch, 2749_backwards_compatible_v3.patch, 2749_backwards_compatible_v4.patch, 2749_backwards_compatible_v4_rebase1.patch, 2749_not_backwards.tar.gz, 2749_proper.tar.gz Currently Cassandra supports multiple data directories but no way to control what sstables are placed where. Particularly for systems with mixed SSDs and rotational disks, it would be nice to pin frequently accessed columnfamilies to the SSDs. Postgresql does this with tablespaces (http://www.postgresql.org/docs/9.0/static/manage-ag-tablespaces.html) but we should probably avoid using that name because of confusing similarity to keyspaces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2749) fine-grained control over data directories
[ https://issues.apache.org/jira/browse/CASSANDRA-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-2749: Attachment: (was: 0003-Fixes.patch) fine-grained control over data directories -- Key: CASSANDRA-2749 URL: https://issues.apache.org/jira/browse/CASSANDRA-2749 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Priority: Minor Fix For: 1.1 Attachments: 0001-2749.patch, 0001-Make-it-possible-to-put-column-families-in-subdirect.patch, 0001-non-backwards-compatible-patch-for-2749-putting-cfs-.patch.gz, 0002-fix-unit-tests.patch, 0003-Fixes.patch, 2749.tar.gz, 2749_backwards_compatible_v1.patch, 2749_backwards_compatible_v2.patch, 2749_backwards_compatible_v3.patch, 2749_backwards_compatible_v4.patch, 2749_backwards_compatible_v4_rebase1.patch, 2749_not_backwards.tar.gz, 2749_proper.tar.gz Currently Cassandra supports multiple data directories but no way to control what sstables are placed where. Particularly for systems with mixed SSDs and rotational disks, it would be nice to pin frequently accessed columnfamilies to the SSDs. Postgresql does this with tablespaces (http://www.postgresql.org/docs/9.0/static/manage-ag-tablespaces.html) but we should probably avoid using that name because of confusing similarity to keyspaces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2749) fine-grained control over data directories
[ https://issues.apache.org/jira/browse/CASSANDRA-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-2749: Attachment: (was: 0001-2749.patch) fine-grained control over data directories -- Key: CASSANDRA-2749 URL: https://issues.apache.org/jira/browse/CASSANDRA-2749 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Jonathan Ellis Priority: Minor Fix For: 1.1 Attachments: 0001-2749.patch, 0001-Make-it-possible-to-put-column-families-in-subdirect.patch, 0001-non-backwards-compatible-patch-for-2749-putting-cfs-.patch.gz, 0002-fix-unit-tests.patch, 0003-Fixes.patch, 2749.tar.gz, 2749_backwards_compatible_v1.patch, 2749_backwards_compatible_v2.patch, 2749_backwards_compatible_v3.patch, 2749_backwards_compatible_v4.patch, 2749_backwards_compatible_v4_rebase1.patch, 2749_not_backwards.tar.gz, 2749_proper.tar.gz Currently Cassandra supports multiple data directories but no way to control what sstables are placed where. Particularly for systems with mixed SSDs and rotational disks, it would be nice to pin frequently accessed columnfamilies to the SSDs. Postgresql does this with tablespaces (http://www.postgresql.org/docs/9.0/static/manage-ag-tablespaces.html) but we should probably avoid using that name because of confusing similarity to keyspaces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[Cassandra Wiki] Update of GitTransition by EricEvans
Dear Wiki user,
You have subscribed to a wiki page or wiki category on Cassandra Wiki for
change notification.
The GitTransition page has been changed by EricEvans:
http://wiki.apache.org/cassandra/GitTransition?action=diffrev1=3rev2=4
{{{
git clone https://username@git-wip-us.apache.org/repos/asf/cassandra.git
}}}
-
- You should see something like:
- {{{
- Cloning into git...
- remote: Counting objects: 81041, done.
- remote: Compressing objects: 100% (11287/11287), done.
- remote: Total 81041 (delta 48412), reused 81038 (delta 48411)
- Receiving objects: 100% (81041/81041), 41.50 MiB | 352 KiB/s, done.
- Resolving deltas: 100% (48412/48412), done.
- warning: remote HEAD refers to nonexistent ref, unable to checkout.
- }}}
-
- The warning is because git is trying to open the default branch 'master'
which does not exist in the repo. '''Therefore you need to checkout the
correct branch before doing anything else.'''
-
- For example:
- {{{
- git checkout trunk
- }}}
-
- Now you can make changes and push as with any other git repository.
== Github ==
git commit: Don't ignore IOException during compaction
Updated Branches:
refs/heads/cassandra-1.0 add9372b0 - 6eac35cfd
Don't ignore IOException during compaction
patch by jbellis; reviewed by slebresne for CASSANDRA-3655
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6eac35cf
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6eac35cf
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6eac35cf
Branch: refs/heads/cassandra-1.0
Commit: 6eac35cfd0bfdf67e96f9147b4aad3ddd24c99c2
Parents: add9372
Author: Sylvain Lebresne sylv...@datastax.com
Authored: Tue Jan 3 16:24:41 2012 +0100
Committer: Sylvain Lebresne sylv...@datastax.com
Committed: Tue Jan 3 16:24:41 2012 +0100
--
.../db/compaction/CompactionIterable.java |2 +-
.../cassandra/db/compaction/PrecompactedRow.java | 10 --
.../cassandra/streaming/IncomingStreamReader.java |4 +---
3 files changed, 10 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6eac35cf/src/java/org/apache/cassandra/db/compaction/CompactionIterable.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/CompactionIterable.java
b/src/java/org/apache/cassandra/db/compaction/CompactionIterable.java
index bb8d2bd..5e0dfa7 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionIterable.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionIterable.java
@@ -95,7 +95,7 @@ public class CompactionIterable extends
AbstractCompactionIterable
protected AbstractCompactedRow getReduced()
{
-assert rows.size() 0;
+assert !rows.isEmpty();
try
{
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6eac35cf/src/java/org/apache/cassandra/db/compaction/PrecompactedRow.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/PrecompactedRow.java
b/src/java/org/apache/cassandra/db/compaction/PrecompactedRow.java
index 6363d9d..5b5ba8d 100644
--- a/src/java/org/apache/cassandra/db/compaction/PrecompactedRow.java
+++ b/src/java/org/apache/cassandra/db/compaction/PrecompactedRow.java
@@ -22,6 +22,7 @@ package org.apache.cassandra.db.compaction;
import java.io.DataOutput;
+import java.io.IOError;
import java.io.IOException;
import java.security.MessageDigest;
import java.util.List;
@@ -56,6 +57,10 @@ public class PrecompactedRow extends AbstractCompactedRow
public static ColumnFamily removeDeletedAndOldShards(DecoratedKey? key,
CompactionController controller, ColumnFamily cf)
{
+assert key != null;
+assert controller != null;
+assert cf != null;
+
// avoid calling shouldPurge unless we actually need to: it can be
very expensive if LCS
// gets behind and has hundreds of overlapping L0 sstables.
Essentially, this method is an
// ugly refactor of
removeDeletedAndOldShards(controller.shouldPurge(key), controller, cf),
@@ -95,6 +100,7 @@ public class PrecompactedRow extends AbstractCompactedRow
private static ColumnFamily merge(ListSSTableIdentityIterator rows)
{
+assert !rows.isEmpty();
ColumnFamily cf = null;
for (SSTableIdentityIterator row : rows)
{
@@ -105,9 +111,9 @@ public class PrecompactedRow extends AbstractCompactedRow
}
catch (IOException e)
{
-logger.error(Skipping row + row.getKey() + in +
row.getPath(), e);
-continue;
+throw new IOError(e);
}
+
if (cf == null)
{
cf = thisCF;
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6eac35cf/src/java/org/apache/cassandra/streaming/IncomingStreamReader.java
--
diff --git a/src/java/org/apache/cassandra/streaming/IncomingStreamReader.java
b/src/java/org/apache/cassandra/streaming/IncomingStreamReader.java
index f70f5ff..e2a618f 100644
--- a/src/java/org/apache/cassandra/streaming/IncomingStreamReader.java
+++ b/src/java/org/apache/cassandra/streaming/IncomingStreamReader.java
@@ -108,7 +108,7 @@ public class IncomingStreamReader
ColumnFamilyStore cfs =
Table.open(localFile.desc.ksname).getColumnFamilyStore(localFile.desc.cfname);
DecoratedKey key;
SSTableWriter writer = new SSTableWriter(localFile.getFilename(),
remoteFile.estimatedKeys);
-CompactionController controller = null;
+CompactionController controller = new CompactionController(cfs,
Collections.SSTableReaderemptyList(), Integer.MIN_VALUE, true);
try
{
@@ -128,8 +128,6 @@ public class
[jira] [Resolved] (CASSANDRA-3655) NPE when running upgradesstables
[ https://issues.apache.org/jira/browse/CASSANDRA-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne resolved CASSANDRA-3655. - Resolution: Fixed Assignee: Jonathan Ellis (was: Tupshin Harper) Committed as even if didn't fixed the issue (which it probably did) it's an improvement. NPE when running upgradesstables Key: CASSANDRA-3655 URL: https://issues.apache.org/jira/browse/CASSANDRA-3655 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.5 Environment: 1.0.5 + patch for https://issues.apache.org/jira/browse/CASSANDRA-3618 Reporter: Tupshin Harper Assignee: Jonathan Ellis Labels: compaction Fix For: 1.0.7 Attachments: 3655.txt Running a test upgrade from 0.7(version f sstables) to 1.0. upgradesstables runs for about 40 minutes and then NPE's when trying to retrieve a key. No files have been succesfully upgraded. Likely related is that scrub (without having run upgrade) consumes all RAM and OOMs. Possible theory is that a lot of paths call IPartitioner's decorateKey, and, at least in the randompartitioner's implementation, if any of those callers pass a null ByteBuffer, they key will be null in the stack trace below. java.util.concurrent.ExecutionException: java.lang.NullPointerException at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.compaction.CompactionManager.performAllSSTableOperation(CompactionManager.java:203) at org.apache.cassandra.db.compaction.CompactionManager.performSSTableRewrite(CompactionManager.java:219) at org.apache.cassandra.db.ColumnFamilyStore.sstablesRewrite(ColumnFamilyStore.java:970) at org.apache.cassandra.service.StorageService.upgradeSSTables(StorageService.java:1540) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.NullPointerException at org.apache.cassandra.db.compaction.PrecompactedRow.removeDeletedAndOldShards(PrecompactedRow.java:65) at org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:92) at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:137) at
[2/2] git commit: Reset min/max compaction threshold when creating SizeTieredCompaction strategy
Reset min/max compaction threshold when creating SizeTieredCompaction strategy
patch by jbellis; reviewed by slebresne for CASSANDRA-3666
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a1f156dd
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a1f156dd
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a1f156dd
Branch: refs/heads/cassandra-1.0
Commit: a1f156ddf80b0ae374028cd1e981400970e208f0
Parents: 6eac35c
Author: Sylvain Lebresne sylv...@datastax.com
Authored: Tue Jan 3 16:29:28 2012 +0100
Committer: Sylvain Lebresne sylv...@datastax.com
Committed: Tue Jan 3 16:29:28 2012 +0100
--
.../compaction/SizeTieredCompactionStrategy.java | 10 ++
1 files changed, 6 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/a1f156dd/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java
b/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java
index e8c5c6f..6f6adbc 100644
---
a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java
+++
b/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java
@@ -40,10 +40,12 @@ public class SizeTieredCompactionStrategy extends
AbstractCompactionStrategy
public SizeTieredCompactionStrategy(ColumnFamilyStore cfs, MapString,
String options)
{
- super(cfs, options);
- this.estimatedRemainingTasks = 0;
- String optionValue = options.get(MIN_SSTABLE_SIZE_KEY);
- minSSTableSize = (null != optionValue) ? Long.parseLong(optionValue) :
DEFAULT_MIN_SSTABLE_SIZE;
+super(cfs, options);
+this.estimatedRemainingTasks = 0;
+String optionValue = options.get(MIN_SSTABLE_SIZE_KEY);
+minSSTableSize = (null != optionValue) ? Long.parseLong(optionValue) :
DEFAULT_MIN_SSTABLE_SIZE;
+
cfs.setMaximumCompactionThreshold(cfs.metadata.getMaxCompactionThreshold());
+
cfs.setMinimumCompactionThreshold(cfs.metadata.getMinCompactionThreshold());
}
public ListAbstractCompactionTask getBackgroundTasks(final int gcBefore)
[1/2] git commit: Fix changelog
Updated Branches: refs/heads/cassandra-1.0 6eac35cfd - 52e671a31 Fix changelog Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/52e671a3 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/52e671a3 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/52e671a3 Branch: refs/heads/cassandra-1.0 Commit: 52e671a315b9fe295402f17d5e95989b32a68e74 Parents: a1f156d Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue Jan 3 16:32:11 2012 +0100 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jan 3 16:32:11 2012 +0100 -- CHANGES.txt |3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/52e671a3/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 76d4981..287fed8 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -13,6 +13,9 @@ (CASSANDRA-3335) * (CQL) compaction_strategy_options and compression_parameters for CREATE COLUMNFAMILY statement (CASSANDRA-3374) + * Reset min/max compaction threshold when creating size tiered compaction + strategy (CASSANDRA-3666) + * Don't ignore IOException during compaction (CASSANDRA-3655) Merged from 0.8: * avoid logging (harmless) exception when GC takes 1ms (CASSANDRA-3656) * prevent new nodes from thinking down nodes are up forever (CASSANDRA-3626)
git commit: Fixed avro after CASSANDRA-3667 new parameter introduction
Updated Branches:
refs/heads/trunk b9813ed38 - 2a7786e45
Fixed avro after CASSANDRA-3667 new parameter introduction
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/2a7786e4
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/2a7786e4
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/2a7786e4
Branch: refs/heads/trunk
Commit: 2a7786e452caa57d650f3aa8a0b509108bbc900e
Parents: b9813ed
Author: Pavel Yaskevich pove...@gmail.com
Authored: Tue Jan 3 18:51:20 2012 +0200
Committer: Pavel Yaskevich pove...@gmail.com
Committed: Tue Jan 3 18:51:20 2012 +0200
--
src/avro/internode.genavro |2 +-
.../org/apache/cassandra/config/CFMetaData.java| 14 +-
2 files changed, 14 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2a7786e4/src/avro/internode.genavro
--
diff --git a/src/avro/internode.genavro b/src/avro/internode.genavro
index d1ea002..c36a83f 100644
--- a/src/avro/internode.genavro
+++ b/src/avro/internode.genavro
@@ -66,7 +66,7 @@ protocol InterNode {
union { null, mapstring } compaction_strategy_options = null;
union { null, mapstring } compression_options = null;
union { null, double } bloom_filter_fp_chance = null;
-string caching = keys_only;
+union { null, string } caching = null;
}
@aliases([org.apache.cassandra.config.avro.KsDef])
http://git-wip-us.apache.org/repos/asf/cassandra/blob/2a7786e4/src/java/org/apache/cassandra/config/CFMetaData.java
--
diff --git a/src/java/org/apache/cassandra/config/CFMetaData.java
b/src/java/org/apache/cassandra/config/CFMetaData.java
index 7a3c266..f86abd9 100644
--- a/src/java/org/apache/cassandra/config/CFMetaData.java
+++ b/src/java/org/apache/cassandra/config/CFMetaData.java
@@ -388,6 +388,17 @@ public final class CFMetaData
throw new RuntimeException(e);
}
+Caching caching;
+
+try
+{
+caching = Caching.fromString(cf.caching.toString());
+}
+catch (ConfigurationException e)
+{
+throw new RuntimeException(e);
+}
+
return newCFMD.comment(cf.comment.toString())
.readRepairChance(cf.read_repair_chance)
.replicateOnWrite(cf.replicate_on_write)
@@ -396,7 +407,8 @@ public final class CFMetaData
.keyValidator(keyValidator)
.columnMetadata(column_metadata)
.compressionParameters(cp)
- .bloomFilterFpChance(cf.bloom_filter_fp_chance);
+ .bloomFilterFpChance(cf.bloom_filter_fp_chance)
+ .caching(caching);
}
public String getComment()
[jira] [Commented] (CASSANDRA-3688) [patch] avoid map lookups in loops by using entrysets
[ https://issues.apache.org/jira/browse/CASSANDRA-3688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13178795#comment-13178795 ] Jonathan Ellis commented on CASSANDRA-3688: --- To be fair, it's more change for the sake of shutting up findbugs, which is good if it reduces the effort we have to do to investigate future reports. This is the right way to loop over maps in Java so I'd be +1 for trunk. [patch] avoid map lookups in loops by using entrysets - Key: CASSANDRA-3688 URL: https://issues.apache.org/jira/browse/CASSANDRA-3688 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 1.0.6 Reporter: Dave Brosius Priority: Trivial Fix For: 1.0.7 Attachments: use_entrysets.diff code loops over the keySet and does gets for the value, just use entrySet() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3634) compare string vs. binary prepared statement parameters
[ https://issues.apache.org/jira/browse/CASSANDRA-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13178818#comment-13178818 ] Jonathan Ellis commented on CASSANDRA-3634: --- Brandon, can you also test increased column size, and not just count? E.g. CFS uses 2MB columns. compare string vs. binary prepared statement parameters --- Key: CASSANDRA-3634 URL: https://issues.apache.org/jira/browse/CASSANDRA-3634 Project: Cassandra Issue Type: Sub-task Components: API, Core Reporter: Eric Evans Assignee: Eric Evans Priority: Minor Labels: cql Fix For: 1.1 Perform benchmarks to compare the performance of string and pre-serialized binary parameters to prepared statements. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3691) LeveledCompactionStrategy is broken because of generation pre-allocation in LeveledManifest.
LeveledCompactionStrategy is broken because of generation pre-allocation in
LeveledManifest.
Key: CASSANDRA-3691
URL: https://issues.apache.org/jira/browse/CASSANDRA-3691
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1
Reporter: Pavel Yaskevich
LeveledManifest constructor has the following code:
{code}
for (int i = 0; i generations.length; i++)
{
generations[i] = new ArrayListSSTableReader();
lastCompactedKeys[i] = new DecoratedKey(cfs.partitioner.getMinimumToken(),
null);
}
{code}
But in the DecoratedKey constructor we have:
{code}
assert token != null key != null key.remaining() 0;
{code}
so when you tried to create a CF with LeveledCompressionStrategy that will
result in
{noformat}
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at
org.apache.cassandra.thrift.CassandraServer.applyMigrationOnStage(CassandraServer.java:865)
at
org.apache.cassandra.thrift.CassandraServer.system_add_keyspace(CassandraServer.java:953)
at
org.apache.cassandra.thrift.Cassandra$Processor$system_add_keyspace.process(Cassandra.java:4103)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:3078)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:188)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException:
java.lang.reflect.InvocationTargetException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.thrift.CassandraServer.applyMigrationOnStage(CassandraServer.java:857)
... 7 more
Caused by: java.lang.RuntimeException:
java.lang.reflect.InvocationTargetException
at
org.apache.cassandra.config.CFMetaData.createCompactionStrategyInstance(CFMetaData.java:770)
at
org.apache.cassandra.db.ColumnFamilyStore.init(ColumnFamilyStore.java:209)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:300)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:281)
at org.apache.cassandra.db.Table.initCf(Table.java:339)
at org.apache.cassandra.db.Table.init(Table.java:288)
at org.apache.cassandra.db.Table.open(Table.java:117)
at
org.apache.cassandra.db.migration.AddKeyspace.applyModels(AddKeyspace.java:72)
at org.apache.cassandra.db.migration.Migration.apply(Migration.java:156)
at
org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:850)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.cassandra.config.CFMetaData.createCompactionStrategyInstance(CFMetaData.java:752)
... 14 more
Caused by: java.lang.AssertionError
at org.apache.cassandra.db.DecoratedKey.init(DecoratedKey.java:55)
at
org.apache.cassandra.db.compaction.LeveledManifest.init(LeveledManifest.java:79)
at
org.apache.cassandra.db.compaction.LeveledManifest.create(LeveledManifest.java:85)
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.init(LeveledCompactionStrategy.java:74)
... 19 more
ERROR 19:52:44,029 Fatal exception in thread Thread[MigrationStage:1,5,main]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2749) fine-grained control over data directories
[
https://issues.apache.org/jira/browse/CASSANDRA-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13178822#comment-13178822
]
Pavel Yaskevich commented on CASSANDRA-2749:
+1 with last nit - Directories.SSTableLister methods
skip{Compacted,Temporary}(boolean) and includeBackups(boolean) ignore argument
and always set corresponding option to true.
fine-grained control over data directories
--
Key: CASSANDRA-2749
URL: https://issues.apache.org/jira/browse/CASSANDRA-2749
Project: Cassandra
Issue Type: New Feature
Components: Core
Reporter: Jonathan Ellis
Priority: Minor
Fix For: 1.1
Attachments: 0001-2749.patch,
0001-Make-it-possible-to-put-column-families-in-subdirect.patch,
0001-non-backwards-compatible-patch-for-2749-putting-cfs-.patch.gz,
0002-fix-unit-tests.patch, 0003-Fixes.patch, 2749.tar.gz,
2749_backwards_compatible_v1.patch, 2749_backwards_compatible_v2.patch,
2749_backwards_compatible_v3.patch, 2749_backwards_compatible_v4.patch,
2749_backwards_compatible_v4_rebase1.patch, 2749_not_backwards.tar.gz,
2749_proper.tar.gz
Currently Cassandra supports multiple data directories but no way to control
what sstables are placed where. Particularly for systems with mixed SSDs and
rotational disks, it would be nice to pin frequently accessed columnfamilies
to the SSDs.
Postgresql does this with tablespaces
(http://www.postgresql.org/docs/9.0/static/manage-ag-tablespaces.html) but we
should probably avoid using that name because of confusing similarity to
keyspaces.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3610) Checksum improvement for CompressedRandomAccessReader
[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13178825#comment-13178825 ] Vijay commented on CASSANDRA-3610: -- for this ticket, Should we just check if it is sunJDK and if true then use the pure java impl else regular CRC32? Checksum improvement for CompressedRandomAccessReader - Key: CASSANDRA-3610 URL: https://issues.apache.org/jira/browse/CASSANDRA-3610 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 1.1 Environment: JVM Reporter: Vijay Assignee: Vijay Priority: Minor Fix For: 1.1 Attachments: 0001-use-pure-java-CRC32-v2.patch, 0001-use-pure-java-CRC32-v3.patch, 0001-use-pure-java-CRC32.patch, TestCrc32Performance.java When compression is on, Currently we see checksum taking up about 40% of the CPU more than snappy library. Looks like hadoop solved it by implementing their own checksum, we can either use it or implement something like that. http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-10132228-phpapp01-slide-15-768.jpg?1321043717 in our test env it provided 50% improvement over native implementation which uses jni to call the OS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3134) Patch Hadoop Streaming Source to Support Cassandra IO
[ https://issues.apache.org/jira/browse/CASSANDRA-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13178885#comment-13178885 ] Brandyn White commented on CASSANDRA-3134: -- Now that CASSANDRA-2799 is in 1.0.6 I can start looking into implementing this. Patch Hadoop Streaming Source to Support Cassandra IO - Key: CASSANDRA-3134 URL: https://issues.apache.org/jira/browse/CASSANDRA-3134 Project: Cassandra Issue Type: New Feature Components: Hadoop Reporter: Brandyn White Priority: Minor Labels: hadoop, hadoop_examples_streaming Original Estimate: 504h Remaining Estimate: 504h (text is a repost from [CASSANDRA-1497|https://issues.apache.org/jira/browse/CASSANDRA-1497]) I'm the author of the Hadoopy http://bwhite.github.com/hadoopy/ python library and I'm interested in taking another stab at streaming support. Hadoopy and Dumbo both use the TypedBytes format that is in CDH for communication with the streaming jar. A simple way to get this to work is modify the streaming code (make hadoop-cassandra-streaming.jar) so that it uses the same TypedBytes communication with streaming programs, but the actual job IO is using the Cassandra IO. The user would have the exact same streaming interface, but the user would specify the keyspace, etc using environmental variables. The benefits of this are 1. Easy implementation: Take the cloudera-patched version of streaming and change the IO, add environmental variable reading. 2. Only Client side: As the streaming jar is included in the job, no server side changes are required. 3. Simple maintenance: If the Hadoop Cassandra interface changes, then this would require the same simple fixup as any other Hadoop job. 4. The TypedBytes format supports all of the necessary Cassandara types (https://issues.apache.org/jira/browse/HADOOP-5450) 5. Compatible with existing streaming libraries: Hadoopy and dumbo would only need to know the path of this new streaming jar 6. No need for avro The negatives of this are 1. Duplicative code: This would be a dupe and patch of the streaming jar. This can be stored itself as a patch. 2. I'd have to check but this solution should work on a stock hadoop (cluster side) but it requires TypedBytes (client side) which can be included in the jar. I can code this up but I wanted to get some feedback from the community first. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-3692) Always use microsecond timestamps in the System Table
Always use microsecond timestamps in the System Table - Key: CASSANDRA-3692 URL: https://issues.apache.org/jira/browse/CASSANDRA-3692 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 1.0.6 Reporter: Aaron Morton Assignee: Aaron Morton Priority: Minor Code in o.a.c.db.SystemTable used a combination of milliseconds, microseconds and 0 for column timestamps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3692) Always use microsecond timestamps in the System Table
[ https://issues.apache.org/jira/browse/CASSANDRA-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Morton updated CASSANDRA-3692: Attachment: 0001-3692.patch Always use microsecond timestamps in the System Table - Key: CASSANDRA-3692 URL: https://issues.apache.org/jira/browse/CASSANDRA-3692 Project: Cassandra Issue Type: Improvement Components: Core Affects Versions: 1.0.6 Reporter: Aaron Morton Assignee: Aaron Morton Priority: Minor Attachments: 0001-3692.patch Code in o.a.c.db.SystemTable used a combination of milliseconds, microseconds and 0 for column timestamps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2474) CQL support for compound columns
[
https://issues.apache.org/jira/browse/CASSANDRA-2474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13178902#comment-13178902
]
Sylvain Lebresne commented on CASSANDRA-2474:
-
Ok, I think I'm really warming up to what we're getting at. I would change the
syntax of the VALUE() thing however. Instead of:
{noformat}
CREATE TABLE timeline (
userid int,
posted_at uuid,
body string,
PRIMARY KEY(user_id, posted_at),
VALUE(body)
)
{noformat}
I would prefer:
{noformat}
CREATE COMPACT TABLE timeline (
userid int,
posted_at uuid,
body string,
PRIMARY KEY(user_id, posted_at),
)
{noformat}
The reasons being that it really influences the implementation layout of the CF
in C*. Namely, the non-compact CF defined by
{noformat}
CREATE TABLE timeline (
userid int,
posted_at uuid,
body string,
PRIMARY KEY(user_id, posted_at),
)
{noformat}
would look in C* like:
{noformat}
userid : {
posted_at:'body' - value
}
{noformat}
while the COMPACT variant would be:
{noformat}
userid : {
posted_at - value
}
{noformat}
which is using the fact that there is only 1 field not part of the key to
optimize the layout. And I believe making the COMPACT part of the CREATE
emphasis better that it's a property of the definition itself (that cannot be
changed) rather than of that specific 'body' field. It also make the rule for
COMPACT table rather simple: a compact table should have only one field not
part of the primary key; you don't have to deal with errors like someone
defining two VALUE() for instance.
That being said, I'd like to try to resume where we're at (including the
COMPACT change above) and add a few random ideas along the way. Please correct
me if I've got something wrong.
I think we have 4 different cases, 2 for 'standard' CF without composites:
* static CFs (the only case CQL handle really well today)
* dynamic CFs (wide rows, time series if you prefer) and 2 for CF with
composite column names:
* 'dense' composite (typically time series but where the key is naturally
multi-parts)
* 'sparse' composite (aka super columns)
Let me try to take an example for which, with how it would translate internally
and example queries.
h3. Standard static CF
For each user, holds his infos
{noformat}
CREATE TABLE users (
userid uuid PRIMARY KEY,
firstname text,
lastname text,
age int
)
{noformat}
In C*:
{noformat}
userid : {
'firstname' - value
'lastname' - value
'age' - value
}
{noformat}
Query:
{noformat}
SELECT firstname, lastname FROM users WHERE userid = '...';
{noformat}
h3. Standard dynamic CF
For each user, keep each url he clicked on with the date of last click
{noformat}
CREATE COMPACT TABLE clicks (
userid uuid,
url text,
timestamp date
PRIMARY KEY (userid, url)
)
{noformat}
In C*:
{noformat}
userid : {
url - timestamp
}
{noformat}
Query:
{noformat}
SELECT url, timestamp FROM clicks WHERE userid = '...';
SELECT timestamp FROM clicks WHERE userid = '...' and url = 'http://...';
{noformat}
h3. 'dense' composite
For each user, keep ip and port from where he connected with the date of last
connection
{noformat}
CREATE COMPACT TABLE connections (
userid uuid,
ip binary,
port int,
timestamp date
PRIMARY KEY (userid, ip, port)
)
{noformat}
In C*:
{noformat}
userid : {
ip:port - timestamp
}
{noformat}
Query:
{noformat}
SELECT ip, port, timestamp FROM connections WHERE userid = '...';
{noformat}
h3. 'sparse' composite
User timeline
{noformat}
CREATE TABLE timeline (
userid uuid,
posted_at date,
body text,
posted_by text,
PRIMARY KEY (user_id, posted_at),
);
{noformat}
In C*:
{noformat}
userid : {
posted_at:'body' - value
posted_at:'posted_by' - value
}
{noformat}
Query:
{noformat}
SELECT body, posted_by FROM timeline WHERE userid = '...' and posted_at = '2
janvier 2010'
{noformat}
Note: I think we really should also be able to do queries like:
{noformat}
SELECT posted_ad, body, posted_by FROM timeline WHERE userid = '...' and
posted_at '2 janvier 2010'
{noformat}
but that's more akin to the modification of the syntax for slices.
h3. Random other ideas
# We could allow something like:
{noformat}
CONSTRAINT key PRIMARY KEY (userid, ip, port)
{noformat}
which would then allow to write
{noformat}
SELECT timestamp FROM users WHERE key = ('...', 192.168.0.1, 80);
{noformat}
(I believe this is the 'standard' notation to name a 'composite' key in SQL)
# Above we're ony handling the use of composites for column names, but they can
be useful for value (and row keys) and it could be nice to have an easy
notation for that (clearly a following ticket however). What about:
{noformat}
CREATE COMPACT TABLE timeline (
userid_part1 text,
userid_part2 int,
posted_at date,
posted_by uuid,
body text
header text
GROUP (userid_part1, userid_part2) AS
[jira] [Updated] (CASSANDRA-3691) LeveledCompactionStrategy is broken because of generation pre-allocation in LeveledManifest.
[
https://issues.apache.org/jira/browse/CASSANDRA-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sylvain Lebresne updated CASSANDRA-3691:
Attachment: 3691.patch
Oops, missed this one. Patch attached to fix.
LeveledCompactionStrategy is broken because of generation pre-allocation in
LeveledManifest.
Key: CASSANDRA-3691
URL: https://issues.apache.org/jira/browse/CASSANDRA-3691
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1
Reporter: Pavel Yaskevich
Attachments: 3691.patch
LeveledManifest constructor has the following code:
{code}
for (int i = 0; i generations.length; i++)
{
generations[i] = new ArrayListSSTableReader();
lastCompactedKeys[i] = new
DecoratedKey(cfs.partitioner.getMinimumToken(), null);
}
{code}
But in the DecoratedKey constructor we have:
{code}
assert token != null key != null key.remaining() 0;
{code}
so when you tried to create a CF with LeveledCompressionStrategy that will
result in
{noformat}
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at
org.apache.cassandra.thrift.CassandraServer.applyMigrationOnStage(CassandraServer.java:865)
at
org.apache.cassandra.thrift.CassandraServer.system_add_keyspace(CassandraServer.java:953)
at
org.apache.cassandra.thrift.Cassandra$Processor$system_add_keyspace.process(Cassandra.java:4103)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:3078)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:188)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.thrift.CassandraServer.applyMigrationOnStage(CassandraServer.java:857)
... 7 more
Caused by: java.lang.RuntimeException:
java.lang.reflect.InvocationTargetException
at
org.apache.cassandra.config.CFMetaData.createCompactionStrategyInstance(CFMetaData.java:770)
at
org.apache.cassandra.db.ColumnFamilyStore.init(ColumnFamilyStore.java:209)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:300)
at
org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:281)
at org.apache.cassandra.db.Table.initCf(Table.java:339)
at org.apache.cassandra.db.Table.init(Table.java:288)
at org.apache.cassandra.db.Table.open(Table.java:117)
at
org.apache.cassandra.db.migration.AddKeyspace.applyModels(AddKeyspace.java:72)
at org.apache.cassandra.db.migration.Migration.apply(Migration.java:156)
at
org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:850)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
... 3 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.cassandra.config.CFMetaData.createCompactionStrategyInstance(CFMetaData.java:752)
... 14 more
Caused by: java.lang.AssertionError
at org.apache.cassandra.db.DecoratedKey.init(DecoratedKey.java:55)
at
org.apache.cassandra.db.compaction.LeveledManifest.init(LeveledManifest.java:79)
at
org.apache.cassandra.db.compaction.LeveledManifest.create(LeveledManifest.java:85)
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.init(LeveledCompactionStrategy.java:74)
... 19 more
ERROR 19:52:44,029 Fatal exception in thread Thread[MigrationStage:1,5,main]
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see:
git commit: fix LeveledCompactionStrategy broken because of generation pre-allocation in LeveledManifest patch by Sylvain Lebresne; reviewed by Pavel Yaskevich for CASSANDRA-3691
Updated Branches:
refs/heads/trunk 2a7786e45 - e75c53ea1
fix LeveledCompactionStrategy broken because of generation pre-allocation in
LeveledManifest
patch by Sylvain Lebresne; reviewed by Pavel Yaskevich for CASSANDRA-3691
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e75c53ea
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e75c53ea
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e75c53ea
Branch: refs/heads/trunk
Commit: e75c53ea14630bb89cde29328723be5e9dcbeb58
Parents: 2a7786e
Author: Pavel Yaskevich pove...@gmail.com
Authored: Tue Jan 3 22:32:44 2012 +0200
Committer: Pavel Yaskevich pove...@gmail.com
Committed: Tue Jan 3 22:34:12 2012 +0200
--
CHANGES.txt|2 ++
.../cassandra/db/compaction/LeveledManifest.java |8
2 files changed, 6 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/e75c53ea/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 24f4656..3be6393 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -37,6 +37,8 @@
* introduce 'crc_check_chance' in CompressionParameters to support
a checksum percentage checking chance similarly to read-repair
(CASSANDRA-3611)
* a way to deactivate global key/row cache on per-CF basis (CASSANDRA-3667)
+ * fix LeveledCompactionStrategy broken because of generation pre-allocation
+ in LeveledManifest (CASSANDRA-3691)
1.0.7
http://git-wip-us.apache.org/repos/asf/cassandra/blob/e75c53ea/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
b/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
index b80aace..bc99e71 100644
--- a/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
+++ b/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java
@@ -32,7 +32,7 @@ import org.slf4j.LoggerFactory;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.ColumnFamilyStore;
-import org.apache.cassandra.db.DecoratedKey;
+import org.apache.cassandra.db.RowPosition;
import org.apache.cassandra.dht.Range;
import org.apache.cassandra.dht.Token;
import org.apache.cassandra.io.sstable.SSTable;
@@ -59,7 +59,7 @@ public class LeveledManifest
private final ColumnFamilyStore cfs;
private final ListSSTableReader[] generations;
-private final DecoratedKey[] lastCompactedKeys;
+private final RowPosition[] lastCompactedKeys;
private final int maxSSTableSizeInMB;
private int levelCount;
@@ -71,11 +71,11 @@ public class LeveledManifest
// allocate enough generations for a PB of data
int n = (int) Math.log10(1000 * 1000 * 1000 / maxSSTableSizeInMB);
generations = new List[n];
-lastCompactedKeys = new DecoratedKey[n];
+lastCompactedKeys = new RowPosition[n];
for (int i = 0; i generations.length; i++)
{
generations[i] = new ArrayListSSTableReader();
-lastCompactedKeys[i] = new
DecoratedKey(cfs.partitioner.getMinimumToken(), null);
+lastCompactedKeys[i] =
cfs.partitioner.getMinimumToken().minKeyBound();
}
}
[jira] [Updated] (CASSANDRA-3507) Proposal: separate cqlsh from CQL drivers
[ https://issues.apache.org/jira/browse/CASSANDRA-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-3507: Priority: Blocker (was: Minor) Fix Version/s: (was: 1.1) 1.0.7 Proposal: separate cqlsh from CQL drivers - Key: CASSANDRA-3507 URL: https://issues.apache.org/jira/browse/CASSANDRA-3507 Project: Cassandra Issue Type: Improvement Components: Packaging, Tools Affects Versions: 1.0.3 Environment: Debian-based systems Reporter: paul cannon Assignee: paul cannon Priority: Blocker Labels: cql, cqlsh Fix For: 1.0.7 Whereas: * It has been shown to be very desirable to decouple the release cycles of Cassandra from the various client CQL drivers, and * It is also desirable to include a good interactive CQL client with releases of Cassandra, and * It is not desirable for Cassandra releases to depend on 3rd-party software which is neither bundled with Cassandra nor readily available for every target platform, but * Any good interactive CQL client will require a CQL driver; Therefore, be it resolved that: * cqlsh will not use an official or supported CQL driver, but will include its own private CQL driver, not intended for use by anything else, and * the Cassandra project will still recommend installing and using a proper CQL driver for client software. To ease maintenance, the private CQL driver included with cqlsh may very well be created by copying the python CQL driver from one directory into another, but the user shouldn't rely on this. Maybe we even ought to take some minor steps to discourage its use for other purposes. Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3507) Proposal: separate cqlsh from CQL drivers
[ https://issues.apache.org/jira/browse/CASSANDRA-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13178981#comment-13178981 ] Sylvain Lebresne commented on CASSANDRA-3507: - So, apparently this is now a blocker because Paul told me that the debian package is broken following the commit of CASSANDRA-3458. I see two possibility: # we revert CASSANDRA-3458 so that we can do a release. # we fix this. My preference goes to reverting CASSANDRA-3458, because I don't really like that we add something we're only mildly getting consensus on to a 1.0 release. As a side note, I would have wished we didn't end up in that state of not being able to release because of some debian problem. Please be as careful with change to the debian packaging in stable release as you are with the rest of the code since it's part of the tree. Proposal: separate cqlsh from CQL drivers - Key: CASSANDRA-3507 URL: https://issues.apache.org/jira/browse/CASSANDRA-3507 Project: Cassandra Issue Type: Improvement Components: Packaging, Tools Affects Versions: 1.0.3 Environment: Debian-based systems Reporter: paul cannon Assignee: paul cannon Priority: Blocker Labels: cql, cqlsh Fix For: 1.0.7 Whereas: * It has been shown to be very desirable to decouple the release cycles of Cassandra from the various client CQL drivers, and * It is also desirable to include a good interactive CQL client with releases of Cassandra, and * It is not desirable for Cassandra releases to depend on 3rd-party software which is neither bundled with Cassandra nor readily available for every target platform, but * Any good interactive CQL client will require a CQL driver; Therefore, be it resolved that: * cqlsh will not use an official or supported CQL driver, but will include its own private CQL driver, not intended for use by anything else, and * the Cassandra project will still recommend installing and using a proper CQL driver for client software. To ease maintenance, the private CQL driver included with cqlsh may very well be created by copying the python CQL driver from one directory into another, but the user shouldn't rely on this. Maybe we even ought to take some minor steps to discourage its use for other purposes. Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3507) Proposal: separate cqlsh from CQL drivers
[ https://issues.apache.org/jira/browse/CASSANDRA-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13178993#comment-13178993 ] paul cannon commented on CASSANDRA-3507: bq. Please be as careful with change to the debian packaging in stable release as you are with the rest of the code since it's part of the tree. That's a little harsh- the debian package and the packaging aren't technically broken, they just require users to add a 3rd-party apt source or install a python-cql deb locally. But yeah, I doubt users really appreciate that distinction much. bq. My preference goes to reverting CASSANDRA-3458, because I don't really like that we add something we're only mildly getting consensus on to a 1.0 release. Do you mean we don't have strong consensus on including cqlsh in the official C* debs? Or that we don't have strong consensus on my proposal for a solution? I haven't seen any signs of disagreement in either case yet. I'm +0 for just shipping a python-cql deb so that these problems get fixed, rather than putting them off, but I understand the need for stability. Proposal: separate cqlsh from CQL drivers - Key: CASSANDRA-3507 URL: https://issues.apache.org/jira/browse/CASSANDRA-3507 Project: Cassandra Issue Type: Improvement Components: Packaging, Tools Affects Versions: 1.0.3 Environment: Debian-based systems Reporter: paul cannon Assignee: paul cannon Priority: Blocker Labels: cql, cqlsh Fix For: 1.0.7 Whereas: * It has been shown to be very desirable to decouple the release cycles of Cassandra from the various client CQL drivers, and * It is also desirable to include a good interactive CQL client with releases of Cassandra, and * It is not desirable for Cassandra releases to depend on 3rd-party software which is neither bundled with Cassandra nor readily available for every target platform, but * Any good interactive CQL client will require a CQL driver; Therefore, be it resolved that: * cqlsh will not use an official or supported CQL driver, but will include its own private CQL driver, not intended for use by anything else, and * the Cassandra project will still recommend installing and using a proper CQL driver for client software. To ease maintenance, the private CQL driver included with cqlsh may very well be created by copying the python CQL driver from one directory into another, but the user shouldn't rely on this. Maybe we even ought to take some minor steps to discourage its use for other purposes. Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3507) Proposal: separate cqlsh from CQL drivers
[ https://issues.apache.org/jira/browse/CASSANDRA-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13179001#comment-13179001 ] Sylvain Lebresne commented on CASSANDRA-3507: - bq. That's a little harsh- the debian package and the packaging aren't technically broken, they just require users to add a 3rd-party apt source or install a python-cql deb locally. But yeah, I doubt users really appreciate that distinction much. I stand corrected. But in a minor release, anything that require more than 'apt-get update' from the user is a regression from his point of view. bq. Do you mean we don't have strong consensus on including cqlsh in the official C* debs? I don't know. I mean that there has been a number of comments on that issue, and some doubt on what we can and cannot do (it would be counter-productive to start doing this some way to change it in two release, so we'd better make sure we're covered on that -- though I'm not saying you're reading of apache rules was non exhaustive in any way) and that right now this is holding the release of 1.0.7. It's kind of annoying given that I don't feel that cqlsh is something current user of 1.0 are really waiting on (and they can get it if they want anyway) -- without making any judgement on the quality of cqlsh btw. Proposal: separate cqlsh from CQL drivers - Key: CASSANDRA-3507 URL: https://issues.apache.org/jira/browse/CASSANDRA-3507 Project: Cassandra Issue Type: Improvement Components: Packaging, Tools Affects Versions: 1.0.3 Environment: Debian-based systems Reporter: paul cannon Assignee: paul cannon Priority: Blocker Labels: cql, cqlsh Fix For: 1.0.7 Whereas: * It has been shown to be very desirable to decouple the release cycles of Cassandra from the various client CQL drivers, and * It is also desirable to include a good interactive CQL client with releases of Cassandra, and * It is not desirable for Cassandra releases to depend on 3rd-party software which is neither bundled with Cassandra nor readily available for every target platform, but * Any good interactive CQL client will require a CQL driver; Therefore, be it resolved that: * cqlsh will not use an official or supported CQL driver, but will include its own private CQL driver, not intended for use by anything else, and * the Cassandra project will still recommend installing and using a proper CQL driver for client software. To ease maintenance, the private CQL driver included with cqlsh may very well be created by copying the python CQL driver from one directory into another, but the user shouldn't rely on this. Maybe we even ought to take some minor steps to discourage its use for other purposes. Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3507) Proposal: separate cqlsh from CQL drivers
[ https://issues.apache.org/jira/browse/CASSANDRA-3507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13179000#comment-13179000 ] Jonathan Ellis commented on CASSANDRA-3507: --- bq. I read through all the rules I can find, and I see nothing prohibiting us from voting on and releasing specific source/binary artifacts of the various cql drivers alongside c*, as long as they follow the ASF licensing restrictions. Really? It feels weird to have a PMC vote on artifacts for something they don't develop. (While there is overlap between the python cql driver and our PMC, it's not entirely the same thing and it's likely to diverge further in the future, or at least that was the idea of hosting drivers out of tree.) Proposal: separate cqlsh from CQL drivers - Key: CASSANDRA-3507 URL: https://issues.apache.org/jira/browse/CASSANDRA-3507 Project: Cassandra Issue Type: Improvement Components: Packaging, Tools Affects Versions: 1.0.3 Environment: Debian-based systems Reporter: paul cannon Assignee: paul cannon Priority: Blocker Labels: cql, cqlsh Fix For: 1.0.7 Whereas: * It has been shown to be very desirable to decouple the release cycles of Cassandra from the various client CQL drivers, and * It is also desirable to include a good interactive CQL client with releases of Cassandra, and * It is not desirable for Cassandra releases to depend on 3rd-party software which is neither bundled with Cassandra nor readily available for every target platform, but * Any good interactive CQL client will require a CQL driver; Therefore, be it resolved that: * cqlsh will not use an official or supported CQL driver, but will include its own private CQL driver, not intended for use by anything else, and * the Cassandra project will still recommend installing and using a proper CQL driver for client software. To ease maintenance, the private CQL driver included with cqlsh may very well be created by copying the python CQL driver from one directory into another, but the user shouldn't rely on this. Maybe we even ought to take some minor steps to discourage its use for other purposes. Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3685) CQL support for non-utf8 column names
[
https://issues.apache.org/jira/browse/CASSANDRA-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13179069#comment-13179069
]
paul cannon commented on CASSANDRA-3685:
I like the latter, if this is going to be syntax that is only supported in
CREATE TABLE. Otherwise, it looks like a generic CQL function. (I know there's
no technical ambiguity here, yet.)
CQL support for non-utf8 column names
-
Key: CASSANDRA-3685
URL: https://issues.apache.org/jira/browse/CASSANDRA-3685
Project: Cassandra
Issue Type: New Feature
Components: API
Reporter: Jonathan Ellis
Labels: cql
Fix For: 1.2
Eric Evans' suggestions from the mailing list:
{code}
CREATE TABLE test (
int(10) text,
uuid(92d21d0a-d6cb-437c-9d3f-b67aa733a19f) bigint
)
{code}
{code}
CREATE TABLE test (
(int)10 text,
(uuid)92d21d0a-d6cb-437c-9d3f-b67aa733a19f bigint
)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2474) CQL support for compound columns
[ https://issues.apache.org/jira/browse/CASSANDRA-2474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13179070#comment-13179070 ] Jonathan Ellis commented on CASSANDRA-2474: --- bq. I would prefer CREATE COMPACT TABLE ... it's a property of the definition itself (that cannot be changed) rather than of that specific 'body' field. It also make the rule for COMPACT table rather simple: a compact table should have only one field not part of the primary key; you don't have to deal with errors like someone defining two VALUE() for instance. Those are good points. Personally I'd still rather have a WITH COMPACT STORAGE syntax *after* CREATE TABLE ( ... ), since that's a more general way to tackle nonstandard options. bq. I think we have 4 different cases You nailed it, that's exactly what I want to solve here. bq. Random other ideas As you say, let's leave these as future work for now, this is a big enough discussion as it is. :) CQL support for compound columns Key: CASSANDRA-2474 URL: https://issues.apache.org/jira/browse/CASSANDRA-2474 Project: Cassandra Issue Type: New Feature Components: API, Core Reporter: Eric Evans Assignee: Pavel Yaskevich Labels: cql Fix For: 1.1 Attachments: 2474-transposed-1.PNG, 2474-transposed-raw.PNG, 2474-transposed-select-no-sparse.PNG, 2474-transposed-select.PNG, raw_composite.txt, screenshot-1.jpg, screenshot-2.jpg For the most part, this boils down to supporting the specification of compound column names (the CQL syntax is colon-delimted terms), and then teaching the decoders (drivers) to create structures from the results. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1956) Convert row cache to row+filter cache
[ https://issues.apache.org/jira/browse/CASSANDRA-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13179146#comment-13179146 ] Pavel Yaskevich commented on CASSANDRA-1956: What I propose is - let's make a row cache more configurable according to the way we handle big rows, we can allow cache to take a per-cf filter class which would have settings for max row size to cache (and probably some other options in the future) so we can feed it a ColumnFamily from getTopColumns and let it decide if we should just keep that ColumnFamily in cache as is, because it's a thin row, or (if we are querying different parts of the big row) keep QueryFilter and ColumnFamily. Convert row cache to row+filter cache - Key: CASSANDRA-1956 URL: https://issues.apache.org/jira/browse/CASSANDRA-1956 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Stu Hood Assignee: Vijay Priority: Minor Fix For: 1.2 Attachments: 0001-1956-cache-updates-v0.patch, 0001-re-factor-row-cache.patch, 0001-row-cache-filter.patch, 0002-1956-updates-to-thrift-and-avro-v0.patch, 0002-add-query-cache.patch Changing the row cache to a row+filter cache would make it much more useful. We currently have to warn against using the row cache with wide rows, where the read pattern is typically a peek at the head, but this usecase would be perfect supported by a cache that stored only columns matching the filter. Possible implementations: * (copout) Cache a single filter per row, and leave the cache key as is * Cache a list of filters per row, leaving the cache key as is: this is likely to have some gotchas for weird usage patterns, and it requires the list overheard * Change the cache key to rowkey+filterid: basically ideal, but you need a secondary index to lookup cache entries by rowkey so that you can keep them in sync with the memtable * others? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1956) Convert row cache to row+filter cache
[ https://issues.apache.org/jira/browse/CASSANDRA-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13179149#comment-13179149 ] Jonathan Ellis commented on CASSANDRA-1956: --- What if we turned it into a real query cache? That way we don't have to predefine filters in the schema, we just use the IFilter objects from the queries involved. Convert row cache to row+filter cache - Key: CASSANDRA-1956 URL: https://issues.apache.org/jira/browse/CASSANDRA-1956 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Stu Hood Assignee: Vijay Priority: Minor Fix For: 1.2 Attachments: 0001-1956-cache-updates-v0.patch, 0001-re-factor-row-cache.patch, 0001-row-cache-filter.patch, 0002-1956-updates-to-thrift-and-avro-v0.patch, 0002-add-query-cache.patch Changing the row cache to a row+filter cache would make it much more useful. We currently have to warn against using the row cache with wide rows, where the read pattern is typically a peek at the head, but this usecase would be perfect supported by a cache that stored only columns matching the filter. Possible implementations: * (copout) Cache a single filter per row, and leave the cache key as is * Cache a list of filters per row, leaving the cache key as is: this is likely to have some gotchas for weird usage patterns, and it requires the list overheard * Change the cache key to rowkey+filterid: basically ideal, but you need a secondary index to lookup cache entries by rowkey so that you can keep them in sync with the memtable * others? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-1956) Convert row cache to row+filter cache
[ https://issues.apache.org/jira/browse/CASSANDRA-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13179155#comment-13179155 ] Pavel Yaskevich commented on CASSANDRA-1956: That would work, actually I think that we can slightly modify IFilter to return number of columns involved or use resulting ColumnFamily.serializedSize() to check for threshold, I just want to make sure that cache has minimal memory overhead associated with keeping filters for thin rows. Convert row cache to row+filter cache - Key: CASSANDRA-1956 URL: https://issues.apache.org/jira/browse/CASSANDRA-1956 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Stu Hood Assignee: Vijay Priority: Minor Fix For: 1.2 Attachments: 0001-1956-cache-updates-v0.patch, 0001-re-factor-row-cache.patch, 0001-row-cache-filter.patch, 0002-1956-updates-to-thrift-and-avro-v0.patch, 0002-add-query-cache.patch Changing the row cache to a row+filter cache would make it much more useful. We currently have to warn against using the row cache with wide rows, where the read pattern is typically a peek at the head, but this usecase would be perfect supported by a cache that stored only columns matching the filter. Possible implementations: * (copout) Cache a single filter per row, and leave the cache key as is * Cache a list of filters per row, leaving the cache key as is: this is likely to have some gotchas for weird usage patterns, and it requires the list overheard * Change the cache key to rowkey+filterid: basically ideal, but you need a secondary index to lookup cache entries by rowkey so that you can keep them in sync with the memtable * others? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[4/11] git commit: Fix changelog
Fix changelog Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/52e671a3 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/52e671a3 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/52e671a3 Branch: refs/heads/trunk Commit: 52e671a315b9fe295402f17d5e95989b32a68e74 Parents: a1f156d Author: Sylvain Lebresne sylv...@datastax.com Authored: Tue Jan 3 16:32:11 2012 +0100 Committer: Sylvain Lebresne sylv...@datastax.com Committed: Tue Jan 3 16:32:11 2012 +0100 -- CHANGES.txt |3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/52e671a3/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 76d4981..287fed8 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -13,6 +13,9 @@ (CASSANDRA-3335) * (CQL) compaction_strategy_options and compression_parameters for CREATE COLUMNFAMILY statement (CASSANDRA-3374) + * Reset min/max compaction threshold when creating size tiered compaction + strategy (CASSANDRA-3666) + * Don't ignore IOException during compaction (CASSANDRA-3655) Merged from 0.8: * avoid logging (harmless) exception when GC takes 1ms (CASSANDRA-3656) * prevent new nodes from thinking down nodes are up forever (CASSANDRA-3626)
[1/11] git commit: merge from 1.0
Updated Branches: refs/heads/3624 [created] aca3d26c6 refs/heads/cassandra-1.0 52e671a31 - 1a3a52642 refs/heads/trunk e75c53ea1 - 18ce201a0 merge from 1.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/18ce201a Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/18ce201a Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/18ce201a Branch: refs/heads/trunk Commit: 18ce201a0787f4b837cd8840a292535a9115a47a Parents: e75c53e 1a3a526 Author: Jonathan Ellis jbel...@apache.org Authored: Tue Jan 3 21:14:47 2012 -0600 Committer: Jonathan Ellis jbel...@apache.org Committed: Tue Jan 3 21:14:47 2012 -0600 -- CHANGES.txt|4 + .../db/compaction/CompactionIterable.java |2 +- .../cassandra/db/compaction/PrecompactedRow.java | 10 +- .../compaction/SizeTieredCompactionStrategy.java | 10 +- .../cassandra/streaming/IncomingStreamReader.java |9 +- .../cassandra/service/ConsistencyLevelTest.java| 191 --- 6 files changed, 24 insertions(+), 202 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/18ce201a/CHANGES.txt -- diff --cc CHANGES.txt index 3be6393,287fed8..15c829a --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,47 -1,5 +1,48 @@@ +1.1-dev + * avoid returning internal Cassandra classes over JMX (CASSANDRA-2805) + * add row-level isolation via SnapTree (CASSANDRA-2893) + * Optimize key count estimation when opening sstable on startup + (CASSANDRA-2988) + * multi-dc replication optimization supporting CL ONE (CASSANDRA-3577) + * add command to stop compactions (CASSANDRA-1740, 3566, 3582) + * multithreaded streaming (CASSANDRA-3494) + * removed in-tree redhat spec (CASSANDRA-3567) + * defragment rows for name-based queries under STCS, again (CASSANDRA-2503) + * Recycle commitlog segments for improved performance + (CASSANDRA-3411, 3543, 3557, 3615) + * update size-tiered compaction to prioritize small tiers (CASSANDRA-2407) + * add message expiration logic to OutboundTcpConnection (CASSANDRA-3005) + * off-heap cache to use sun.misc.Unsafe instead of JNA (CASSANDRA-3271) + * EACH_QUORUM is only supported for writes (CASSANDRA-3272) + * replace compactionlock use in schema migration by checking CFS.isValid + (CASSANDRA-3116) + * recognize that SELECT first ... * isn't really SELECT * (CASSANDRA-3445) + * Use faster bytes comparison (CASSANDRA-3434) + * Bulk loader is no longer a fat client, (HADOOP) bulk load output format + (CASSANDRA-3045) + * remove assumption that keys and token are in bijection + (CASSANDRA-1034, 3574, 3604) + * always remove endpoints from delevery queue in HH (CASSANDRA-3546) + * fix race between cf flush and its 2ndary indexes flush (CASSANDRA-3547) + * fix potential race in AES when a repair fails (CASSANDRA-3548) + * Remove columns shadowed by a deleted container even when we cannot purge + (CASSANDRA-3538) + * Improve memtable slice iteration performance (CASSANDRA-3545) + * more efficient allocation of small bloom filters (CASSANDRA-3618) + * Use separate writer thread in SSTableSimpleUnsortedWriter (CASSANDRA-3619) + * fsync the directory after new sstable or commitlog segment are created (CASSANDRA-3250) + * fix minor issues reported by FindBugs (CASSANDRA-3658) + * global key/row caches (CASSANDRA-3143) + * optimize memtable iteration during range scan (CASSANDRA-3638) + * introduce 'crc_check_chance' in CompressionParameters to support + a checksum percentage checking chance similarly to read-repair (CASSANDRA-3611) + * a way to deactivate global key/row cache on per-CF basis (CASSANDRA-3667) + * fix LeveledCompactionStrategy broken because of generation pre-allocation + in LeveledManifest (CASSANDRA-3691) + + 1.0.7 + * retry failed stream on IOException (CASSANDRA-3686) * allow configuring bloom_filter_fp_chance (CASSANDRA-3497) * attempt hint delivery every ten minutes, or when failure detector notifies us that a node is back up, whichever comes first. hint http://git-wip-us.apache.org/repos/asf/cassandra/blob/18ce201a/src/java/org/apache/cassandra/db/compaction/PrecompactedRow.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/18ce201a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java -- diff --cc src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java index 171692e,6f6adbc..95db17d --- a/src/java/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategy.java +++
[2/11] git commit: remove ConsistencyLevelTest.java patch by slebresne; reviewed by jbellis for CASSANDRA-3531
remove ConsistencyLevelTest.java
patch by slebresne; reviewed by jbellis for CASSANDRA-3531
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1a3a5264
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1a3a5264
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1a3a5264
Branch: refs/heads/cassandra-1.0
Commit: 1a3a52642c6e72666beb20346093cc3cf2e374f5
Parents: 52e671a
Author: Jonathan Ellis jbel...@apache.org
Authored: Tue Jan 3 21:12:33 2012 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Tue Jan 3 21:12:33 2012 -0600
--
.../cassandra/service/ConsistencyLevelTest.java| 193 ---
1 files changed, 0 insertions(+), 193 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/1a3a5264/test/unit/org/apache/cassandra/service/ConsistencyLevelTest.java
--
diff --git a/test/unit/org/apache/cassandra/service/ConsistencyLevelTest.java
b/test/unit/org/apache/cassandra/service/ConsistencyLevelTest.java
deleted file mode 100644
index af65779..000
--- a/test/unit/org/apache/cassandra/service/ConsistencyLevelTest.java
+++ /dev/null
@@ -1,193 +0,0 @@
-package org.apache.cassandra.service;
-/*
- *
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- *
- */
-
-
-import java.net.InetAddress;
-import java.util.ArrayList;
-import java.util.List;
-
-import org.apache.cassandra.config.Schema;
-import org.junit.Test;
-
-import org.apache.cassandra.CleanupHelper;
-import org.apache.cassandra.Util;
-import org.apache.cassandra.config.ConfigurationException;
-import org.apache.cassandra.config.KSMetaData;
-import org.apache.cassandra.db.Row;
-import org.apache.cassandra.dht.IPartitioner;
-import org.apache.cassandra.dht.RandomPartitioner;
-import org.apache.cassandra.dht.Token;
-import org.apache.cassandra.locator.AbstractReplicationStrategy;
-import org.apache.cassandra.locator.SimpleSnitch;
-import org.apache.cassandra.locator.TokenMetadata;
-import org.apache.cassandra.thrift.ConsistencyLevel;
-import org.apache.cassandra.thrift.UnavailableException;
-import org.apache.cassandra.utils.ByteBufferUtil;
-
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
-
-public class ConsistencyLevelTest extends CleanupHelper
-{
-@Test
-public void testReadWriteConsistencyChecks() throws Exception
-{
-StorageService ss = StorageService.instance;
-final int RING_SIZE = 3;
-
-TokenMetadata tmd = ss.getTokenMetadata();
-tmd.clearUnsafe();
-IPartitioner partitioner = new RandomPartitioner();
-
-ss.setPartitionerUnsafe(partitioner);
-
-ArrayListToken endpointTokens = new ArrayListToken();
-ArrayListToken keyTokens = new ArrayListToken();
-ListInetAddress hostsInUse = new ArrayListInetAddress();
-ListInetAddress hosts = new ArrayListInetAddress();
-
-Util.createInitialRing(ss, partitioner, endpointTokens, keyTokens,
hosts, RING_SIZE);
-
-AbstractReplicationStrategy strategy;
-
-for (final String table : Schema.instance.getNonSystemTables())
-{
-strategy = getStrategy(table, tmd);
-StorageService.calculatePendingRanges(strategy, table);
-int replicationFactor = strategy.getReplicationFactor();
-if (replicationFactor 2)
-continue;
-
-for (ConsistencyLevel c : ConsistencyLevel.values())
-{
-
-if (c == ConsistencyLevel.EACH_QUORUM || c ==
ConsistencyLevel.LOCAL_QUORUM)
-continue;
-
-for (int i = 0; i replicationFactor; i++)
-{
-hostsInUse.clear();
-for (int j = 0 ; j i ; j++)
-{
-hostsInUse.add(hosts.get(j));
-}
-
-if (hostsInUse.isEmpty())
-{
-// We skip this case as it means RF =
[6/11] git commit: Don't ignore IOException during compaction
Don't ignore IOException during compaction
patch by jbellis; reviewed by slebresne for CASSANDRA-3655
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/6eac35cf
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/6eac35cf
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/6eac35cf
Branch: refs/heads/trunk
Commit: 6eac35cfd0bfdf67e96f9147b4aad3ddd24c99c2
Parents: add9372
Author: Sylvain Lebresne sylv...@datastax.com
Authored: Tue Jan 3 16:24:41 2012 +0100
Committer: Sylvain Lebresne sylv...@datastax.com
Committed: Tue Jan 3 16:24:41 2012 +0100
--
.../db/compaction/CompactionIterable.java |2 +-
.../cassandra/db/compaction/PrecompactedRow.java | 10 --
.../cassandra/streaming/IncomingStreamReader.java |4 +---
3 files changed, 10 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6eac35cf/src/java/org/apache/cassandra/db/compaction/CompactionIterable.java
--
diff --git
a/src/java/org/apache/cassandra/db/compaction/CompactionIterable.java
b/src/java/org/apache/cassandra/db/compaction/CompactionIterable.java
index bb8d2bd..5e0dfa7 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionIterable.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionIterable.java
@@ -95,7 +95,7 @@ public class CompactionIterable extends
AbstractCompactionIterable
protected AbstractCompactedRow getReduced()
{
-assert rows.size() 0;
+assert !rows.isEmpty();
try
{
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6eac35cf/src/java/org/apache/cassandra/db/compaction/PrecompactedRow.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/PrecompactedRow.java
b/src/java/org/apache/cassandra/db/compaction/PrecompactedRow.java
index 6363d9d..5b5ba8d 100644
--- a/src/java/org/apache/cassandra/db/compaction/PrecompactedRow.java
+++ b/src/java/org/apache/cassandra/db/compaction/PrecompactedRow.java
@@ -22,6 +22,7 @@ package org.apache.cassandra.db.compaction;
import java.io.DataOutput;
+import java.io.IOError;
import java.io.IOException;
import java.security.MessageDigest;
import java.util.List;
@@ -56,6 +57,10 @@ public class PrecompactedRow extends AbstractCompactedRow
public static ColumnFamily removeDeletedAndOldShards(DecoratedKey? key,
CompactionController controller, ColumnFamily cf)
{
+assert key != null;
+assert controller != null;
+assert cf != null;
+
// avoid calling shouldPurge unless we actually need to: it can be
very expensive if LCS
// gets behind and has hundreds of overlapping L0 sstables.
Essentially, this method is an
// ugly refactor of
removeDeletedAndOldShards(controller.shouldPurge(key), controller, cf),
@@ -95,6 +100,7 @@ public class PrecompactedRow extends AbstractCompactedRow
private static ColumnFamily merge(ListSSTableIdentityIterator rows)
{
+assert !rows.isEmpty();
ColumnFamily cf = null;
for (SSTableIdentityIterator row : rows)
{
@@ -105,9 +111,9 @@ public class PrecompactedRow extends AbstractCompactedRow
}
catch (IOException e)
{
-logger.error(Skipping row + row.getKey() + in +
row.getPath(), e);
-continue;
+throw new IOError(e);
}
+
if (cf == null)
{
cf = thisCF;
http://git-wip-us.apache.org/repos/asf/cassandra/blob/6eac35cf/src/java/org/apache/cassandra/streaming/IncomingStreamReader.java
--
diff --git a/src/java/org/apache/cassandra/streaming/IncomingStreamReader.java
b/src/java/org/apache/cassandra/streaming/IncomingStreamReader.java
index f70f5ff..e2a618f 100644
--- a/src/java/org/apache/cassandra/streaming/IncomingStreamReader.java
+++ b/src/java/org/apache/cassandra/streaming/IncomingStreamReader.java
@@ -108,7 +108,7 @@ public class IncomingStreamReader
ColumnFamilyStore cfs =
Table.open(localFile.desc.ksname).getColumnFamilyStore(localFile.desc.cfname);
DecoratedKey key;
SSTableWriter writer = new SSTableWriter(localFile.getFilename(),
remoteFile.estimatedKeys);
-CompactionController controller = null;
+CompactionController controller = new CompactionController(cfs,
Collections.SSTableReaderemptyList(), Integer.MIN_VALUE, true);
try
{
@@ -128,8 +128,6 @@ public class IncomingStreamReader
if (cached != null remoteFile.type ==
[11/11] git commit: dynamic HH page size
dynamic HH page size
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8e528c8a
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8e528c8a
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8e528c8a
Branch: refs/heads/3624
Commit: 8e528c8abfc2f62cb10fb8de21e1cb58f70d281e
Parents: 02346a1
Author: Jonathan Ellis jbel...@apache.org
Authored: Wed Dec 28 20:58:22 2011 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Wed Dec 28 20:58:22 2011 -0600
--
.../apache/cassandra/db/HintedHandOffManager.java | 14 --
1 files changed, 12 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/8e528c8a/src/java/org/apache/cassandra/db/HintedHandOffManager.java
--
diff --git a/src/java/org/apache/cassandra/db/HintedHandOffManager.java
b/src/java/org/apache/cassandra/db/HintedHandOffManager.java
index a028bdf..0ac93d8 100644
--- a/src/java/org/apache/cassandra/db/HintedHandOffManager.java
+++ b/src/java/org/apache/cassandra/db/HintedHandOffManager.java
@@ -87,7 +87,7 @@ public class HintedHandOffManager implements
HintedHandOffManagerMBean
public static final String HINTS_CF = HintsColumnFamily;
private static final Logger logger_ =
LoggerFactory.getLogger(HintedHandOffManager.class);
-private static final int PAGE_SIZE = 1024;
+private static final int PAGE_SIZE = 128;
private static final int LARGE_NUMBER = 65536; // 64k nodes ought to be
enough for anybody.
// in 0.8, subcolumns were KS-CF bytestrings, and the data was stored in
the normal storage there.
@@ -270,10 +270,20 @@ public class HintedHandOffManager implements
HintedHandOffManagerMBean
int rowsReplayed = 0;
ByteBuffer startColumn = ByteBufferUtil.EMPTY_BYTE_BUFFER;
+int pageSize = PAGE_SIZE;
+// read less columns (mutations) per page if they are very large
+if (hintStore.getMeanColumns() 0)
+{
+int averageColumnSize = (int) (hintStore.getMeanRowSize() /
hintStore.getMeanColumns());
+pageSize = Math.min(PAGE_SIZE,
DatabaseDescriptor.getInMemoryCompactionLimit() / averageColumnSize);
+pageSize = Math.max(2, pageSize); // page size of 1 does not allow
actual paging b/c of = behavr on startColumn
+logger_.debug(average hinted-row column size is {}; using
pageSize of {}, averageColumnSize, pageSize);
+}
+
delivery:
while (true)
{
-QueryFilter filter = QueryFilter.getSliceFilter(epkey, new
QueryPath(HINTS_CF), startColumn, ByteBufferUtil.EMPTY_BYTE_BUFFER, false,
PAGE_SIZE);
+QueryFilter filter = QueryFilter.getSliceFilter(epkey, new
QueryPath(HINTS_CF), startColumn, ByteBufferUtil.EMPTY_BYTE_BUFFER, false,
pageSize);
ColumnFamily hintsPage =
ColumnFamilyStore.removeDeleted(hintStore.getColumnFamily(filter),
Integer.MAX_VALUE);
if (pagingFinished(hintsPage, startColumn))
break;
[10/11] git commit: don't clear queuedDeliveries entry until actually done
don't clear queuedDeliveries entry until actually done
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/aca3d26c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/aca3d26c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/aca3d26c
Branch: refs/heads/3624
Commit: aca3d26c667676c2ac2cc871f3436b11356bf181
Parents: 8e528c8
Author: Jonathan Ellis jbel...@apache.org
Authored: Wed Dec 28 21:05:15 2011 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Wed Dec 28 21:05:15 2011 -0600
--
.../apache/cassandra/db/HintedHandOffManager.java | 41 +++
1 files changed, 19 insertions(+), 22 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/aca3d26c/src/java/org/apache/cassandra/db/HintedHandOffManager.java
--
diff --git a/src/java/org/apache/cassandra/db/HintedHandOffManager.java
b/src/java/org/apache/cassandra/db/HintedHandOffManager.java
index 0ac93d8..e4a3afd 100644
--- a/src/java/org/apache/cassandra/db/HintedHandOffManager.java
+++ b/src/java/org/apache/cassandra/db/HintedHandOffManager.java
@@ -229,31 +229,28 @@ public class HintedHandOffManager implements
HintedHandOffManagerMBean
private void deliverHintsToEndpoint(InetAddress endpoint) throws
IOException, DigestMismatchException, InvalidRequestException,
TimeoutException, InterruptedException
{
ColumnFamilyStore hintStore =
Table.open(Table.SYSTEM_TABLE).getColumnFamilyStore(HINTS_CF);
-try
+if (hintStore.isEmpty())
{
-if (hintStore.isEmpty())
-return; // nothing to do, don't confuse users by logging a
no-op handoff
-
-logger_.debug(Checking remote({}) schema before delivering
hints, endpoint);
-int waited = waitForSchemaAgreement(endpoint);
-// sleep a random amount to stagger handoff delivery from
different replicas.
-// (if we had to wait, then gossiper randomness took care of that
for us already.)
-if (waited == 0) {
-// use a 'rounded' sleep interval because of a strange bug
with windows: CASSANDRA-3375
-int sleep = FBUtilities.threadLocalRandom().nextInt(2000) * 30;
-logger_.debug(Sleeping {}ms to stagger hint delivery, sleep);
-Thread.sleep(sleep);
-}
+queuedDeliveries.remove(endpoint);
+return; // nothing to do, don't confuse users by logging a no-op
handoff
+}
-if (!FailureDetector.instance.isAlive(endpoint))
-{
-logger_.info(Endpoint {} died before hint delivery,
aborting, endpoint);
-return;
-}
+logger_.debug(Checking remote({}) schema before delivering hints,
endpoint);
+int waited = waitForSchemaAgreement(endpoint);
+// sleep a random amount to stagger handoff delivery from different
replicas.
+// (if we had to wait, then gossiper randomness took care of that for
us already.)
+if (waited == 0) {
+// use a 'rounded' sleep interval because of a strange bug with
windows: CASSANDRA-3375
+int sleep = FBUtilities.threadLocalRandom().nextInt(2000) * 30;
+logger_.debug(Sleeping {}ms to stagger hint delivery, sleep);
+Thread.sleep(sleep);
}
-finally
+
+if (!FailureDetector.instance.isAlive(endpoint))
{
+logger_.info(Endpoint {} died before hint delivery, aborting,
endpoint);
queuedDeliveries.remove(endpoint);
+return;
}
// 1. Get the key of the endpoint we need to handoff
@@ -341,8 +338,8 @@ public class HintedHandOffManager implements
HintedHandOffManagerMBean
}
}
-logger_.info(String.format(Finished hinted handoff of %s rows to
endpoint %s,
- rowsReplayed, endpoint));
+logger_.info(String.format(Finished hinted handoff of %s rows to
endpoint %s, rowsReplayed, endpoint));
+queuedDeliveries.remove(endpoint);
}
/**
[9/11] git commit: -javadoc and -sources jars for clientutil
-javadoc and -sources jars for clientutil
Patch by satish babu krishnamoorthy w/ minor changes; reviewed by eevans for
CASSANDRA-3426
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/8703f745
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/8703f745
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/8703f745
Branch: refs/heads/trunk
Commit: 8703f745fede8705d8e186c0bf6e91a32159701c
Parents: 02346a1
Author: Eric Evans eev...@apache.org
Authored: Thu Dec 29 15:33:45 2011 -0600
Committer: Eric Evans eev...@apache.org
Committed: Thu Dec 29 16:38:24 2011 -0600
--
build.xml | 46 ++
1 files changed, 38 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/8703f745/build.xml
--
diff --git a/build.xml b/build.xml
index 70aa7e5..0cb7f9e 100644
--- a/build.xml
+++ b/build.xml
@@ -790,17 +790,29 @@ url=${svn.entry.url}?pathrev=${svn.entry.commit.revision}
create-javadoc destdir=${javadoc.jars.dir}/main
filesets
-fileset dir=${build.src.java} defaultexcludes=yes
- include name=org/apache/**/*.java/
-/fileset
-fileset dir=${build.src.gen-java} defaultexcludes=yes
- include name=org/apache/**/*.java/
-/fileset
+ fileset dir=${build.src.java} defaultexcludes=yes
+include name=org/apache/**/*.java/
+ /fileset
+ fileset dir=${build.src.gen-java} defaultexcludes=yes
+include name=org/apache/**/*.java/
+ /fileset
/filesets
/create-javadoc
jar jarfile=${build.dir}/${final.name}-javadoc.jar
basedir=${javadoc.jars.dir}/main/
+ create-javadoc destdir=${javadoc.jars.dir}/clientutil
+filesets
+ fileset dir=${build.src.java} defaultexcludes=yes
+include name=org/apache/cassandra/cql/jdbc/* /
+include name=org/apache/cassandra/utils/ByteBufferUtil*.java /
+include name=org/apache/cassandra/utils/Hex.java /
+include name=org/apache/cassandra/utils/UUIDGen*.java /
+ /fileset
+/filesets
+ /create-javadoc
+ jar
jarfile=${build.dir}/${ant.project.name}-clientutil-${version}-javadoc.jar
+ basedir=${javadoc.jars.dir}/clientutil/
!-- javadoc task always rebuilds so might as well remove the generated
docs to prevent
being pulled into the distribution by accident --
delete quiet=true dir=${javadoc.jars.dir}/
@@ -823,6 +835,14 @@ url=${svn.entry.url}?pathrev=${svn.entry.commit.revision}
include name=org/apache/**/*.java/
/fileset
/jar
+ jar
jarfile=${build.dir}/${ant.project.name}-clientutil-${version}-sources.jar
+fileset dir=${build.src.java} defaultexcludes=yes
+ include name=org/apache/cassandra/cql/jdbc/* /
+ include name=org/apache/cassandra/utils/ByteBufferUtil*.java /
+ include name=org/apache/cassandra/utils/Hex.java /
+ include name=org/apache/cassandra/utils/UUIDGen*.java /
+/fileset
+ /jar
/target
!-- creates release tarballs --
@@ -1253,6 +1273,12 @@ url=${svn.entry.url}?pathrev=${svn.entry.commit.revision}
!-- the cassandra-clientutil jar --
install
pomFile=${build.dir}/${ant.project.name}-clientutil-${version}.pom
file=${build.dir}/${ant.project.name}-clientutil-${version}.jar/
+install
pomFile=${build.dir}/${ant.project.name}-clientutil-${version}.pom
+
file=${build.dir}/${ant.project.name}-clientutil-${version}-sources.jar
+ classifier=sources/
+install
pomFile=${build.dir}/${ant.project.name}-clientutil-${version}.pom
+
file=${build.dir}/${ant.project.name}-clientutil-${version}-javadoc.jar
+ classifier=javadoc/
!-- the cassandra-all jar --
install pomFile=${build.dir}/${final.name}.pom
@@ -1301,7 +1327,12 @@ url=${svn.entry.url}?pathrev=${svn.entry.commit.revision}
!-- the cassandra-clientutil jar --
deploy
pomFile=${build.dir}/${ant.project.name}-clientutil-${version}.pom
file=${build.dir}/${ant.project.name}-clientutil-${version}.jar/
-
+deploy
pomFile=${build.dir}/${ant.project.name}-clientutil-${version}.pom
+
file=${build.dir}/${ant.project.name}-clientutil-${version}-sources.jar
+ classifier=sources/
+deploy
pomFile=${build.dir}/${ant.project.name}-clientutil-${version}.pom
+
file=${build.dir}/${ant.project.name}-clientutil-${version}-javadoc.jar
+ classifier=javadoc/
!-- the cassandra-all jar --
deploy pomFile=${build.dir}/${final.name}.pom
[8/11] git commit: retry failed stream on IOException patch by yukim; reviewed by jbellis for CASSANDRA-3686
retry failed stream on IOException
patch by yukim; reviewed by jbellis for CASSANDRA-3686
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7efab48e
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7efab48e
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7efab48e
Branch: refs/heads/trunk
Commit: 7efab48e72247734ae38eabfc61a9ff0a5aa87ec
Parents: 02346a1
Author: Jonathan Ellis jbel...@apache.org
Authored: Fri Dec 30 23:04:43 2011 -0600
Committer: Jonathan Ellis jbel...@apache.org
Committed: Fri Dec 30 23:04:43 2011 -0600
--
CHANGES.txt|1 +
.../cassandra/streaming/IncomingStreamReader.java |5 -
2 files changed, 5 insertions(+), 1 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/7efab48e/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index bd35149..76d4981 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
1.0.7
+ * retry failed stream on IOException (CASSANDRA-3686)
* allow configuring bloom_filter_fp_chance (CASSANDRA-3497)
* attempt hint delivery every ten minutes, or when failure detector
notifies us that a node is back up, whichever comes first. hint
http://git-wip-us.apache.org/repos/asf/cassandra/blob/7efab48e/src/java/org/apache/cassandra/streaming/IncomingStreamReader.java
--
diff --git a/src/java/org/apache/cassandra/streaming/IncomingStreamReader.java
b/src/java/org/apache/cassandra/streaming/IncomingStreamReader.java
index 8c89873..f70f5ff 100644
--- a/src/java/org/apache/cassandra/streaming/IncomingStreamReader.java
+++ b/src/java/org/apache/cassandra/streaming/IncomingStreamReader.java
@@ -158,7 +158,10 @@ public class IncomingStreamReader
catch (Exception e)
{
writer.abort();
-throw FBUtilities.unchecked(e);
+if (e instanceof IOException)
+throw (IOException) e;
+else
+throw FBUtilities.unchecked(e);
}
}
[7/11] git commit: Merge branch 'cassandra-1.0' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-1.0
Merge branch 'cassandra-1.0' of https://git-wip-us.apache.org/repos/asf/cassandra into cassandra-1.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/add9372b Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/add9372b Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/add9372b Branch: refs/heads/trunk Commit: add9372b027ca5d5433e5da03ca35a0ea94dc887 Parents: 7efab48 8703f74 Author: Jonathan Ellis jbel...@apache.org Authored: Sat Dec 31 10:07:54 2011 -0600 Committer: Jonathan Ellis jbel...@apache.org Committed: Sat Dec 31 10:07:54 2011 -0600 -- build.xml | 46 ++ 1 files changed, 38 insertions(+), 8 deletions(-) --
[jira] [Resolved] (CASSANDRA-3531) Fix crack-smoking in ConsistencyLevelTest
[ https://issues.apache.org/jira/browse/CASSANDRA-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-3531. --- Resolution: Fixed Reviewer: jbellis Assignee: Sylvain Lebresne You're right: while it's reasonable to unit-test assureSufficientLiveNodes, the right place to do that is just with IWriteResponseHandler objects instead of mocking up a ring. This test also pre-dated NTS and the different IWRH for that, so it's pretty fragile. Went ahead and deleted it per your suggestion. Fix crack-smoking in ConsistencyLevelTest -- Key: CASSANDRA-3531 URL: https://issues.apache.org/jira/browse/CASSANDRA-3531 Project: Cassandra Issue Type: Bug Components: Tests Affects Versions: 1.0.4 Reporter: Sylvain Lebresne Assignee: Sylvain Lebresne Priority: Minor Fix For: 1.0.7 First, let's note that this test fails in current 1.0 branch. It was broken (emphasis on the quotes) by CASSANDRA-3529. But it's not CASSANDRA-3529 fault, it's only that the use of NonBlockingHashMap changed the order of the tables returned by Schema.instance.getNonSystemTables(). *And*, it turns out that ConsistencyLevelTest bails out as soon as it has found one keyspace with rf = 2 due to a misplaced return. So it use to be that ConsistencyLevelTest was only ran for Keyspace5 (whose RF is 2) for which the test work. But for any RF 2, the test fails. The reason of this failing is that the test creates a 3 node cluster for whom only 1 node is alive as far as the failure detector is concerned. So for RF=3 and CL=QUORUM, the writes are unavailable (the failure detector is queried), while for reads we pretend two nodes are alive so we end up with a case where isWriteUnavailable != isReadUnavailable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3624) Hinted Handoff - related OOM
[ https://issues.apache.org/jira/browse/CASSANDRA-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-3624: -- Reviewer: brandon.williams (was: krummas) Hinted Handoff - related OOM Key: CASSANDRA-3624 URL: https://issues.apache.org/jira/browse/CASSANDRA-3624 Project: Cassandra Issue Type: Bug Affects Versions: 1.0.0 Reporter: Marcus Eriksson Assignee: Jonathan Ellis Labels: hintedhandoff Fix For: 1.0.7 Attachments: 3624-rebased.txt, 3624.txt One of our nodes had collected alot of hints for another node, so when the dead node came back and the row mutations were read back from disk, the node died with an OOM-exception (and kept dying after restart, even with increased heap (from 8G to 12G)). The heap dump contained alot of SuperColumns and our application does not use those (but HH does). I'm guessing that each mutation is big so that PAGE_SIZE*mutation_size does not fit in memory (will check this tomorrow) A simple fix (if my assumption above is correct) would be to reduce the PAGE_SIZE in HintedHandOffManager.java to something like 10 (or even 1?) to reduce the memory pressure. The performance hit would be small since we are doing the hinted handoff throttle delay sleep before sending every *mutation* anyway (not every page), thoughts? If anyone runs in to the same problem, I got the node started again by simply removing the HintsColumnFamily* files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Git Push Summary
Updated Branches: refs/heads/3624 [deleted] aca3d26c6
[jira] [Reopened] (CASSANDRA-3497) BloomFilter FP ratio should be configurable or size-restricted some other way
[ https://issues.apache.org/jira/browse/CASSANDRA-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis reopened CASSANDRA-3497: --- BloomFilter FP ratio should be configurable or size-restricted some other way - Key: CASSANDRA-3497 URL: https://issues.apache.org/jira/browse/CASSANDRA-3497 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Brandon Williams Assignee: Yuki Morishita Priority: Minor Fix For: 1.0.7 Attachments: 0001-give-default-val-to-fp_chance.patch, 3497-v3.txt, 3497-v4.txt, CASSANDRA-1.0-3497.txt When you have a live dc and purely analytical dc, in many situations you can have less nodes on the analytical side, but end up getting restricted by having the BloomFilters in-memory, even though you have absolutely no use for them. It would be nice if you could reduce this memory requirement by tuning the desired FP ratio, or even just disabling them altogether. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3642) Failed to delete any Index.db on Windows
[
https://issues.apache.org/jira/browse/CASSANDRA-3642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13179253#comment-13179253
]
Jonathan Ellis commented on CASSANDRA-3642:
---
The Sun/Oracle JVM does include the cleaner method on Windows as well, so
unless Viktor is running another JVM that's not going to fix it.
But now I find I can't reproduce my own test failures either, with or without
this patch. I get a bunch of failures like this with SSTableReaderTest:
{noformat}
[junit] java.io.IOException: Exception while executing the command: cmd /c
mklink /H
C:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\backups\Standard1-hc-2-Statistics.db
c:\Users\Jonathan\projects\cassandra\git\build\test\cassandra\data\Keyspace1\Standard1-hc-2-Statistics.db,
command error Code: 1, command output: Cannot create a file when that file
already exists.
[junit]
[junit] at org.apache.cassandra.utils.CLibrary.exec(CLibrary.java:213)
[junit] at
org.apache.cassandra.utils.CLibrary.createHardLinkWithExec(CLibrary.java:188)
[junit] at
org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:151)
[junit] at
org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:833)
[junit] at
org.apache.cassandra.db.DataTracker$1.runMayThrow(DataTracker.java:161)
[junit] at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
[junit] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
[junit] at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit] at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
[junit] at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
[junit] at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[junit] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[junit] at java.lang.Thread.run(Thread.java:662)
{noformat}
... but rolling back to 1.0.2 (before the sha was introduced) doesn't help
reproduce it either. Weird.
Failed to delete any Index.db on Windows
Key: CASSANDRA-3642
URL: https://issues.apache.org/jira/browse/CASSANDRA-3642
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.0.0
Environment: Windows Server 2008 R2 64bit
Java SE 7u1 64bit
Reporter: Viktor Jevdokimov
Assignee: Sylvain Lebresne
Priority: Minor
Fix For: 1.0.7
Attachments: 3642.patch
After upgrade Cassandra 0.8.7 to 1.0.6 under Windows Server 2008 R2 64bit
with disk acces mode mmap_index_only, Cassandra failing to delete any
*-Index.db files after compaction or scrub:
ERROR 13:43:17,490 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
java.lang.RuntimeException: java.io.IOException: Failed to delete
D:\cassandra\data\data\system\LocationInfo-g-29-Index.db
at
org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:689)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown
Source)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source) Caused by:
java.io.IOException: Failed to delete
D:\cassandra\data\data\system\LocationInfo-g-29-Index.db
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54)
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:44)
at org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:141)
at
org.apache.cassandra.io.sstable.SSTableDeletingTask.runMayThrow(SSTableDeletingTask.java:81)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 8 more
ERROR 17:20:09,701 Fatal exception in thread Thread[NonPeriodicTasks:1,5,main]
java.lang.RuntimeException: java.io.IOException: Failed to delete
D:\cassandra\data\data\Keyspace1\ColumnFamily1-hc-840-Index.db
[jira] [Commented] (CASSANDRA-3625) Do something about DynamicCompositeType
[
https://issues.apache.org/jira/browse/CASSANDRA-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13179259#comment-13179259
]
Ed Anuff commented on CASSANDRA-3625:
-
Thanks for taking this one on, I'd hoped to get a chance to work on it but
wasn't able to. This looks like it will solve the problem nicely by doing the
comparison on classnames. I'll try it out, is there a test case for it?
Do something about DynamicCompositeType
---
Key: CASSANDRA-3625
URL: https://issues.apache.org/jira/browse/CASSANDRA-3625
Project: Cassandra
Issue Type: Improvement
Components: Core
Reporter: Sylvain Lebresne
Attachments: 0001-allow-comparing-different-types.patch
Currently, DynamicCompositeType is a super dangerous type. We cannot leave it
that way or people will get hurt.
Let's recall that DynamicCompositeType allows composite column names without
any limitation on what each component type can be. It was added to basically
allow to use different rows of the same column family to each store a
different index. So for instance you would have:
{noformat}
index1: {
bar:24 - someval
bar:42 - someval
foo:12 - someval
...
}
index2: {
0:uuid1:3.2 - someval
1:uuid2:2.2 - someval
...
}
{noformat}
where index1, index2, ... are rows.
So each row have columns whose names have similar structure (so they can be
compared), but between rows the structure can be different (we neve compare
two columns from two different rows).
But the problem is the following: what happens if in the index1 row above,
you insert a column whose name is 0:uuid1 ? There is no really meaningful way
to compare bar:24 and 0:uuid1. The current implementation of
DynamicCompositeType, when confronted with this, says that it is a user error
and throw a MarshalException.
The problem with that is that the exception is not throw at insert time, and
it *cannot* be because of the dynamic nature of the comparator. But that
means that if you do insert the wrong column in the wrong row, you end up
*corrupting* a sstable.
It is too dangerous a behavior. And it's probably made worst by the fact that
some people probably think that DynamicCompositeType should be superior to
CompositeType since you know, it's dynamic.
One solution to that problem could be to decide of some random (but
predictable) order between two incomparable component. For example we could
design that IntType LongType StringType ...
Note that even if we do that, I would suggest renaming the
DynamicCompositeType to something that suggest that CompositeType is always
preferable to DynamicCompositeType unless you're really doing very advanced
stuffs.
Opinions?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
