TLS version error when using cqlsh with SSL
Hi, I'm getting the following error on the node when trying to connect to Cassandra through cqlsh with SSL enabled: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported I'm running the C* code (based on trunk from July 5th 2017) in Intellij. After having run 'ant realclean' the issue temporarily disappeared, but now it's back again and 'realclean' isn't addressing the issue anymore. My Netty version is 4.0.44.Final, and the server/client encryption settings from the cassandra.yaml file are as follows: -- server_encryption_options: cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA256] internode_encryption: all require_client_auth: true protocol: TLSv1.2 client_encryption_options: enabled: true optional: false protocol: TLSv1.2 require_client_auth: true cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA256] -- Things that haven't helped: * Changing the TLS version to either TLSv1 or TLSv1.1. * Using Cassandra 3.0 * Googling the issue (others have had the same error message in other contexts, but those solutions don't seem to apply here) * Using other certificates * Running Cassandra outside of Intellij (starting the /bin/cassandra script) Any help is really appreciated. Thanks! BR, Fredrik
high WMI CPU usage on Windows
Hi Does anyone know what Cassandra uses WMI for when running on Windows? We have some issues with high WMI CPU usage.. Yours sincerely / Med venlig hilsen Fredrik Løkke (FRSKL) Backend Architect, Application Development Plant Application (PA) Technology & Service Solutions (TSS) Vestas Technology R M +45 5215 7675 fr...@vestas.com<mailto:fr...@vestas.com> http://www.vestas.com<http://www.vestas.com/> Company reg. name: Vestas Wind Systems A/S This e-mail is subject to our e-mail disclaimer statement. Please refer to www.vestas.com/legal/notice<http://www.vestas.com/legal/notice> If you have received this e-mail in error please contact the sender.
Corrupt SSTables
Hi all. Having two FSReadErrors: FSReadError in ..\..\data\system\compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca\system-compaction_history-ka-329-CompressionInfo.db FSReadError in ..\..\data\system\sstable_activity-5a1ff267ace03f128563cfae6103c65e\system-sstable_activity-ka-475-CompressionInfo.db >From my understanding, the system tables are local (not replicated) which means that removing those sstables and then run repair wont help. If running scrub on those sstables doesn't help would it be safe to delete those SSTables? /Fred
Re: Upgrade from 2.0.9 to 2.1.3
Thanks for the advice. /Fredrik 7 mar 2015 kl. 02:25 skrev graham sanderson gra...@vast.com: Note for anyone who accidentally or otherwise ends up with 2.1.3 in a situation they cannot downgrade, feel free to look at https://github.com/vast-engineering/cassandra/tree/vast-cassandra-2.1.3 https://github.com/vast-engineering/cassandra/tree/vast-cassandra-2.1.3 We sometimes make custom versions incorporating as many important patches as we reasonably can that we need to run a newer C* environment successfully. Obviously use at your own risk blah blah… basically install procedure would be to replace the main cassandra jar on a 2.1.3 node while it is down. On Mar 6, 2015, at 3:15 PM, Robert Coli rc...@eventbrite.com mailto:rc...@eventbrite.com wrote: On Fri, Mar 6, 2015 at 6:25 AM, graham sanderson gra...@vast.com mailto:gra...@vast.com wrote: I would definitely wait for at least 2.1.4 +1 https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ =Rob
Upgrade from 2.0.9 to 2.1.3
What’s the recommended way of upgrading from 2.0.9 to 2.1.3? Is upgradeSSTables required? According to http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeC_c.html http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeC_c.html it should be possible to just start the on 2.1.3 directly after 2.0.9. Regards Fredrik
Re: Upgrade from 2.0.9 to 2.1.3
So no upgradeSSTables are required? /Fredrik 6 mar 2015 kl. 15:11 skrev Carlos Rolo r...@pythian.com: I would not recommend an upgrade to 2.1.x for now. Do you have any specific reason to upgrade? For upgrading from 2.0.9 you can just do a direct upgrade. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo Tel: 1649 www.pythian.com http://www.pythian.com/ On Fri, Mar 6, 2015 at 3:03 PM, Fredrik Larsson Stigbäck fredrik.l.stigb...@sitevision.se mailto:fredrik.l.stigb...@sitevision.se wrote: What’s the recommended way of upgrading from 2.0.9 to 2.1.3? Is upgradeSSTables required? According to http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeC_c.html http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeC_c.html it should be possible to just start the on 2.1.3 directly after 2.0.9. Regards Fredrik --
Cassandra 2.1.3, Windows 7 clear snapshot
What is the current status of clearing snapshots on Windows? When running Cassandra 2.1.3, trying manually to run clearSnapshot I get: FSWriteError… Caused by: java.nio.file.FileSystemException… File is used by another process I know there’s been numerous issues in JIRA trying to fix similar problems e.g. https://issues.apache.org/jira/browse/CASSANDRA-6283 https://issues.apache.org/jira/browse/CASSANDRA-6283 Are there any outstanding issues in 2.1.3 which specifically pinpoints manually clearing snapshots on Windows? Regards Fredrik
Snappy 1.1.0 Cassandra 2.1.2 compability
Is it safe to replace Snappy 1.0.5 in a Cassandra 2.1.2 environment with Snappy 1.1.0? I’ve tried running with 1.1.0 and Cassandra seems to run with no issues and according to this post https://github.com/xerial/snappy-java/issues/60 https://github.com/xerial/snappy-java/issues/60 1.1.0 is compatible with 1.0.5. But might there be problems/data incompatibility in the future when upgrading Cassandra to a never version regarding *CompressionInfo.db files etc..? /Fredrik
Does Cassandra support running on Java 8?
Are there any official recomendations, validations/tests done with Cassandra = 2.0 on Java 8? Regards /Fredrik
mmaped files and swap
I've got a question regarding understanding the recomendation to disable swap. Since Cassandra uses mlockall to lock the heap in RAM what is the reason for disabling swap? My guess is that is has to do with memory mapped files but as of my understanding, accessing pages of memory mapped files, those pages are never put in swap since they're backed by files on disk and the OS writes those pages to the memory mapped file instead of swap. We've seen on Cassandra installations on Linux with swap enabled that parts of the java process is swaped out and increasing. So what's swaped out? Regards /Fredrik
Re: mmaped files and swap
Well, we've seen a Cassandra process swap out 500 MB on a Linux OS with plenty of RAM, so I was just curious as why the OS thinks it should use the swap at all. 2013/3/13 karim duran karim.du...@gmail.com: I agree with Edward Capriolo, Even when swap is enabled on your system, swaping rarely occurs on OS today...(except for very loaded systems). But, take care that some 32 bits system kernels allows only 2^32 bits memory mapped file length ( ~ 2 Go ). It could be a limitation for NoSQL databases. It's the case for MongoDB on 32 bits OS. I don't know how to avoid swaping if Cassandra exceeds these limitation when this case occurs. 2013/3/13 Edward Capriolo edlinuxg...@gmail.com You really can not control what the OS-swaps out. java has other memory usage outside the heap, and native memory. best to turn swap off. Swap is kinda old school anyway at this point. It made sense when machines had 32MB RAM. Keeping your read 95th percentile low is mostly about removing deviations that cause requests to slow down, swap is one of the things that cause fluctuation becuase it is not predictable. On Wed, Mar 13, 2013 at 10:39 AM, Fredrik fredrik.l.stigb...@sitevision.se wrote: I've got a question regarding understanding the recomendation to disable swap. Since Cassandra uses mlockall to lock the heap in RAM what is the reason for disabling swap? My guess is that is has to do with memory mapped files but as of my understanding, accessing pages of memory mapped files, those pages are never put in swap since they're backed by files on disk and the OS writes those pages to the memory mapped file instead of swap. We've seen on Cassandra installations on Linux with swap enabled that parts of the java process is swaped out and increasing. So what's swaped out? Regards /Fredrik -- Fredrik Larsson Stigbäck SiteVision AB Vasagatan 10, 107 10 Örebro 019-17 30 30
Denormalization
Hi. Since denormalized data is first-class citizen in Cassandra, how to handle updating denormalized data. E.g. If we have a USER cf with name, email etc. and denormalize user data into many other CF:s and then update the information about a user (name, email...). What is the best way to handle updating those user data properties which might be spread out over many cf:s and many rows? Regards /Fredrik
Re: Denormalization
I don't have a current use-case. I was just curious how applications handle and how to think when modelling, since I guess denormalization might increase the complexity of the application. Fredrik 2013/1/27 Hiller, Dean dean.hil...@nrel.gov: There is a really a mix of denormalization and normalization. It really depends on specific use-cases. To get better help on the email list, a more specific use case may be appropriate. Dean On 1/27/13 2:03 PM, Fredrik Stigbäck fredrik.l.stigb...@sitevision.se wrote: Hi. Since denormalized data is first-class citizen in Cassandra, how to handle updating denormalized data. E.g. If we have a USER cf with name, email etc. and denormalize user data into many other CF:s and then update the information about a user (name, email...). What is the best way to handle updating those user data properties which might be spread out over many cf:s and many rows? Regards /Fredrik -- Fredrik Larsson Stigbäck SiteVision AB Vasagatan 10, 107 10 Örebro 019-17 30 30
Question regarding hinted handoffs and restoring backup in cluster
When restoring a backup for the entire cluster my understanding is that you must shutdown the entire cluster and then restore the backup and then start up all nodes again. http://www.datastax.com/docs/1.0/operations/backup_restore But how should I handle hinted handoffs (Hints CF). Since they're stored in the system keyspace and according to the docs I only need to restore the specific keyspace not the system keyspace. Won't these hinted handoffs, which isn't based on the backup, be delivered and applied as soon as one of the node which they're aimed for comes up and thus be applied to the backuped data. What is the recomended way to handle this situation? Removing the hints cf from the system tables before restart of the cluster nodes? Regards /Fredrik
Re: Remove node from cluster and have it run as a single node cluster by itself
I guess that the other nodes still gossips about the removed node. The node isn't removed from gossiper in the cluster until some amount of time have elapsed. My guess is that you haven't changed the cluster_name property in the cassandra.yaml on the removed node. Xu, Zaili skrev 2012-09-28 20:53: I have an existing Cassandra Cluster. I removed a node from the cluster. Then I decommissioned the removed node, stopped it, updated its config so that it only has itself as the seed and in the cassandra-topology.properties file, even deleted the data, commitlog, and saved_caches. But as soon as I start it backup it is able to join back to the cluster. How does this node know the information of the existing cluster and was able to join it ?
Re: Removed node, jumps back into the cluster
Wrong assumption of me. I found the answer in GossipDigestSynVerbHandler. I forgot to change the cluster name of the new cluster. /Fredrik 2012/9/11 Fredrik fredrik.l.stigb...@sitevision.se: I've tested a scenario where I wanted to reuse a removed node in a new cluster with same IP, maybe not very common but anyway, found some strange behaviour in Gossiper. Here is what I think/see happening: - Cassandra 1.1. Three node cluster A, B and C. - Shutdown node C and remove token for node C. - Everything looks ok in logs, reporting that node C is removed etc.. - Node A and B still sends Gossip digest about the removed node, but I guess that's ok since they know about it (Gossiper.endpointStateMap). - Node C has status removed when checking in JMX console. - Checked in LocationInfo that Ring only contains token/IP for node A and B. - Removed system/data tables for C. - Changed seed on C to point to itself. - Startup node C, node C only gossips itself and node A and B doesn't recognize that node C is running, which is correct. - Restart e.g. node A. Now node A will loose all gossip information (Gossiper.endpointStateMap) about node C. Node A will request information from LocationInfo and ask node B about endpoint states. Node A will receive information from node B about node C, this will trigger Gossiper.handleMajorStateChange and node C will be first marked as unreachable because it's in dead state (removed), node A will try to Gossip (unreachable endpoints) to node C, which will reply that it's up and node C becomes incorporated into the old cluster again. Is this a a bug or is it a requirement that if you take a node out of the cluster you must change IP on the removed node if you want to use it in another cluster? Please enlight me. Regards /Fredrik -- Fredrik Larsson Stigbäck SiteVision AB Vasagatan 10, 107 10 Örebro 019-17 30 30
Removed node, jumps back into the cluster
I've tested a scenario where I wanted to reuse a removed node in a new cluster with same IP, maybe not very common but anyway, found some strange behaviour in Gossiper. Here is what I think/see happening: - Cassandra 1.1. Three node cluster A, B and C. - Shutdown node C and remove token for node C. - Everything looks ok in logs, reporting that node C is removed etc.. - Node A and B still sends Gossip digest about the removed node, but I guess that's ok since they know about it (Gossiper.endpointStateMap). - Node C has status removed when checking in JMX console. - Checked in LocationInfo that Ring only contains token/IP for node A and B. - Removed system/data tables for C. - Changed seed on C to point to itself. - Startup node C, node C only gossips itself and node A and B doesn't recognize that node C is running, which is correct. - Restart e.g. node A. Now node A will loose all gossip information (Gossiper.endpointStateMap) about node C. Node A will request information from LocationInfo and ask node B about endpoint states. Node A will receive information from node B about node C, this will trigger Gossiper.handleMajorStateChange and node C will be first marked as unreachable because it's in dead state (removed), node A will try to Gossip (unreachable endpoints) to node C, which will reply that it's up and node C becomes incorporated into the old cluster again. Is this a a bug or is it a requirement that if you take a node out of the cluster you must change IP on the removed node if you want to use it in another cluster? Please enlight me. Regards /Fredrik
Question regarding tombstone removal and compaction
We've had a bug that caused one of our column families to grow very big 280 GB on a 500 GB disk. We're using size tiered compaction. Since it's only append data I've now issued deletes of 260 GB of superflous data. 1. There are som quite large SSTables (80 GB, 40 GB etc..). If I run a major compaction before GC grace, which is 6 hours, will the compaction succeed or will it fail due to the GC grace hasn't elapsed and thus major compaction will ignore the tombstones and then fail due to insufficient disk space? 2. If I wait until GC grace has elapsed, will it be possible to run a major compaction since there are only deletes which doesn't require double amount of SStable size when merging tombstones with the large SSTables? Regards /Fredrik
JNA on Windows
Hello. I have a question regarding JNA and Windows. I read about the problem that when taking snapshots might require the process space x 2 due to how hardlinks are created. Is JNA for Windows supported? Looking at jira issue https://issues.apache.org/jira/browse/CASSANDRA-1371 looks like it but checking in the Cassandra code base org.apache.cassandra.utils.CLibrary the only thing I see, is Native.register(c) which tries to load the c-library but I think doesn't exists on Windows which will result in creating links with cmd or fsutil and which might then triggger these extensive memory requirements. I'd be happy if someone could shed some light on this issue. Regards /Fredrik
Running Cassandra with IPv6
Anyone know if there are any specific outstanding issues running Cassandra on a ipv6 network (defining cassandra.yaml with ipv6 addresses, communication in cluster between nodes defined with ipv4 and nodes defined with ipv6 addresses, etc.)? The only thing I'm aware about so far is using ipv6 addresses with NIO on Windows with Java jdk7, (https://issues.apache.org/jira/browse/CASSANDRA-4359) which isn't a Cassandra issue anyway. Regards /Fredrik
Re: Question regarding major compaction.
Thank you Aaron. That explanation cleared things up. 2012/4/30 aaron morton aa...@thelastpickle.com: Depends on your definition of significantly, there are a few things to consider. * Reading from SSTables for a request is a serial operation. Reading from 2 SSTables will take twice as long as 1. * If the data in the One Big File™ has been overwritten, reading it is a waste of time. And it will continue to be read until it the row is compacted away. * You will need to get min_compaction_threshold (CF setting) SSTables that big before automatic compaction will pickup the big file. On the other side: Some people do report getting value from nightly major compactions. They also manage their cluster to reduce the impact of performing the compactions. Hope that helps. - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 26/04/2012, at 9:37 PM, Fredrik wrote: Exactly, but why would reads be significantly slower over time when including just one more, although sometimes large, SSTable in the read? Ji Cheng skrev 2012-04-26 11:11: I'm also quite interested in this question. Here's my understanding on this problem. 1. If your workload is append-only, doing a major compaction shouldn't affect the read performance too much, because each row appears in one sstable anyway. 2. If your workload is mostly updating existing rows, then more and more columns will be obsoleted in that big sstable created by major compaction. And that super big sstable won't be compacted until you either have another 3 similar-sized sstables or start another major compaction. But I am not very sure whether this will be a major problem, because you only end up with reading one more sstable. Using size-tiered compaction against mostly-update workload itself may result in reading multiple sstables for a single row key. Please correct me if I am wrong. Cheng On Thu, Apr 26, 2012 at 3:50 PM, Fredrik fredrik.l.stigb...@sitevision.se wrote: In the tuning documentation regarding Cassandra, it's recomended not to run major compactions. I understand what a major compaction is all about but I'd like an in depth explanation as to why reads will continually degrade until the next major compaction is manually invoked. From the doc: So while read performance will be good immediately following a major compaction, it will continually degrade until the next major compaction is manually invoked. For this reason, major compaction is NOT recommended by DataStax. Regards /Fredrik -- Fredrik Larsson Stigbäck SiteVision AB Vasagatan 10, 107 10 Örebro 019-17 30 30
Question regarding major compaction.
In the tuning documentation regarding Cassandra, it's recomended not to run major compactions. I understand what a major compaction is all about but I'd like an in depth explanation as to why reads will continually degrade until the next major compaction is manually invoked. From the doc: So while read performance will be good immediately following a major compaction, it will continually degrade until the next major compaction is manually invoked. For this reason, major compaction is NOT recommended by DataStax. Regards /Fredrik
Re: Question regarding major compaction.
Exactly, but why would reads be significantly slower over time when including just one more, although sometimes large, SSTable in the read? Ji Cheng skrev 2012-04-26 11:11: I'm also quite interested in this question. Here's my understanding on this problem. 1. If your workload is append-only, doing a major compaction shouldn't affect the read performance too much, because each row appears in one sstable anyway. 2. If your workload is mostly updating existing rows, then more and more columns will be obsoleted in that big sstable created by major compaction. And that super big sstable won't be compacted until you either have another 3 similar-sized sstables or start another major compaction. But I am not very sure whether this will be a major problem, because you only end up with reading one more sstable. Using size-tiered compaction against mostly-update workload itself may result in reading multiple sstables for a single row key. Please correct me if I am wrong. Cheng On Thu, Apr 26, 2012 at 3:50 PM, Fredrik fredrik.l.stigb...@sitevision.se mailto:fredrik.l.stigb...@sitevision.se wrote: In the tuning documentation regarding Cassandra, it's recomended not to run major compactions. I understand what a major compaction is all about but I'd like an in depth explanation as to why reads will continually degrade until the next major compaction is manually invoked. From the doc: So while read performance will be good immediately following a major compaction, it will continually degrade until the next major compaction is manually invoked. For this reason, major compaction is NOT recommended by DataStax. Regards /Fredrik
Hinted handoff bug?
Hi,
We,re running cassandra 1.0.3.
I've done some testing with 2 nodes (node A, node B), replication factor 2.
I take node A down, writing some data to node B and then take node A up.
Sometimes hints aren't delivered when node A comes up.
I've done some debugging in org.apache.cassandra.db.HintedHandOffManager
and sometimes node B ends up in a strange state in method
org.apache.cassandra.db.HintedHandOffManager.deliverHints(final
InetAddress to), where
org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries already
has node A in it's Set and therefore no hints will ever be delivered to
node A.
The only reason for this that I can see is that in
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(InetAddress
endpoint) the hintStore.isEmpty() check returns true and the endpoint
(node A) isn't removed from
org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries. Then no
hints will ever be delivered again until node B is restarted.
During what conditions will hintStore.isEmpty() return true?
Shouldn't the hintStore.isEmpty() check be inside the try {} finally{}
clause, removing the endpoint from queuedDeliveries in the finally block?
public void deliverHints(final InetAddress to)
{
logger_.debug(deliverHints to {}, to);
*if (!queuedDeliveries.add(to))*
return;
...
}
private void deliverHintsToEndpoint(InetAddress endpoint) throws
IOException, DigestMismatchException, InvalidRequestException,
TimeoutException,
{
ColumnFamilyStore hintStore =
Table.open(Table.SYSTEM_TABLE).getColumnFamilyStore(HINTS_CF);
* if (hintStore.isEmpty())*
return; // nothing to do, don't confuse users by logging a
no-op handoff
try
{
..
}
finally
{
*queuedDeliveries.remove(endpoint);*
}
}
Regards
/Fredrik
Re: Hinted handoff bug?
Yes, I'll do that.
/Fredrik
Sylvain Lebresne skrev 2011-12-01 11:10:
You're right, good catch.
Do you mind opening a ticket on jira
(https://issues.apache.org/jira/browse/CASSANDRA)?
--
Sylvain
On Thu, Dec 1, 2011 at 10:03 AM, Fredrik L Stigbäck
fredrik.l.stigb...@sitevision.se wrote:
Hi,
We,re running cassandra 1.0.3.
I've done some testing with 2 nodes (node A, node B), replication factor 2.
I take node A down, writing some data to node B and then take node A up.
Sometimes hints aren't delivered when node A comes up.
I've done some debugging in org.apache.cassandra.db.HintedHandOffManager and
sometimes node B ends up in a strange state in method
org.apache.cassandra.db.HintedHandOffManager.deliverHints(final InetAddress
to), where org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries
already has node A in it's Set and therefore no hints will ever be delivered
to node A.
The only reason for this that I can see is that in
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(InetAddress
endpoint) the hintStore.isEmpty() check returns true and the endpoint (node
A) isn't removed from
org.apache.cassandra.db.HintedHandOffManager.queuedDeliveries. Then no hints
will ever be delivered again until node B is restarted.
During what conditions will hintStore.isEmpty() return true?
Shouldn't the hintStore.isEmpty() check be inside the try {} finally{}
clause, removing the endpoint from queuedDeliveries in the finally block?
public void deliverHints(final InetAddress to)
{
logger_.debug(deliverHints to {}, to);
if (!queuedDeliveries.add(to))
return;
...
}
private void deliverHintsToEndpoint(InetAddress endpoint) throws
IOException, DigestMismatchException, InvalidRequestException,
TimeoutException,
{
ColumnFamilyStore hintStore =
Table.open(Table.SYSTEM_TABLE).getColumnFamilyStore(HINTS_CF);
if (hintStore.isEmpty())
return; // nothing to do, don't confuse users by logging a no-op
handoff
try
{
..
}
finally
{
queuedDeliveries.remove(endpoint);
}
}
Regards
/Fredrik
Clear snapshot on windows
Clearing snapshots on Windows with nodetool sometimes fails. The reason is because the snapshot files hardlinks into corresponding files in the data directory. When compaction runs, most of the files are merged and the corresponding snapshot files can be deleted but often .e.g. the secondary index files isn't compacted. If one of those undeletable files is the first in the loop to be deleted then the whole clear snapshot fails, and deletable files in the snapshot isn't deleted. I guess all files in data directory is eventually compacted and will then be eligible for deletion but would this be problem if we have seldom updated column family that wont be compacted very often? Is this considered a bug or is it something that won't be fixed due to the way Windows handles hard links? Regards /Fredrik
Commit log grows and grows
I have a problem or at least something that I have missed to understand regarding commit log removal. I'm running cassandra 1.0.1 but have seen the same thing in 1.0. I thought that when a node is flushed or drained, it's memtable is flushed to disk and the commit log is removed and new commit logs are created as write requests comes in. I've setup a small test, adding column family data through cassandra-cli then running both flush and drain but the commit log isn't removed. There is only one commit log in the commit log directory. I've turned on debug and the log always reports Not safe to delete...: DEBUG [COMMIT-LOG-WRITER] 2011-11-01 20:51:19,195 CommitLog.java (line 458) discard completed log segments for ReplayPosition(segmentId=1320176934665, position=340), column family 7. DEBUG [COMMIT-LOG-WRITER] 2011-11-01 20:51:19,196 CommitLog.java (line 497) Not safe to delete commit log CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1320176934665.log); dirty is ; hasNext: false Could someone please clarify the relationship between commit log creation, removal/rotation and flush, drain. Regards /Fredrik
replication factor no nodes
What is the implication of having a replication factor greater that the number of nodes in cluster? We're changing replication factor at runtime and sometimes when a node is removed/decomissioned from the cluster, replication factor will be greater than number of nodes since we have replication factor == no nodes in cluster. Regards /Fredrik
Re: NullPointerException doing cleanup
Thanks. Will upgrade to 0.8.5. Regards /Fredrik 2011/9/13 Sylvain Lebresne sylv...@datastax.com This should be fixed in 0.8.5 (more precisely by https://issues.apache.org/jira/browse/CASSANDRA-3039) -- Sylvain On Tue, Sep 13, 2011 at 12:53 PM, Fredrik Stigbäck fredrik.l.stigb...@sitevision.se wrote: Trying to do a nodetool cleanup but gets a NullPointerException. I've done some debugging and found out that the property org.apache.cassandra.db.compaction.PrecompactedRow.compactedCf is initilaized to null since ColumnFamilyStore.removeDeleted(ColumnFamily, int) returns null which I interpret as the row is marked for delete and has no columns. This will cause org.apache.cassandra.db.ColumnIndexer.serializeInternal to get a IIterableColumns == null. Any ideas? Corrupted SStable or a bug or something else? Running Cassandra 0.8.4 on a Windows 7 but have seen the same behaviour on Linux. 09:09:50 ERROR [AbstractCassandraDaemon] [] Fatal exception in thread Thread[CompactionExecutor:8,1,main] java.lang.NullPointerException at org.apache.cassandra.db.ColumnIndexer.serializeInternal(ColumnIndexer.java:60) at org.apache.cassandra.db.ColumnIndexer.serialize(ColumnIndexer.java:50) at org.apache.cassandra.db.compaction.PrecompactedRow.write(PrecompactedRow.java:110) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:132) at org.apache.cassandra.db.compaction.CompactionManager.doCleanupCompaction(CompactionManager.java:866) at org.apache.cassandra.db.compaction.CompactionManager.access$500(CompactionManager.java:65) at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:204) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) /Regards Fredrik -- Fredrik Larsson Stigbäck SiteVision AB Vasagatan 10, 107 10 Örebro 019-17 30 30
Reading quorum
Does reading quorum mean only waiting for quorum respones or does it mean quorum respones with same latest timestamp? Regards /Fredrik
