Converting from Apache Cassandra 2.2.6 to 3.x

[William Boutin]

We are converting our product from Apache Cassandra 2.2.6 to 3.x. What issues 
may we run into when we convert?


[Ericsson]

WILLIAM L. BOUTIN
Engineer IV - Sftwr
BMDA PADB DSE DU CC NGEE

Ericsson
1 Ericsson Drive, US PI06 1.S747
Piscataway, NJ, 08854, USA
Phone (913) 241-5574
Mobile (732) 213-1368
Emergency (732) 354-1263
william.bou...@ericsson.com
www.ericsson.com
[http://www.ericsson.com/current_campaign]
Legal entity: EUS - ERICSSON INC., registered office in US PI01 4A242. This 
Communication is Confidential. We only send and receive email on the basis of 
the terms set out at 
www.ericsson.com/email_disclaimer

RE: Cassandra 2.2.6 Fails to Boot Up correctly - JNA Class

[William Boutin]

Thank you for your help.
We have been using jna-4.0.0.jar since using Cassandra 2.2(.6). Until last 
week, we had no issues. Now, we are experiencing the exception that I 
identified. We only have jna-4.0.0.jar loaded on our machines and the CLASSPATH 
that Cassandra builds only uses the jar from the /usr/share/cassandra/lib 
directory. Jna-3.2.4.jar was only introduced when I was debugging the original 
issue and will never be loaded on our machines. See the output of ps below.

Should I be looking elsewhere? Some have brought up that low memory on a JVM or 
LINUX machine could cause jar loading issues. I have tried doubling -Xss256k to 
-Xss512k.

Thanks

Billy S. Boutin 
Office Phone No. (913) 241-5574 
Cell Phone No. (732) 213-1368 

496   3318 1  0 Jul19 ?00:06:20 /usr/java/latest/bin/java 
-Dorg.xerial.snappy.tempdir=/var/tmp -Djava.io.tmpdir=/var/tmp -ea 
-javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar 
-XX:+CMSClassUnloadingEnabled -XX:+UseThreadPriorities 
-XX:ThreadPriorityPolicy=42 -Xms1968M -Xmx1968M -Xmn200M 
-XX:+HeapDumpOnOutOfMemoryError -Xss256k -XX:StringTableSize=103 
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled 
-XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly 
-XX:+UseTLAB -XX:+PerfDisableSharedMem 
-XX:CompileCommandFile=/etc/cassandra/conf/hotspot_compiler 
-XX:CMSWaitDuration=1 -XX:+CMSParallelInitialMarkEnabled 
-XX:+CMSEdenChunksRecordAlways -XX:CMSWaitDuration=1 -XX:+UseCondCardMark 
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC 
-XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime 
-XX:+PrintPromotionFailure -Xloggc:/usr/share/cassandra/logs/gc.log 
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M 
-Djava.net.preferIPv4Stack=true -Dcassandra.jmx.local.port=7199 
-XX:+DisableExplicitGC -Djava.library.path=/usr/share/cassandra/lib/sigar-bin 
-Dlogback.configurationFile=logback.xml 
-Dcassandra.logdir=/usr/share/cassandra/logs -Dcassandra.storagedir= 
-Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp 
/etc/cassandra/conf:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.2.6-E002.jar:/usr/share/cassandra/lib/cassandra-driver-core-2.2.0-rc2-SNAPSHOT-20150617-shaded.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share
/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/crc32ex-0.1.1.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/joda-time-2.4.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/logback-classic-1.1.3.jar:/usr/share/cassandra/lib/logback-core-1.1.3.jar:/usr/share/cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.jar:/usr/share/cassandra/lib/metrics-logback-3.1.0.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/ohc-core-0.3.4.jar:/usr/share/cassandra/lib/ohc-core-j8-0.3.4.jar:/usr/share/cassandra/lib/reporter-config3-3.0.0.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.0.jar:/usr/share/cassandra/lib/sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.7.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.7.jar:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/super-c
sv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.2.6-E002.jar:/usr/share/cassandra/apache-cassandra-thrift-2.2.6-E002.jar:/usr/share/cassandra/stress.jar:
 org.apache.cassandra.service.CassandraDaemon

-Original Message-
From: Jeff Jirsa [mailto:jji...@apache.org] 
Sent: Wednesday, July 19, 2017 11:47 PM
To: user@cassandra.apache.org
Subject: Re: Cassandra 2.2.6 Fails to Boot Up correctly - JNA Class



On 2017-07-19 10:41 (-0700), William Boutin <william.bou...@ericsson.com> 
wrote: 
> We are running apache-cassandra-2.2.6 for months with no JNA startup issues.
> Recently, we have updated some of our cassandra machines and we ran into a 
> Cassandra startup issue with JNA. See the stack trace 1 below

Cassandra 2.2.6 Fails to Boot Up correctly - JNA Class

[William Boutin]

We are running apache-cassandra-2.2.6 for months with no JNA startup issues.
Recently, we have updated some of our cassandra machines and we ran into a 
Cassandra startup issue with JNA. See the stack trace 1 below.

Question 1: Our Cassandra package contains jna-4.0.0.jar in the 
/usr/share/cassandra/lib/ directory. Does apache-cassandra-2.2.6 specifically 
package A-C* with the jna-4.0.0.jar ?

I tried to circumvent the exception by yum installing  jna-3.2.4.jar for RHEL 
6.4 and adding it to the /usr/share/cassandra/lib/ directory. This got me past 
the JNA mlockall failure with "INFO  16:56:47 JNA mlockall successful" but when 
the MemtableFlushWriter requested memory allocation, I now get the STACK TRACE 
2 error. "nodetool status" still reports the node as incomplete.

Any Suggestions?

Thank you

STACK TRACE 1:
ERROR [MemtableFlushWriter:1] 2017-07-17 22:33:13,376 CassandraDaemon.java:185 
- Exception in thread Thread[MemtableFlushWriter:1,5,main]
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
at 
org.apache.cassandra.utils.memory.MemoryUtil.allocate(MemoryUtil.java:82) 
~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.util.Memory.(Memory.java:74) 
~[apache-cassandra-clientutil-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.util.SafeMemory.(SafeMemory.java:32) 
~[apache-cassandra-clientutil-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.io.compress.CompressionMetadata$Writer.(CompressionMetadata.java:304)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.io.compress.CompressionMetadata$Writer.open(CompressionMetadata.java:318)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.io.compress.CompressedSequentialWriter.(CompressedSequentialWriter.java:73)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:168) 
~[apache-cassandra-clientutil-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.io.sstable.format.big.BigTableWriter.(BigTableWriter.java:75)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:107)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:84)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memtable.java:433)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:376)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:361) 
~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) 
~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
 ~[guava-16.0.jar:na]
at 
org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1146)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
~[na:1.7.0_141]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
~[na:1.7.0_141]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_141]

STACK TRACE 2:
ERROR 16:56:51 Exception in thread Thread[MemtableFlushWriter:1,5,main]
java.lang.NoSuchMethodError: com.sun.jna.Native.malloc(J)J
at 
org.apache.cassandra.utils.memory.MemoryUtil.allocate(MemoryUtil.java:82) 
~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.util.Memory.(Memory.java:74) 
~[apache-cassandra-clientutil-2.2.6-E002.jar:2.2.6-E002]
at org.apache.cassandra.io.util.SafeMemory.(SafeMemory.java:32) 
~[apache-cassandra-clientutil-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.io.compress.CompressionMetadata$Writer.(CompressionMetadata.java:304)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.io.compress.CompressionMetadata$Writer.open(CompressionMetadata.java:318)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.io.compress.CompressedSequentialWriter.(CompressedSequentialWriter.java:73)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:168) 
~[apache-cassandra-clientutil-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.io.sstable.format.big.BigTableWriter.(BigTableWriter.java:75)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:107)
 

Strange Cassandra startup error

[William Boutin]

I've been running apache-cassandra 2.2.6 for a few years. Recently when I 
configured a new cassandra node and started Cassandra via "service start 
cassandra" on LINUX 6.5, I got the following exception. After cleaning out my 
data_file_directories I was able to successfully restart. Any ideas? Thanks
ERROR [main] 2017-05-10 02:12:09,935 CassandraDaemon.java:638 - Exception 
encountered during startup
org.apache.cassandra.exceptions.ConfigurationException: Unable to check disk 
space available to /usr/share/cassandra/data/commitlog. Perhaps the Cassandra 
user does not have the necessary permissions
at 
org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:522)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:119)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:491) 
[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:625) 
[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
Caused by: java.io.IOException: Mount point not found
at sun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:91) 
~[na:1.7.0_141]
at sun.nio.fs.UnixFileStore.(UnixFileStore.java:65) 
~[na:1.7.0_141]
at sun.nio.fs.LinuxFileStore.(LinuxFileStore.java:44) 
~[na:1.7.0_141]
at 
sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:49)
 ~[na:1.7.0_141]
at 
sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:37)
 ~[na:1.7.0_141]
at 
sun.nio.fs.UnixFileSystemProvider.getFileStore(UnixFileSystemProvider.java:368) 
~[na:1.7.0_141]
at java.nio.file.Files.getFileStore(Files.java:1413) ~[na:1.7.0_141]
at 
org.apache.cassandra.config.DatabaseDescriptor.guessFileStore(DatabaseDescriptor.java:685)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
at 
org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:517)
 ~[apache-cassandra-2.2.6-E002.jar:2.2.6-E002]
... 3 common frames omitted

[Ericsson]

WILLIAM L. BOUTIN
Engineer IV - Sftwr
BMDA PADB DSE DU CC NGEE

Ericsson
1 Ericsson Drive, US PI06 1.S747
Piscataway, NJ, 08854, USA
Phone (913) 241-5574
Mobile (732) 213-1368
Emergency (732) 354-1263
william.bou...@ericsson.com
www.ericsson.com
[http://www.ericsson.com/current_campaign]
Legal entity: EUS - ERICSSON INC., registered office in US PI01 4A242. This 
Communication is Confidential. We only send and receive email on the basis of 
the terms set out at 
www.ericsson.com/email_disclaimer

Cassandra and LINUX CPU Context Switches

[William Boutin]

I've noticed that my apache-cassandra 2.2.6 process is consistently performing 
CPU Context Switches above 10,000 per second.
Is this to be expected or should I be looking into ways to lower the number of 
context switches done on my Cassandra cluster?
Thanks in advance.

[Ericsson]

WILLIAM L. BOUTIN
Engineer IV - Sftwr
BMDA PADB DSE DU CC NGEE

Ericsson
1 Ericsson Drive, US PI06 1.S747
Piscataway, NJ, 08854, USA
Phone (913) 241-5574
Mobile (732) 213-1368
Emergency (732) 354-1263
william.bou...@ericsson.com
www.ericsson.com
[http://www.ericsson.com/current_campaign]
Legal entity: EUS - ERICSSON INC., registered office in US PI01 4A242. This 
Communication is Confidential. We only send and receive email on the basis of 
the terms set out at 
www.ericsson.com/email_disclaimer

Making a Cassandra node cluster unique

[William Boutin]

Someone on my team asked me a question that I could not find an easy answer and 
I was hoping someone could answer for me.
When we configure Cassandra, we use the Cluster Name, Data Center, and Rack to 
define the group of Cassandra nodes involved in holding our keyspace records. 
If a second set of nodes had the same Cluster Name, Data Center, and Rack 
values, is there a chance that CRUD actions directed at the first cluster of 
nodes could somehow end up at the second cluster of nodes?
Thank you in advance.


[Ericsson]

WILLIAM L. BOUTIN
Engineer IV - Sftwr
BMDA PADB DSE DU CC NGEE

Ericsson
1 Ericsson Drive, US PI06 1.S747
Piscataway, NJ, 08854, USA
Phone (913) 241-5574
Mobile (732) 213-1368
Emergency (732) 354-1263
william.bou...@ericsson.com
www.ericsson.com
[http://www.ericsson.com/current_campaign]
Legal entity: EUS - ERICSSON INC., registered office in US PI01 4A242. This 
Communication is Confidential. We only send and receive email on the basis of 
the terms set out at 
www.ericsson.com/email_disclaimer