Hi Ellie,
Thanks a lot for the response!
I modified Sprout and Chronos configurations. They are working correctly
now!
I checked Cassandra on Homestead node. The log does show a few errors
during initialization. But it started successfully finally. The Cassandra,
Homestead and Homestead-prov logs are attached.
Actually, I did run into the same problem before. But after rebooting
Homestead node a few times, it works fine. So, I didn't dig into it.
Is it possible that the problem is due to some starting conflicts among
Cassandra, Homestead and Homestead-prov?
Thanks,
Lianjie
On Wed, Feb 4, 2015 at 3:25 PM, Eleanor Merry <[email protected]>
wrote:
> Hi Lianjie,
>
> Your configuration files aren't quite right.
>
> The cluster_settings file should have the form servers=<address>,<address>
> - so in your case it would be "servers=192.168.1.21:11211,
> 192.168.1.22:11211". This file should be identical on each Sprout node
> (so the sprouts must be in same order on each node).
>
> The chronos.conf file should have one localhost entry, which is set to the
> IP address of the local node, and multiple node entries, which are set to
> the IP addresses of each node in the cluster. In your case, this would be
> (on sprout 1):
>
> [cluster]
> localhost = 192.168.1.21
> node = 192.168.1.21
> node = 192.168.1.22
>
> The order of the nodes must be the same on each node - so the file on
> sprout 2 should be:
>
> [cluster]
> localhost = 192.168.1.22
> node = 192.168.1.21
> node = 192.168.1.22
>
> Can you make these changes to the config files, and then reload Sprout and
> Chronos (sudo service <service> reload)?
>
> In the logs below, Homestead has stopped because it couldn't contact
> cassandra:
>
> 04-02-2015 18:42:19.616 UTC Error cassandra_store.cpp:207: Cache caught
> TTransportException: connect() failed: Connection refused
> 04-02-2015 18:42:19.616 UTC Error main.cpp:550: Failed to initialize cache
> - rc 3
> 04-02-2015 18:42:19.616 UTC Status cassandra_store.cpp:185: Stopping cache
>
> Can you check whether Cassandra is running reliably on the Homestead node?
> Does /var/monit/monit.log show that monit is restarting it, and are there
> any logs in /var/log/cassandra?
>
> Ellie
>
> -----Original Message-----
> From: [email protected] [mailto:
> [email protected]] On Behalf Of Lianjie Cao
> Sent: 04 February 2015 19:37
> To: [email protected]
> Subject: [Clearwater] Problems with Sprout clustering and Homestead failure
>
> Hi,
>
> We recently built a Clearwater deployment with one Bono node, two Sprout
> nodes, one Homestead node, one Homer node and one Ralf node. Howerver, we
> ran into some problems related to Homestead start failure and Sprout
> clustering.
>
> *Sprout clustering:*
> The manual installation instruction shows for the latest version Sprout
> clustering is done by Chronos. To add or remove a Sprout node,
> /etc/chronos/chronos.conf needs to modified correspondingly.
> However, we found that when we don't have chronos.conf file, the two
> Sprout nodes seems working fine by adding IPs of the two Sprout nodes to
> /etc/clearwater/cluster_settings.
>
> [sprout]cw@sprout-2:~$ cat /etc/clearwater/cluster_settings
> servers=192.168.1.21:11211
> servers=192.168.1.22:11211
>
> But, if we do add /etc/chronos/chronos.conf with the information of two
> Sprout nodes as below, Chronos failed and no new log files found under
> /var/log/chronos.
>
> [sprout]cw@sprout-1:/var/log/chronos$ cat /etc/chronos/chronos.conf
> [http] bind-address = 0.0.0.0 bind-port = 7253
>
> [logging]
> folder = /var/log/chronos
> level = 5
>
> [cluster]
> localhost = 192.168.1.21
> node = localhost
>
> sprout-2 = 192.168.1.22
> node = sprout-2
>
> [alarms]
> enabled = true
>
>
> [sprout]cw@sprout-1:~$ sudo monit status The Monit daemon 5.8.1 uptime: 0m
>
> Program 'poll_sprout'
> status Status ok
> monitoring status Monitored
> last started Wed, 04 Feb 2015 11:20:36
> last exit value 0
> data collected Wed, 04 Feb 2015 11:20:36
>
> Process 'sprout'
> status Running
> monitoring status Monitored
> pid 1157
> parent pid 1
> uid 999
> effective uid 999
> gid 999
> uptime 1m
> children 0
> memory kilobytes 42412
> memory kilobytes total 42412
> memory percent 1.0%
> memory percent total 1.0%
> cpu percent 0.4%
> cpu percent total 0.4%
> data collected Wed, 04 Feb 2015 11:20:36
>
> Program 'poll_memcached'
> status Status ok
> monitoring status Monitored
> last started Wed, 04 Feb 2015 11:20:36
> last exit value 0
> data collected Wed, 04 Feb 2015 11:20:36
>
> Process 'memcached'
> status Running
> monitoring status Monitored
> pid 1092
> parent pid 1
> uid 108
> effective uid 108
> gid 114
> uptime 1m
> children 0
> memory kilobytes 1180
> memory kilobytes total 1180
> memory percent 0.0%
> memory percent total 0.0%
> cpu percent 0.0%
> cpu percent total 0.0%
> data collected Wed, 04 Feb 2015 11:20:36
>
> Process 'clearwater_diags_monitor'
> status Running
> monitoring status Monitored
> pid 1072
> parent pid 1
> uid 0
> effective uid 0
> gid 0
> uptime 1m
> children 1
> memory kilobytes 1796
> memory kilobytes total 2172
> memory percent 0.0%
> memory percent total 0.0%
> cpu percent 0.0%
> cpu percent total 0.0%
> data collected Wed, 04 Feb 2015 11:20:36
>
> Process 'chronos'
> status Execution failed
> monitoring status Monitored
> data collected Wed, 04 Feb 2015 11:20:26
>
> System 'sprout-1'
> status Running
> monitoring status Monitored
> load average [0.20] [0.09] [0.04]
> cpu 6.8%us 1.1%sy 0.0%wa
> memory usage 116944 kB [2.8%]
> swap usage 0 kB [0.0%]
> data collected Wed, 04 Feb 2015 11:20:26
>
>
> Is it because we are not using Chronos in the right way or there are other
> settings we need to do?
>
>
> *Homestead Failure:*
>
> When we use SIPp to perform user registration tests, we receive “403
> Forbidden" response and we observed error on both sprout nodes.
>
> [sprout]cw@sprout-1:~$ cat /var/log/sprout/sprout_current.txt
> 04-02-2015 18:54:50.884 UTC Warning acr.cpp:627: Failed to send Ralf ACR
> message (0x7fce241cd780), rc = 400
> 04-02-2015 18:54:51.083 UTC Error httpconnection.cpp:573:
>
> http://hs.hp-clearwater.com:8888/impi/6500000008%40hp-clearwater.com/av?impu=sip%3A6500000008%40hp-clearwater.com
> failed at server 192.168.1.31 : Timeout was reached (28) : fatal
> 04-02-2015 18:54:51.083 UTC Error httpconnection.cpp:688: cURL failure
> with cURL error code 28 (see man 3 libcurl-errors) and HTTP error code 500
> 04-02-2015 18:54:51.083 UTC Error hssconnection.cpp:145: Failed to get
> Authentication Vector for [email protected]
> 04-02-2015 18:54:51.086 UTC Error httpconnection.cpp:688: cURL failure
> with cURL error code 0 (see man 3 libcurl-errors) and HTTP error code 400
> 04-02-2015 18:54:51.086 UTC Warning acr.cpp:627: Failed to send Ralf ACR
> message (0x14322c0), rc = 400
> 04-02-2015 18:54:51.282 UTC Error httpconnection.cpp:573:
>
> http://hs.hp-clearwater.com:8888/impi/6500000009%40hp-clearwater.com/av?impu=sip%3A6500000009%40hp-clearwater.com
> failed at server 192.168.1.31 : Timeout was reached (28) : fatal
> 04-02-2015 18:54:51.283 UTC Error httpconnection.cpp:688: cURL failure
> with cURL error code 28 (see man 3 libcurl-errors) and HTTP error code 500
> 04-02-2015 18:54:51.283 UTC Error hssconnection.cpp:145: Failed to get
> Authentication Vector for [email protected]
> 04-02-2015 18:54:51.286 UTC Error httpconnection.cpp:688: cURL failure
> with cURL error code 0 (see man 3 libcurl-errors) and HTTP error code 400
> 04-02-2015 18:54:51.286 UTC Warning acr.cpp:627: Failed to send Ralf ACR
> message (0x7fce1c1fdef0), rc = 400 ....
>
>
> It seems like Homestead is unreachable.
> Then on Homestead node, if we check status using monit:
>
> [homestead]cw@homestead-1:~$ sudo monit status The Monit daemon 5.8.1
> uptime: 15m
>
> Process 'nginx'
> status Running
> monitoring status Monitored
> pid 1044
> parent pid 1
> uid 0
> effective uid 0
> gid 0
> uptime 15m
> children 4
> memory kilobytes 1240
> memory kilobytes total 8448
> memory percent 0.0%
> memory percent total 0.2%
> cpu percent 0.0%
> cpu percent total 0.0%
> port response time 0.000s to 127.0.0.1:80/ping [HTTP via
> TCP]
> data collected Wed, 04 Feb 2015 10:58:02
>
> Program 'poll_homestead'
> status Status failed
> monitoring status Monitored
> last started Wed, 04 Feb 2015 10:58:02
> last exit value 1
> data collected Wed, 04 Feb 2015 10:58:02
>
> Process 'homestead'
> status Does not exist
> monitoring status Monitored
> data collected Wed, 04 Feb 2015 10:58:02
>
> Program 'poll_homestead-prov'
> status Status ok
> monitoring status Monitored
> last started Wed, 04 Feb 2015 10:58:02
> last exit value 0
> data collected Wed, 04 Feb 2015 10:58:02
>
> Process 'homestead-prov'
> status Execution failed
> monitoring status Monitored
> data collected Wed, 04 Feb 2015 10:58:32
>
> Process 'clearwater_diags_monitor'
> status Running
> monitoring status Monitored
> pid 1027
> parent pid 1
> uid 0
> effective uid 0
> gid 0
> uptime 16m
> children 1
> memory kilobytes 1664
> memory kilobytes total 2040
> memory percent 0.0%
> memory percent total 0.0%
> cpu percent 0.0%
> cpu percent total 0.0%
> data collected Wed, 04 Feb 2015 10:58:32
>
> Program 'poll_cassandra_ring'
> status Status ok
> monitoring status Monitored
> last started Wed, 04 Feb 2015 10:58:32
> last exit value 0
> data collected Wed, 04 Feb 2015 10:58:32
>
> Process 'cassandra'
> status Running
> monitoring status Monitored
> pid 1280
> parent pid 1277
> uid 106
> effective uid 106
> gid 113
> uptime 16m
> children 0
> memory kilobytes 1388648
> memory kilobytes total 1388648
> memory percent 34.3%
> memory percent total 34.3%
> cpu percent 0.4%
> cpu percent total 0.4%
> data collected Wed, 04 Feb 2015 10:58:32
>
> System 'homestead-1'
> status Running
> monitoring status Monitored
> load average [0.00] [0.04] [0.05]
> cpu 3.0%us 0.8%sy 0.0%wa
> memory usage 1505324 kB [37.1%]
> swap usage 0 kB [0.0%]
> data collected Wed, 04 Feb 2015 10:58:32
>
>
> And log file shows:
>
> [homestead]cw@homestead-1:~$ cat
> /var/log/homestead-prov/homestead-prov-err.log
> Traceback (most recent call last):
> File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
> File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
> File
>
> "/usr/share/clearwater/homestead/env/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/main.py",
> line 156, in <module>
> standalone()
> File
>
> "/usr/share/clearwater/homestead/env/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/main.py",
> line 119, in standalone
> reactor.listenUNIX(unix_sock_name, application)
> File
>
> "/usr/share/clearwater/homestead/env/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/internet/posixbase.py",
> line 413, in listenUNIX
> p.startListening()
> File
>
> "/usr/share/clearwater/homestead/env/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/internet/unix.py",
> line 293, in startListening
> raise CannotListenError, (None, self.port, le)
> twisted.internet.error.CannotListenError: Couldn't listen on
> any:/tmp/.homestead-prov-sock-0: [Errno 98] Address already in use.
> ......
>
> [homestead]cw@homestead-1:~$ cat
> /var/log/homestead-prov/homestead-prov-0.log
> 2015-02-04 18:42:23,476 UTC INFO main:118 Going to listen for HTTP on UNIX
> socket /tmp/.homestead-prov-sock-0
> 2015-02-04 18:42:24,087 UTC INFO main:118 Going to listen for HTTP on UNIX
> socket /tmp/.homestead-prov-sock-0
> 2015-02-04 18:42:35,826 UTC INFO main:118 Going to listen for HTTP on UNIX
> socket /tmp/.homestead-prov-sock-0
> 2015-02-04 18:43:16,205 UTC INFO main:118 Going to listen for HTTP on UNIX
> socket /tmp/.homestead-prov-sock-0 ......
>
> homestead_20150204T180000Z.txt homestead_current.txt
> [homestead]cw@homestead-1:~$ cat /var/log/homestead/homestead_current.txt
> 04-02-2015 18:42:19.586 UTC Status main.cpp:468: Log level set to 2
> 04-02-2015 18:42:19.602 UTC Status main.cpp:489: Access logging enabled to
> /var/log/homestead
> 04-02-2015 18:42:19.614 UTC Status load_monitor.cpp:93: Constructing
> LoadMonitor
> 04-02-2015 18:42:19.614 UTC Status load_monitor.cpp:94: Target latency
> (usecs) : 100000
> 04-02-2015 18:42:19.614 UTC Status load_monitor.cpp:95: Max bucket size
> : 20
> 04-02-2015 18:42:19.614 UTC Status load_monitor.cpp:96: Initial token
> fill rate/s: 10.000000
> 04-02-2015 18:42:19.614 UTC Status load_monitor.cpp:97: Min token fill
> rate/s : 10.000000
> 04-02-2015 18:42:19.614 UTC Status dnscachedresolver.cpp:90: Creating
> Cached Resolver using server 127.0.0.1
> 04-02-2015 18:42:19.614 UTC Status httpresolver.cpp:50: Created HTTP
> resolver
> 04-02-2015 18:42:19.614 UTC Status cassandra_store.cpp:145: Configuring
> store
> 04-02-2015 18:42:19.614 UTC Status cassandra_store.cpp:146: Hostname:
> localhost
> 04-02-2015 18:42:19.614 UTC Status cassandra_store.cpp:147: Port:
> 9160
> 04-02-2015 18:42:19.614 UTC Status cassandra_store.cpp:148: Threads: 10
> 04-02-2015 18:42:19.614 UTC Status cassandra_store.cpp:149: Max Queue: 0
> 04-02-2015 18:42:19.614 UTC Status cassandra_store.cpp:199: Starting store
> 04-02-2015 18:42:19.616 UTC Error cassandra_store.cpp:207: Cache caught
> TTransportException: connect() failed: Connection refused
> 04-02-2015 18:42:19.616 UTC Error main.cpp:550: Failed to initialize cache
> - rc 3
> 04-02-2015 18:42:19.616 UTC Status cassandra_store.cpp:185: Stopping cache
> 04-02-2015 18:42:19.616 UTC Status cassandra_store.cpp:226: Waiting for
> cache to stop ......
>
> And the port usage is:
>
> [homestead]cw@homestead-1:~$ sudo netstat -tulpn Active Internet
> connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> PID/Program name
> tcp 0 0 127.0.0.1:9042 0.0.0.0:* LISTEN
> 1280/jsvc.exec
> tcp 0 0 0.0.0.0:53 0.0.0.0:* LISTEN
> 952/dnsmasq
> tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
> 827/sshd
> tcp 0 0 127.0.0.1:7000 0.0.0.0:* LISTEN
> 1280/jsvc.exec
> tcp 0 0 127.0.0.1:2812 0.0.0.0:* LISTEN
> 1036/monit
> tcp 0 0 0.0.0.0:37791 0.0.0.0:* LISTEN
> 1280/jsvc.exec
> tcp 0 0 0.0.0.0:7199 0.0.0.0:* LISTEN
> 1280/jsvc.exec
> tcp 0 0 0.0.0.0:53313 0.0.0.0:* LISTEN
> 1280/jsvc.exec
> tcp 0 0 127.0.0.1:9160 0.0.0.0:* LISTEN
> 1280/jsvc.exec
> tcp6 0 0 :::53 :::* LISTEN
> 952/dnsmasq
> tcp6 0 0 :::22 :::* LISTEN
> 827/sshd
> tcp6 0 0 :::8889 :::* LISTEN
> 1044/nginx
> tcp6 0 0 :::80 :::* LISTEN
> 1044/nginx
> udp 0 0 0.0.0.0:13344 0.0.0.0:*
> 952/dnsmasq
> udp 0 0 0.0.0.0:48567 0.0.0.0:*
> 952/dnsmasq
> udp 0 0 0.0.0.0:53 0.0.0.0:*
> 952/dnsmasq
> udp 0 0 0.0.0.0:41016 0.0.0.0:*
> 952/dnsmasq
> udp 0 0 0.0.0.0:68 0.0.0.0:*
> 634/dhclient3
> udp 0 0 192.168.1.31:123 0.0.0.0:*
> 791/ntpd
> udp 0 0 127.0.0.1:123 0.0.0.0:*
> 791/ntpd
> udp 0 0 0.0.0.0:123 0.0.0.0:*
> 791/ntpd
> udp6 0 0 :::53 :::*
> 952/dnsmasq
> udp6 0 0 fe80::f816:3eff:fe7:123 :::*
> 791/ntpd
> udp6 0 0 ::1:123 :::*
> 791/ntpd
> udp6 0 0 :::123 :::*
> 791/ntpd
>
>
>
> So, how should we fix the problems with Homestead and Homestead-prov?
>
> Best regards,
> Lianjie
> _______________________________________________
> Clearwater mailing list
> [email protected]
> http://lists.projectclearwater.org/listinfo/clearwater
>
[homestead]cw@homestead-1:~$ sudo cat /var/log/cassandra/output.log
INFO 11:30:50,639 Logging initialized
INFO 11:30:50,677 JVM vendor/version: OpenJDK 64-Bit Server VM/1.6.0_34
WARN 11:30:50,677 OpenJDK is not recommended. Please upgrade to the newest
Oracle Java release
INFO 11:30:50,678 Heap size: 1063256064/1063256064
INFO 11:30:50,678 Classpath:
/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.6.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-13.0.1.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.1.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.2.12.jar:/usr/share/cassandra/apache-cassandra-thrift-1.2.12.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
INFO 11:30:53,269 JNA mlockall successful
INFO 11:30:53,324 Loading settings from file:/etc/cassandra/cassandra.yaml
INFO 11:30:54,102 Data files directories: [/var/lib/cassandra/data]
INFO 11:30:54,102 Commit log directory: /var/lib/cassandra/commitlog
INFO 11:30:54,102 DiskAccessMode 'auto' determined to be mmap, indexAccessMode
is mmap
INFO 11:30:54,103 disk_failure_policy is stop
INFO 11:30:54,111 Global memtable threshold is enabled at 338MB
INFO 11:30:54,297 Not using multi-threaded compaction
INFO 11:30:54,650 Initializing key cache with capacity of 50 MBs.
INFO 11:30:54,661 Scheduling key cache save to each 14400 seconds (going to
save all keys).
INFO 11:30:54,662 Initializing row cache with capacity of 0 MBs and provider
org.apache.cassandra.cache.SerializingCacheProvider
INFO 11:30:54,670 Scheduling row cache save to each 0 seconds (going to save
all keys).
INFO 11:30:54,839 Initializing system.batchlog
INFO 11:30:54,859 Initializing system.peer_events
INFO 11:30:54,865 Initializing system.hints
INFO 11:30:54,873 Initializing system.Schema
INFO 11:30:54,885 Initializing system.schema_keyspaces
INFO 11:30:54,904 Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-23
(220 bytes)
INFO 11:30:54,904 Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-23
(220 bytes)
INFO 11:30:54,952 Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-21
(220 bytes)
INFO 11:30:54,952 Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-21
(220 bytes)
INFO 11:30:54,973 Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-16
(317 bytes)
INFO 11:30:54,973 Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-16
(317 bytes)
ERROR 11:30:54,988 Exception in thread Thread[SSTableBatchOpen:1,5,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:108)
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:63)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:418)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:209)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:157)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:273)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
at java.io.DataInputStream.readUTF(DataInputStream.java:589)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:83)
... 12 more
INFO 11:30:54,990 Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-22
(220 bytes)
INFO 11:30:54,992 Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-22
(220 bytes)
INFO 11:30:55,025 Initializing system.range_xfers
INFO 11:30:55,027 Initializing system.HintsColumnFamily
INFO 11:30:55,032 Initializing system.schema_columnfamilies
INFO 11:30:55,035 Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-36
(4593 bytes)
INFO 11:30:55,035 Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-36
(4593 bytes)
INFO 11:30:55,044 Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-31
(6226 bytes)
INFO 11:30:55,044 Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-31
(6226 bytes)
ERROR 11:30:55,047 Exception in thread Thread[SSTableBatchOpen:1,5,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:108)
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:63)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:418)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:209)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:157)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:273)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
at java.io.DataInputStream.readUTF(DataInputStream.java:589)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:83)
... 12 more
INFO 11:30:55,049 Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-38
(4593 bytes)
INFO 11:30:55,050 Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-38
(4593 bytes)
INFO 11:30:55,054 Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-37
(4593 bytes)
INFO 11:30:55,055 Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-37
(4593 bytes)
INFO 11:30:55,067 Initializing system.NodeIdInfo
INFO 11:30:55,079 Initializing system.schema_columns
INFO 11:30:55,081 Opening
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-37 (4165
bytes)
INFO 11:30:55,081 Opening
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-37 (4165
bytes)
INFO 11:30:55,096 Opening
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-31 (3305
bytes)
INFO 11:30:55,096 Opening
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-31 (3305
bytes)
ERROR 11:30:55,099 Exception in thread Thread[SSTableBatchOpen:1,5,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:108)
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:63)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:418)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:209)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:157)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:273)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
at java.io.DataInputStream.readUTF(DataInputStream.java:589)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:83)
... 12 more
INFO 11:30:55,099 Opening
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-38 (4165
bytes)
INFO 11:30:55,099 Opening
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-38 (4165
bytes)
INFO 11:30:55,110 Initializing system.IndexInfo
INFO 11:30:55,119 Initializing system.Migrations
INFO 11:30:55,122 Initializing system.peers
INFO 11:30:55,127 Initializing system.local
INFO 11:30:55,133 Opening
/var/lib/cassandra/data/system/local/system-local-ic-2 (120 bytes)
INFO 11:30:55,133 Opening
/var/lib/cassandra/data/system/local/system-local-ic-2 (120 bytes)
ERROR 11:30:55,141 Exception in thread Thread[SSTableBatchOpen:1,5,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:108)
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:63)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:418)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:209)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:157)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:273)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
at java.io.DataInputStream.readUTF(DataInputStream.java:589)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:83)
... 12 more
INFO 11:30:55,141 Opening
/var/lib/cassandra/data/system/local/system-local-ic-92 (121 bytes)
INFO 11:30:55,142 Opening
/var/lib/cassandra/data/system/local/system-local-ic-92 (121 bytes)
INFO 11:30:55,149 Opening
/var/lib/cassandra/data/system/local/system-local-ic-1 (357 bytes)
INFO 11:30:55,150 Opening
/var/lib/cassandra/data/system/local/system-local-ic-1 (357 bytes)
ERROR 11:30:55,157 Exception in thread Thread[SSTableBatchOpen:1,5,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:108)
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:63)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:418)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:209)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:157)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:273)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
at java.io.DataInputStream.readUTF(DataInputStream.java:589)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:83)
... 12 more
INFO 11:30:55,158 Opening
/var/lib/cassandra/data/system/local/system-local-ic-3 (109 bytes)
INFO 11:30:55,159 Opening
/var/lib/cassandra/data/system/local/system-local-ic-3 (109 bytes)
ERROR 11:30:55,163 Exception in thread Thread[SSTableBatchOpen:1,5,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:108)
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:63)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:418)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:209)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:157)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:273)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
at java.io.DataInputStream.readUTF(DataInputStream.java:589)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:83)
... 12 more
INFO 11:30:55,168 Opening
/var/lib/cassandra/data/system/local/system-local-ic-91 (451 bytes)
INFO 11:30:55,168 Opening
/var/lib/cassandra/data/system/local/system-local-ic-91 (451 bytes)
INFO 11:30:55,179 Opening
/var/lib/cassandra/data/system/local/system-local-ic-93 (109 bytes)
INFO 11:30:55,179 Opening
/var/lib/cassandra/data/system/local/system-local-ic-93 (109 bytes)
INFO 11:30:55,191 reading saved cache
/var/lib/cassandra/saved_caches/system-local-KeyCache-b.db
INFO 11:30:55,217 Initializing system.LocationInfo
INFO 11:30:55,775 Initializing system_traces.sessions
INFO 11:30:55,790 Initializing system_traces.events
INFO 11:30:55,796 completed pre-loading (3 keys) key cache.
INFO 11:30:55,825 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312419.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312420.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312421.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312422.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312423.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312424.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312425.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312426.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312427.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312428.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312429.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312430.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312431.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312432.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312433.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312434.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312435.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312436.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312437.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312438.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312439.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312440.log
INFO 11:30:55,835 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312419.log
INFO 11:30:55,957 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312419.log
INFO 11:30:55,957 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312420.log
INFO 11:30:55,964 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312420.log
INFO 11:30:55,964 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312421.log
INFO 11:30:55,967 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312421.log
INFO 11:30:55,967 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312422.log
INFO 11:30:55,981 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312422.log
INFO 11:30:55,981 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312423.log
INFO 11:30:55,986 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312423.log
INFO 11:30:55,986 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312424.log
INFO 11:30:55,989 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312424.log
INFO 11:30:55,990 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312425.log
INFO 11:30:55,996 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312425.log
INFO 11:30:55,996 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312426.log
INFO 11:30:56,002 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312426.log
INFO 11:30:56,002 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312427.log
INFO 11:30:56,007 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312427.log
INFO 11:30:56,007 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312428.log
INFO 11:30:56,012 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312428.log
INFO 11:30:56,012 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312429.log
INFO 11:30:56,016 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312429.log
INFO 11:30:56,017 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312430.log
INFO 11:30:56,020 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312430.log
INFO 11:30:56,020 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312431.log
INFO 11:30:56,026 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312431.log
INFO 11:30:56,026 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312432.log
INFO 11:30:56,029 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312432.log
INFO 11:30:56,030 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312433.log
INFO 11:30:56,040 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312433.log
INFO 11:30:56,041 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312434.log
INFO 11:30:56,049 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312434.log
INFO 11:30:56,049 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312435.log
INFO 11:30:56,059 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312435.log
INFO 11:30:56,059 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312436.log
INFO 11:30:56,064 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312436.log
INFO 11:30:56,064 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312437.log
INFO 11:30:56,074 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312437.log
INFO 11:30:56,074 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312438.log
INFO 11:30:56,078 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312438.log
INFO 11:30:56,078 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312439.log
INFO 11:30:56,084 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312439.log
INFO 11:30:56,084 Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312440.log
INFO 11:30:56,091 Finished reading
/var/lib/cassandra/commitlog/CommitLog-2-1423085312440.log
INFO 11:30:56,096 Enqueuing flush of Memtable-local@399520043(52/520
serialized/live bytes, 2 ops)
INFO 11:30:56,099 Writing Memtable-local@399520043(52/520 serialized/live
bytes, 2 ops)
INFO 11:30:56,105 Enqueuing flush of
Memtable-schema_keyspaces@39144552(251/2510 serialized/live bytes, 8 ops)
INFO 11:30:56,105 Enqueuing flush of
Memtable-schema_columns@772489482(23741/237410 serialized/live bytes, 386 ops)
INFO 11:30:56,105 Enqueuing flush of
Memtable-schema_columnfamilies@1692077859(22187/221870 serialized/live bytes,
369 ops)
INFO 11:30:56,123 Completed flushing
/var/lib/cassandra/data/system/local/system-local-ic-94-Data.db (84 bytes) for
commitlog position ReplayPosition(segmentId=1423251055622, position=142)
INFO 11:30:56,140 Writing Memtable-schema_keyspaces@39144552(251/2510
serialized/live bytes, 8 ops)
INFO 11:30:56,159 Completed flushing
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-24-Data.db
(220 bytes) for commitlog position ReplayPosition(segmentId=1423251055622,
position=142)
INFO 11:30:56,160 Writing Memtable-schema_columns@772489482(23741/237410
serialized/live bytes, 386 ops)
INFO 11:30:56,186 Completed flushing
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-39-Data.db
(4167 bytes) for commitlog position ReplayPosition(segmentId=1423251055622,
position=142)
INFO 11:30:56,187 Writing
Memtable-schema_columnfamilies@1692077859(22187/221870 serialized/live bytes,
369 ops)
INFO 11:30:56,216 Completed flushing
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-39-Data.db
(4593 bytes) for commitlog position ReplayPosition(segmentId=1423251055622,
position=142)
INFO 11:30:56,218 Log replay complete, 12 replayed mutations
INFO 11:30:56,607 Cassandra version: 1.2.12
INFO 11:30:56,607 Thrift API version: 19.36.1
INFO 11:30:56,608 CQL supported versions: 2.0.0,3.0.5 (default: 3.0.5)
INFO 11:30:56,633 Loading persisted ring state
INFO 11:30:56,652 Starting up server gossip
INFO 11:30:56,665 Enqueuing flush of Memtable-local@1392725193(252/2520
serialized/live bytes, 9 ops)
INFO 11:30:56,666 Writing Memtable-local@1392725193(252/2520 serialized/live
bytes, 9 ops)
INFO 11:30:56,676 Completed flushing
/var/lib/cassandra/data/system/local/system-local-ic-95-Data.db (243 bytes) for
commitlog position ReplayPosition(segmentId=1423251055622, position=52890)
INFO 11:30:56,705 Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-95-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-94-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-92-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-93-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-91-Data.db')]
INFO 11:30:56,855 Compacted 5 sstables to
[/var/lib/cassandra/data/system/local/system-local-ic-96,]. 1,008 bytes to 451
(~44% of original) in 148ms = 0.002906MB/s. 5 total rows, 1 unique. Row merge
counts were {1:0, 2:0, 3:0, 4:0, 5:1, }
INFO 11:30:56,905 Starting Messaging Service on port 7000
INFO 11:30:56,944 Using saved token [2069380001304406395]
INFO 11:30:56,945 Enqueuing flush of Memtable-local@1528877937(84/840
serialized/live bytes, 4 ops)
INFO 11:30:56,945 Writing Memtable-local@1528877937(84/840 serialized/live
bytes, 4 ops)
INFO 11:30:56,954 Completed flushing
/var/lib/cassandra/data/system/local/system-local-ic-97-Data.db (120 bytes) for
commitlog position ReplayPosition(segmentId=1423251055622, position=53165)
INFO 11:30:56,961 Enqueuing flush of Memtable-local@702560738(50/500
serialized/live bytes, 2 ops)
INFO 11:30:56,961 Writing Memtable-local@702560738(50/500 serialized/live
bytes, 2 ops)
INFO 11:30:56,970 Completed flushing
/var/lib/cassandra/data/system/local/system-local-ic-98-Data.db (109 bytes) for
commitlog position ReplayPosition(segmentId=1423251055622, position=53339)
INFO 11:30:56,983 Node localhost/127.0.0.1 state jump to normal
INFO 11:30:56,987 Startup completed! Now serving reads.
INFO 11:30:57,134 Starting listening for CQL clients on
localhost/127.0.0.1:9042...
INFO 11:30:57,150 Binding thrift service to localhost/127.0.0.1:9160
INFO 11:30:57,192 Using TFramedTransport with a max frame size of 15728640
bytes.
INFO 11:30:57,198 Using synchronous/threadpool thrift server on localhost :
9160
INFO 11:30:57,198 Listening for thrift clients...
[homestead]cw@homestead-1:~$ sudo cat /var/log/cassandra/system.log
INFO [main] 2015-02-06 11:30:50,639 CassandraDaemon.java (line 119) Logging
initialized
INFO [main] 2015-02-06 11:30:50,677 CassandraDaemon.java (line 144) JVM
vendor/version: OpenJDK 64-Bit Server VM/1.6.0_34
WARN [main] 2015-02-06 11:30:50,677 CassandraDaemon.java (line 149) OpenJDK is
not recommended. Please upgrade to the newest Oracle Java release
INFO [main] 2015-02-06 11:30:50,678 CassandraDaemon.java (line 182) Heap size:
1063256064/1063256064
INFO [main] 2015-02-06 11:30:50,678 CassandraDaemon.java (line 183) Classpath:
/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.6.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-13.0.1.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.1.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.2.12.jar:/usr/share/cassandra/apache-cassandra-thrift-1.2.12.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
INFO [main] 2015-02-06 11:30:53,269 CLibrary.java (line 125) JNA mlockall
successful
INFO [main] 2015-02-06 11:30:53,324 DatabaseDescriptor.java (line 141) Loading
settings from file:/etc/cassandra/cassandra.yaml
INFO [main] 2015-02-06 11:30:54,102 DatabaseDescriptor.java (line 159) Data
files directories: [/var/lib/cassandra/data]
INFO [main] 2015-02-06 11:30:54,102 DatabaseDescriptor.java (line 160) Commit
log directory: /var/lib/cassandra/commitlog
INFO [main] 2015-02-06 11:30:54,102 DatabaseDescriptor.java (line 200)
DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO [main] 2015-02-06 11:30:54,103 DatabaseDescriptor.java (line 214)
disk_failure_policy is stop
INFO [main] 2015-02-06 11:30:54,111 DatabaseDescriptor.java (line 285) Global
memtable threshold is enabled at 338MB
INFO [main] 2015-02-06 11:30:54,297 DatabaseDescriptor.java (line 416) Not
using multi-threaded compaction
INFO [main] 2015-02-06 11:30:54,650 CacheService.java (line 111) Initializing
key cache with capacity of 50 MBs.
INFO [main] 2015-02-06 11:30:54,661 CacheService.java (line 140) Scheduling
key cache save to each 14400 seconds (going to save all keys).
INFO [main] 2015-02-06 11:30:54,662 CacheService.java (line 154) Initializing
row cache with capacity of 0 MBs and provider
org.apache.cassandra.cache.SerializingCacheProvider
INFO [main] 2015-02-06 11:30:54,670 CacheService.java (line 166) Scheduling
row cache save to each 0 seconds (going to save all keys).
INFO [main] 2015-02-06 11:30:54,839 ColumnFamilyStore.java (line 223)
Initializing system.batchlog
INFO [main] 2015-02-06 11:30:54,859 ColumnFamilyStore.java (line 223)
Initializing system.peer_events
INFO [main] 2015-02-06 11:30:54,865 ColumnFamilyStore.java (line 223)
Initializing system.hints
INFO [main] 2015-02-06 11:30:54,873 ColumnFamilyStore.java (line 223)
Initializing system.Schema
INFO [main] 2015-02-06 11:30:54,885 ColumnFamilyStore.java (line 223)
Initializing system.schema_keyspaces
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:54,904 SSTableReader.java (line
192) Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-23
(220 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:54,904 SSTableReader.java (line
232) Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-23
(220 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:54,952 SSTableReader.java (line
192) Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-21
(220 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:54,952 SSTableReader.java (line
232) Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-21
(220 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:54,973 SSTableReader.java (line
192) Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-16
(317 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:54,973 SSTableReader.java (line
232) Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-16
(317 bytes)
ERROR [SSTableBatchOpen:1] 2015-02-06 11:30:54,988 CassandraDaemon.java (line
191) Exception in thread Thread[SSTableBatchOpen:1,5,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:108)
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:63)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:418)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:209)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:157)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:273)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
at java.io.DataInputStream.readUTF(DataInputStream.java:589)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:83)
... 12 more
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:54,990 SSTableReader.java (line
192) Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-22
(220 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:54,992 SSTableReader.java (line
232) Opening
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-22
(220 bytes)
INFO [main] 2015-02-06 11:30:55,025 ColumnFamilyStore.java (line 223)
Initializing system.range_xfers
INFO [main] 2015-02-06 11:30:55,027 ColumnFamilyStore.java (line 223)
Initializing system.HintsColumnFamily
INFO [main] 2015-02-06 11:30:55,032 ColumnFamilyStore.java (line 223)
Initializing system.schema_columnfamilies
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,035 SSTableReader.java (line
192) Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-36
(4593 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,035 SSTableReader.java (line
232) Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-36
(4593 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,044 SSTableReader.java (line
192) Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-31
(6226 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,044 SSTableReader.java (line
232) Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-31
(6226 bytes)
ERROR [SSTableBatchOpen:1] 2015-02-06 11:30:55,047 CassandraDaemon.java (line
191) Exception in thread Thread[SSTableBatchOpen:1,5,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:108)
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:63)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:418)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:209)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:157)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:273)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
at java.io.DataInputStream.readUTF(DataInputStream.java:589)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:83)
... 12 more
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,049 SSTableReader.java (line
192) Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-38
(4593 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,050 SSTableReader.java (line
232) Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-38
(4593 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,054 SSTableReader.java (line
192) Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-37
(4593 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,055 SSTableReader.java (line
232) Opening
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-37
(4593 bytes)
INFO [main] 2015-02-06 11:30:55,067 ColumnFamilyStore.java (line 223)
Initializing system.NodeIdInfo
INFO [main] 2015-02-06 11:30:55,079 ColumnFamilyStore.java (line 223)
Initializing system.schema_columns
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,081 SSTableReader.java (line
192) Opening
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-37 (4165
bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,081 SSTableReader.java (line
232) Opening
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-37 (4165
bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,096 SSTableReader.java (line
192) Opening
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-31 (3305
bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,096 SSTableReader.java (line
232) Opening
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-31 (3305
bytes)
ERROR [SSTableBatchOpen:1] 2015-02-06 11:30:55,099 CassandraDaemon.java (line
191) Exception in thread Thread[SSTableBatchOpen:1,5,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:108)
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:63)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:418)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:209)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:157)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:273)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
at java.io.DataInputStream.readUTF(DataInputStream.java:589)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:83)
... 12 more
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,099 SSTableReader.java (line
192) Opening
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-38 (4165
bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,099 SSTableReader.java (line
232) Opening
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-38 (4165
bytes)
INFO [main] 2015-02-06 11:30:55,110 ColumnFamilyStore.java (line 223)
Initializing system.IndexInfo
INFO [main] 2015-02-06 11:30:55,119 ColumnFamilyStore.java (line 223)
Initializing system.Migrations
INFO [main] 2015-02-06 11:30:55,122 ColumnFamilyStore.java (line 223)
Initializing system.peers
INFO [main] 2015-02-06 11:30:55,127 ColumnFamilyStore.java (line 223)
Initializing system.local
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,133 SSTableReader.java (line
192) Opening /var/lib/cassandra/data/system/local/system-local-ic-2 (120 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,133 SSTableReader.java (line
232) Opening /var/lib/cassandra/data/system/local/system-local-ic-2 (120 bytes)
ERROR [SSTableBatchOpen:1] 2015-02-06 11:30:55,141 CassandraDaemon.java (line
191) Exception in thread Thread[SSTableBatchOpen:1,5,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:108)
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:63)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:418)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:209)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:157)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:273)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
at java.io.DataInputStream.readUTF(DataInputStream.java:589)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:83)
... 12 more
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,141 SSTableReader.java (line
192) Opening /var/lib/cassandra/data/system/local/system-local-ic-92 (121 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,142 SSTableReader.java (line
232) Opening /var/lib/cassandra/data/system/local/system-local-ic-92 (121 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,149 SSTableReader.java (line
192) Opening /var/lib/cassandra/data/system/local/system-local-ic-1 (357 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,150 SSTableReader.java (line
232) Opening /var/lib/cassandra/data/system/local/system-local-ic-1 (357 bytes)
ERROR [SSTableBatchOpen:1] 2015-02-06 11:30:55,157 CassandraDaemon.java (line
191) Exception in thread Thread[SSTableBatchOpen:1,5,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:108)
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:63)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:418)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:209)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:157)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:273)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
at java.io.DataInputStream.readUTF(DataInputStream.java:589)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:83)
... 12 more
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,158 SSTableReader.java (line
192) Opening /var/lib/cassandra/data/system/local/system-local-ic-3 (109 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,159 SSTableReader.java (line
232) Opening /var/lib/cassandra/data/system/local/system-local-ic-3 (109 bytes)
ERROR [SSTableBatchOpen:1] 2015-02-06 11:30:55,163 CassandraDaemon.java (line
191) Exception in thread Thread[SSTableBatchOpen:1,5,main]
org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:108)
at
org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:63)
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:42)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:418)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:209)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:157)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:273)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: java.io.EOFException
at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340)
at java.io.DataInputStream.readUTF(DataInputStream.java:589)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at
org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:83)
... 12 more
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,168 SSTableReader.java (line
192) Opening /var/lib/cassandra/data/system/local/system-local-ic-91 (451 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,168 SSTableReader.java (line
232) Opening /var/lib/cassandra/data/system/local/system-local-ic-91 (451 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,179 SSTableReader.java (line
192) Opening /var/lib/cassandra/data/system/local/system-local-ic-93 (109 bytes)
INFO [SSTableBatchOpen:1] 2015-02-06 11:30:55,179 SSTableReader.java (line
232) Opening /var/lib/cassandra/data/system/local/system-local-ic-93 (109 bytes)
INFO [main] 2015-02-06 11:30:55,191 AutoSavingCache.java (line 140) reading
saved cache /var/lib/cassandra/saved_caches/system-local-KeyCache-b.db
INFO [main] 2015-02-06 11:30:55,217 ColumnFamilyStore.java (line 223)
Initializing system.LocationInfo
INFO [main] 2015-02-06 11:30:55,775 ColumnFamilyStore.java (line 223)
Initializing system_traces.sessions
INFO [main] 2015-02-06 11:30:55,790 ColumnFamilyStore.java (line 223)
Initializing system_traces.events
INFO [main] 2015-02-06 11:30:55,796 CassandraDaemon.java (line 286) completed
pre-loading (3 keys) key cache.
INFO [main] 2015-02-06 11:30:55,825 CommitLog.java (line 125) Replaying
/var/lib/cassandra/commitlog/CommitLog-2-1423085312419.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312420.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312421.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312422.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312423.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312424.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312425.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312426.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312427.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312428.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312429.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312430.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312431.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312432.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312433.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312434.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312435.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312436.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312437.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312438.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312439.log,
/var/lib/cassandra/commitlog/CommitLog-2-1423085312440.log
INFO [main] 2015-02-06 11:30:55,835 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312419.log
INFO [main] 2015-02-06 11:30:55,957 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312419.log
INFO [main] 2015-02-06 11:30:55,957 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312420.log
INFO [main] 2015-02-06 11:30:55,964 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312420.log
INFO [main] 2015-02-06 11:30:55,964 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312421.log
INFO [main] 2015-02-06 11:30:55,967 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312421.log
INFO [main] 2015-02-06 11:30:55,967 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312422.log
INFO [main] 2015-02-06 11:30:55,981 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312422.log
INFO [main] 2015-02-06 11:30:55,981 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312423.log
INFO [main] 2015-02-06 11:30:55,986 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312423.log
INFO [main] 2015-02-06 11:30:55,986 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312424.log
INFO [main] 2015-02-06 11:30:55,989 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312424.log
INFO [main] 2015-02-06 11:30:55,990 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312425.log
INFO [main] 2015-02-06 11:30:55,996 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312425.log
INFO [main] 2015-02-06 11:30:55,996 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312426.log
INFO [main] 2015-02-06 11:30:56,002 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312426.log
INFO [main] 2015-02-06 11:30:56,002 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312427.log
INFO [main] 2015-02-06 11:30:56,007 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312427.log
INFO [main] 2015-02-06 11:30:56,007 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312428.log
INFO [main] 2015-02-06 11:30:56,012 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312428.log
INFO [main] 2015-02-06 11:30:56,012 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312429.log
INFO [main] 2015-02-06 11:30:56,016 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312429.log
INFO [main] 2015-02-06 11:30:56,017 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312430.log
INFO [main] 2015-02-06 11:30:56,020 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312430.log
INFO [main] 2015-02-06 11:30:56,020 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312431.log
INFO [main] 2015-02-06 11:30:56,026 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312431.log
INFO [main] 2015-02-06 11:30:56,026 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312432.log
INFO [main] 2015-02-06 11:30:56,029 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312432.log
INFO [main] 2015-02-06 11:30:56,030 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312433.log
INFO [main] 2015-02-06 11:30:56,040 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312433.log
INFO [main] 2015-02-06 11:30:56,041 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312434.log
INFO [main] 2015-02-06 11:30:56,049 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312434.log
INFO [main] 2015-02-06 11:30:56,049 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312435.log
INFO [main] 2015-02-06 11:30:56,059 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312435.log
INFO [main] 2015-02-06 11:30:56,059 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312436.log
INFO [main] 2015-02-06 11:30:56,064 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312436.log
INFO [main] 2015-02-06 11:30:56,064 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312437.log
INFO [main] 2015-02-06 11:30:56,074 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312437.log
INFO [main] 2015-02-06 11:30:56,074 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312438.log
INFO [main] 2015-02-06 11:30:56,078 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312438.log
INFO [main] 2015-02-06 11:30:56,078 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312439.log
INFO [main] 2015-02-06 11:30:56,084 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312439.log
INFO [main] 2015-02-06 11:30:56,084 CommitLogReplayer.java (line 116)
Replaying /var/lib/cassandra/commitlog/CommitLog-2-1423085312440.log
INFO [main] 2015-02-06 11:30:56,091 CommitLogReplayer.java (line 292) Finished
reading /var/lib/cassandra/commitlog/CommitLog-2-1423085312440.log
INFO [main] 2015-02-06 11:30:56,096 ColumnFamilyStore.java (line 632)
Enqueuing flush of Memtable-local@399520043(52/520 serialized/live bytes, 2 ops)
INFO [FlushWriter:1] 2015-02-06 11:30:56,099 Memtable.java (line 400) Writing
Memtable-local@399520043(52/520 serialized/live bytes, 2 ops)
INFO [main] 2015-02-06 11:30:56,105 ColumnFamilyStore.java (line 632)
Enqueuing flush of Memtable-schema_keyspaces@39144552(251/2510 serialized/live
bytes, 8 ops)
INFO [main] 2015-02-06 11:30:56,105 ColumnFamilyStore.java (line 632)
Enqueuing flush of Memtable-schema_columns@772489482(23741/237410
serialized/live bytes, 386 ops)
INFO [main] 2015-02-06 11:30:56,105 ColumnFamilyStore.java (line 632)
Enqueuing flush of Memtable-schema_columnfamilies@1692077859(22187/221870
serialized/live bytes, 369 ops)
INFO [FlushWriter:1] 2015-02-06 11:30:56,123 Memtable.java (line 438)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-ic-94-Data.db (84 bytes) for
commitlog position ReplayPosition(segmentId=1423251055622, position=142)
INFO [FlushWriter:1] 2015-02-06 11:30:56,140 Memtable.java (line 400) Writing
Memtable-schema_keyspaces@39144552(251/2510 serialized/live bytes, 8 ops)
INFO [FlushWriter:1] 2015-02-06 11:30:56,159 Memtable.java (line 438)
Completed flushing
/var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-24-Data.db
(220 bytes) for commitlog position ReplayPosition(segmentId=1423251055622,
position=142)
INFO [FlushWriter:1] 2015-02-06 11:30:56,160 Memtable.java (line 400) Writing
Memtable-schema_columns@772489482(23741/237410 serialized/live bytes, 386 ops)
INFO [FlushWriter:1] 2015-02-06 11:30:56,186 Memtable.java (line 438)
Completed flushing
/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-39-Data.db
(4167 bytes) for commitlog position ReplayPosition(segmentId=1423251055622,
position=142)
INFO [FlushWriter:1] 2015-02-06 11:30:56,187 Memtable.java (line 400) Writing
Memtable-schema_columnfamilies@1692077859(22187/221870 serialized/live bytes,
369 ops)
INFO [FlushWriter:1] 2015-02-06 11:30:56,216 Memtable.java (line 438)
Completed flushing
/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-39-Data.db
(4593 bytes) for commitlog position ReplayPosition(segmentId=1423251055622,
position=142)
INFO [main] 2015-02-06 11:30:56,218 CommitLog.java (line 127) Log replay
complete, 12 replayed mutations
INFO [main] 2015-02-06 11:30:56,607 StorageService.java (line 487) Cassandra
version: 1.2.12
INFO [main] 2015-02-06 11:30:56,607 StorageService.java (line 488) Thrift API
version: 19.36.1
INFO [main] 2015-02-06 11:30:56,608 StorageService.java (line 489) CQL
supported versions: 2.0.0,3.0.5 (default: 3.0.5)
INFO [main] 2015-02-06 11:30:56,633 StorageService.java (line 514) Loading
persisted ring state
INFO [main] 2015-02-06 11:30:56,652 StorageService.java (line 619) Starting up
server gossip
INFO [main] 2015-02-06 11:30:56,665 ColumnFamilyStore.java (line 632)
Enqueuing flush of Memtable-local@1392725193(252/2520 serialized/live bytes, 9
ops)
INFO [FlushWriter:1] 2015-02-06 11:30:56,666 Memtable.java (line 400) Writing
Memtable-local@1392725193(252/2520 serialized/live bytes, 9 ops)
INFO [FlushWriter:1] 2015-02-06 11:30:56,676 Memtable.java (line 438)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-ic-95-Data.db (243 bytes) for
commitlog position ReplayPosition(segmentId=1423251055622, position=52890)
INFO [CompactionExecutor:1] 2015-02-06 11:30:56,705 CompactionTask.java (line
105) Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-95-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-94-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-92-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-93-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ic-91-Data.db')]
INFO [CompactionExecutor:1] 2015-02-06 11:30:56,855 CompactionTask.java (line
262) Compacted 5 sstables to
[/var/lib/cassandra/data/system/local/system-local-ic-96,]. 1,008 bytes to 451
(~44% of original) in 148ms = 0.002906MB/s. 5 total rows, 1 unique. Row merge
counts were {1:0, 2:0, 3:0, 4:0, 5:1, }
INFO [main] 2015-02-06 11:30:56,905 MessagingService.java (line 455) Starting
Messaging Service on port 7000
INFO [main] 2015-02-06 11:30:56,944 StorageService.java (line 822) Using saved
token [2069380001304406395]
INFO [main] 2015-02-06 11:30:56,945 ColumnFamilyStore.java (line 632)
Enqueuing flush of Memtable-local@1528877937(84/840 serialized/live bytes, 4
ops)
INFO [FlushWriter:1] 2015-02-06 11:30:56,945 Memtable.java (line 400) Writing
Memtable-local@1528877937(84/840 serialized/live bytes, 4 ops)
INFO [FlushWriter:1] 2015-02-06 11:30:56,954 Memtable.java (line 438)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-ic-97-Data.db (120 bytes) for
commitlog position ReplayPosition(segmentId=1423251055622, position=53165)
INFO [main] 2015-02-06 11:30:56,961 ColumnFamilyStore.java (line 632)
Enqueuing flush of Memtable-local@702560738(50/500 serialized/live bytes, 2 ops)
INFO [FlushWriter:1] 2015-02-06 11:30:56,961 Memtable.java (line 400) Writing
Memtable-local@702560738(50/500 serialized/live bytes, 2 ops)
INFO [FlushWriter:1] 2015-02-06 11:30:56,970 Memtable.java (line 438)
Completed flushing
/var/lib/cassandra/data/system/local/system-local-ic-98-Data.db (109 bytes) for
commitlog position ReplayPosition(segmentId=1423251055622, position=53339)
INFO [main] 2015-02-06 11:30:56,983 StorageService.java (line 1383) Node
localhost/127.0.0.1 state jump to normal
INFO [main] 2015-02-06 11:30:56,987 StorageService.java (line 835) Startup
completed! Now serving reads.
INFO [main] 2015-02-06 11:30:57,134 Server.java (line 133) Starting listening
for CQL clients on localhost/127.0.0.1:9042...
INFO [main] 2015-02-06 11:30:57,150 ThriftServer.java (line 90) Binding thrift
service to localhost/127.0.0.1:9160
INFO [main] 2015-02-06 11:30:57,192 ThriftServer.java (line 102) Using
TFramedTransport with a max frame size of 15728640 bytes.
INFO [main] 2015-02-06 11:30:57,198 TServerCustomFactory.java (line 47) Using
synchronous/threadpool thrift server on localhost : 9160
INFO [Thread-2] 2015-02-06 11:30:57,198 ThriftServer.java (line 110) Listening
for thrift clients...
[homestead]cw@homestead-1:~$ sudo cat /var/log/homestead/homestead_current.txt
06-02-2015 19:26:04.749 UTC Status main.cpp:468: Log level set to 2
06-02-2015 19:26:04.749 UTC Status main.cpp:489: Access logging enabled to
/var/log/homestead
06-02-2015 19:26:04.750 UTC Status load_monitor.cpp:93: Constructing LoadMonitor
06-02-2015 19:26:04.750 UTC Status load_monitor.cpp:94: Target latency
(usecs) : 100000
06-02-2015 19:26:04.750 UTC Status load_monitor.cpp:95: Max bucket size
: 20
06-02-2015 19:26:04.750 UTC Status load_monitor.cpp:96: Initial token fill
rate/s: 10.000000
06-02-2015 19:26:04.750 UTC Status load_monitor.cpp:97: Min token fill
rate/s : 10.000000
06-02-2015 19:26:04.750 UTC Status dnscachedresolver.cpp:90: Creating Cached
Resolver using server 127.0.0.1
06-02-2015 19:26:04.750 UTC Status httpresolver.cpp:50: Created HTTP resolver
06-02-2015 19:26:04.750 UTC Status cassandra_store.cpp:145: Configuring store
06-02-2015 19:26:04.750 UTC Status cassandra_store.cpp:146: Hostname:
localhost
06-02-2015 19:26:04.750 UTC Status cassandra_store.cpp:147: Port: 9160
06-02-2015 19:26:04.750 UTC Status cassandra_store.cpp:148: Threads: 10
06-02-2015 19:26:04.750 UTC Status cassandra_store.cpp:149: Max Queue: 0
06-02-2015 19:26:04.750 UTC Status cassandra_store.cpp:199: Starting store
06-02-2015 19:26:04.752 UTC Error cassandra_store.cpp:217: Cache caught unknown
exception!
06-02-2015 19:26:04.752 UTC Error main.cpp:550: Failed to initialize cache - rc
5
06-02-2015 19:26:04.752 UTC Status cassandra_store.cpp:185: Stopping cache
06-02-2015 19:26:04.752 UTC Status cassandra_store.cpp:226: Waiting for cache
to stop
06-02-2015 19:30:48.463 UTC Status main.cpp:468: Log level set to 2
06-02-2015 19:30:48.464 UTC Status main.cpp:489: Access logging enabled to
/var/log/homestead
06-02-2015 19:30:48.466 UTC Status load_monitor.cpp:93: Constructing LoadMonitor
06-02-2015 19:30:48.466 UTC Status load_monitor.cpp:94: Target latency
(usecs) : 100000
06-02-2015 19:30:48.466 UTC Status load_monitor.cpp:95: Max bucket size
: 20
06-02-2015 19:30:48.466 UTC Status load_monitor.cpp:96: Initial token fill
rate/s: 10.000000
06-02-2015 19:30:48.466 UTC Status load_monitor.cpp:97: Min token fill
rate/s : 10.000000
06-02-2015 19:30:48.466 UTC Status dnscachedresolver.cpp:90: Creating Cached
Resolver using server 127.0.0.1
06-02-2015 19:30:48.467 UTC Status httpresolver.cpp:50: Created HTTP resolver
06-02-2015 19:30:48.467 UTC Status cassandra_store.cpp:145: Configuring store
06-02-2015 19:30:48.467 UTC Status cassandra_store.cpp:146: Hostname:
localhost
06-02-2015 19:30:48.467 UTC Status cassandra_store.cpp:147: Port: 9160
06-02-2015 19:30:48.467 UTC Status cassandra_store.cpp:148: Threads: 10
06-02-2015 19:30:48.467 UTC Status cassandra_store.cpp:149: Max Queue: 0
06-02-2015 19:30:48.467 UTC Status cassandra_store.cpp:199: Starting store
06-02-2015 19:30:48.480 UTC Error cassandra_store.cpp:207: Cache caught
TTransportException: connect() failed: Connection refused
06-02-2015 19:30:48.480 UTC Error main.cpp:550: Failed to initialize cache - rc
3
06-02-2015 19:30:48.480 UTC Status cassandra_store.cpp:185: Stopping cache
06-02-2015 19:30:48.480 UTC Status cassandra_store.cpp:226: Waiting for cache
to stop
06-02-2015 19:30:50.516 UTC Status main.cpp:468: Log level set to 2
06-02-2015 19:30:50.516 UTC Status main.cpp:489: Access logging enabled to
/var/log/homestead
06-02-2015 19:30:50.516 UTC Status load_monitor.cpp:93: Constructing LoadMonitor
06-02-2015 19:30:50.516 UTC Status load_monitor.cpp:94: Target latency
(usecs) : 100000
06-02-2015 19:30:50.516 UTC Status load_monitor.cpp:95: Max bucket size
: 20
06-02-2015 19:30:50.516 UTC Status load_monitor.cpp:96: Initial token fill
rate/s: 10.000000
06-02-2015 19:30:50.516 UTC Status load_monitor.cpp:97: Min token fill
rate/s : 10.000000
06-02-2015 19:30:50.516 UTC Status dnscachedresolver.cpp:90: Creating Cached
Resolver using server 127.0.0.1
06-02-2015 19:30:50.516 UTC Status httpresolver.cpp:50: Created HTTP resolver
06-02-2015 19:30:50.516 UTC Status cassandra_store.cpp:145: Configuring store
06-02-2015 19:30:50.516 UTC Status cassandra_store.cpp:146: Hostname:
localhost
06-02-2015 19:30:50.516 UTC Status cassandra_store.cpp:147: Port: 9160
06-02-2015 19:30:50.516 UTC Status cassandra_store.cpp:148: Threads: 10
06-02-2015 19:30:50.516 UTC Status cassandra_store.cpp:149: Max Queue: 0
06-02-2015 19:30:50.516 UTC Status cassandra_store.cpp:199: Starting store
06-02-2015 19:30:50.520 UTC Error cassandra_store.cpp:207: Cache caught
TTransportException: connect() failed: Connection refused
06-02-2015 19:30:50.522 UTC Error main.cpp:550: Failed to initialize cache - rc
3
06-02-2015 19:30:50.522 UTC Status cassandra_store.cpp:185: Stopping cache
06-02-2015 19:30:50.522 UTC Status cassandra_store.cpp:226: Waiting for cache
to stop
06-02-2015 19:31:03.024 UTC Status main.cpp:468: Log level set to 2
06-02-2015 19:31:03.024 UTC Status main.cpp:489: Access logging enabled to
/var/log/homestead
06-02-2015 19:31:03.025 UTC Status load_monitor.cpp:93: Constructing LoadMonitor
06-02-2015 19:31:03.025 UTC Status load_monitor.cpp:94: Target latency
(usecs) : 100000
06-02-2015 19:31:03.025 UTC Status load_monitor.cpp:95: Max bucket size
: 20
06-02-2015 19:31:03.025 UTC Status load_monitor.cpp:96: Initial token fill
rate/s: 10.000000
06-02-2015 19:31:03.025 UTC Status load_monitor.cpp:97: Min token fill
rate/s : 10.000000
06-02-2015 19:31:03.025 UTC Status dnscachedresolver.cpp:90: Creating Cached
Resolver using server 127.0.0.1
06-02-2015 19:31:03.025 UTC Status httpresolver.cpp:50: Created HTTP resolver
06-02-2015 19:31:03.025 UTC Status cassandra_store.cpp:145: Configuring store
06-02-2015 19:31:03.025 UTC Status cassandra_store.cpp:146: Hostname:
localhost
06-02-2015 19:31:03.025 UTC Status cassandra_store.cpp:147: Port: 9160
06-02-2015 19:31:03.025 UTC Status cassandra_store.cpp:148: Threads: 10
06-02-2015 19:31:03.025 UTC Status cassandra_store.cpp:149: Max Queue: 0
06-02-2015 19:31:03.025 UTC Status cassandra_store.cpp:199: Starting store
06-02-2015 19:31:03.034 UTC Error cassandra_store.cpp:217: Cache caught unknown
exception!
06-02-2015 19:31:03.034 UTC Error main.cpp:550: Failed to initialize cache - rc
5
06-02-2015 19:31:03.034 UTC Status cassandra_store.cpp:185: Stopping cache
06-02-2015 19:31:03.034 UTC Status cassandra_store.cpp:226: Waiting for cache
to stop
06-02-2015 19:31:43.448 UTC Status main.cpp:468: Log level set to 2
06-02-2015 19:31:43.448 UTC Status main.cpp:489: Access logging enabled to
/var/log/homestead
06-02-2015 19:31:43.448 UTC Status load_monitor.cpp:93: Constructing LoadMonitor
06-02-2015 19:31:43.448 UTC Status load_monitor.cpp:94: Target latency
(usecs) : 100000
06-02-2015 19:31:43.448 UTC Status load_monitor.cpp:95: Max bucket size
: 20
06-02-2015 19:31:43.448 UTC Status load_monitor.cpp:96: Initial token fill
rate/s: 10.000000
06-02-2015 19:31:43.448 UTC Status load_monitor.cpp:97: Min token fill
rate/s : 10.000000
06-02-2015 19:31:43.448 UTC Status dnscachedresolver.cpp:90: Creating Cached
Resolver using server 127.0.0.1
06-02-2015 19:31:43.448 UTC Status httpresolver.cpp:50: Created HTTP resolver
06-02-2015 19:31:43.448 UTC Status cassandra_store.cpp:145: Configuring store
06-02-2015 19:31:43.448 UTC Status cassandra_store.cpp:146: Hostname:
localhost
06-02-2015 19:31:43.448 UTC Status cassandra_store.cpp:147: Port: 9160
06-02-2015 19:31:43.448 UTC Status cassandra_store.cpp:148: Threads: 10
06-02-2015 19:31:43.448 UTC Status cassandra_store.cpp:149: Max Queue: 0
06-02-2015 19:31:43.448 UTC Status cassandra_store.cpp:199: Starting store
06-02-2015 19:31:43.450 UTC Error cassandra_store.cpp:217: Cache caught unknown
exception!
06-02-2015 19:31:43.450 UTC Error main.cpp:550: Failed to initialize cache - rc
5
06-02-2015 19:31:43.450 UTC Status cassandra_store.cpp:185: Stopping cache
06-02-2015 19:31:43.450 UTC Status cassandra_store.cpp:226: Waiting for cache
to stop
06-02-2015 19:32:23.677 UTC Status main.cpp:468: Log level set to 2
06-02-2015 19:32:23.677 UTC Status main.cpp:489: Access logging enabled to
/var/log/homestead
06-02-2015 19:32:23.678 UTC Status load_monitor.cpp:93: Constructing LoadMonitor
06-02-2015 19:32:23.678 UTC Status load_monitor.cpp:94: Target latency
(usecs) : 100000
06-02-2015 19:32:23.678 UTC Status load_monitor.cpp:95: Max bucket size
: 20
06-02-2015 19:32:23.678 UTC Status load_monitor.cpp:96: Initial token fill
rate/s: 10.000000
06-02-2015 19:32:23.678 UTC Status load_monitor.cpp:97: Min token fill
rate/s : 10.000000
06-02-2015 19:32:23.683 UTC Status dnscachedresolver.cpp:90: Creating Cached
Resolver using server 127.0.0.1
06-02-2015 19:32:23.684 UTC Status httpresolver.cpp:50: Created HTTP resolver
06-02-2015 19:32:23.684 UTC Status cassandra_store.cpp:145: Configuring store
06-02-2015 19:32:23.684 UTC Status cassandra_store.cpp:146: Hostname:
localhost
06-02-2015 19:32:23.684 UTC Status cassandra_store.cpp:147: Port: 9160
06-02-2015 19:32:23.684 UTC Status cassandra_store.cpp:148: Threads: 10
06-02-2015 19:32:23.684 UTC Status cassandra_store.cpp:149: Max Queue: 0
06-02-2015 19:32:23.684 UTC Status cassandra_store.cpp:199: Starting store
06-02-2015 19:32:23.685 UTC Error cassandra_store.cpp:217: Cache caught unknown
exception!
06-02-2015 19:32:23.685 UTC Error main.cpp:550: Failed to initialize cache - rc
5
06-02-2015 19:32:23.685 UTC Status cassandra_store.cpp:185: Stopping cache
06-02-2015 19:32:23.685 UTC Status cassandra_store.cpp:226: Waiting for cache
to stop
06-02-2015 19:33:04.028 UTC Status main.cpp:468: Log level set to 2
06-02-2015 19:33:04.028 UTC Status main.cpp:489: Access logging enabled to
/var/log/homestead
06-02-2015 19:33:04.028 UTC Status load_monitor.cpp:93: Constructing LoadMonitor
06-02-2015 19:33:04.028 UTC Status load_monitor.cpp:94: Target latency
(usecs) : 100000
06-02-2015 19:33:04.028 UTC Status load_monitor.cpp:95: Max bucket size
: 20
06-02-2015 19:33:04.028 UTC Status load_monitor.cpp:96: Initial token fill
rate/s: 10.000000
06-02-2015 19:33:04.028 UTC Status load_monitor.cpp:97: Min token fill
rate/s : 10.000000
06-02-2015 19:33:04.028 UTC Status dnscachedresolver.cpp:90: Creating Cached
Resolver using server 127.0.0.1
06-02-2015 19:33:04.028 UTC Status httpresolver.cpp:50: Created HTTP resolver
06-02-2015 19:33:04.028 UTC Status cassandra_store.cpp:145: Configuring store
06-02-2015 19:33:04.028 UTC Status cassandra_store.cpp:146: Hostname:
localhost
06-02-2015 19:33:04.028 UTC Status cassandra_store.cpp:147: Port: 9160
06-02-2015 19:33:04.028 UTC Status cassandra_store.cpp:148: Threads: 10
06-02-2015 19:33:04.028 UTC Status cassandra_store.cpp:149: Max Queue: 0
06-02-2015 19:33:04.028 UTC Status cassandra_store.cpp:199: Starting store
06-02-2015 19:33:04.030 UTC Error cassandra_store.cpp:217: Cache caught unknown
exception!
06-02-2015 19:33:04.030 UTC Error main.cpp:550: Failed to initialize cache - rc
5
06-02-2015 19:33:04.030 UTC Status cassandra_store.cpp:185: Stopping cache
06-02-2015 19:33:04.030 UTC Status cassandra_store.cpp:226: Waiting for cache
to stop
06-02-2015 19:33:44.356 UTC Status main.cpp:468: Log level set to 2
06-02-2015 19:33:44.356 UTC Status main.cpp:489: Access logging enabled to
/var/log/homestead
06-02-2015 19:33:44.357 UTC Status load_monitor.cpp:93: Constructing LoadMonitor
06-02-2015 19:33:44.357 UTC Status load_monitor.cpp:94: Target latency
(usecs) : 100000
06-02-2015 19:33:44.357 UTC Status load_monitor.cpp:95: Max bucket size
: 20
06-02-2015 19:33:44.357 UTC Status load_monitor.cpp:96: Initial token fill
rate/s: 10.000000
06-02-2015 19:33:44.357 UTC Status load_monitor.cpp:97: Min token fill
rate/s : 10.000000
06-02-2015 19:33:44.357 UTC Status dnscachedresolver.cpp:90: Creating Cached
Resolver using server 127.0.0.1
06-02-2015 19:33:44.357 UTC Status httpresolver.cpp:50: Created HTTP resolver
06-02-2015 19:33:44.357 UTC Status cassandra_store.cpp:145: Configuring store
06-02-2015 19:33:44.357 UTC Status cassandra_store.cpp:146: Hostname:
localhost
06-02-2015 19:33:44.357 UTC Status cassandra_store.cpp:147: Port: 9160
06-02-2015 19:33:44.357 UTC Status cassandra_store.cpp:148: Threads: 10
06-02-2015 19:33:44.357 UTC Status cassandra_store.cpp:149: Max Queue: 0
06-02-2015 19:33:44.357 UTC Status cassandra_store.cpp:199: Starting store
06-02-2015 19:33:44.359 UTC Error cassandra_store.cpp:217: Cache caught unknown
exception!
06-02-2015 19:33:44.359 UTC Error main.cpp:550: Failed to initialize cache - rc
5
06-02-2015 19:33:44.359 UTC Status cassandra_store.cpp:185: Stopping cache
06-02-2015 19:33:44.359 UTC Status cassandra_store.cpp:226: Waiting for cache
to stop
06-02-2015 19:34:24.672 UTC Status main.cpp:468: Log level set to 2
06-02-2015 19:34:24.672 UTC Status main.cpp:489: Access logging enabled to
/var/log/homestead
06-02-2015 19:34:24.673 UTC Status load_monitor.cpp:93: Constructing LoadMonitor
06-02-2015 19:34:24.673 UTC Status load_monitor.cpp:94: Target latency
(usecs) : 100000
06-02-2015 19:34:24.673 UTC Status load_monitor.cpp:95: Max bucket size
: 20
06-02-2015 19:34:24.673 UTC Status load_monitor.cpp:96: Initial token fill
rate/s: 10.000000
06-02-2015 19:34:24.673 UTC Status load_monitor.cpp:97: Min token fill
rate/s : 10.000000
06-02-2015 19:34:24.673 UTC Status dnscachedresolver.cpp:90: Creating Cached
Resolver using server 127.0.0.1
06-02-2015 19:34:24.673 UTC Status httpresolver.cpp:50: Created HTTP resolver
06-02-2015 19:34:24.673 UTC Status cassandra_store.cpp:145: Configuring store
06-02-2015 19:34:24.673 UTC Status cassandra_store.cpp:146: Hostname:
localhost
06-02-2015 19:34:24.673 UTC Status cassandra_store.cpp:147: Port: 9160
06-02-2015 19:34:24.673 UTC Status cassandra_store.cpp:148: Threads: 10
06-02-2015 19:34:24.673 UTC Status cassandra_store.cpp:149: Max Queue: 0
06-02-2015 19:34:24.673 UTC Status cassandra_store.cpp:199: Starting store
06-02-2015 19:34:24.676 UTC Error cassandra_store.cpp:217: Cache caught unknown
exception!
06-02-2015 19:34:24.676 UTC Error main.cpp:550: Failed to initialize cache - rc
5
06-02-2015 19:34:24.676 UTC Status cassandra_store.cpp:185: Stopping cache
06-02-2015 19:34:24.676 UTC Status cassandra_store.cpp:226: Waiting for cache
to stop
06-02-2015 19:35:05.028 UTC Status main.cpp:468: Log level set to 2
06-02-2015 19:35:05.028 UTC Status main.cpp:489: Access logging enabled to
/var/log/homestead
06-02-2015 19:35:05.029 UTC Status load_monitor.cpp:93: Constructing LoadMonitor
06-02-2015 19:35:05.029 UTC Status load_monitor.cpp:94: Target latency
(usecs) : 100000
06-02-2015 19:35:05.029 UTC Status load_monitor.cpp:95: Max bucket size
: 20
06-02-2015 19:35:05.029 UTC Status load_monitor.cpp:96: Initial token fill
rate/s: 10.000000
06-02-2015 19:35:05.029 UTC Status load_monitor.cpp:97: Min token fill
rate/s : 10.000000
06-02-2015 19:35:05.029 UTC Status dnscachedresolver.cpp:90: Creating Cached
Resolver using server 127.0.0.1
06-02-2015 19:35:05.029 UTC Status httpresolver.cpp:50: Created HTTP resolver
06-02-2015 19:35:05.029 UTC Status cassandra_store.cpp:145: Configuring store
06-02-2015 19:35:05.029 UTC Status cassandra_store.cpp:146: Hostname:
localhost
06-02-2015 19:35:05.029 UTC Status cassandra_store.cpp:147: Port: 9160
06-02-2015 19:35:05.029 UTC Status cassandra_store.cpp:148: Threads: 10
06-02-2015 19:35:05.029 UTC Status cassandra_store.cpp:149: Max Queue: 0
06-02-2015 19:35:05.029 UTC Status cassandra_store.cpp:199: Starting store
06-02-2015 19:35:05.031 UTC Error cassandra_store.cpp:217: Cache caught unknown
exception!
06-02-2015 19:35:05.031 UTC Error main.cpp:550: Failed to initialize cache - rc
5
06-02-2015 19:35:05.031 UTC Status cassandra_store.cpp:185: Stopping cache
06-02-2015 19:35:05.031 UTC Status cassandra_store.cpp:226: Waiting for cache
to stop
[homestead]cw@homestead-1:~$ sudo cat
/var/log/homestead-prov/homestead-prov-err.log
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File
"/usr/share/clearwater/homestead/env/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/main.py",
line 156, in <module>
standalone()
File
"/usr/share/clearwater/homestead/env/lib/python2.7/site-packages/crest-0.1-py2.7.egg/metaswitch/crest/main.py",
line 119, in standalone
reactor.listenUNIX(unix_sock_name, application)
File
"/usr/share/clearwater/homestead/env/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/internet/posixbase.py",
line 413, in listenUNIX
p.startListening()
File
"/usr/share/clearwater/homestead/env/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-linux-x86_64.egg/twisted/internet/unix.py",
line 293, in startListening
raise CannotListenError, (None, self.port, le)
twisted.internet.error.CannotListenError: Couldn't listen on
any:/tmp/.homestead-prov-sock-0: [Errno 98] Address already in use.
[homestead]cw@homestead-1:~$ sudo cat
/var/log/homestead-prov/homestead-prov-0.log
2015-02-06 19:30:52,617 UTC INFO main:118 Going to listen for HTTP on UNIX
socket /tmp/.homestead-prov-sock-0
2015-02-06 19:30:53,167 UTC INFO main:118 Going to listen for HTTP on UNIX
socket /tmp/.homestead-prov-sock-0
2015-02-06 19:31:03,913 UTC INFO main:118 Going to listen for HTTP on UNIX
socket /tmp/.homestead-prov-sock-0
2015-02-06 19:31:44,284 UTC INFO main:118 Going to listen for HTTP on UNIX
socket /tmp/.homestead-prov-sock-0
2015-02-06 19:32:24,541 UTC INFO main:118 Going to listen for HTTP on UNIX
socket /tmp/.homestead-prov-sock-0
2015-02-06 19:33:04,912 UTC INFO main:118 Going to listen for HTTP on UNIX
socket /tmp/.homestead-prov-sock-0
2015-02-06 19:33:45,238 UTC INFO main:118 Going to listen for HTTP on UNIX
socket /tmp/.homestead-prov-sock-0
2015-02-06 19:34:25,422 UTC INFO main:118 Going to listen for HTTP on UNIX
socket /tmp/.homestead-prov-sock-0
2015-02-06 19:35:05,886 UTC INFO main:118 Going to listen for HTTP on UNIX
socket /tmp/.homestead-prov-sock-0
2015-02-06 19:35:46,247 UTC INFO main:118 Going to listen for HTTP on UNIX
socket /tmp/.homestead-prov-sock-0
2015-02-06 19:36:26,570 UTC INFO main:118 Going to listen for HTTP on UNIX
socket /tmp/.homestead-prov-sock-0
_______________________________________________
Clearwater mailing list
[email protected]
http://lists.projectclearwater.org/listinfo/clearwater
