Re: run cassandra on a small instance
You might also have some gains setting in_memory_compaction_limit_in_mb
to something very low to force Cassandra to use on disk compaction rather
than doing it in memory.
Cool Ben.. thanks I'll add that to my config as well.
Glad that helped. Thanks for reporting back!
No problem, Nate! That's the least I can do. All I can hope is that this
thread adds to the overall fund of knowledge for the list.
Cheers,
Tim
On Mon, Feb 23, 2015 at 11:46 AM, Nate McCall n...@thelastpickle.com
wrote:
Glad that helped. Thanks for reporting back!
On Sun, Feb 22, 2015 at 9:12 PM, Tim Dunphy bluethu...@gmail.com wrote:
Nate,
Definitely thank you for this advice. After leaving the new Cassandra
node running on the 2GB instance for the past couple of days, I think I've
had ample reason to report complete success in getting it stabilized on
that instance! Here are the changes I've been able to make:
I think manipulating the key cache and other stuff like concurrent
writes and some of the other stuff I worked on based on that thread from
the cassandra list definitely was key in getting Cassandra to work on the
new instance.
Check out the before and after (before working/ after working):
Before in cassandra-env.sh:
MAX_HEAP_SIZE=800M
HEAP_NEWSIZE=200M
After:
MAX_HEAP_SIZE=512M
HEAP_NEWSIZE=100M
And before in the cassandra.yaml file:
concurrent_writes: 32
compaction_throughput_mb_per_sec: 16
key_cache_size_in_mb:
key_cache_save_period: 14400
# native_transport_max_threads: 128
And after:
concurrent_writes: 2
compaction_throughput_mb_per_sec: 8
key_cache_size_in_mb: 4
key_cache_save_period: 0
native_transport_max_threads: 4
That really made the difference. I'm a puppet user, so these changes are
in puppet. So any new 2GB instances I should bring up on Digital Ocean
should absolutely work the way the first 2GB node does, there. But I was
able to make enough sense of your chef recipe to adapt what you were
showing me.
Thanks again!
Tim
On Fri, Feb 20, 2015 at 10:31 PM, Tim Dunphy bluethu...@gmail.com
wrote:
The most important things to note:
- don't include JNA (it needs to lock pages larger than what will be
available)
- turn down threadpools for transports
- turn compaction throughput way down
- make concurrent reads and writes very small
I have used the above run a healthy 5 node clusters locally in it's own
private network with a 6th monitoring server for light to moderate local
testing in 16g of laptop ram. YMMV but it is possible.
Thanks!! That was very helpful. I just tried applying your suggestions
to my cassandra.yaml file. I used the info from your chef recipe. Well like
I've been saying typically it takes about 5 hours or so for this situation
to shake itself out. I'll provide an update to the list once I have a
better idea of how this is working.
Thanks again!
Tim
On Fri, Feb 20, 2015 at 9:37 PM, Nate McCall n...@thelastpickle.com
wrote:
I frequently test with multi-node vagrant-based clusters locally. The
following chef attributes should give you an idea of what to turn down in
cassandra.yaml and cassandra-env.sh to build a decent testing cluster:
:cassandra = {'cluster_name' = 'VerifyCluster',
'package_name' = 'dsc20',
'version' = '2.0.11',
'release' = '1',
'setup_jna' = false,
'max_heap_size' = '512M',
'heap_new_size' = '100M',
'initial_token' = server['initial_token'],
'seeds' = 192.168.33.10,
'listen_address' = server['ip'],
'broadcast_address' = server['ip'],
'rpc_address' = server['ip'],
'conconcurrent_reads' = 2,
'concurrent_writes' = 2,
'memtable_flush_queue_size' = 2,
'compaction_throughput_mb_per_sec' = 8,
'key_cache_size_in_mb' = 4,
'key_cache_save_period' = 0,
'native_transport_min_threads' = 2,
'native_transport_max_threads' = 4,
'notify_restart' = true,
'reporter' = {
'riemann' = {
'enable' = true,
'host' = '192.168.33.51'
},
'graphite' = {
'enable' = true,
'host' = '192.168.33.51'
}
}
},
The most important things to note:
- don't include JNA (it needs to lock pages larger than what will be
available)
- turn down threadpools for
Re: run cassandra on a small instance
You might also have some gains setting in_memory_compaction_limit_in_mb to
something very low to force Cassandra to use on disk compaction rather than
doing it in memory.
On 23 February 2015 at 14:12, Tim Dunphy bluethu...@gmail.com wrote:
Nate,
Definitely thank you for this advice. After leaving the new Cassandra
node running on the 2GB instance for the past couple of days, I think I've
had ample reason to report complete success in getting it stabilized on
that instance! Here are the changes I've been able to make:
I think manipulating the key cache and other stuff like concurrent writes
and some of the other stuff I worked on based on that thread from the
cassandra list definitely was key in getting Cassandra to work on the new
instance.
Check out the before and after (before working/ after working):
Before in cassandra-env.sh:
MAX_HEAP_SIZE=800M
HEAP_NEWSIZE=200M
After:
MAX_HEAP_SIZE=512M
HEAP_NEWSIZE=100M
And before in the cassandra.yaml file:
concurrent_writes: 32
compaction_throughput_mb_per_sec: 16
key_cache_size_in_mb:
key_cache_save_period: 14400
# native_transport_max_threads: 128
And after:
concurrent_writes: 2
compaction_throughput_mb_per_sec: 8
key_cache_size_in_mb: 4
key_cache_save_period: 0
native_transport_max_threads: 4
That really made the difference. I'm a puppet user, so these changes are
in puppet. So any new 2GB instances I should bring up on Digital Ocean
should absolutely work the way the first 2GB node does, there. But I was
able to make enough sense of your chef recipe to adapt what you were
showing me.
Thanks again!
Tim
On Fri, Feb 20, 2015 at 10:31 PM, Tim Dunphy bluethu...@gmail.com wrote:
The most important things to note:
- don't include JNA (it needs to lock pages larger than what will be
available)
- turn down threadpools for transports
- turn compaction throughput way down
- make concurrent reads and writes very small
I have used the above run a healthy 5 node clusters locally in it's own
private network with a 6th monitoring server for light to moderate local
testing in 16g of laptop ram. YMMV but it is possible.
Thanks!! That was very helpful. I just tried applying your suggestions to
my cassandra.yaml file. I used the info from your chef recipe. Well like
I've been saying typically it takes about 5 hours or so for this situation
to shake itself out. I'll provide an update to the list once I have a
better idea of how this is working.
Thanks again!
Tim
On Fri, Feb 20, 2015 at 9:37 PM, Nate McCall n...@thelastpickle.com
wrote:
I frequently test with multi-node vagrant-based clusters locally. The
following chef attributes should give you an idea of what to turn down in
cassandra.yaml and cassandra-env.sh to build a decent testing cluster:
:cassandra = {'cluster_name' = 'VerifyCluster',
'package_name' = 'dsc20',
'version' = '2.0.11',
'release' = '1',
'setup_jna' = false,
'max_heap_size' = '512M',
'heap_new_size' = '100M',
'initial_token' = server['initial_token'],
'seeds' = 192.168.33.10,
'listen_address' = server['ip'],
'broadcast_address' = server['ip'],
'rpc_address' = server['ip'],
'conconcurrent_reads' = 2,
'concurrent_writes' = 2,
'memtable_flush_queue_size' = 2,
'compaction_throughput_mb_per_sec' = 8,
'key_cache_size_in_mb' = 4,
'key_cache_save_period' = 0,
'native_transport_min_threads' = 2,
'native_transport_max_threads' = 4,
'notify_restart' = true,
'reporter' = {
'riemann' = {
'enable' = true,
'host' = '192.168.33.51'
},
'graphite' = {
'enable' = true,
'host' = '192.168.33.51'
}
}
},
The most important things to note:
- don't include JNA (it needs to lock pages larger than what will be
available)
- turn down threadpools for transports
- turn compaction throughput way down
- make concurrent reads and writes very small
I have used the above run a healthy 5 node clusters locally in it's own
private network with a 6th monitoring server for light to moderate local
testing in 16g of laptop ram. YMMV but it is possible.
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
--
Re: run cassandra on a small instance
Nate,
Definitely thank you for this advice. After leaving the new Cassandra node
running on the 2GB instance for the past couple of days, I think I've had
ample reason to report complete success in getting it stabilized on that
instance! Here are the changes I've been able to make:
I think manipulating the key cache and other stuff like concurrent writes
and some of the other stuff I worked on based on that thread from the
cassandra list definitely was key in getting Cassandra to work on the new
instance.
Check out the before and after (before working/ after working):
Before in cassandra-env.sh:
MAX_HEAP_SIZE=800M
HEAP_NEWSIZE=200M
After:
MAX_HEAP_SIZE=512M
HEAP_NEWSIZE=100M
And before in the cassandra.yaml file:
concurrent_writes: 32
compaction_throughput_mb_per_sec: 16
key_cache_size_in_mb:
key_cache_save_period: 14400
# native_transport_max_threads: 128
And after:
concurrent_writes: 2
compaction_throughput_mb_per_sec: 8
key_cache_size_in_mb: 4
key_cache_save_period: 0
native_transport_max_threads: 4
That really made the difference. I'm a puppet user, so these changes are in
puppet. So any new 2GB instances I should bring up on Digital Ocean should
absolutely work the way the first 2GB node does, there. But I was able to
make enough sense of your chef recipe to adapt what you were showing me.
Thanks again!
Tim
On Fri, Feb 20, 2015 at 10:31 PM, Tim Dunphy bluethu...@gmail.com wrote:
The most important things to note:
- don't include JNA (it needs to lock pages larger than what will be
available)
- turn down threadpools for transports
- turn compaction throughput way down
- make concurrent reads and writes very small
I have used the above run a healthy 5 node clusters locally in it's own
private network with a 6th monitoring server for light to moderate local
testing in 16g of laptop ram. YMMV but it is possible.
Thanks!! That was very helpful. I just tried applying your suggestions to
my cassandra.yaml file. I used the info from your chef recipe. Well like
I've been saying typically it takes about 5 hours or so for this situation
to shake itself out. I'll provide an update to the list once I have a
better idea of how this is working.
Thanks again!
Tim
On Fri, Feb 20, 2015 at 9:37 PM, Nate McCall n...@thelastpickle.com
wrote:
I frequently test with multi-node vagrant-based clusters locally. The
following chef attributes should give you an idea of what to turn down in
cassandra.yaml and cassandra-env.sh to build a decent testing cluster:
:cassandra = {'cluster_name' = 'VerifyCluster',
'package_name' = 'dsc20',
'version' = '2.0.11',
'release' = '1',
'setup_jna' = false,
'max_heap_size' = '512M',
'heap_new_size' = '100M',
'initial_token' = server['initial_token'],
'seeds' = 192.168.33.10,
'listen_address' = server['ip'],
'broadcast_address' = server['ip'],
'rpc_address' = server['ip'],
'conconcurrent_reads' = 2,
'concurrent_writes' = 2,
'memtable_flush_queue_size' = 2,
'compaction_throughput_mb_per_sec' = 8,
'key_cache_size_in_mb' = 4,
'key_cache_save_period' = 0,
'native_transport_min_threads' = 2,
'native_transport_max_threads' = 4,
'notify_restart' = true,
'reporter' = {
'riemann' = {
'enable' = true,
'host' = '192.168.33.51'
},
'graphite' = {
'enable' = true,
'host' = '192.168.33.51'
}
}
},
The most important things to note:
- don't include JNA (it needs to lock pages larger than what will be
available)
- turn down threadpools for transports
- turn compaction throughput way down
- make concurrent reads and writes very small
I have used the above run a healthy 5 node clusters locally in it's own
private network with a 6th monitoring server for light to moderate local
testing in 16g of laptop ram. YMMV but it is possible.
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
Re: run cassandra on a small instance
The most important things to note:
- don't include JNA (it needs to lock pages larger than what will be
available)
- turn down threadpools for transports
- turn compaction throughput way down
- make concurrent reads and writes very small
I have used the above run a healthy 5 node clusters locally in it's own
private network with a 6th monitoring server for light to moderate local
testing in 16g of laptop ram. YMMV but it is possible.
Thanks!! That was very helpful. I just tried applying your suggestions to
my cassandra.yaml file. I used the info from your chef recipe. Well like
I've been saying typically it takes about 5 hours or so for this situation
to shake itself out. I'll provide an update to the list once I have a
better idea of how this is working.
Thanks again!
Tim
On Fri, Feb 20, 2015 at 9:37 PM, Nate McCall n...@thelastpickle.com wrote:
I frequently test with multi-node vagrant-based clusters locally. The
following chef attributes should give you an idea of what to turn down in
cassandra.yaml and cassandra-env.sh to build a decent testing cluster:
:cassandra = {'cluster_name' = 'VerifyCluster',
'package_name' = 'dsc20',
'version' = '2.0.11',
'release' = '1',
'setup_jna' = false,
'max_heap_size' = '512M',
'heap_new_size' = '100M',
'initial_token' = server['initial_token'],
'seeds' = 192.168.33.10,
'listen_address' = server['ip'],
'broadcast_address' = server['ip'],
'rpc_address' = server['ip'],
'conconcurrent_reads' = 2,
'concurrent_writes' = 2,
'memtable_flush_queue_size' = 2,
'compaction_throughput_mb_per_sec' = 8,
'key_cache_size_in_mb' = 4,
'key_cache_save_period' = 0,
'native_transport_min_threads' = 2,
'native_transport_max_threads' = 4,
'notify_restart' = true,
'reporter' = {
'riemann' = {
'enable' = true,
'host' = '192.168.33.51'
},
'graphite' = {
'enable' = true,
'host' = '192.168.33.51'
}
}
},
The most important things to note:
- don't include JNA (it needs to lock pages larger than what will be
available)
- turn down threadpools for transports
- turn compaction throughput way down
- make concurrent reads and writes very small
I have used the above run a healthy 5 node clusters locally in it's own
private network with a 6th monitoring server for light to moderate local
testing in 16g of laptop ram. YMMV but it is possible.
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
Re: run cassandra on a small instance
What does your schema look like, your total data size and your read/write
patterns? Maybe you are simply doing a heavier workload than a small
instance can handle.
Hi Mark,
OK well as mentioned this is all test data with almost literally no
workload. So I doubt it's the data and/ or workload that's causing it to
crash on the 2GB instance after 5 hours.
But when I describe the schema with my test data this is what I see:
cqlsh use joke_fire1
... ;
cqlsh:joke_fire1 describe schema;
CREATE KEYSPACE joke_fire1 WITH replication = {'class': 'SimpleStrategy',
'replication_factor': '3'} AND durable_writes = true;
'module' object has no attribute 'UserTypesMeta'
If I take a look at the size of the total amount of data this is what I see:
[root@beta-new:/etc/alternatives/cassandrahome/data] #du -hs data
17M data
Which includes the system keyspace. But the test data that I created for my
use is only 15MB:
[root@beta-new:/etc/alternatives/cassandrahome/data/data] #du -hs
joke_fire1/
15M joke_fire1/
But just to see if it's my data that could be causing the problem, I tried
removing it all, and setting the IP of the 2GB instance itself as the seed
node. I'll try running that for a while and seeing if it crashes.
Also I tried just installing a plain cassandra 2.1.3 onto a plain CentOS
6.6 instance on the AWS free tier. It's a t.2 micro instance. So far it's
running. I'll keep an eye on both. At this point, I'm thinking that there
might be something about my data that could be causing it to fail after 5
or so hours.
However I might need some help diagnosing the data, as I'm not familiar on
how to do that with cassandra.
Thanks!
Tim
On Thu, Feb 19, 2015 at 3:51 AM, Mark Reddy mark.l.re...@gmail.com wrote:
What does your schema look like, your total data size and your read/write
patterns? Maybe you are simply doing a heavier workload than a small
instance can handle.
Regards,
Mark
On 19 February 2015 at 08:40, Carlos Rolo r...@pythian.com wrote:
I have Cassandra instances running on VMs with smaller RAM (1GB even) and
I don't go OOM when testing them. Although I use them in AWS and other
providers, never tried Digital Ocean.
Does Cassandra just fails after some time running or it is failing on
some specific read/write?
Regards,
Carlos Juzarte Rolo
Cassandra Consultant
Pythian - Love your data
rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
http://linkedin.com/in/carlosjuzarterolo*
Tel: 1649
www.pythian.com
On Thu, Feb 19, 2015 at 7:16 AM, Tim Dunphy bluethu...@gmail.com wrote:
Hey guys,
After the upgrade to 2.1.3, and after almost exactly 5 hours running
cassandra did indeed crash again on the 2GB ram VM.
This is how the memory on the VM looked after the crash:
[root@web2:~] #free -m
total used free sharedbuffers cached
Mem: 2002 1227774 8 45386
-/+ buffers/cache:794 1207
Swap:0 0 0
And that's with this set in the cassandra-env.sh file:
MAX_HEAP_SIZE=800M
HEAP_NEWSIZE=200M
So I'm thinking now, do I just have to abandon this idea I have of
running Cassandra on a 2GB instance? Or is this something we can all agree
can be done? And if so, how can we do that? :)
Thanks
Tim
On Wed, Feb 18, 2015 at 8:39 PM, Jason Kushmaul | WDA
jason.kushm...@wda.com wrote:
I asked this previously when a similar message came through, with a
similar response.
planetcassandra seems to have it “right”, in that stable=2.0,
development=2.1, whereas the apache site says stable is 2.1.
“Right” in they assume latest minor version is development. Why not
have the apache site do the same? That’s just my lowly non-contributing
opinion though.
*Jason *
*From:* Andrew [mailto:redmu...@gmail.com]
*Sent:* Wednesday, February 18, 2015 8:26 PM
*To:* Robert Coli; user@cassandra.apache.org
*Subject:* Re: run cassandra on a small instance
Robert,
Let me know if I’m off base about this—but I feel like I see a lot of
posts that are like this (i.e., use this arbitrary version, not this other
arbitrary version). Why are releases going out if they’re “broken”? This
seems like a very confusing way for new (and existing) users to approach
versions...
Andrew
On February 18, 2015 at 5:16:27 PM, Robert Coli (rc...@eventbrite.com)
wrote:
On Wed, Feb 18, 2015 at 5:09 PM, Tim Dunphy bluethu...@gmail.com
wrote:
I'm attempting to run Cassandra 2.1.2 on a smallish 2.GB ram instance
over at Digital Ocean. It's a CentOS 7 host.
2.1.2 is IMO broken and should not be used for any purpose.
Use 2.1.1 or 2.1.3.
https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/
=Rob
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
--
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
Re: run cassandra on a small instance
On Wed, Feb 18, 2015 at 5:26 PM, Andrew redmu...@gmail.com wrote: Let me know if I’m off base about this—but I feel like I see a lot of posts that are like this (i.e., use this arbitrary version, not this other arbitrary version). Why are releases going out if they’re “broken”? This seems like a very confusing way for new (and existing) users to approach versions... In my opinion and in no way speaking for or representing Apache Cassandra, Datastax, or anyone else : I think it's a problem of messaging, and a mismatch of expectations between the development team and operators. I think the stable versions are stable by the dev team's standards, and not by operators' standards. While testing has historically been IMO insufficient for a data-store (where correctness really matters) there are also various issues which probably can not realistically be detected in testing. Of course, operators need to be willing to operate (ideally in non-production) near the cutting edge in order to assist in the detection and resolution of these bugs, but I think the project does itself a disservice by encouraging noobs to run these versions. You only get one chance to make a first impression, as the saying goes. My ideal messaging would probably say something like versions near the cutting edge should be treated cautiously, conservative operators should run mature point releases in production and only upgrade to near the cutting edge after extended burn-in in dev/QA/stage environments. A fair response to this critique is that operators should know better than to trust that x.y.0-5 release versions of any open source software are likely to be production ready, even if the website says stable next to the download. Trust, but verify? =Rob
Re: run cassandra on a small instance
I have Cassandra instances running on VMs with smaller RAM (1GB even) and I don't go OOM when testing them. Although I use them in AWS and other providers, never tried Digital Ocean. Does Cassandra just fails after some time running or it is failing on some specific read/write? Hi Carlos, Ok, that's really interesting. So I have to ask, did you have to do anything special to get Cassandra to run on those 1GB AWS instances? I'd love to do the same. I even tried there as well and failed due to lack of memory to run it. And there is no specific reason other than lack of memory that I can tell for it to fail. And it doesn's seem to matter what data I use either. Because even if I remove the data directory with rm -rf, the phenomenon is the same. It'll run for a while, usually about 5 hours and then just crash with the word 'killed' as the last line of output. Thanks Tim On Thu, Feb 19, 2015 at 3:40 AM, Carlos Rolo r...@pythian.com wrote: I have Cassandra instances running on VMs with smaller RAM (1GB even) and I don't go OOM when testing them. Although I use them in AWS and other providers, never tried Digital Ocean. Does Cassandra just fails after some time running or it is failing on some specific read/write? Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Feb 19, 2015 at 7:16 AM, Tim Dunphy bluethu...@gmail.com wrote: Hey guys, After the upgrade to 2.1.3, and after almost exactly 5 hours running cassandra did indeed crash again on the 2GB ram VM. This is how the memory on the VM looked after the crash: [root@web2:~] #free -m total used free sharedbuffers cached Mem: 2002 1227774 8 45386 -/+ buffers/cache:794 1207 Swap:0 0 0 And that's with this set in the cassandra-env.sh file: MAX_HEAP_SIZE=800M HEAP_NEWSIZE=200M So I'm thinking now, do I just have to abandon this idea I have of running Cassandra on a 2GB instance? Or is this something we can all agree can be done? And if so, how can we do that? :) Thanks Tim On Wed, Feb 18, 2015 at 8:39 PM, Jason Kushmaul | WDA jason.kushm...@wda.com wrote: I asked this previously when a similar message came through, with a similar response. planetcassandra seems to have it “right”, in that stable=2.0, development=2.1, whereas the apache site says stable is 2.1. “Right” in they assume latest minor version is development. Why not have the apache site do the same? That’s just my lowly non-contributing opinion though. *Jason * *From:* Andrew [mailto:redmu...@gmail.com] *Sent:* Wednesday, February 18, 2015 8:26 PM *To:* Robert Coli; user@cassandra.apache.org *Subject:* Re: run cassandra on a small instance Robert, Let me know if I’m off base about this—but I feel like I see a lot of posts that are like this (i.e., use this arbitrary version, not this other arbitrary version). Why are releases going out if they’re “broken”? This seems like a very confusing way for new (and existing) users to approach versions... Andrew On February 18, 2015 at 5:16:27 PM, Robert Coli (rc...@eventbrite.com) wrote: On Wed, Feb 18, 2015 at 5:09 PM, Tim Dunphy bluethu...@gmail.com wrote: I'm attempting to run Cassandra 2.1.2 on a smallish 2.GB ram instance over at Digital Ocean. It's a CentOS 7 host. 2.1.2 is IMO broken and should not be used for any purpose. Use 2.1.1 or 2.1.3. https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ =Rob -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B -- -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
Re: run cassandra on a small instance
What I normally do is install plain CentOS (Not any AMI build for Cassandra) and I don't use them for production! I run them for testing, fire drills and some cassandra-stress benchmarks. I will look if I had more than 5h Cassandra uptime. I can even put one up now and do the test and get the results back to you. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Feb 19, 2015 at 6:41 PM, Tim Dunphy bluethu...@gmail.com wrote: I have Cassandra instances running on VMs with smaller RAM (1GB even) and I don't go OOM when testing them. Although I use them in AWS and other providers, never tried Digital Ocean. Does Cassandra just fails after some time running or it is failing on some specific read/write? Hi Carlos, Ok, that's really interesting. So I have to ask, did you have to do anything special to get Cassandra to run on those 1GB AWS instances? I'd love to do the same. I even tried there as well and failed due to lack of memory to run it. And there is no specific reason other than lack of memory that I can tell for it to fail. And it doesn's seem to matter what data I use either. Because even if I remove the data directory with rm -rf, the phenomenon is the same. It'll run for a while, usually about 5 hours and then just crash with the word 'killed' as the last line of output. Thanks Tim On Thu, Feb 19, 2015 at 3:40 AM, Carlos Rolo r...@pythian.com wrote: I have Cassandra instances running on VMs with smaller RAM (1GB even) and I don't go OOM when testing them. Although I use them in AWS and other providers, never tried Digital Ocean. Does Cassandra just fails after some time running or it is failing on some specific read/write? Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Feb 19, 2015 at 7:16 AM, Tim Dunphy bluethu...@gmail.com wrote: Hey guys, After the upgrade to 2.1.3, and after almost exactly 5 hours running cassandra did indeed crash again on the 2GB ram VM. This is how the memory on the VM looked after the crash: [root@web2:~] #free -m total used free sharedbuffers cached Mem: 2002 1227774 8 45386 -/+ buffers/cache:794 1207 Swap:0 0 0 And that's with this set in the cassandra-env.sh file: MAX_HEAP_SIZE=800M HEAP_NEWSIZE=200M So I'm thinking now, do I just have to abandon this idea I have of running Cassandra on a 2GB instance? Or is this something we can all agree can be done? And if so, how can we do that? :) Thanks Tim On Wed, Feb 18, 2015 at 8:39 PM, Jason Kushmaul | WDA jason.kushm...@wda.com wrote: I asked this previously when a similar message came through, with a similar response. planetcassandra seems to have it “right”, in that stable=2.0, development=2.1, whereas the apache site says stable is 2.1. “Right” in they assume latest minor version is development. Why not have the apache site do the same? That’s just my lowly non-contributing opinion though. *Jason * *From:* Andrew [mailto:redmu...@gmail.com] *Sent:* Wednesday, February 18, 2015 8:26 PM *To:* Robert Coli; user@cassandra.apache.org *Subject:* Re: run cassandra on a small instance Robert, Let me know if I’m off base about this—but I feel like I see a lot of posts that are like this (i.e., use this arbitrary version, not this other arbitrary version). Why are releases going out if they’re “broken”? This seems like a very confusing way for new (and existing) users to approach versions... Andrew On February 18, 2015 at 5:16:27 PM, Robert Coli (rc...@eventbrite.com) wrote: On Wed, Feb 18, 2015 at 5:09 PM, Tim Dunphy bluethu...@gmail.com wrote: I'm attempting to run Cassandra 2.1.2 on a smallish 2.GB ram instance over at Digital Ocean. It's a CentOS 7 host. 2.1.2 is IMO broken and should not be used for any purpose. Use 2.1.1 or 2.1.3. https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ =Rob -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B -- -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B -- --
Re: run cassandra on a small instance
What I normally do is install plain CentOS (Not any AMI build for Cassandra) and I don't use them for production! I run them for testing, fire drills and some cassandra-stress benchmarks. I will look if I had more than 5h Cassandra uptime. I can even put one up now and do the test and get the results back to you. Hey thanks for letting me know that. And yep! Same here. It's just a plain CentOS 7 VM I've been using. None of this is for production. I also have an AWS account that I use only for testing. I can try setting it up there to and get back to you with my results. Thank you! Tim On Thu, Feb 19, 2015 at 12:52 PM, Carlos Rolo r...@pythian.com wrote: What I normally do is install plain CentOS (Not any AMI build for Cassandra) and I don't use them for production! I run them for testing, fire drills and some cassandra-stress benchmarks. I will look if I had more than 5h Cassandra uptime. I can even put one up now and do the test and get the results back to you. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Feb 19, 2015 at 6:41 PM, Tim Dunphy bluethu...@gmail.com wrote: I have Cassandra instances running on VMs with smaller RAM (1GB even) and I don't go OOM when testing them. Although I use them in AWS and other providers, never tried Digital Ocean. Does Cassandra just fails after some time running or it is failing on some specific read/write? Hi Carlos, Ok, that's really interesting. So I have to ask, did you have to do anything special to get Cassandra to run on those 1GB AWS instances? I'd love to do the same. I even tried there as well and failed due to lack of memory to run it. And there is no specific reason other than lack of memory that I can tell for it to fail. And it doesn's seem to matter what data I use either. Because even if I remove the data directory with rm -rf, the phenomenon is the same. It'll run for a while, usually about 5 hours and then just crash with the word 'killed' as the last line of output. Thanks Tim On Thu, Feb 19, 2015 at 3:40 AM, Carlos Rolo r...@pythian.com wrote: I have Cassandra instances running on VMs with smaller RAM (1GB even) and I don't go OOM when testing them. Although I use them in AWS and other providers, never tried Digital Ocean. Does Cassandra just fails after some time running or it is failing on some specific read/write? Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Feb 19, 2015 at 7:16 AM, Tim Dunphy bluethu...@gmail.com wrote: Hey guys, After the upgrade to 2.1.3, and after almost exactly 5 hours running cassandra did indeed crash again on the 2GB ram VM. This is how the memory on the VM looked after the crash: [root@web2:~] #free -m total used free sharedbuffers cached Mem: 2002 1227774 8 45 386 -/+ buffers/cache:794 1207 Swap:0 0 0 And that's with this set in the cassandra-env.sh file: MAX_HEAP_SIZE=800M HEAP_NEWSIZE=200M So I'm thinking now, do I just have to abandon this idea I have of running Cassandra on a 2GB instance? Or is this something we can all agree can be done? And if so, how can we do that? :) Thanks Tim On Wed, Feb 18, 2015 at 8:39 PM, Jason Kushmaul | WDA jason.kushm...@wda.com wrote: I asked this previously when a similar message came through, with a similar response. planetcassandra seems to have it “right”, in that stable=2.0, development=2.1, whereas the apache site says stable is 2.1. “Right” in they assume latest minor version is development. Why not have the apache site do the same? That’s just my lowly non-contributing opinion though. *Jason * *From:* Andrew [mailto:redmu...@gmail.com] *Sent:* Wednesday, February 18, 2015 8:26 PM *To:* Robert Coli; user@cassandra.apache.org *Subject:* Re: run cassandra on a small instance Robert, Let me know if I’m off base about this—but I feel like I see a lot of posts that are like this (i.e., use this arbitrary version, not this other arbitrary version). Why are releases going out if they’re “broken”? This seems like a very confusing way for new (and existing) users to approach versions... Andrew On February 18, 2015 at 5:16:27 PM, Robert Coli (rc...@eventbrite.com) wrote: On Wed, Feb 18, 2015 at 5:09 PM, Tim Dunphy bluethu...@gmail.com wrote: I'm attempting to run Cassandra 2.1.2 on a smallish 2.GB ram instance over at Digital Ocean. It's a CentOS 7 host. 2.1.2 is IMO broken and should not be used for any purpose. Use 2.1.1 or 2.1.3
Re: run cassandra on a small instance
2.1.2 is IMO broken and should not be used for any purpose. Use 2.1.1 or 2.1.3. https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ =Rob Cool man. Thanks for the info. I just upgraded to 2.1.3. We'll see how that goes. I can let you know more once it's been running for a while. Thanks Tim On Wed, Feb 18, 2015 at 8:16 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Feb 18, 2015 at 5:09 PM, Tim Dunphy bluethu...@gmail.com wrote: I'm attempting to run Cassandra 2.1.2 on a smallish 2.GB ram instance over at Digital Ocean. It's a CentOS 7 host. 2.1.2 is IMO broken and should not be used for any purpose. Use 2.1.1 or 2.1.3. https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ =Rob -- GPG me!! gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
Re: run cassandra on a small instance
Robert, Let me know if I’m off base about this—but I feel like I see a lot of posts that are like this (i.e., use this arbitrary version, not this other arbitrary version). Why are releases going out if they’re “broken”? This seems like a very confusing way for new (and existing) users to approach versions... Andrew On February 18, 2015 at 5:16:27 PM, Robert Coli (rc...@eventbrite.com) wrote: On Wed, Feb 18, 2015 at 5:09 PM, Tim Dunphy bluethu...@gmail.com wrote: I'm attempting to run Cassandra 2.1.2 on a smallish 2.GB ram instance over at Digital Ocean. It's a CentOS 7 host. 2.1.2 is IMO broken and should not be used for any purpose. Use 2.1.1 or 2.1.3. https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ =Rob
