Re: storing indexes on ssd

2018-02-13 Thread Oleksandr Shulgin 
On Tue, Feb 13, 2018 at 10:46 PM, Dan Kinder  wrote:

> On a single node that's a bit less than half full, the index files are 87G.
>

That's not small, true.  Out of curiosity: how much data per node do you
have in total?

How will OS disk cache know to keep the index file blocks cached but not
> cache blocks from the data files? As far as I know it is not smart enough
> to handle that gracefully.
>

Given the total size of index files it sounds unlikely that your indexes
will reside fully in memory, unless you can afford to have much smaller
nodes.  In that light lvmcache could be the solution you're looking for
indeed.

Re: ram expensiveness, see https://www.extremetech.com/computing/263031-ram-
> prices-roof-stuck-way -- it's really not an important point though, ram
> is still far more expensive than disk, regardless of whether the price has
> been going up.
>

That's an interesting perspective, thanks for sharing.

--
Alex

Re: storing indexes on ssd

2018-02-13 Thread Jon Haddad 
It seems like cart-before-horse decision to assume you want to keep your index 
files cached but not your data files.  Why not rely on lvmcache’s statistics 
about file access to determine what to keep and what not to?  It’s going to 
keep your most heavily hit blocks in the cache and your least hit blocks get 
evicted.  

lvmcache is build on top of dm-cache, which uses a “hotspot” queue, promoting 
and demoting blocks based on least-recently used metrics.  Read up on the smq 
policy, which is the default.

http://man7.org/linux/man-pages/man7/lvmcache.7.html 


Jon

> On Feb 13, 2018, at 1:46 PM, Dan Kinder  wrote:
> 
> On a single node that's a bit less than half full, the index files are 87G.
> 
> How will OS disk cache know to keep the index file blocks cached but not 
> cache blocks from the data files? As far as I know it is not smart enough to 
> handle that gracefully.
> 
> Re: ram expensiveness, see 
> https://www.extremetech.com/computing/263031-ram-prices-roof-stuck-way 
>  -- 
> it's really not an important point though, ram is still far more expensive 
> than disk, regardless of whether the price has been going up.
> 
> On Tue, Feb 13, 2018 at 12:02 AM, Oleksandr Shulgin 
> mailto:oleksandr.shul...@zalando.de>> wrote:
> On Tue, Feb 13, 2018 at 1:30 AM, Dan Kinder  > wrote:
> Created https://issues.apache.org/jira/browse/CASSANDRA-14229 
> 
> 
> This is confusing.  You've already started the conversation here...
> 
> How big are your index files in the end?  Even if Cassandra doesn't cache 
> them in or (off-) heap, they might as well just fit into the OS disk cache.
> 
> From your ticket description:
> > ... as ram continues to get more expensive,..
> 
> Where did you get that from?  I would expect quite the opposite.
> 
> Regards,
> --
> Alex
> 
> 
> 
> 
> -- 
> Dan Kinder
> Principal Software Engineer
> Turnitin – www.turnitin.com 
> dkin...@turnitin.com

Re: storing indexes on ssd

2018-02-13 Thread Dan Kinder 
On a single node that's a bit less than half full, the index files are 87G.

How will OS disk cache know to keep the index file blocks cached but not
cache blocks from the data files? As far as I know it is not smart enough
to handle that gracefully.

Re: ram expensiveness, see
https://www.extremetech.com/computing/263031-ram-prices-roof-stuck-way --
it's really not an important point though, ram is still far more expensive
than disk, regardless of whether the price has been going up.

On Tue, Feb 13, 2018 at 12:02 AM, Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

> On Tue, Feb 13, 2018 at 1:30 AM, Dan Kinder  wrote:
>
>> Created https://issues.apache.org/jira/browse/CASSANDRA-14229
>>
>
> This is confusing.  You've already started the conversation here...
>
> How big are your index files in the end?  Even if Cassandra doesn't cache
> them in or (off-) heap, they might as well just fit into the OS disk cache.
>
> From your ticket description:
> > ... as ram continues to get more expensive,..
>
> Where did you get that from?  I would expect quite the opposite.
>
> Regards,
> --
> Alex
>
>


-- 
Dan Kinder
Principal Software Engineer
Turnitin – www.turnitin.com
dkin...@turnitin.com

Re: if the heap size exceeds 32GB..

2018-02-13 Thread Thakrar, Jayesh 
Sure, here are our settings:

#
# Simpler, new generation G1GC settings.
#
JVM_OPTS="$JVM_OPTS -XX:+UseG1GC"
JVM_OPTS="$JVM_OPTS -XX:+UnlockExperimentalVMOptions"
JVM_OPTS="$JVM_OPTS -XX:+ParallelRefProcEnabled"
JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=50"
JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=2"
#

# GC logging options -- uncomment to enable
JVM_OPTS="$JVM_OPTS -XX:+PrintGCDetails"
JVM_OPTS="$JVM_OPTS -XX:+PrintGCDateStamps"
JVM_OPTS="$JVM_OPTS -XX:+PrintGCTimeStamps"
JVM_OPTS="$JVM_OPTS -XX:+PrintHeapAtGC"
JVM_OPTS="$JVM_OPTS -XX:+PrintTenuringDistribution"
JVM_OPTS="$JVM_OPTS -XX:+PrintGCApplicationStoppedTime"
JVM_OPTS="$JVM_OPTS -XX:+PrintPromotionFailure"
JVM_OPTS="$JVM_OPTS -Xloggc:/home/vchadoop/var/logs/cassandra/cassandra-gc.log"
JVM_OPTS="$JVM_OPTS -XX:+UseGCLogFileRotation"
JVM_OPTS="$JVM_OPTS -XX:NumberOfGCLogFiles=10"
JVM_OPTS="$JVM_OPTS -XX:GCLogFileSize=1M"

#
MAX_HEAP_SIZE="84G"
HEAP_NEWSIZE="2G"
#

The only issue that we currently have and are looking to fix it soon is the 
need to upgrade our old JDK version and to set metaspace to a higher value.
We found that when the Java runtime reaches the high watermark, it induces a a 
full GC even if there is plenty of memory to expand the heap.

{Heap before GC invocations=1 (full 1):
garbage-first heap   total 88080384K, used 655025K [0x7fdd6000, 
0x7ff26000, 0x7ff26000)
  region size 32768K, 20 young (655360K), 0 survivors (0K)
Metaspace   used 34166K, capacity 35325K, committed 35328K, reserved 36864K
2018-01-05T08:10:31.491+: 81.789: [Full GC (Metadata GC Threshold)  
651M->30M(84G), 0.6598667 secs]
   [Eden: 640.0M(2048.0M)->0.0B(2048.0M) Survivors: 0.0B->0.0B Heap: 
651.4M(84.0G)->30.4M(84.0G)], [Metaspace: 34166K->34162K(36864K)]
Heap after GC invocations=2 (full 2):
garbage-first heap   total 88080384K, used 31140K [0x7fdd6000, 
0x7ff26000, 0x7ff26000)
  region size 32768K, 0 young (0K), 0 survivors (0K)
Metaspace   used 34162K, capacity 35315K, committed 35328K, reserved 36864K
}
[Times: user=0.67 sys=0.00, real=0.66 secs]



From: Jeff Jirsa 
Date: Tuesday, February 13, 2018 at 11:40 AM
To: cassandra 
Cc: "Steinmaurer, Thomas" 
Subject: Re: if the heap size exceeds 32GB..

I'm not Jayesh, but I assume they're using G1GC (or one of the proprietary 
collectors) that does a lot more concurrent marking without STW.

If you ran 84G heaps with CMS, you'd either need to dramatically tune your CMS 
initiating occupancy, or you'd probably see horrible, horrible pauses.





On Tue, Feb 13, 2018 at 8:44 AM, James Rothering 
mailto:jrother...@codojo.me>> wrote:
Wow, an 84GB heap! Would you mind disclosing the kind of data requirements 
behind this choice? And what kind of STW GC pauses do you see?





On Tue, Feb 13, 2018 at 10:06 AM, Thakrar, Jayesh 
mailto:jthak...@conversantmedia.com>> wrote:
In most cases, Cassandra is pretty efficient about memory usage.
However, if your use case does require/need/demand more memory for your 
workload, I would not hesitate to use heap > 32 GB.
FYI, we have configured our heap for 84 GB.
However there's more tuning that we have done beyond just the heap, so make 
sure you are aware of what else needs to be done.

From: "Steinmaurer, Thomas" 
mailto:thomas.steinmau...@dynatrace.com>>
Date: Tuesday, February 13, 2018 at 1:49 AM
To: "user@cassandra.apache.org" 
mailto:user@cassandra.apache.org>>
Subject: RE: if the heap size exceeds 32GB..

Stick with 31G in your case. Another article on compressed Oops: 
https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/

Thomas

From: Eunsu Kim [mailto:eunsu.bil...@gmail.com]
Sent: Dienstag, 13. Februar 2018 08:09
To: user@cassandra.apache.org
Subject: if the heap size exceeds 32GB..

https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html#compressed_oops

According to the article above, if the heap size of the JVM is about 32GB, it 
is a waste of memory because it can not use the compress object pointer. (Of 
course talking about ES)

But if this is a general theory about the JVM, does that apply to Cassandra as 
well?

I am using a 64 GB physical memory server and I am concerned about heap size 
allocation.

Thank you.
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Re: if the heap size exceeds 32GB..

2018-02-13 Thread Jeff Jirsa 
I'm not Jayesh, but I assume they're using G1GC (or one of the proprietary
collectors) that does a lot more concurrent marking without STW.

If you ran 84G heaps with CMS, you'd either need to dramatically tune your
CMS initiating occupancy, or you'd probably see horrible, horrible pauses.





On Tue, Feb 13, 2018 at 8:44 AM, James Rothering 
wrote:

> Wow, an 84GB heap! Would you mind disclosing the kind of data requirements
> behind this choice? And what kind of STW GC pauses do you see?
>
>
>
>
>
> On Tue, Feb 13, 2018 at 10:06 AM, Thakrar, Jayesh <
> jthak...@conversantmedia.com> wrote:
>
>> In most cases, Cassandra is pretty efficient about memory usage.
>>
>> However, if your use case does require/need/demand more memory for your
>> workload, I would not hesitate to use heap > 32 GB.
>>
>> FYI, we have configured our heap for 84 GB.
>>
>> However there's more tuning that we have done beyond just the heap, so
>> make sure you are aware of what else needs to be done.
>>
>>
>>
>> *From: *"Steinmaurer, Thomas" 
>> *Date: *Tuesday, February 13, 2018 at 1:49 AM
>> *To: *"user@cassandra.apache.org" 
>> *Subject: *RE: if the heap size exceeds 32GB..
>>
>>
>>
>> Stick with 31G in your case. Another article on compressed Oops:
>> https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-
>> java-jvm-memory-oddities/
>>
>>
>>
>> Thomas
>>
>>
>>
>> *From:* Eunsu Kim [mailto:eunsu.bil...@gmail.com]
>> *Sent:* Dienstag, 13. Februar 2018 08:09
>> *To:* user@cassandra.apache.org
>> *Subject:* if the heap size exceeds 32GB..
>>
>>
>>
>> https://www.elastic.co/guide/en/elasticsearch/guide/current/
>> heap-sizing.html#compressed_oops
>>
>>
>>
>> According to the article above, if the heap size of the JVM is about
>> 32GB, it is a waste of memory because it can not use the compress object
>> pointer. (Of course talking about ES)
>>
>>
>>
>> But if this is a general theory about the JVM, does that apply to
>> Cassandra as well?
>>
>>
>>
>> I am using a 64 GB physical memory server and I am concerned about heap
>> size allocation.
>>
>>
>>
>> Thank you.
>>
>> The contents of this e-mail are intended for the named addressee only. It
>> contains information that may be confidential. Unless you are the named
>> addressee or an authorized designee, you may not copy or use it, or
>> disclose it to anyone else. If you received it in error please notify us
>> immediately and then destroy it. Dynatrace Austria GmbH (registration
>> number FN 91482h) is a company registered in Linz whose registered office
>> is at 4040 Linz, Austria, Freistädterstraße 313
>>
>
>

Re: if the heap size exceeds 32GB..

2018-02-13 Thread James Rothering 
Wow, an 84GB heap! Would you mind disclosing the kind of data requirements
behind this choice? And what kind of STW GC pauses do you see?





On Tue, Feb 13, 2018 at 10:06 AM, Thakrar, Jayesh <
jthak...@conversantmedia.com> wrote:

> In most cases, Cassandra is pretty efficient about memory usage.
>
> However, if your use case does require/need/demand more memory for your
> workload, I would not hesitate to use heap > 32 GB.
>
> FYI, we have configured our heap for 84 GB.
>
> However there's more tuning that we have done beyond just the heap, so
> make sure you are aware of what else needs to be done.
>
>
>
> *From: *"Steinmaurer, Thomas" 
> *Date: *Tuesday, February 13, 2018 at 1:49 AM
> *To: *"user@cassandra.apache.org" 
> *Subject: *RE: if the heap size exceeds 32GB..
>
>
>
> Stick with 31G in your case. Another article on compressed Oops:
> https://blog.codecentric.de/en/2014/02/35gb-heap-less-
> 32gb-java-jvm-memory-oddities/
>
>
>
> Thomas
>
>
>
> *From:* Eunsu Kim [mailto:eunsu.bil...@gmail.com]
> *Sent:* Dienstag, 13. Februar 2018 08:09
> *To:* user@cassandra.apache.org
> *Subject:* if the heap size exceeds 32GB..
>
>
>
> https://www.elastic.co/guide/en/elasticsearch/guide/
> current/heap-sizing.html#compressed_oops
>
>
>
> According to the article above, if the heap size of the JVM is about 32GB,
> it is a waste of memory because it can not use the compress object pointer.
> (Of course talking about ES)
>
>
>
> But if this is a general theory about the JVM, does that apply to
> Cassandra as well?
>
>
>
> I am using a 64 GB physical memory server and I am concerned about heap
> size allocation.
>
>
>
> Thank you.
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
>

Re: Cassandra is not running

2018-02-13 Thread Irtiza Ali 
Thankyou everyone

On 13 Feb 2018 8:50 p.m., "Michael Shuler"  wrote:

> On 02/13/2018 09:45 AM, Jürgen Albersdorfer wrote:
> > 1.8.0_161 is not yet supported - try 1.8.0_151
> >
> >> Am 13.02.2018 um 16:44 schrieb Irtiza Ali  >> >:
> >>
> >> 1.8.0_161
> >
>
> Or install the 3.11.2 tentative release, which should work fine on
> 1.8.0_161.
>
> https://lists.apache.org/thread.html/6c38262e25cfca27b4e697aa429fe1
> c51cb83b8fa341ec8a860c272b@%3Cdev.cassandra.apache.org%3E
>
> Looks like a packaged install, so you can find those at:
> https://people.apache.org/~mshuler/
>
> --
> Kind regards,
> Michael
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: Cassandra is not running

2018-02-13 Thread Michael Shuler 
On 02/13/2018 09:45 AM, Jürgen Albersdorfer wrote:
> 1.8.0_161 is not yet supported - try 1.8.0_151
> 
>> Am 13.02.2018 um 16:44 schrieb Irtiza Ali > >:
>>
>> 1.8.0_161
> 

Or install the 3.11.2 tentative release, which should work fine on
1.8.0_161.

https://lists.apache.org/thread.html/6c38262e25cfca27b4e697aa429fe1c51cb83b8fa341ec8a860c272b@%3Cdev.cassandra.apache.org%3E

Looks like a packaged install, so you can find those at:
https://people.apache.org/~mshuler/

-- 
Kind regards,
Michael

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

RE: Cassandra is not running

2018-02-13 Thread Haro, Vanessa (Nokia - US/Naperville) 
Adding known issue reference and fix versions:
CASSANDRA-14173
Fix Version/s: 3.11.2/4.0

From: Jürgen Albersdorfer [mailto:jalbersdor...@gmail.com]
Sent: Tuesday, February 13, 2018 9:45 AM
To: user@cassandra.apache.org
Subject: Re: Cassandra is not running

1.8.0_161 is not yet supported - try 1.8.0_151


Am 13.02.2018 um 16:44 schrieb Irtiza Ali mailto:i...@an10.io>>:

1.8.0_161

Re: Cassandra is not running

2018-02-13 Thread @Nandan@ 
Downgrade your Java version please

On Feb 13, 2018 11:46 PM, "Irtiza Ali"  wrote:

> Thank you
>
> On 13 Feb 2018 20:45, "Jürgen Albersdorfer" 
> wrote:
>
>> 1.8.0_161 is not yet supported - try 1.8.0_151
>>
>> Am 13.02.2018 um 16:44 schrieb Irtiza Ali :
>>
>> 1.8.0_161
>>
>>
>>

Re: Cassandra is not running

2018-02-13 Thread Irtiza Ali 
Thank you

On 13 Feb 2018 20:45, "Jürgen Albersdorfer"  wrote:

> 1.8.0_161 is not yet supported - try 1.8.0_151
>
> Am 13.02.2018 um 16:44 schrieb Irtiza Ali :
>
> 1.8.0_161
>
>
>

Re: Cassandra is not running

2018-02-13 Thread Jürgen Albersdorfer 
1.8.0_161 is not yet supported - try 1.8.0_151

> Am 13.02.2018 um 16:44 schrieb Irtiza Ali :
> 
> 1.8.0_161

Re: Cassandra is not running

2018-02-13 Thread Irtiza Ali 
output of the cassandra system log file:

INFO  [main] 2018-02-13 20:25:40,973 DatabaseDescriptor.java:367 -
DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO  [main] 2018-02-13 20:25:40,973 DatabaseDescriptor.java:421 - Global
memtable on-heap threshold is enabled at 480MB
INFO  [main] 2018-02-13 20:25:40,974 DatabaseDescriptor.java:425 - Global
memtable off-heap threshold is enabled at 480MB
INFO  [main] 2018-02-13 20:25:41,294 RateBasedBackPressure.java:123 -
Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST,
window size: 2000.
INFO  [main] 2018-02-13 20:25:41,295 DatabaseDescriptor.java:725 -
Back-pressure is disabled with strategy
org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5,
flow=FAST}.
ERROR [main] 2018-02-13 20:25:41,546 CassandraDaemon.java:706 - Exception
encountered during startup
java.lang.AbstractMethodError:
org.apache.cassandra.utils.JMXServerUtils$Exporter.exportObject(Ljava/rmi/Remote;ILjava/rmi/server/RMIClientSocketFactory;Ljava/rmi/server/RMIServerSocketFactory;Lsun/misc/ObjectInputFilter;)Ljava/rmi/Remote;
at
javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:150)
~[na:1.8.0_161]
at
javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:135)
~[na:1.8.0_161]
at
javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:405)
~[na:1.8.0_161]
at
org.apache.cassandra.utils.JMXServerUtils.createJMXServer(JMXServerUtils.java:104)
~[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.service.CassandraDaemon.maybeInitJmx(CassandraDaemon.java:143)
[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188)
[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
[apache-cassandra-3.11.1.jar:3.11.1]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689)
[apache-cassandra-3.11.1.jar:3.11.1]
INFO  [main] 2018-02-13 20:29:58,199 YamlConfigurationLoader.java:89 -
Configuration location: file:/etc/cassandra/cassandra.yaml



On Tue, Feb 13, 2018 at 8:37 PM, @Nandan@ 
wrote:

> What error message or WARN you got in system.log file and also check
> output.log file ..
>
>
> On Feb 13, 2018 11:34 PM, "Irtiza Ali"  wrote:
>
>> What should I do now?
>>
>> On 13 Feb 2018 20:21, "Irtiza Ali"  wrote:
>>
>>> Thank you i will check it
>>>
>>>
>>> On 13 Feb 2018 20:16, "Nicolas Guyomar" 
>>> wrote:
>>>
 Hi,

 Such a quick failure might indicate that you are trying to start
 Cassandra with the latest JDK which is not yet supported.

 You should have a look at the /var/log/system.log

 On 13 February 2018 at 16:03, Irtiza Ali  wrote:

> Hello everyone:
>
> Whenever I try to run the Cassandra it stop with status:
>
> result of [sudo service cassandra status] command:
>
> ● cassandra.service - LSB: distributed storage system for structured
> data
>Loaded: loaded (/etc/init.d/cassandra; bad; vendor preset: enabled)
>Active: active (exited) since 2018-02-13 19:57:51 PKT; 31s ago
>  Docs: man:systemd-sysv-generator(8)
>   Process: 28844 ExecStop=/etc/init.d/cassandra stop (code=exited,
> status=0/SUCCESS)
>   Process: 28929 ExecStart=/etc/init.d/cassandra start (code=exited,
> status=0/SUCCESS)
>
> Anyone knows why cassandra is not running properly?
>
> With Regards
> Irtiza Ali
>

Re: Cassandra is not running

2018-02-13 Thread @Nandan@ 
What error message or WARN you got in system.log file and also check
output.log file ..


On Feb 13, 2018 11:34 PM, "Irtiza Ali"  wrote:

> What should I do now?
>
> On 13 Feb 2018 20:21, "Irtiza Ali"  wrote:
>
>> Thank you i will check it
>>
>>
>> On 13 Feb 2018 20:16, "Nicolas Guyomar" 
>> wrote:
>>
>>> Hi,
>>>
>>> Such a quick failure might indicate that you are trying to start
>>> Cassandra with the latest JDK which is not yet supported.
>>>
>>> You should have a look at the /var/log/system.log
>>>
>>> On 13 February 2018 at 16:03, Irtiza Ali  wrote:
>>>
 Hello everyone:

 Whenever I try to run the Cassandra it stop with status:

 result of [sudo service cassandra status] command:

 ● cassandra.service - LSB: distributed storage system for structured
 data
Loaded: loaded (/etc/init.d/cassandra; bad; vendor preset: enabled)
Active: active (exited) since 2018-02-13 19:57:51 PKT; 31s ago
  Docs: man:systemd-sysv-generator(8)
   Process: 28844 ExecStop=/etc/init.d/cassandra stop (code=exited,
 status=0/SUCCESS)
   Process: 28929 ExecStart=/etc/init.d/cassandra start (code=exited,
 status=0/SUCCESS)

 Anyone knows why cassandra is not running properly?

 With Regards
 Irtiza Ali

>>>
>>>

Re: Cassandra is not running

2018-02-13 Thread Irtiza Ali 
What should I do now?

On 13 Feb 2018 20:21, "Irtiza Ali"  wrote:

> Thank you i will check it
>
>
> On 13 Feb 2018 20:16, "Nicolas Guyomar"  wrote:
>
>> Hi,
>>
>> Such a quick failure might indicate that you are trying to start
>> Cassandra with the latest JDK which is not yet supported.
>>
>> You should have a look at the /var/log/system.log
>>
>> On 13 February 2018 at 16:03, Irtiza Ali  wrote:
>>
>>> Hello everyone:
>>>
>>> Whenever I try to run the Cassandra it stop with status:
>>>
>>> result of [sudo service cassandra status] command:
>>>
>>> ● cassandra.service - LSB: distributed storage system for structured data
>>>Loaded: loaded (/etc/init.d/cassandra; bad; vendor preset: enabled)
>>>Active: active (exited) since 2018-02-13 19:57:51 PKT; 31s ago
>>>  Docs: man:systemd-sysv-generator(8)
>>>   Process: 28844 ExecStop=/etc/init.d/cassandra stop (code=exited,
>>> status=0/SUCCESS)
>>>   Process: 28929 ExecStart=/etc/init.d/cassandra start (code=exited,
>>> status=0/SUCCESS)
>>>
>>> Anyone knows why cassandra is not running properly?
>>>
>>> With Regards
>>> Irtiza Ali
>>>
>>
>>

Re: Cassandra is not running

2018-02-13 Thread Irtiza Ali 
Thank you i will check it


On 13 Feb 2018 20:16, "Nicolas Guyomar"  wrote:

> Hi,
>
> Such a quick failure might indicate that you are trying to start Cassandra
> with the latest JDK which is not yet supported.
>
> You should have a look at the /var/log/system.log
>
> On 13 February 2018 at 16:03, Irtiza Ali  wrote:
>
>> Hello everyone:
>>
>> Whenever I try to run the Cassandra it stop with status:
>>
>> result of [sudo service cassandra status] command:
>>
>> ● cassandra.service - LSB: distributed storage system for structured data
>>Loaded: loaded (/etc/init.d/cassandra; bad; vendor preset: enabled)
>>Active: active (exited) since 2018-02-13 19:57:51 PKT; 31s ago
>>  Docs: man:systemd-sysv-generator(8)
>>   Process: 28844 ExecStop=/etc/init.d/cassandra stop (code=exited,
>> status=0/SUCCESS)
>>   Process: 28929 ExecStart=/etc/init.d/cassandra start (code=exited,
>> status=0/SUCCESS)
>>
>> Anyone knows why cassandra is not running properly?
>>
>> With Regards
>> Irtiza Ali
>>
>
>

Re: Cassandra is not running

2018-02-13 Thread Nicolas Guyomar 
Hi,

Such a quick failure might indicate that you are trying to start Cassandra
with the latest JDK which is not yet supported.

You should have a look at the /var/log/system.log

On 13 February 2018 at 16:03, Irtiza Ali  wrote:

> Hello everyone:
>
> Whenever I try to run the Cassandra it stop with status:
>
> result of [sudo service cassandra status] command:
>
> ● cassandra.service - LSB: distributed storage system for structured data
>Loaded: loaded (/etc/init.d/cassandra; bad; vendor preset: enabled)
>Active: active (exited) since 2018-02-13 19:57:51 PKT; 31s ago
>  Docs: man:systemd-sysv-generator(8)
>   Process: 28844 ExecStop=/etc/init.d/cassandra stop (code=exited,
> status=0/SUCCESS)
>   Process: 28929 ExecStart=/etc/init.d/cassandra start (code=exited,
> status=0/SUCCESS)
>
> Anyone knows why cassandra is not running properly?
>
> With Regards
> Irtiza Ali
>

Re: node restart causes application latency

2018-02-13 Thread Mike Torra 
Then could it be that calling `nodetool drain` after calling `nodetool
disablegossip` is what causes the problem?

On Mon, Feb 12, 2018 at 6:12 PM, kurt greaves  wrote:

>
> ​Actually, it's not really clear to me why disablebinary and thrift are
> necessary prior to drain, because they happen in the same order during
> drain anyway. It also really doesn't make sense that disabling gossip after
> drain would make a difference here, because it should be already stopped.
> This is all assuming drain isn't erroring out.
>

Re: if the heap size exceeds 32GB..

2018-02-13 Thread Thakrar, Jayesh 
In most cases, Cassandra is pretty efficient about memory usage.
However, if your use case does require/need/demand more memory for your 
workload, I would not hesitate to use heap > 32 GB.
FYI, we have configured our heap for 84 GB.
However there's more tuning that we have done beyond just the heap, so make 
sure you are aware of what else needs to be done.

From: "Steinmaurer, Thomas" 
Date: Tuesday, February 13, 2018 at 1:49 AM
To: "user@cassandra.apache.org" 
Subject: RE: if the heap size exceeds 32GB..

Stick with 31G in your case. Another article on compressed Oops: 
https://blog.codecentric.de/en/2014/02/35gb-heap-less-32gb-java-jvm-memory-oddities/

Thomas

From: Eunsu Kim [mailto:eunsu.bil...@gmail.com]
Sent: Dienstag, 13. Februar 2018 08:09
To: user@cassandra.apache.org
Subject: if the heap size exceeds 32GB..

https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html#compressed_oops

According to the article above, if the heap size of the JVM is about 32GB, it 
is a waste of memory because it can not use the compress object pointer. (Of 
course talking about ES)

But if this is a general theory about the JVM, does that apply to Cassandra as 
well?

I am using a 64 GB physical memory server and I am concerned about heap size 
allocation.

Thank you.
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or disclose it 
to anyone else. If you received it in error please notify us immediately and 
then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a 
company registered in Linz whose registered office is at 4040 Linz, Austria, 
Freistädterstraße 313

Cassandra is not running

2018-02-13 Thread Irtiza Ali 
Hello everyone:

Whenever I try to run the Cassandra it stop with status:

result of [sudo service cassandra status] command:

● cassandra.service - LSB: distributed storage system for structured data
   Loaded: loaded (/etc/init.d/cassandra; bad; vendor preset: enabled)
   Active: active (exited) since 2018-02-13 19:57:51 PKT; 31s ago
 Docs: man:systemd-sysv-generator(8)
  Process: 28844 ExecStop=/etc/init.d/cassandra stop (code=exited,
status=0/SUCCESS)
  Process: 28929 ExecStart=/etc/init.d/cassandra start (code=exited,
status=0/SUCCESS)

Anyone knows why cassandra is not running properly?

With Regards
Irtiza Ali

Re: Old tombstones not being cleaned up

2018-02-13 Thread Bo Finnerup Madsen 
Hi Eric,

I had not seen your talk, it was very informative thank you! :)

Based on your talk, I can see how tombstones might noget get removed during
normal operations under certain conditions. But I am not sure our scenario
fit those conditions.

We have less than 100.000 live rows in the table in question, and when
flushed the table is roughly 60Mb. Using "nodetool compact" I did several
full compactions of the table. How ever, I always ended up with two
sstables as Jeff mentions, so perhaps some kind of issue with the
incremental repair...

man. 12. feb. 2018 kl. 15.46 skrev Eric Stevens :

> Hi,
>
> Just in case you haven't seen it, I gave a talk last year at the summit.
> In the first part of the talk I speak for a while about the lifecycle of a
> tombstone, and how they don't always get cleaned up when you might expect.
>
> https://youtu.be/BhGkSnBZgJA
>
> It looks like you're deleting cluster keys on a partition that you always
> append to?  If so those tombstones can never be cleaned up - see the talk.
> I don't know if this is what's affecting you or not, but it might be
> worthwhile to consider.
>
> On Mon, Feb 12, 2018, 3:17 AM Jeff Jirsa  wrote:
>
>> When you force compacted, did you end up with 1 sstable or 2?
>>
>> If 2, did you ever run (incremental) repair on some of the data? If so,
>> it moves the repaired sstable to a different compaction manager, which
>> means it won’t purge the tombstone if it shadows data in the unrepaired set
>>
>>
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Feb 12, 2018, at 12:46 AM, Bo Finnerup Madsen 
>> wrote:
>>
>> Well for anyone having the same issue, I "fixed" it by dropping and
>> re-creating the table.
>>
>> fre. 2. feb. 2018 kl. 07.29 skrev Steinmaurer, Thomas <
>> thomas.steinmau...@dynatrace.com>:
>>
>>> Right. In this case, cleanup should have done the necessary work here.
>>>
>>>
>>>
>>> Thomas
>>>
>>>
>>>
>>> *From:* Bo Finnerup Madsen [mailto:bo.gunder...@gmail.com]
>>> *Sent:* Freitag, 02. Februar 2018 06:59
>>>
>>>
>>> *To:* user@cassandra.apache.org
>>> *Subject:* Re: Old tombstones not being cleaned up
>>>
>>>
>>>
>>> We did start with a 3 node cluster and a RF of 3, then added another 3
>>> nodes and again another 3 nodes. So it is a good guess :)
>>>
>>> But I have run both repair and cleanup against the table on all nodes,
>>> would that not have removed any stray partitions?
>>>
>>> tor. 1. feb. 2018 kl. 22.31 skrev Steinmaurer, Thomas <
>>> thomas.steinmau...@dynatrace.com>:
>>>
>>> Did you started with a 9 node cluster from the beginning or did you
>>> extend / scale out your cluster (with vnodes) beyond the replication factor?
>>>
>>>
>>>
>>> If later applies and if you are deleting by explicit deletes and not via
>>> TTL, then nodes might not see the deletes anymore, as a node might not own
>>> the partition anymore after a topology change (e.g. scale out beyond the
>>> keyspace RF).
>>>
>>>
>>>
>>> Just a very wild guess.
>>>
>>>
>>>
>>> Thomas
>>>
>>>
>>>
>>> *From:* Bo Finnerup Madsen [mailto:bo.gunder...@gmail.com]
>>> *Sent:* Donnerstag, 01. Februar 2018 22:14
>>>
>>>
>>> *To:* user@cassandra.apache.org
>>> *Subject:* Re: Old tombstones not being cleaned up
>>>
>>>
>>>
>>> We do not use TTL anywhere...records are inserted and deleted "manually"
>>> by our software.
>>>
>>> tor. 1. feb. 2018 kl. 18.29 skrev Jonathan Haddad :
>>>
>>> Changing the defaul TTL doesn’t change the TTL on the existing data,
>>> only new data. It’s only set if you don’t supply one yourself.
>>>
>>>
>>>
>>> On Wed, Jan 31, 2018 at 11:35 PM Bo Finnerup Madsen <
>>> bo.gunder...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>>
>>>
>>> We are running a small 9 node Cassandra v2.1.17 cluster. The cluster
>>> generally runs fine, but we have one table that are causing OOMs because an
>>> enormous amount of tombstones.
>>>
>>> Looking at the data in the table (sstable2json), the first of the
>>> tombstones are almost a year old. The table was initially created with a
>>> gc_grace_period of 10 days, but I have now lowered it to 1 hour.
>>>
>>> I have run a full repair of the table across all nodes. I have forced
>>> several major compactions of the table by using "nodetool compact", and
>>> also tried to switch from LeveledCompaction to SizeTierCompaction and back.
>>>
>>>
>>>
>>> What could cause cassandra to keep these tombstones?
>>>
>>>
>>>
>>> sstable2json:
>>>
>>> {"key": "foo",
>>>
>>>  "cells":
>>> [["082f-25ef-4324-bb8a-8cf013c823c1:_","082f-25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135],
>>>
>>>
>>>  
>>> ["10f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","10f3-c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731],
>>>
>>>
>>>  
>>> ["1d7a-ce95-4c74-b67e-f8cdffec4f85:_","1d7a-ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102],
>>>
>>>
>>>  
>>> ["1dd3-ae22-4f6e-944a-8cfa147cde68:_","1dd3-ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006],
>>>
>>>
>>>  
>>> ["22cc-d69c-4596-89e5-3e976c0cb9a8:_","2

RE: LWT broken?

2018-02-13 Thread Jacques-Henri Berthemet 
Yes, non-applied LWT will return the row of the winning result. I agree, in 
theory I’d expect your code to have a correct behavior.

You could also check release notes of later Cassandra versions for LWT related 
bugs. If your ids are timeUUID you could try to extract the time when the 
inconsistencies happened and check corresponding Cassandra logs to see what 
happened.
--
Jacques-Henri Berthemet

From: Mahdi Ben Hamida [mailto:ma...@signalfx.com]
Sent: Monday, February 12, 2018 8:45 PM
To: user@cassandra.apache.org
Subject: Re: LWT broken?

On 2/12/18 2:04 AM, Jacques-Henri Berthemet wrote:

Mahdi, you don’t need to re-read at CL ONE on line 9. When a LWT statement is 
not applied, the values that prevented the LWT are returned as part of the 
response, I’d expect them to be more consistent than your read. I’m not 100% 
sure it’s the case for 2.0.x but it’s the case for Cassandra 2.2.

Yes. That's an optimization that can be added. I need to check that it works 
properly with the version of cassandra that I'm running. Right now, we have 
line 9 done at a SERIAL consistency and the issue still happens.



And it’s the same for line 1, you should only keep your LWT statement unless 
you have a huge performance benefit of doing. In Cassandra doing a read before 
write is a bad pattern.
I'll be trying this next and seeing if the issue disappears when we change it 
to serial. Although, I still don't understand how this would cause any 
inconsistencies. In the worst case, a non serial read would return no rows for 
the specified primary key which I handle by trying to do an LWT insert. If it's 
returning a result, I assume that result will be the row that the winning 
lightweight transaction has written. I think that assumption may not be correct 
all the time and I would love to understand why that is the case.

--
Mahdi.


AFAIK a LWT statement is always executed as SERIAL, the only choice you have is 
between SERIAL and LOCAL_SERIAL.

Regards,
--
Jacques-Henri Berthemet

From: DuyHai Doan [mailto:doanduy...@gmail.com]
Sent: Sunday, February 11, 2018 6:11 PM
To: user 
Subject: Re: LWT broken?

Mahdi , the issue in your code is here:

else // we lost LWT, fetch the winning value
 9existing_id = SELECT id FROM hash_id WHERE hash=computed_hash | 
consistency = ONE

You lost LWT, it means that there is a concurrent LWT that has won the Paxos 
round and has applied the value using QUORUM/SERIAL.

In best case, it means that the won LWT value has been applied to at least 2 
replicas out of 3 (assuming RF=3)
In worst case, the won LWT value has not been applied yet or is pending to be 
applied to any replica

Now, if you immediately read with CL=ONE, you may:

1) Read the staled value on the 3rd replica which has not yet received the 
correct won LWT value
2) Or worst, read a staled value because the won LWT is being applied when the 
read operation is made

That's the main reason reading with CL=SERIAL is recommended (CL=QUORUM is not 
sufficient enough)

Reading with CL=SERIAL will:

a. like QUORUM, contact strict majority of replicas
b. unlike QUORUM, look for validated (but not yet applied) previous Paxos round 
value and force-applied it before actually reading the new value




On Sun, Feb 11, 2018 at 5:36 PM, Mahdi Ben Hamida 
mailto:ma...@signalfx.com>> wrote:

Totally understood that it's not worth (or it's rather incorrect) to mix serial 
and non serial operations for LWT tables. It would be highly satisfying to my 
engineer mind if someone can explain why that would cause issues in this 
particular situation. The only explanation I have is that a non serial read may 
cause a read repair to happen and that could interfere with a concurrent serial 
write, although I still can't explain how that would cause two different 
"insert if not exist" transactions to both succeed.

--

Mahdi.
On 2/9/18 2:40 PM, Jonathan Haddad wrote:
If you want consistent reads you have to use the CL that enforces it. There’s 
no way around it.
On Fri, Feb 9, 2018 at 2:35 PM Mahdi Ben Hamida 
mailto:ma...@signalfx.com>> wrote:

In this case, we only write using CAS (code guarantees that). We also never 
update, just insert if not exist. Once a hash exists, it never changes (it may 
get deleted later and that'll be a CAS delete as well).

--

Mahdi.
On 2/9/18 1:38 PM, Jeff Jirsa wrote:


On Fri, Feb 9, 2018 at 1:33 PM, Mahdi Ben Hamida 
mailto:ma...@signalfx.com>> wrote:

 Under what circumstances would we be reading inconsistent results ? Is there a 
case where we end up reading a value that actually end up not being written ?




If you ever write the same value with CAS and without CAS (different code paths 
both updating the same value), you're using CAS wrong, and inconsistencies can 
happen.

Re: storing indexes on ssd

2018-02-13 Thread Oleksandr Shulgin 
On Tue, Feb 13, 2018 at 1:30 AM, Dan Kinder  wrote:

> Created https://issues.apache.org/jira/browse/CASSANDRA-14229
>

This is confusing.  You've already started the conversation here...

How big are your index files in the end?  Even if Cassandra doesn't cache
them in or (off-) heap, they might as well just fit into the OS disk cache.

>From your ticket description:
> ... as ram continues to get more expensive,..

Where did you get that from?  I would expect quite the opposite.

Regards,
--
Alex