Re: Node crashes on repair (Cassandra 3.11.1)

[Javier Canillas]

Christian,

I'm not an expert, but maybe the merkle tree is too big to transfer between
nodes and that's why it times out. How many nodes do you have and what's
the size of the keyspace? Have you ever done a successfully repair before?

Cassandra reaper does repair based on tokenrange (or even part of it),
that's why it can manage to require a small merkle tree.

Regards,

Javier.

2017-11-30 6:48 GMT-03:00 Christian Lorenz :

> Hello,
>
>
>
> after updating our cluster to Cassandra 3.11.1 (previously 3.9) running a
> ‘nodetool repair –full’ leads to the node crashing.
>
> Logfile showed the following Exception:
>
> ERROR [ReadRepairStage:36] 2017-11-30 07:42:06,439
> CassandraDaemon.java:228 - Exception in thread Thread[ReadRepairStage:36,5,
> main]
>
> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out
> - received only 0 responses.
>
> at org.apache.cassandra.service.DataResolver$
> RepairMergeListener.close(DataResolver.java:199)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
>
> at org.apache.cassandra.db.partitions.
> UnfilteredPartitionIterators$2.close(UnfilteredPartitionIterators.java:175)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
>
> at 
> org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:92)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
>
> at 
> org.apache.cassandra.service.DataResolver.compareResponses(DataResolver.java:76)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
>
> at 
> org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:50)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
>
> at 
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
>
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[na:1.8.0_151]
>
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[na:1.8.0_151]
>
> at org.apache.cassandra.concurrent.NamedThreadFactory.
> lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> ~[apache-cassandra-3.11.1.jar:3.11.1]
>
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_151]
>
>
>
> The node datasize is ~270GB.  A repair with Cassandra reaper works fine
> though.
>
>
>
> Any idea why this could be happening?
>
>
>
> Regards,
>
> Christian
>

Re: Best approach to prepare to shutdown a cassandra node

[Javier Canillas]

So, just to clarify.. a good approach to shutdown an instance of cassandra
should be:

*# Drain all information from commitlog into sstables*
*bin/nodetool stopdaemon*
*cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep -v "grep"|awk
'{print $2}'`*
*if [ "$?" -ne 0 ]; then*
*echo "Cassandra stopdaemon fail? Please check logs"*
*if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne "1" ]; then*
*echo "Cassandra is still running, killing it gracefully"*
*kill $cassandra_pid*
*echo -n "+ Checking it is down. "*
*counter=10*
*while [ "$counter" -ne 0 -a ! kill -0 $cassandra_pid >
/dev/null 2>&1 ]*
*do*
*  echo -n ". "*
*  ((counter--))*
*  sleep 1s*
*done*
*echo ""*
*if ! kill -0 $cassandra_pid > /dev/null 2>&1; then*
*echo "+ Its down."*
*else*
*echo "- Killing forcefully Cassandra."*
*kill -9 $cassandra_pid*
*fi*
*   else*
*echo "Care there was a problem finding Cassandra PID, it
might be still running"*
*exit 1*
*   fi*
*  else*
*   echo "Cassandra stopped"*
*fi*

2017-10-20 9:04 GMT-03:00 Lutaya Shafiq Holmes <lutayasha...@gmail.com>:

> Looking at the code in trunk, the stopdemon command invokes the
> CassandraDaemon.stop() function which does a graceful shutdown by
> stopping jmxServer and drains the node by the shutdown hook.
>
>
> On 10/20/17, Simon Fontana Oscarsson
> <simon.fontana.oscars...@ericsson.com> wrote:
> > Yes, drain will always be run when Cassandra exits normally.
> >
> > On 2017-10-20 00:57, Varun Gupta wrote:
> >> Does, nodetool stopdaemon, implicitly drain too? or we should invoke
> >> drain and then stopdaemon?
> >>
> >> On Mon, Oct 16, 2017 at 4:54 AM, Simon Fontana Oscarsson
> >> <simon.fontana.oscars...@ericsson.com
> >> <mailto:simon.fontana.oscars...@ericsson.com>> wrote:
> >>
> >> Looking at the code in trunk, the stopdemon command invokes the
> >> CassandraDaemon.stop() function which does a graceful shutdown by
> >> stopping jmxServer and drains the node by the shutdown hook.
> >>
> >> /Simon
> >>
> >>
> >> On 2017-10-13 20:42, Javier Canillas wrote:
> >>> As far as I know, the nodetool stopdaemon is doing a "kill -9".
> >>>
> >>> Or did it change?
> >>>
> >>> 2017-10-12 23:49 GMT-03:00 Anshu Vajpayee
> >>> <anshu.vajpa...@gmail.com <mailto:anshu.vajpa...@gmail.com>>:
> >>>
> >>> Why are you killing when we have nodetool stopdaemon ?
> >>>
> >>> On Fri, Oct 13, 2017 at 1:49 AM, Javier Canillas
> >>> <javier.canil...@gmail.com
> >>> <mailto:javier.canil...@gmail.com>> wrote:
> >>>
> >>> That's what I thought.
> >>>
> >>> Thanks!
> >>>
> >>> 2017-10-12 14:26 GMT-03:00 Hannu Kröger
> >>> <hkro...@gmail.com <mailto:hkro...@gmail.com>>:
> >>>
> >>> Hi,
> >>>
> >>> Drain should be enough.  It stops accepting writes
> >>> and after that cassandra can be safely shut down.
> >>>
> >>> Hannu
> >>>
> >>> On 12 October 2017 at 20:24:41, Javier Canillas
> >>> (javier.canil...@gmail.com
> >>> <mailto:javier.canil...@gmail.com>) wrote:
> >>>
> >>>> Hello everyone,
> >>>>
> >>>> I have some time working with Cassandra, but every
> >>>> time I need to shutdown a node (for any reason like
> >>>> upgrading version or moving instance to another
> >>>> host) I see several errors on the client
> >>>> applications (yes, I'm using the official java
> driver).
> >>>>
> >>>> By the way, I'm starting C* as a stand-alone process
> >>>>
>

Re: Best approach to prepare to shutdown a cassandra node

[Javier Canillas]

As far as I know, the nodetool stopdaemon is doing a "kill -9".

Or did it change?

2017-10-12 23:49 GMT-03:00 Anshu Vajpayee <anshu.vajpa...@gmail.com>:

> Why are you killing when we have nodetool stopdaemon ?
>
> On Fri, Oct 13, 2017 at 1:49 AM, Javier Canillas <
> javier.canil...@gmail.com> wrote:
>
>> That's what I thought.
>>
>> Thanks!
>>
>> 2017-10-12 14:26 GMT-03:00 Hannu Kröger <hkro...@gmail.com>:
>>
>>> Hi,
>>>
>>> Drain should be enough.  It stops accepting writes and after that
>>> cassandra can be safely shut down.
>>>
>>> Hannu
>>>
>>> On 12 October 2017 at 20:24:41, Javier Canillas (
>>> javier.canil...@gmail.com) wrote:
>>>
>>> Hello everyone,
>>>
>>> I have some time working with Cassandra, but every time I need to
>>> shutdown a node (for any reason like upgrading version or moving instance
>>> to another host) I see several errors on the client applications (yes, I'm
>>> using the official java driver).
>>>
>>> By the way, I'm starting C* as a stand-alone process
>>> <https://docs.datastax.com/en/cassandra/3.0/cassandra/initialize/referenceStartCprocess.html?hl=start>,
>>> and C* version is 3.11.0.
>>>
>>> The way I have implemented the shutdown process is something like the
>>> following:
>>>
>>> *# Drain all information from commitlog into sstables*
>>>
>>> *bin/nodetool drain*
>>>
>>>
>>> *cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep -v "grep"|awk
>>> '{print $2}'`*
>>> *if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne "1" ]; then*
>>> *echo "Asking Cassandra to shutdown (nodetool drain doesn't stop
>>> cassandra)"*
>>> *kill $cassandra_pid*
>>>
>>> *echo -n "+ Checking it is down. "*
>>> *counter=10*
>>> *while [ "$counter" -ne 0 -a ! kill -0 $cassandra_pid >
>>> /dev/null 2>&1 ]*
>>> *do*
>>> *echo -n ". "*
>>> *((counter--))*
>>> *sleep 1s*
>>> *done*
>>> *echo ""*
>>> *if ! kill -0 $cassandra_pid > /dev/null 2>&1; then*
>>> *echo "+ Its down."*
>>> *else*
>>> *echo "- Killing Cassandra."*
>>> *kill -9 $cassandra_pid*
>>> *fi*
>>> *else*
>>> *echo "Care there was a problem finding Cassandra PID"*
>>> *fi*
>>>
>>> Should I add at the beginning the following lines?
>>>
>>> echo "shutdowing cassandra gracefully with: nodetool disable gossip"
>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablegossip
>>> echo "shutdowing cassandra gracefully with: nodetool disable binary
>>> protocol"
>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablebinary
>>> echo "shutdowing cassandra gracefully with: nodetool thrift"
>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablethrift
>>>
>>> The shutdown log is the following:
>>>
>>> *WARN  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,343
>>> StorageService.java:321 - Stopping gossip by operator request*
>>> *INFO  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,344
>>> Gossiper.java:1532 - Announcing shutdown*
>>> *INFO  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,355
>>> StorageService.java:2268 - Node /10.254.169.36 <http://10.254.169.36> state
>>> jump to shutdown*
>>> *INFO  [RMI TCP Connection(12)-127.0.0.1] 2017-10-12 14:20:56,141
>>> Server.java:176 - Stop listening for CQL clients*
>>> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,472
>>> StorageService.java:1442 - DRAINING: starting drain process*
>>> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,474
>>> HintsService.java:220 - Paused hints dispatch*
>>> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,477
>>> Gossiper.java:1532 - Announcing shutdown*
>>> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,480
>>> StorageService.java:2268 - Node /127.0.0.1 <http://127.0.0.1> state jump to
>>> shutdown*
>>> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:01,483
>>> MessagingService.java:984 - Waiting for messaging service to quiesce*
>>> *INFO  [ACCEPT-/192.168.6.174 <http://192.168.6.174>] 2017-10-12
>>> 14:21:01,485 MessagingService.java:1338 - MessagingService has terminated
>>> the accept() thread*
>>> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,095
>>> HintsService.java:220 - Paused hints dispatch*
>>> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,111
>>> StorageService.java:1442 - DRAINED*
>>>
>>> Disabling Gossip seemed a good idea, but watching the logs, it may use
>>> it to gracefully telling the other nodes he is going down, so I don't know
>>> if it's good or bad idea.
>>>
>>> Disabling Thrift and Binary protocol should only avoid new connections,
>>> but the one stablished and running should be attempted to finish.
>>>
>>> Any thoughts or comments?
>>>
>>> Thanks
>>>
>>> Javier.
>>>
>>>
>>>
>>
>
>
> --
> *Regards,*
> *Anshu *
>
>
>

Re: Best approach to prepare to shutdown a cassandra node

[Javier Canillas]

That's what I thought.

Thanks!

2017-10-12 14:26 GMT-03:00 Hannu Kröger <hkro...@gmail.com>:

> Hi,
>
> Drain should be enough.  It stops accepting writes and after that
> cassandra can be safely shut down.
>
> Hannu
>
> On 12 October 2017 at 20:24:41, Javier Canillas (javier.canil...@gmail.com)
> wrote:
>
> Hello everyone,
>
> I have some time working with Cassandra, but every time I need to shutdown
> a node (for any reason like upgrading version or moving instance to another
> host) I see several errors on the client applications (yes, I'm using the
> official java driver).
>
> By the way, I'm starting C* as a stand-alone process
> <https://docs.datastax.com/en/cassandra/3.0/cassandra/initialize/referenceStartCprocess.html?hl=start>,
> and C* version is 3.11.0.
>
> The way I have implemented the shutdown process is something like the
> following:
>
> *# Drain all information from commitlog into sstables*
>
> *bin/nodetool drain*
>
>
> *cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep -v "grep"|awk
> '{print $2}'`*
> *if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne "1" ]; then*
> *echo "Asking Cassandra to shutdown (nodetool drain doesn't stop
> cassandra)"*
> *kill $cassandra_pid*
>
> *echo -n "+ Checking it is down. "*
> *counter=10*
> *while [ "$counter" -ne 0 -a ! kill -0 $cassandra_pid > /dev/null
> 2>&1 ]*
> *do*
> *echo -n ". "*
> *((counter--))*
> *sleep 1s*
> *done*
> *echo ""*
> *if ! kill -0 $cassandra_pid > /dev/null 2>&1; then*
> *echo "+ Its down."*
> *else*
> *echo "- Killing Cassandra."*
> *kill -9 $cassandra_pid*
> *fi*
> *else*
> *echo "Care there was a problem finding Cassandra PID"*
> *fi*
>
> Should I add at the beginning the following lines?
>
> echo "shutdowing cassandra gracefully with: nodetool disable gossip"
> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablegossip
> echo "shutdowing cassandra gracefully with: nodetool disable binary
> protocol"
> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablebinary
> echo "shutdowing cassandra gracefully with: nodetool thrift"
> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablethrift
>
> The shutdown log is the following:
>
> *WARN  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,343
> StorageService.java:321 - Stopping gossip by operator request*
> *INFO  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,344
> Gossiper.java:1532 - Announcing shutdown*
> *INFO  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,355
> StorageService.java:2268 - Node /10.254.169.36 <http://10.254.169.36> state
> jump to shutdown*
> *INFO  [RMI TCP Connection(12)-127.0.0.1] 2017-10-12 14:20:56,141
> Server.java:176 - Stop listening for CQL clients*
> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,472
> StorageService.java:1442 - DRAINING: starting drain process*
> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,474
> HintsService.java:220 - Paused hints dispatch*
> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,477
> Gossiper.java:1532 - Announcing shutdown*
> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,480
> StorageService.java:2268 - Node /127.0.0.1 <http://127.0.0.1> state jump to
> shutdown*
> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:01,483
> MessagingService.java:984 - Waiting for messaging service to quiesce*
> *INFO  [ACCEPT-/192.168.6.174 <http://192.168.6.174>] 2017-10-12
> 14:21:01,485 MessagingService.java:1338 - MessagingService has terminated
> the accept() thread*
> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,095
> HintsService.java:220 - Paused hints dispatch*
> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,111
> StorageService.java:1442 - DRAINED*
>
> Disabling Gossip seemed a good idea, but watching the logs, it may use it
> to gracefully telling the other nodes he is going down, so I don't know if
> it's good or bad idea.
>
> Disabling Thrift and Binary protocol should only avoid new connections,
> but the one stablished and running should be attempted to finish.
>
> Any thoughts or comments?
>
> Thanks
>
> Javier.
>
>
>

Best approach to prepare to shutdown a cassandra node

[Javier Canillas]

Hello everyone,

I have some time working with Cassandra, but every time I need to shutdown
a node (for any reason like upgrading version or moving instance to another
host) I see several errors on the client applications (yes, I'm using the
official java driver).

By the way, I'm starting C* as a stand-alone process
,
and C* version is 3.11.0.

The way I have implemented the shutdown process is something like the
following:

*# Drain all information from commitlog into sstables*

*bin/nodetool drain*


*cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep -v "grep"|awk
'{print $2}'`*
*if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne "1" ]; then*
*echo "Asking Cassandra to shutdown (nodetool drain doesn't stop
cassandra)"*
*kill $cassandra_pid*

*echo -n "+ Checking it is down. "*
*counter=10*
*while [ "$counter" -ne 0 -a ! kill -0 $cassandra_pid > /dev/null
2>&1 ]*
*do*
*echo -n ". "*
*((counter--))*
*sleep 1s*
*done*
*echo ""*
*if ! kill -0 $cassandra_pid > /dev/null 2>&1; then*
*echo "+ Its down."*
*else*
*echo "- Killing Cassandra."*
*kill -9 $cassandra_pid*
*fi*
*else*
*echo "Care there was a problem finding Cassandra PID"*
*fi*

Should I add at the beginning the following lines?

echo "shutdowing cassandra gracefully with: nodetool disable gossip"
$CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablegossip
echo "shutdowing cassandra gracefully with: nodetool disable binary
protocol"
$CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablebinary
echo "shutdowing cassandra gracefully with: nodetool thrift"
$CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablethrift

The shutdown log is the following:

*WARN  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,343
StorageService.java:321 - Stopping gossip by operator request*
*INFO  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,344
Gossiper.java:1532 - Announcing shutdown*
*INFO  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,355
StorageService.java:2268 - Node /10.254.169.36  state
jump to shutdown*
*INFO  [RMI TCP Connection(12)-127.0.0.1] 2017-10-12 14:20:56,141
Server.java:176 - Stop listening for CQL clients*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,472
StorageService.java:1442 - DRAINING: starting drain process*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,474
HintsService.java:220 - Paused hints dispatch*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,477
Gossiper.java:1532 - Announcing shutdown*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,480
StorageService.java:2268 - Node /127.0.0.1  state jump to
shutdown*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:01,483
MessagingService.java:984 - Waiting for messaging service to quiesce*
*INFO  [ACCEPT-/192.168.6.174 ] 2017-10-12
14:21:01,485 MessagingService.java:1338 - MessagingService has terminated
the accept() thread*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,095
HintsService.java:220 - Paused hints dispatch*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,111
StorageService.java:1442 - DRAINED*

Disabling Gossip seemed a good idea, but watching the logs, it may use it
to gracefully telling the other nodes he is going down, so I don't know if
it's good or bad idea.

Disabling Thrift and Binary protocol should only avoid new connections, but
the one stablished and running should be attempted to finish.

Any thoughts or comments?

Thanks

Javier.

Re: LWT and non-LWT mixed

[Javier Canillas]

Daniel,

Cassandra is "eventually consistent". This means that the DELETE can go to
a different coordinator than the INSERT... IF NOT EXISTS. Being so, each
coordinator enters a race condition than can make the INSERT...IF NOT
EXISTS failed reading data that the DELETE will destroy. Even on the same
coordinator, each statement is treated on different threads.

You can play around with CONSISTENCY LEVEL, applying both statements with
ALL may reduce the chance of failure, but it won't make it go away.

What you are willing to do is to "lock" the row on delete, that's something
you can do on SQL engines but not on C*.

Regards,

2017-10-10 5:22 GMT-03:00 Daniel Woo :

> The document explains you cannot mix them
> http://docs.datastax.com/en/archived/cassandra/2.2/cassandra/dml/
> dmlLtwtTransactions.html
>
> But what happens under the hood if I do? e.g,
>
> DELETE 
> INSERT ... IF NOT EXISTS
>
> The coordinator has 4 steps to do the second statement (INSERT)
> 1. prepare/promise a ballot
> 2. read current row from replicas
> 3. propose new value along with the ballot to replicas
> 4. commit and wait for ack from replicas
>
> My question is, once the row is DELETed, the next INSERT LWT should be
> able to see that row's tombstone in step 2, then successfully inserts the
> new value. But my tests shows that this often fails, does anybody know why?
>
> --
> Thanks & Regards,
> Daniel
>

Re: table repair question

[Javier Canillas]

I'm doing it on a daily basis. If the repaired percentage drops from 80%,
then I do a repair over that keyspace-table. Seems working fine for the
last months without problems and avoids huge merkle tree building. No data
loss so far.

I'm running 3.11.0 btw.

2017-10-04 17:44 GMT-03:00 Blake Eggleston <beggles...@apple.com>:

> Incremental repairs should also update the percentage, although I'd
> recommend not using incremental repair before 4.0. Just want to point out
> that running repairs based on repaired % isn't necessarily a bad thing, but
> it should be a secondary consideration. The important thing is to repair
> data more frequently than your gc grace period.
>
>
> On October 4, 2017 at 1:33:57 PM, Javier Canillas (
> javier.canil...@gmail.com) wrote:
>
> That percentage will only be updated if you do a full repair. If you do
> repairs on local dc or with -pr, that percentage will not be updated.
>
> I scripted a regular repair on each node based on if this percentage is
> below some threshold. It has been running fine since several months ago.
>
> 2017-10-04 12:46 GMT-03:00 Blake Eggleston <beggles...@apple.com>:
>
>> Not really no. There's a repaired % in nodetool tablestats if you're
>> using incremental repair (and you probably shouldn't be before 4.0 comes
>> out), but I wouldn't make any decisions based off it's value.
>>
>>
>> On October 4, 2017 at 8:05:44 AM, ZAIDI, ASAD A (az1...@att.com) wrote:
>>
>> Hello folk,
>>
>>
>>
>> I’m wondering if there is way to find out list of table(s) which need
>> repair OR if there Is way to find out what %age of data would need to be
>> repaired on a table?  Is such information available from Cassandra  db
>> engine through some other means?
>>
>>
>>
>> TIA~ Asad
>>
>>
>>
>>
>>
>>
>>
>>
>

Re: CREATE INDEX without IF NOT EXISTS when snapshoting

[Javier Canillas]

Well,

Patches submitted for version 3.0, 3.11 and trunk (although 3.0 patch can
be applied to 3.11). Feel free to review and comment.

Thanks!

2017-10-04 16:41 GMT-03:00 Javier Canillas <javier.canil...@gmail.com>:

> Kurt,
>
> Thanks for your response. Created this ticket
> <https://issues.apache.org/jira/browse/CASSANDRA-13935>. Feel free to add
> anything to it that seems legit.
>
> Downloading Cassandra code right now.
>
> Fix seems quite simple. Expect a pull-request soon xD
>
> 2017-10-03 20:19 GMT-03:00 kurt greaves <k...@instaclustr.com>:
>
>> Certainly would make sense and should be trivial.  here
>> <https://github.com/apache/cassandra/blob/982ab93a2f8a0f5c56af9378f65d3e9e43b9/src/java/org/apache/cassandra/db/ColumnFamilyStoreCQLHelper.java#L251>
>>  is
>> where you want to look. Just create a ticket for it and prod here for a
>> reviewer once you've got a change.​
>>
>
>

Re: table repair question

[Javier Canillas]

That percentage will only be updated if you do a full repair. If you do
repairs on local dc or with -pr, that percentage will not be updated.

I scripted a regular repair on each node based on if this percentage is
below some threshold. It has been running fine since several months ago.

2017-10-04 12:46 GMT-03:00 Blake Eggleston :

> Not really no. There's a repaired % in nodetool tablestats if you're using
> incremental repair (and you probably shouldn't be before 4.0 comes out),
> but I wouldn't make any decisions based off it's value.
>
>
> On October 4, 2017 at 8:05:44 AM, ZAIDI, ASAD A (az1...@att.com) wrote:
>
> Hello folk,
>
>
>
> I’m wondering if there is way to find out list of table(s) which need
> repair OR if there Is way to find out what %age of data would need to be
> repaired on a table?  Is such information available from Cassandra  db
> engine through some other means?
>
>
>
> TIA~ Asad
>
>
>
>
>
>
>
>

Re: CREATE INDEX without IF NOT EXISTS when snapshoting

[Javier Canillas]

Kurt,

Thanks for your response. Created this ticket
. Feel free to add
anything to it that seems legit.

Downloading Cassandra code right now.

Fix seems quite simple. Expect a pull-request soon xD

2017-10-03 20:19 GMT-03:00 kurt greaves :

> Certainly would make sense and should be trivial.  here
> 
>  is
> where you want to look. Just create a ticket for it and prod here for a
> reviewer once you've got a change.​
>

CREATE INDEX without IF NOT EXISTS when snapshoting

[Javier Canillas]

Hi everyone,

I came across something that bothers me a lot. I'm using snapshots to
backup data from my Cassandra cluster in case something really bad happens
(like dropping a table or a keyspace).

Exercising the recovery actions from those backups, I discover that the
schema put on the file "schema.cql" as a result of the snapshot has the
"CREATE IF NOT EXISTS" for the table, but not for the indexes.

When restoring from snapshots, and relying on the execution of these
schemas to build up the table structure, everything seems fine for tables
without secondary indexes, but for the ones that make use of them, the
execution of these statements fail miserably.

Here I paste a generated schema.cql content for a table with indexes:

CREATE TABLE IF NOT EXISTS keyspace1.table1 (
id text PRIMARY KEY,
content text,
last_update_date date,
last_update_date_time timestamp)
WITH ID = f1045fc0-2f59-11e7-95ec-295c3c064920
AND bloom_filter_fp_chance = 0.01
AND dclocal_read_repair_chance = 0.1
AND crc_check_chance = 1.0
AND default_time_to_live = 864
AND gc_grace_seconds = 864000
AND min_index_interval = 128
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE'
AND caching = { 'keys': 'NONE', 'rows_per_partition': 'NONE' }
AND compaction = { 'max_threshold': '32', 'min_threshold': '4', 'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy' }
AND compression = { 'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor' }
AND cdc = false
AND extensions = {  };
*CREATE INDEX table1_last_update_date_idx ON keyspace1.table1
(last_update_date);*

I think the last part should be:

*CREATE INDEX IF NOT EXISTS table1_last_update_date_idx ON keyspace1.table1
(last_update_date);*

Any ideas? Have you wrote this part of the snapshot behavior for a
particular reason I'm not seeing?

I'm willing to help on coding (as I have done before xD) if you consider
this a trivial bug, but something that should be address.

Kind regards,

Javier.

Re: Repairing question

[Javier Canillas]

Thanks for all the responses. It's much clearer now.

2017-06-26 0:59 GMT-03:00 Paulo Motta <pauloricard...@gmail.com>:

> > Not sure since what version, but in 3.10 at least (I think its since 3.x
> started) full repair does do anti-compactions and marks sstables as
> repaired.
>
> Thanks for the correction, anti-compaction after full repairs was
> added on 2.2 CASSANDRA-7586 but removed on 4.0 by CASSANDRA-9143. Just
> for completeness, anti-compaction is not run when the following
> options are specified:
> -st/-et
> --local or --dc
> --full on 4.0+
>
> 2017-06-25 16:35 GMT-05:00 Cameron Zemek <came...@instaclustr.com>:
> >> When you perform a non-incremental repair data is repaired but not
> marked
> >> as repaired since this require anti-compaction to be run.
> >
> > Not sure since what version, but in 3.10 at least (I think its since 3.x
> > started) full repair does do anti-compactions and marks sstables as
> > repaired.
> >
> > On 23 June 2017 at 06:30, Paulo Motta <pauloricard...@gmail.com> wrote:
> >>
> >> > This attribute seems to be only modified when executing "nodetool
> repair
> >> > [keyspace] [table]", but not when executing with other options
> like
> >> > --in-local-dc or --pr.
> >>
> >> This is correct behavior because this metric actually represent the
> >> percentage of SSTables incrementally repaired - and marked as repaired
> >> - which doesn't happen when you execute a non-incremental repair
> >> (--full, --in-local-dc, --pr). When you perform a non-incremental
> >> repair data is repaired but not marked as repaired since this require
> >> anti-compaction to be run.
> >>
> >> Actually this "percent repaired" display name is a bit misleading,
> >> since it sounds like data needs to be repaired while you could be
> >> running non-incremental repairs and still have data 100% repaired, so
> >> we should probably open a ticket to rename that to "Percent
> >> incrementally repaired" or similar.
> >>
> >>
> >> 2017-06-22 14:38 GMT-05:00 Javier Canillas <javier.canil...@gmail.com>:
> >> > Hi,
> >> >
> >> > I have been thinking about scheduling a daily routine to force repairs
> >> > on a
> >> > cluster to maintain its health.
> >> >
> >> > I saw that by running a nodetool tablestats [keyspace] there is an
> >> > attribute
> >> > called "Percent repaired" that show the percentage of data repaired on
> >> > the
> >> > each table.
> >> >
> >> > This attribute seems to be only modified when executing "nodetool
> repair
> >> > [keyspace] [table]", but not when executing with other options
> like
> >> > --in-local-dc or --pr.
> >> >
> >> > My main concern is about building the whole MERKLE tree for a big
> table.
> >> > I
> >> > have also check to repair by token ranges, but this also seems not to
> >> > modify
> >> > this attribute of the table.
> >> >
> >> > Is this an expected behavior? Or there is something missing on the
> code
> >> > that
> >> > needs to be fixed?
> >> >
> >> > My "maintenance" script would be calling nodetool tablestats per each
> >> > keyspace that has replication_factor > 0 to check for the value of the
> >> > "Percent repaired" of each table and, in case it is below some
> >> > threshold, I
> >> > would execute a repair on it.
> >> >
> >> > Any ideas?
> >> >
> >> > Thanks in advance.
> >> >
> >> > Javier.
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>
> >
>

Re: Repairing question

[Javier Canillas]

Paulo,

Thanks for your explanation.

2017-06-22 17:30 GMT-03:00 Paulo Motta <pauloricard...@gmail.com>:

> > This attribute seems to be only modified when executing "nodetool repair
> [keyspace] [table]", but not when executing with other options like
> --in-local-dc or --pr.
>
> This is correct behavior because this metric actually represent the
> percentage of SSTables incrementally repaired - and marked as repaired
> - which doesn't happen when you execute a non-incremental repair
> (--full, --in-local-dc, --pr). When you perform a non-incremental
> repair data is repaired but not marked as repaired since this require
> anti-compaction to be run.
>
> Actually this "percent repaired" display name is a bit misleading,
> since it sounds like data needs to be repaired while you could be
> running non-incremental repairs and still have data 100% repaired, so
> we should probably open a ticket to rename that to "Percent
> incrementally repaired" or similar.
>
>
> 2017-06-22 14:38 GMT-05:00 Javier Canillas <javier.canil...@gmail.com>:
> > Hi,
> >
> > I have been thinking about scheduling a daily routine to force repairs
> on a
> > cluster to maintain its health.
> >
> > I saw that by running a nodetool tablestats [keyspace] there is an
> attribute
> > called "Percent repaired" that show the percentage of data repaired on
> the
> > each table.
> >
> > This attribute seems to be only modified when executing "nodetool repair
> > [keyspace] [table]", but not when executing with other options like
> > --in-local-dc or --pr.
> >
> > My main concern is about building the whole MERKLE tree for a big table.
> I
> > have also check to repair by token ranges, but this also seems not to
> modify
> > this attribute of the table.
> >
> > Is this an expected behavior? Or there is something missing on the code
> that
> > needs to be fixed?
> >
> > My "maintenance" script would be calling nodetool tablestats per each
> > keyspace that has replication_factor > 0 to check for the value of the
> > "Percent repaired" of each table and, in case it is below some
> threshold, I
> > would execute a repair on it.
> >
> > Any ideas?
> >
> > Thanks in advance.
> >
> > Javier.
>

Repairing question

[Javier Canillas]

Hi,

I have been thinking about scheduling a daily routine to force repairs on a
cluster to maintain its health.

I saw that by running a nodetool tablestats [keyspace] there is an
attribute called "Percent repaired" that show the percentage of data
repaired on the each table.

This attribute seems to be only modified when executing "nodetool repair
[keyspace] [table]", but not when executing with other options like
--in-local-dc or --pr.

My main concern is about building the whole MERKLE tree for a big table. I
have also check to repair by token ranges, but this also seems not to
modify this attribute of the table.

Is this an expected behavior? Or there is something missing on the code
that needs to be fixed?

My "maintenance" script would be calling nodetool tablestats per each
keyspace that has replication_factor > 0 to check for the value of the
"Percent repaired" of each table and, in case it is below some threshold, I
would execute a repair on it.

Any ideas?

Thanks in advance.

Javier.

Cassandra Driver Configurer

[Javier Canillas]

Hey guys,

I have following Cassandra since its preliminar 0,6.X versions and develop
against it from different platforms like .NET, Java and Scala.

I found it very difficult to properly configure my cassandra driver for the
client application since there is no straight way to change a value without
releasing a new version since it lacks the support for an external
configuration file (That was the only flaw I found).

Well, instead of making only a critiscm, I publish on
https://github.com/javiercanillas/cassandra-driver-configurer my own
configurer for datastax cassandra driver. It relies only on TypeSafe.Config
structure.

Feel free to take a look and use it. Any comments are welcome.

PD: I will try to push it into Maven Central if you really find it useful.

Cheers,

Javier.

Re: UnavailableException with 1 node down and RF=2?

[Javier Canillas]

What the problem might be is that you are setting the Consistency Level to a
value bigger than 1. In such cases, Cassandra will respond you with an
UnavailableException since it can't achieve the level of consistency you are
asking for.

Remember that, when you have RF=2, CS values as ALL and QUORUM are the same.

Regards,

Javier.

On Thu, Oct 27, 2011 at 1:23 PM, Jonathan Ellis jbel...@gmail.com wrote:

 Ha.  On the one hand, good on you for searching the list archives for
 similar problems.  On the other hand, after over a year it's probably
 worth starting a new thread. :)

 Standard questions:

 - What Cassandra version are you running?
 - Are there exceptions in the log for the machine still running?
 - What does not responding anymore mean?  Reporting timeouts,
 reporting unavailable, refusing client connections, ... ?

 On Thu, Oct 27, 2011 at 10:22 AM, RobinUs2 ro...@us2.nl wrote:
  I'm currently having a similar problem with a 2-node cluster. When 1
 shutdown
  one of the nodes, the other isn't responding any more.
 
  Did you found a solution for your problem?
 
  /I'm new to mailing lists, if it's inappropriate to reply here, please
 let
  me know../
 
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/2-node-cluster-1-node-down-overall-failure-td6936722.html
 
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/2-node-cluster-1-node-down-overall-failure-td6936722.html
 
  --
  View this message in context:
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/UnavailableException-with-1-node-down-and-RF-2-tp5242055p6936767.html
  Sent from the cassandra-u...@incubator.apache.org mailing list archive
 at Nabble.com.
 



 --
 Jonathan Ellis
 Project Chair, Apache Cassandra
 co-founder of DataStax, the source for professional Cassandra support
 http://www.datastax.com

Re: Columns values(integer) need frequent updates/ increments

[Javier Canillas]

By comparing timestamp. The highest value is supposed to be the most updated

Sent from my iPhone

On 09/04/2011, at 21:02, mcasandra mohitanch...@gmail.com wrote:

 If there are multiple updates to same columns and scattered accross multiple
 sstables then how does cassandra know which sstable has the most recent
 value.

 --
 View this message in context: 
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Columns-values-integer-need-frequent-updates-increments-tp6251464p6257960.html
 Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
 Nabble.com.

Re: Understanding Indexes

[Javier Canillas]

I dont think i got the point in your question. But if you are thinking
about key indexes (like PKs), take in mind that cassandra will manage
keys using the partition strategy. By doing so, it will be able to
determine on which node the row with such key should be hold.
So, in another words, inside cassandra, each column family is treated
as a big table (hashtable). Taking this last in mind, there is no need
to have an index by key. Would you put an index over a hashtable's
keys??

Enviado desde mi iPhone

El 23/02/2011, a las 19:50, mcasandra mohitanch...@gmail.com escribió:


 So far my understanding about indexes is that you can create indexes only on
 column values (username in below eg).

 Does it make sense to also have index on the keys that columnFamily uses to
 store rows (row keys abc in below example). I am thinking in an event rows
 keep growing would search be fast if there is an index on row keys if you
 want to retrieve for eg def only out of tons of rows?

 UserProfile = { // this is a ColumnFamily
abc: {   // this is the key to this Row inside the CF
// now we have an infinite # of columns in this row
username: phatduckk,
email: phatdu...@example.com,
phone: (900) 976-
}, // end row
def: {   // this is the key to another row in the CF
// now we have another infinite # of columns in this row
username: ieure,
email: ie...@example.com,
phone: (888) 555-1212
age: 66,
gender: undecided
},
 }


 2) Is the hash of column key used or row key used by RandomPartitioner to
 distribute it accross the cassandra nodes?
 --
 View this message in context: 
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Understanding-Indexes-tp6058238p6058238.html
 Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
 Nabble.com.

Re: Understand eventually consistent

[Javier Canillas]

First of all, in your example W=CL?

If it so, then the success of any read / write operarion will be
determine by if the CL required can be satisfied in that moment.

If you write with CL ONE over a CF with RF 3 when 1 node of the
replicas is down, then the operarion will success and HitedHandOff
will manage to propagate the op through the falling node when it comes
up.

Instead, when you execute the same OP using CL QUORUM, then it means
RF /2+1, it will try to write on the coordinator node and replica.
Considering only 1 replica is down, the OP will success too.

Now consider same OP but with CL ALL, it will fail since it cant
assure that coordinador and both replicas are updated.

Hope you can understand the relation between CL and RF

Enviado desde mi iPhone

El 23/02/2011, a las 21:43, mcasandra mohitanch...@gmail.com escribió:


 I am reading this again http://wiki.apache.org/cassandra/HintedHandoff and
 got little confused. This is my understanding about how HH should work based
 on what I read in Dynamo Paper:

 1) Say node A, B, C, D, E are in the cluster in a ring (in that order).
 2) For a given key K RF=3.
 3) Node B holds theyhash of that key K. Which means when K is written it
 will be written to B (owner of the hash) + C + D since RF = 3
 4) If Node D goes down and there is a write again to key K then this time
 key K row will be written with W=1 to B (owner) + C + E (HH) since RF=3
 needs to be satisfied. Is this correct?
 5) In above scenario where node D is down and if we are reading at W=2 and
 R=2 would it fail even though original nodes B + C are up? Here I am
 thinking W=2 and R=2 means that 2 nodes that hold the key K are up so it
 satisfies the CL and thus writes and read will not fail.
 --
 View this message in context: 
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Understand-eventually-consistent-tp6038330p6058576.html
 Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
 Nabble.com.

Re: Understanding Indexes

[Javier Canillas]

I really don't see the point.. Again, suppose a cluster with 3 nodes, where
there is a ColumnFamily that will hold data which key is basically consisted
on a word of 2 letters (pretty simple). That's make a total of 729 posible
keys.

RandomPartitioner then will tokenize each key and assign them to a node
within the cluster. Then, each node will handle 243 keys each (plus
replication, of course).

ok, Now suppose that you need to look for data on key AG, the node that
you ask, will then use RandomPartitioner to tokenize the key and determine
which node is the coordinator for that key and proceed to ask that node for
the data (and ask the replicas an md5 version of the data to compare). So,
each node will only need to look for over 1/3 of the stored keys.

How do you think an Index is implemented? As far as I know, a simple index
is básically a HashTable that has the Index value as Key, and the position
as value. How do you think a search within the Index (Hashcode) is
implemented?

I don't know, maybe there is some magic behind indexes (I know there are
some complex indexes that hold some B-Tree, etc; like the one used over SQL
solutions), but I think all the whole thing will only add more complexity
over a more straight solution. How big should be the CF (in terms of keys)
to be able to present latency when searching over hashcodes? And then think,
if I need to add a new Key, what's the cost in the whole process? Now, lets
assume you can make the whole B-Tree in first place (even for the keys that
does not exists), how much memory would that cost? There should be some
papers that discuss this problem somewhere.

I would definitly make some volume calculations and some stress test over
this at least to be sure there is a problem before attempting any kind of
solution.

PD: I feel this is like the problem I present about TTL values, saying
basically, that a TTL value past 2050 year would throw an exception. Who
will be alive after 2012 doomsday? :)


On Thu, Feb 24, 2011 at 3:18 PM, mcasandra mohitanch...@gmail.com wrote:


 What I am trying to ask is that what if there are billions of row keys (eg:
 abc, def, xyz in below eg.) and then client does a lookup/query on a row
 say
 xyz (get all cols for row xyz). Now since there are billions of rows look
 up
 using Hash mechanism, is it going to be slow? What algorithm will be used
 to
 retrieve row xyz which could be anywhere in those billion rows on a
 particular node.

 Is it going to help if there is an index on row keys (eg: abc, xyz)?

  UserProfile = { // this is a ColumnFamily
 abc: {   // this is the key to this Row inside the CF
 // now we have an infinite # of columns in this row
 username: phatduckk,
 email: phatdu...@example.com,
 phone: (900) 976-
 }, // end row
 def: {   // this is the key to another row in the CF
 // now we have another infinite # of columns in this row
 username: ieure,
 email: ie...@example.com,
 phone: (888) 555-1212
 age: 66,
 gender: undecided
 },
  }
 --
 View this message in context:
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Understanding-Indexes-tp6058238p6061356.html
 Sent from the cassandra-u...@incubator.apache.org mailing list archive at
 Nabble.com.

Re: Understand eventually consistent

[Javier Canillas]

Well, it will need all nodes that are required on the operation to be up,
and to response in a timely fashion, even a time-out rpc of 1 replica will
get you a fail response.

CL is calculated based on the RF configured for the ColumnFamily.

The ConsistencyLevel is an enum that controls both read and write behavior
based on ReplicationFactor in your storage-conf.xml.

QUORUM = RF / 2 +1;
ALL = RF
ONE = 1
ANY = 0

Then, on a column family configured with RF = 6, QUORUM means be sure to
write at least over 4 nodes before responding, but on a column family
configured with RF = 3, QUORUM means be sure to write on 2 at least. In
cases where RF is 1 or 2, then QUORUM is like ALL (be sure to write on all
nodes involved).


On Thu, Feb 24, 2011 at 3:29 PM, mcasandra mohitanch...@gmail.com wrote:



 Javier Canillas wrote:
 
  Instead, when you execute the same OP using CL QUORUM, then it means
  RF /2+1, it will try to write on the coordinator node and replica.
  Considering only 1 replica is down, the OP will success too.
 

 I am assuming even read will succeed when CL QUORUM and RF=3 and 1 node is
 down.


 Javier Canillas wrote:
 
  Now consider same OP but with CL ALL, it will fail since it cant
  assure that coordinador and both replicas are updated.
 

 Can you please explain this little more? I thought CL ALL will fail because
 it needs all the nodes to be up.
 http://wiki.apache.org/cassandra/API

 --
 View this message in context:
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Understand-eventually-consistent-tp6038330p6061399.html
 Sent from the cassandra-u...@incubator.apache.org mailing list archive at
 Nabble.com.

Re: Understanding Indexes

[Javier Canillas]

I don't say you shouldn't. In case you feel like there is a problem, you may
think of splitting column families into N. But I think you won't get that
problem. You can read about RowCacheSize and KeyCache support on 0.7.X of
Cassandra, if you rows are small, you may cache a lot of them and avoid a
lot of latency issues when reading writing.

On Thu, Feb 24, 2011 at 4:18 PM, mcasandra mohitanch...@gmail.com wrote:


 Thanks! I am thinking more in terms where you have millions of keys (rows).
 For eg: UUID as a row key. or there could millions of users.

 So are we saying that we should NOT create column families with these many
 keys? What are the other options in such cases?

 UserProfile = { // this is a ColumnFamily
 1 {   // this is the key to this Row inside the CF
 // now we have an infinite # of columns in this row
 username: phatduckk,
 email: [hidden email],
 phone: (900) 976-
 }, // end row
 2 {   // this is the key to another row in the CF
 // now we have another infinite # of columns in this row
 username: ieure,
 email: [hidden email],
 phone: (888) 555-1212
 age: 66,
 gender: undecided
 },
  }

 --
 View this message in context:
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Understanding-Indexes-tp6058238p6061574.html
 Sent from the cassandra-u...@incubator.apache.org mailing list archive at
 Nabble.com.

Re: Understand eventually consistent

[Javier Canillas]

HH is some kind of write repair, so it has nothing to do with CL that is a
requirement of the operation; and it won't be used over reads.

In your example QUORUM is the same as ALL, since you only have 1 RF (only
the data holder - coordinator). If that node fails, all read / writes will
fail.

Now, on another scenario, with RF = 3 and 1 node down:

CL = QUORUM. Will work, but the coordination will mark an HH over the write
and attempt to do it for some time over the failed node. Despite this, the
operation will success for the client.
CL = ALL. Will fail.
CL = ONE. Will work. 2 HH will be sent to replicas to perform the update.

*Consider CL is the client minimum requirement over an operation to succeed*.
If the cluster can assure that value, then the operation will succeed and
returned to the client (despite some HH work needs to be done after), if not
an error response will be returned.


On Thu, Feb 24, 2011 at 4:26 PM, mcasandra mohitanch...@gmail.com wrote:


 Does HH count towards QUORUM? Say  RF=1 and CL of W=QUORUM and one node
 that
 owns the key dies. Would subsequent write operations for that key be
 successful? I am guessing it will not succeed.
 --
 View this message in context:
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Understand-eventually-consistent-tp6038330p6061593.html
 Sent from the cassandra-u...@incubator.apache.org mailing list archive at
 Nabble.com.

Re: Understand eventually consistent

[Javier Canillas]

No, since you are intentionally asking that at least a QUORUM of the RFs are
written. So in your scenario, only 1 node is up of 3, and QUORUM value is 2.
So that operation will fail, no HH is made.

A read won't succedd either, since you are asking that the data to be
returned must be validated at least by 2 nodes.

HH only takes place on write operations and when the OP succeded because the
CL can be satisfied and other replicas are down. Then the coordinator uses
HH to perform the updates on the failed replicas (as soon as they get up).

On Thu, Feb 24, 2011 at 5:13 PM, mcasandra mohitanch...@gmail.com wrote:



 Javier Canillas wrote:
 
  HH is some kind of write repair, so it has nothing to do with CL that is
 a
  requirement of the operation; and it won't be used over reads.
 
  In your example QUORUM is the same as ALL, since you only have 1 RF (only
  the data holder - coordinator). If that node fails, all read / writes
 will
  fail.
 
  Now, on another scenario, with RF = 3 and 1 node down:
 
  CL = QUORUM. Will work, but the coordination will mark an HH over the
  write
  and attempt to do it for some time over the failed node. Despite this,
 the
  operation will success for the client.
  CL = ALL. Will fail.
  CL = ONE. Will work. 2 HH will be sent to replicas to perform the update.
 
  *Consider CL is the client minimum requirement over an operation to
  succeed*.
  If the cluster can assure that value, then the operation will succeed and
  returned to the client (despite some HH work needs to be done after), if
  not
  an error response will be returned.
 
 
  On Thu, Feb 24, 2011 at 4:26 PM, mcasandra mohitanch...@gmail.com
 wrote:
 
 
  Does HH count towards QUORUM? Say  RF=1 and CL of W=QUORUM and one node
  that
  owns the key dies. Would subsequent write operations for that key be
  successful? I am guessing it will not succeed.
  --
  View this message in context:
 
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Understand-eventually-consistent-tp6038330p6061593.html
  Sent from the cassandra-u...@incubator.apache.org mailing list archive
 at
  Nabble.com.
 
 
 

 Thanks! In above scenario what happens if 2 nodes die and RF=3, CL of
 W=QUORUM. Would a write succeed since one write can be made to coordinator
 node with HH and other to the replica node that is up.

 And similarly in above scenario would read succeed. Would HH be considered
 towards CL in this case?
 --
 View this message in context:
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Understand-eventually-consistent-tp6038330p6061772.html
 Sent from the cassandra-u...@incubator.apache.org mailing list archive at
 Nabble.com.

Re: Timeout

[Javier Canillas]

Why don't you post some details about your Cassandra Cluster, version,
information about the keyspace you are creating (for example which is the
replication factor within)? It might be of help.

Besides, I don't fully understand your code. First you drop KEYSPACE, then
create it again with a column Family CF_NAME, then you add a Column to
CF_NAME's column definitions (Am I right?) and finally you display
KEYSPACE's information.

After that, you create your own structure by passing again another object.
What does actually do that method createKeyspace()?

If you can add more details, it might be of help.

On Fri, Feb 18, 2011 at 10:02 PM, mcasandra mohitanch...@gmail.com wrote:


 I have this below code and what I see is that when I run this below code
 there is a timeout that occurs when I try to insert a column. But when I
 comment out first 4 lines (drop to display) then it works without any
 issues. I am trying to understand why. If required I can sleep and then
 insert. Is it because it's getting insert too fast before Cassandra is able
 to persist keyspace info accross nodes?




hUtil.dropKeyspace(c, KEYSPACE); //drop on the server
 hUtil.createKeyspace(c, KEYSPACE, CF_NAME); //Create on the
 server
hUtil.addColumn(c, KEYSPACE, CF_NAME); // Add on the server
hUtil.display(c, KEYSPACE); //Display keyspace info

ExampleDaoV2 ed = new ExampleDaoV2(createKeyspace(KEYSPACE, c));
ed.insert(key1, value2, StringSerializer.get());
System.out.println(ed.get(key1, StringSerializer.get()));


 Caused by: TimedOutException()
at

 org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:16493)
at

 org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:916)
at

 org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:890)
at

 me.prettyprint.cassandra.service.KeyspaceServiceImpl$1.execute(KeyspaceServiceImpl.java:93)
... 14 more

 --
 View this message in context:
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Timeout-tp6042052p6042052.html
 Sent from the cassandra-u...@incubator.apache.org mailing list archive at
 Nabble.com.

Re: What is the most solid version of Cassandra? No secondary indexes needed.

[Javier Canillas]

We have been running a 0.6.3 with some custom features for more than 1 month
and it has been running fine. we are planning on moving to 0.7.1 in about 1
month from now if it past our stress tests.

If you are really going from scratch to production environment, I would
definetly go with 0.7.1 after some heavy stress test, it will give you a
better migration process to any new version.

Javier Canillas

On Tue, Feb 15, 2011 at 5:03 PM, buddhasystem potek...@bnl.gov wrote:


 Thank you! It's just that 7.1 seems the bleeding edge now (a serious bug
 fixed today). Would you still trust it as a production-level service? I'm
 just slightly concerned. I don't want to create a perception among our IT
 that the product is not ready for prime time.
 --
 View this message in context:
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/What-is-the-most-solid-version-of-Cassandra-No-secondary-indexes-needed-tp6028966p6029047.html
 Sent from the cassandra-u...@incubator.apache.org mailing list archive at
 Nabble.com.

Re: Indexes and hard disk

[Javier Canillas]

As for indexes, cassandra supports columns as indexes (there is a
property called some sort of type that can have the value KEYS.
Start your search in that direction (this is a feature from 0.7)
introduce when you create dinamically columnfamilies and you declare
its columns

On the other hand, if your application will use heavily cassandra i
recommend making a calcularon of the data volumen that will handle and
using a partition that is actually 3 to 5 times bigger. Doing so you
will be able to play around with compactation strategy to avoid
latency when your application is under heavy load.

Hope this information is useful to you

Regards,

Javier canillas

El 12/02/2011, a las 04:24, mcasandra mohitanch...@gmail.com escribió:


 Are indexes supported in Cassandra? If yes then what kind? Also, if it's
 supported then please point me to the place that give more information about
 it.

 Are there any kind of hard disk in particular recommended by Cassandra? We
 generally get only 500GB hard disks on our virtual machines. But I read that
 it's better to have 2TB HDD.
 --
 View this message in context: 
 http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Indexes-and-hard-disk-tp6018234p6018234.html
 Sent from the cassandra-u...@incubator.apache.org mailing list archive at 
 Nabble.com.

Re: Compression in Cassandra

[Javier Canillas]

How do you calculate your 40g data? When you insert it into Cassandra, you
need to convert the data into a Byte[], maybe your problem is there.

On Thu, Jan 20, 2011 at 10:02 AM, akshatbakli...@gmail.com 
akshatbakli...@gmail.com wrote:

 Hi all,

 I am experiencing a unique situation. I loaded some data onto Cassandra.
 my data was about 40 GB but when loaded to Cassandra the data directory
 size is almost 170GB.

 This means the **data got inflated**.

 Is it the case just with me or some else is also facing the inflation or
 its the general behavior of Cassandra.

 I am using Cassandra 0.6.8. on Ubuntu 10.10

 --
 Akshat Bakliwal
 Search Information and Extraction Lab
 IIIT-Hyderabad
 09963885762
 WebPage
 http://web.iiit.ac.in/%7Eakshat_b

Re: can't re-insert row after deleting it?

[Javier Canillas]

As Jonathan has told you, Please double check the Timestamp of the column
you are adding the second time. Probably this timestamp is smaller (or
lowerer) than the timestamp given when the deletion happens.

On Thu, Dec 30, 2010 at 1:27 AM, Jonathan Ellis jbel...@gmail.com wrote:

 deletes win races, so if 3) happens at the same timestamp as 2) then
 the delete takes precedence.

 On Wed, Dec 29, 2010 at 9:58 PM, mike dooley doo...@apple.com wrote:
  sorry, i forgot to mention that i am using 0.7-rc2
 
  On Dec 29, 2010, at 7:56 PM, mike dooley wrote:
 
  things have not been going very well for me today, so maybe there's
 something
  that i'm missing, but here is a little test program that just:
 
  1) inserts a single column in a row,
  2) deletes the row,
  3) re-inserts the row
 
  but it seems that the second insert fails to actually insert the data.
 here is the output
  of the test program
 
  === starting
  === inserting row: row-a, value: value-a
  === about to retrieve data
  === got (dummy-col,value-a)
  === deleting row: row-a
  === data not found, as expected
  === re-inserting row: row-a
  ===  re-inserted data not found
  === crush all boxes
 
  and subsequent inserts also fail (if you run the program again).
 
  here is the test program
 
  CClient.java
 
  i am just using a single node, with the following schema
 
   create keyspace Harvest with replication_factor = 1 and
 placement_strategy = 'org.apache.cassandra.locator.SimpleStrategy';
   use Harvest;
   create column family Test1 with column_type = 'Standard' and comparator
 = 'UTF8Type';
 
  thanks in advance.
  -mike
 
 



 --
 Jonathan Ellis
 Project Chair, Apache Cassandra
 co-founder of Riptano, the source for professional Cassandra support
 http://riptano.com