I quick glimpse at the code. It seems that truncate only cares about if any node goes down. It does not check the replication strategy.
Boris On Thu, Nov 14, 2013 at 9:30 PM, Cyril Scetbon <cyril.scet...@free.fr>wrote: > anyone ? > > -- > Cyril SCETBON > > On 10 Nov 2013, at 10:48, Cyril Scetbon <cyril.scet...@free.fr> wrote: > > > Are you really sure of that ? cause Tracing says the opposite when I do > it on one node : > > > > cqlsh:system> TRUNCATE hints ; > > > > Tracing session: fe0dbc80-49dd-11e3-a39b-e367b713537d > > > > activity > | timestamp | source | source_elapsed > > > ----------------------------------------------------------------------------------------+--------------+------------+---------------- > > > execute_cql3_query | 07:59:15,019 | 10.0.0.37 | 0 > > Parsing > TRUNCATE hints ; | 07:59:15,020 | 10.0.0.37 | 991 > > > Preparing statement | 07:59:15,021 | 10.0.0.37 | 2009 > > Enqueuing truncate messages to hosts [/10.0.0.185, /10.0.0.33, / > 10.0.0.17, /10.0.0.37] | 07:59:15,023 | 10.0.0.37 | 4459 > > Message received > from /10.0.0.37 | 07:59:15,186 | 10.0.0.33 | 121 > > Applying truncation > of system.hints | 07:59:15,200 | 10.0.0.33 | 11064 > > Sending message > to /10.0.0.37 | 07:59:15,214 | 10.0.0.37 | 195660 > > Sending message > to /10.0.0.33 | 07:59:15,217 | 10.0.0.37 | 198664 > > Message received > from /10.0.0.37 | 07:59:15,230 | 10.0.0.185 | 149 > > Applying truncation > of system.hints | 07:59:15,240 | 10.0.0.185 | 6248 > > Message received > from /10.0.0.37 | 07:59:15,244 | 10.0.0.17 | 117 > > Sending message > to /10.0.0.185 | 07:59:15,247 | 10.0.0.37 | 134087 > > Sending message > to /10.0.0.17 | 07:59:15,248 | 10.0.0.37 | 134760 > > Message received > from /10.0.0.37 | 07:59:15,251 | 10.0.0.37 | 232058 > > Applying truncation > of system.hints | 07:59:15,254 | 10.0.0.37 | 234904 > > Applying truncation > of system.hints | 07:59:15,259 | 10.0.0.17 | 8574 > > Enqueuing response to truncate operation > to /10.0.0.37 | 07:59:15,388 | 10.0.0.17 | 144491 > > Sending message > to /10.0.0.37 | 07:59:15,389 | 10.0.0.17 | 145341 > > Message received > from /10.0.0.17 | 07:59:15,461 | 10.0.0.37 | 441863 > > Processing response > from /10.0.0.17 | 07:59:15,502 | 10.0.0.37 | 482942 > > Enqueuing response to truncate operation > to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 391940 > > Sending message > to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 392364 > > Message received > from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613172 > > Processing response > from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613303 > > Enqueuing response to truncate operation > to /10.0.0.37 | 07:59:15,736 | 10.0.0.37 | 717066 > > Sending message > to /10.0.0.37 | 07:59:15,738 | 10.0.0.37 | 719165 > > Message received > from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null > > Processing response > from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null > > Enqueuing response to truncate operation > to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685116 > > Sending message > to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685519 > > Message received > from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null > > Processing response > from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null > > > Request complete | 07:59:15,936 | 10.0.0.37 | 917846 > > > > I also tried to generate different hints on different nodes, and > truncating the table on one node removed data everywhere : > > > > > > node 1 > > > > at t1 > > > > cqlsh> select * from system.hints ; > > > > target_id | hint_id > | message_version | mutation > > > --------------------------------------+--------------------------------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > 748cb454-da16-4218-b6ac-728f0fe0cf41 | > 9f152ea0-49e9-11e3-af88-ed7ea188ea44 | 6 | > 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf278eb3a0000000000006000376616c00000004eacf278eb3a0000000040000270f > > 78078974-1f02-4da8-9902-cca56291a9dc | > 474e4c30-49ec-11e3-af88-ed7ea188ea44 | 6 | > 0x0006706e735f6672000868706172736f6e7300000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6b945e10000000000006000376616c00000004eacf6b945e1000000004000010f7 > > 78078974-1f02-4da8-9902-cca56291a9dc | > 4e64df70-49ec-11e3-af88-ed7ea188ea44 | 6 | > 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6c49f270000000000006000376616c00000004eacf6c49f2700000000400001618 > > > > at t2 > > > > cqlsh:pns_fr> TRUNCATE system.hints ; > > cqlsh:pns_fr> select * from system.hints ; > > > > node 2 > > > > at t1 > > > > cqlsh:pns_fr> select * from system.hints ; > > > > target_id | hint_id > | message_version | mutation > > > --------------------------------------+--------------------------------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > 78078974-1f02-4da8-9902-cca56291a9dc | > aa8f4da0-49ea-11e3-a8dc-e367b713537d | 6 | > 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf424d88b0000000000006000376616c00000004eacf424d88b000000004000004bc > > > > at t2 > > > > cqlsh:pns_fr> select * from system.hints ; > > > > Regards > > -- > > Cyril SCETBON > > > > On 09 Nov 2013, at 22:48, Jonathan Ellis <jbel...@gmail.com> wrote: > > > >> Hints are not replicated (technically: replicated with LocalStrategy), > >> so truncating hints will always only affect the local node. > >> > >> On Sat, Nov 9, 2013 at 3:44 PM, Cyril Scetbon <cyril.scet...@free.fr> > wrote: > >>> Hi, > >>> > >>> I would like to add a feature request for local truncates of hints but > I want to know your opinion before. > >>> > >>> We faced an issue last week concerning 3 new nodes that where not seen > by some other nodes in the ring. a week later we resolved the issues but it > was taking days with a high CPU load (85%) to apply GBytes of Hints. I > noticed that CF Hints is defined with LocalStategy and so thought the > truncate was only local but that wasn't. We had to remove sstables, > commitlog and cached to remove all that old hints stored but not removed > after max_hint_window_in_ms as target nodes where not dead (they were only > not accepting hints). > >>> > >>> So, I suggest that the truncate of hints stay local (maybe you need it > to stay global per default ?) or that we could do it locally (a truncate > argument or a new command like the one used for CONSISTENCY) > >>> > >>> thanks > >>> -- > >>> Cyril SCETBON > >>> > >> > >> > >> > >> -- > >> Jonathan Ellis > >> Project Chair, Apache Cassandra > >> co-founder, http://www.datastax.com > >> @spyced > > > >
