Right. On Nov 17, 2013 11:00 AM, "Cyril Scetbon" <cyril.scet...@free.fr> wrote:
> and/or a nodetool purgelocalhints command ? > > On 15 Nov 2013, at 05:56, Jonathan Ellis <jbel...@gmail.com> wrote: > > > You're right. You'd want to expose the node-local truncate over JMX. > > > > On Sun, Nov 10, 2013 at 3:48 AM, Cyril Scetbon <cyril.scet...@free.fr> > wrote: > >> Are you really sure of that ? cause Tracing says the opposite when I do > it on one node : > >> > >> cqlsh:system> TRUNCATE hints ; > >> > >> Tracing session: fe0dbc80-49dd-11e3-a39b-e367b713537d > >> > >> activity > | timestamp | source | source_elapsed > >> > ----------------------------------------------------------------------------------------+--------------+------------+---------------- > >> > execute_cql3_query | 07:59:15,019 | 10.0.0.37 | 0 > >> Parsing > TRUNCATE hints ; | 07:59:15,020 | 10.0.0.37 | 991 > >> > Preparing statement | 07:59:15,021 | 10.0.0.37 | 2009 > >> Enqueuing truncate messages to hosts [/10.0.0.185, /10.0.0.33, / > 10.0.0.17, /10.0.0.37] | 07:59:15,023 | 10.0.0.37 | 4459 > >> Message received > from /10.0.0.37 | 07:59:15,186 | 10.0.0.33 | 121 > >> Applying truncation > of system.hints | 07:59:15,200 | 10.0.0.33 | 11064 > >> Sending > message to /10.0.0.37 | 07:59:15,214 | 10.0.0.37 | 195660 > >> Sending > message to /10.0.0.33 | 07:59:15,217 | 10.0.0.37 | 198664 > >> Message received > from /10.0.0.37 | 07:59:15,230 | 10.0.0.185 | 149 > >> Applying truncation > of system.hints | 07:59:15,240 | 10.0.0.185 | 6248 > >> Message received > from /10.0.0.37 | 07:59:15,244 | 10.0.0.17 | 117 > >> Sending message > to /10.0.0.185 | 07:59:15,247 | 10.0.0.37 | 134087 > >> Sending > message to /10.0.0.17 | 07:59:15,248 | 10.0.0.37 | 134760 > >> Message received > from /10.0.0.37 | 07:59:15,251 | 10.0.0.37 | 232058 > >> Applying truncation > of system.hints | 07:59:15,254 | 10.0.0.37 | 234904 > >> Applying truncation > of system.hints | 07:59:15,259 | 10.0.0.17 | 8574 > >> Enqueuing response to truncate > operation to /10.0.0.37 | 07:59:15,388 | 10.0.0.17 | 144491 > >> Sending > message to /10.0.0.37 | 07:59:15,389 | 10.0.0.17 | 145341 > >> Message received > from /10.0.0.17 | 07:59:15,461 | 10.0.0.37 | 441863 > >> Processing response > from /10.0.0.17 | 07:59:15,502 | 10.0.0.37 | 482942 > >> Enqueuing response to truncate > operation to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 391940 > >> Sending > message to /10.0.0.37 | 07:59:15,578 | 10.0.0.33 | 392364 > >> Message received > from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613172 > >> Processing response > from /10.0.0.33 | 07:59:15,632 | 10.0.0.37 | 613303 > >> Enqueuing response to truncate > operation to /10.0.0.37 | 07:59:15,736 | 10.0.0.37 | 717066 > >> Sending > message to /10.0.0.37 | 07:59:15,738 | 10.0.0.37 | 719165 > >> Message received > from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null > >> Processing response > from /10.0.0.37 | 07:59:15,739 | 10.0.0.37 | null > >> Enqueuing response to truncate > operation to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685116 > >> Sending > message to /10.0.0.37 | 07:59:15,915 | 10.0.0.185 | 685519 > >> Message received > from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null > >> Processing response > from /10.0.0.185 | 07:59:15,936 | 10.0.0.37 | null > >> > Request complete | 07:59:15,936 | 10.0.0.37 | 917846 > >> > >> I also tried to generate different hints on different nodes, and > truncating the table on one node removed data everywhere : > >> > >> > >> node 1 > >> > >> at t1 > >> > >> cqlsh> select * from system.hints ; > >> > >> target_id | hint_id > | message_version | mutation > >> > --------------------------------------+--------------------------------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >> 748cb454-da16-4218-b6ac-728f0fe0cf41 | > 9f152ea0-49e9-11e3-af88-ed7ea188ea44 | 6 | > 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf278eb3a0000000000006000376616c00000004eacf278eb3a0000000040000270f > >> 78078974-1f02-4da8-9902-cca56291a9dc | > 474e4c30-49ec-11e3-af88-ed7ea188ea44 | 6 | > 0x0006706e735f6672000868706172736f6e7300000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6b945e10000000000006000376616c00000004eacf6b945e1000000004000010f7 > >> 78078974-1f02-4da8-9902-cca56291a9dc | > 4e64df70-49ec-11e3-af88-ed7ea188ea44 | 6 | > 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf6c49f270000000000006000376616c00000004eacf6c49f2700000000400001618 > >> > >> at t2 > >> > >> cqlsh:pns_fr> TRUNCATE system.hints ; > >> cqlsh:pns_fr> select * from system.hints ; > >> > >> node 2 > >> > >> at t1 > >> > >> cqlsh:pns_fr> select * from system.hints ; > >> > >> target_id | hint_id > | message_version | mutation > >> > --------------------------------------+--------------------------------------+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > >> 78078974-1f02-4da8-9902-cca56291a9dc | > aa8f4da0-49ea-11e3-a8dc-e367b713537d | 6 | > 0x0006706e735f667200086373636574626f6e00000001019cad5323f2a6389388a31004124bdf2d7fffffff800000000000000000000000000000020003000000000004eacf424d88b0000000000006000376616c00000004eacf424d88b000000004000004bc > >> > >> at t2 > >> > >> cqlsh:pns_fr> select * from system.hints ; > >> > >> Regards > >> -- > >> Cyril SCETBON > >> > >> On 09 Nov 2013, at 22:48, Jonathan Ellis <jbel...@gmail.com> wrote: > >> > >>> Hints are not replicated (technically: replicated with LocalStrategy), > >>> so truncating hints will always only affect the local node. > >>> > >>> On Sat, Nov 9, 2013 at 3:44 PM, Cyril Scetbon <cyril.scet...@free.fr> > wrote: > >>>> Hi, > >>>> > >>>> I would like to add a feature request for local truncates of hints > but I want to know your opinion before. > >>>> > >>>> We faced an issue last week concerning 3 new nodes that where not > seen by some other nodes in the ring. a week later we resolved the issues > but it was taking days with a high CPU load (85%) to apply GBytes of Hints. > I noticed that CF Hints is defined with LocalStategy and so thought the > truncate was only local but that wasn't. We had to remove sstables, > commitlog and cached to remove all that old hints stored but not removed > after max_hint_window_in_ms as target nodes where not dead (they were only > not accepting hints). > >>>> > >>>> So, I suggest that the truncate of hints stay local (maybe you need > it to stay global per default ?) or that we could do it locally (a truncate > argument or a new command like the one used for CONSISTENCY) > >>>> > >>>> thanks > >>>> -- > >>>> Cyril SCETBON > >>>> > >>> > >>> > >>> > >>> -- > >>> Jonathan Ellis > >>> Project Chair, Apache Cassandra > >>> co-founder, http://www.datastax.com > >>> @spyced > >> > > > > > > > > -- > > Jonathan Ellis > > Project Chair, Apache Cassandra > > co-founder, http://www.datastax.com > > @spyced > >
