Re: force processing of pending hinted handoffs
I encountered this situation also once or twice but didn't succeed. I just deleted the old hints and ran a repair Am 13.04.2017 10:35 schrieb "Roland Otta": > unfortunately it does not. > > i guess this is intended for resuming hinted handoff handling in case it > hase been paused with the pausehandoff before. > i have tested it (resuming .. pausing & resuming) but it has no effect on > those old hints > > On Thu, 2017-04-13 at 10:27 +0200, benjamin roth wrote: > > There is a nodetool command to resume hints. Maybe that helps? > > Am 13.04.2017 09:42 schrieb "Roland Otta" : > > oh ... the operation is deprecated according to the docs ... > > > On Thu, 2017-04-13 at 07:40 +, Roland Otta wrote: > > i figured out that there is an mbean > > org.apache.cassandra.db.type=HintedHandoffManager with the operation > > scheduleHintDelivery > > > > i guess thats what i would need in that case. at least the docs let > > me > > think so http://javadox.com/org.apache.cassandra/cassandra-all/3.0.0/ > > or > > g/apache/cassandra/db/HintedHandOffManagerMBean.html > > > > but everytime i try invoking that operation i get an > > UnsupportedOperationException (tried it with hostname, ip and host-id > > as parameters - everytime the same exception) > > > > > > > > On Tue, 2017-04-11 at 07:40 +, Roland Otta wrote: > > > hi, > > > > > > sometimes we have the problem that we have hinted handoffs (for > > > example > > > because auf network problems between 2 DCs) that do not get > > > processed > > > even if the connection problem between the dcs recovers. Some of > > > the > > > files stay in the hints directory until we restart the node that > > > contains the hints. > > > > > > after the restart of cassandra we can see the proper messages for > > > the > > > hints handling > > > > > > Apr 11 09:28:56 bigd006 cassandra: INFO 07:28:56 Deleted hint file > > > c429ad19-ee9f-4b5a-abcd-1da1516d1003-1491895717182-1.hints > > > Apr 11 09:28:56 bigd006 cassandra: INFO 07:28:56 Finished hinted > > > handoff of file c429ad19-ee9f-4b5a-abcd-1da1516d1003-1491895717182- > > > 1.hints to endpoint c429ad19-ee9f-4b5a-abcd-1da1516d1003 > > > > > > is there a way (for example via jmx) to force a node to process > > > outstanding hints instead of restarting the node? > > > does anyone know whats the cause for not retrying to process those > > > hints automatically? > > > > > > br, > > > roland > > > > >
Re: force processing of pending hinted handoffs
unfortunately it does not. i guess this is intended for resuming hinted handoff handling in case it hase been paused with the pausehandoff before. i have tested it (resuming .. pausing & resuming) but it has no effect on those old hints On Thu, 2017-04-13 at 10:27 +0200, benjamin roth wrote: There is a nodetool command to resume hints. Maybe that helps? Am 13.04.2017 09:42 schrieb "Roland Otta">: oh ... the operation is deprecated according to the docs ... On Thu, 2017-04-13 at 07:40 +, Roland Otta wrote: > i figured out that there is an mbean > org.apache.cassandra.db.type=HintedHandoffManager with the operation > scheduleHintDelivery > > i guess thats what i would need in that case. at least the docs let > me > think so http://javadox.com/org.apache.cassandra/cassandra-all/3.0.0/ > or > g/apache/cassandra/db/HintedHandOffManagerMBean.html > > but everytime i try invoking that operation i get an > UnsupportedOperationException (tried it with hostname, ip and host-id > as parameters - everytime the same exception) > > > > On Tue, 2017-04-11 at 07:40 +, Roland Otta wrote: > > hi, > > > > sometimes we have the problem that we have hinted handoffs (for > > example > > because auf network problems between 2 DCs) that do not get > > processed > > even if the connection problem between the dcs recovers. Some of > > the > > files stay in the hints directory until we restart the node that > > contains the hints. > > > > after the restart of cassandra we can see the proper messages for > > the > > hints handling > > > > Apr 11 09:28:56 bigd006 cassandra: INFO 07:28:56 Deleted hint file > > c429ad19-ee9f-4b5a-abcd-1da1516d1003-1491895717182-1.hints > > Apr 11 09:28:56 bigd006 cassandra: INFO 07:28:56 Finished hinted > > handoff of file c429ad19-ee9f-4b5a-abcd-1da1516d1003-1491895717182- > > 1.hints to endpoint c429ad19-ee9f-4b5a-abcd-1da1516d1003 > > > > is there a way (for example via jmx) to force a node to process > > outstanding hints instead of restarting the node? > > does anyone know whats the cause for not retrying to process those > > hints automatically? > > > > br, > > roland > >
Re: force processing of pending hinted handoffs
There is a nodetool command to resume hints. Maybe that helps? Am 13.04.2017 09:42 schrieb "Roland Otta": > oh ... the operation is deprecated according to the docs ... > > > On Thu, 2017-04-13 at 07:40 +, Roland Otta wrote: > > i figured out that there is an mbean > > org.apache.cassandra.db.type=HintedHandoffManager with the operation > > scheduleHintDelivery > > > > i guess thats what i would need in that case. at least the docs let > > me > > think so http://javadox.com/org.apache.cassandra/cassandra-all/3.0.0/ > > or > > g/apache/cassandra/db/HintedHandOffManagerMBean.html > > > > but everytime i try invoking that operation i get an > > UnsupportedOperationException (tried it with hostname, ip and host-id > > as parameters - everytime the same exception) > > > > > > > > On Tue, 2017-04-11 at 07:40 +, Roland Otta wrote: > > > hi, > > > > > > sometimes we have the problem that we have hinted handoffs (for > > > example > > > because auf network problems between 2 DCs) that do not get > > > processed > > > even if the connection problem between the dcs recovers. Some of > > > the > > > files stay in the hints directory until we restart the node that > > > contains the hints. > > > > > > after the restart of cassandra we can see the proper messages for > > > the > > > hints handling > > > > > > Apr 11 09:28:56 bigd006 cassandra: INFO 07:28:56 Deleted hint file > > > c429ad19-ee9f-4b5a-abcd-1da1516d1003-1491895717182-1.hints > > > Apr 11 09:28:56 bigd006 cassandra: INFO 07:28:56 Finished hinted > > > handoff of file c429ad19-ee9f-4b5a-abcd-1da1516d1003-1491895717182- > > > 1.hints to endpoint c429ad19-ee9f-4b5a-abcd-1da1516d1003 > > > > > > is there a way (for example via jmx) to force a node to process > > > outstanding hints instead of restarting the node? > > > does anyone know whats the cause for not retrying to process those > > > hints automatically? > > > > > > br, > > > roland > > >
Re: force processing of pending hinted handoffs
oh ... the operation is deprecated according to the docs ... On Thu, 2017-04-13 at 07:40 +, Roland Otta wrote: > i figured out that there is an mbean > org.apache.cassandra.db.type=HintedHandoffManager with the operation > scheduleHintDelivery > > i guess thats what i would need in that case. at least the docs let > me > think so http://javadox.com/org.apache.cassandra/cassandra-all/3.0.0/ > or > g/apache/cassandra/db/HintedHandOffManagerMBean.html > > but everytime i try invoking that operation i get an > UnsupportedOperationException (tried it with hostname, ip and host-id > as parameters - everytime the same exception) > > > > On Tue, 2017-04-11 at 07:40 +, Roland Otta wrote: > > hi, > > > > sometimes we have the problem that we have hinted handoffs (for > > example > > because auf network problems between 2 DCs) that do not get > > processed > > even if the connection problem between the dcs recovers. Some of > > the > > files stay in the hints directory until we restart the node that > > contains the hints. > > > > after the restart of cassandra we can see the proper messages for > > the > > hints handling > > > > Apr 11 09:28:56 bigd006 cassandra: INFO 07:28:56 Deleted hint file > > c429ad19-ee9f-4b5a-abcd-1da1516d1003-1491895717182-1.hints > > Apr 11 09:28:56 bigd006 cassandra: INFO 07:28:56 Finished hinted > > handoff of file c429ad19-ee9f-4b5a-abcd-1da1516d1003-1491895717182- > > 1.hints to endpoint c429ad19-ee9f-4b5a-abcd-1da1516d1003 > > > > is there a way (for example via jmx) to force a node to process > > outstanding hints instead of restarting the node? > > does anyone know whats the cause for not retrying to process those > > hints automatically? > > > > br, > > roland > >
Re: force processing of pending hinted handoffs
i figured out that there is an mbean org.apache.cassandra.db.type=HintedHandoffManager with the operation scheduleHintDelivery i guess thats what i would need in that case. at least the docs let me think so http://javadox.com/org.apache.cassandra/cassandra-all/3.0.0/or g/apache/cassandra/db/HintedHandOffManagerMBean.html but everytime i try invoking that operation i get an UnsupportedOperationException (tried it with hostname, ip and host-id as parameters - everytime the same exception) On Tue, 2017-04-11 at 07:40 +, Roland Otta wrote: > hi, > > sometimes we have the problem that we have hinted handoffs (for > example > because auf network problems between 2 DCs) that do not get processed > even if the connection problem between the dcs recovers. Some of the > files stay in the hints directory until we restart the node that > contains the hints. > > after the restart of cassandra we can see the proper messages for the > hints handling > > Apr 11 09:28:56 bigd006 cassandra: INFO 07:28:56 Deleted hint file > c429ad19-ee9f-4b5a-abcd-1da1516d1003-1491895717182-1.hints > Apr 11 09:28:56 bigd006 cassandra: INFO 07:28:56 Finished hinted > handoff of file c429ad19-ee9f-4b5a-abcd-1da1516d1003-1491895717182- > 1.hints to endpoint c429ad19-ee9f-4b5a-abcd-1da1516d1003 > > is there a way (for example via jmx) to force a node to process > outstanding hints instead of restarting the node? > does anyone know whats the cause for not retrying to process those > hints automatically? > > br, > roland > >