[jira] [Commented] (CASSANDRA-11168) Hint Metrics are updated even if hinted_hand-offs=false
[ https://issues.apache.org/jira/browse/CASSANDRA-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193558#comment-15193558 ] Joel Knighton commented on CASSANDRA-11168: --- +1. CI looks clean - I ran for 2.2, 3.0, 3.5, and trunk. For committer: there are two patches to be applied. One for 2.2 and one for 3.0+. These are the two most recent patches attached to the issue. The 2.2 patch is attached to the issue as 0001-Hinted-Handoff-fix-2_2.patch. The 3.0+ patch is attached to the issue as 0001-Hinted-handoffs-fix.patch. This patch should merge forward cleanly up to trunk. > Hint Metrics are updated even if hinted_hand-offs=false > --- > > Key: CASSANDRA-11168 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11168 > Project: Cassandra > Issue Type: Bug >Reporter: Anubhav Kale >Assignee: Anubhav Kale >Priority: Minor > Attachments: 0001-Hinted-Handoff-Fix.patch, > 0001-Hinted-Handoff-fix-2_2.patch, 0001-Hinted-handoff-metrics.patch, > 0001-Hinted-handoffs-fix.patch > > > In our PROD logs, we noticed a lot of hint metrics even though we have > disabled hinted handoffs. > The reason is StorageProxy.ShouldHint has an inverted if condition. > We should also wrap the if (hintWindowExpired) block in if > (DatabaseDescriptor.hintedHandoffEnabled()). > The fix is easy, and I can provide a patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11168) Hint Metrics are updated even if hinted_hand-offs=false
[ https://issues.apache.org/jira/browse/CASSANDRA-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189855#comment-15189855 ] Anubhav Kale commented on CASSANDRA-11168: -- My bad on the trunk patch. Updated. > Hint Metrics are updated even if hinted_hand-offs=false > --- > > Key: CASSANDRA-11168 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11168 > Project: Cassandra > Issue Type: Bug >Reporter: Anubhav Kale >Assignee: Anubhav Kale >Priority: Minor > Attachments: 0001-Hinted-Handoff-Fix.patch, > 0001-Hinted-Handoff-fix-2_2.patch, 0001-Hinted-handoff-metrics.patch, > 0001-Hinted-handoffs-fix.patch > > > In our PROD logs, we noticed a lot of hint metrics even though we have > disabled hinted handoffs. > The reason is StorageProxy.ShouldHint has an inverted if condition. > We should also wrap the if (hintWindowExpired) block in if > (DatabaseDescriptor.hintedHandoffEnabled()). > The fix is easy, and I can provide a patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11168) Hint Metrics are updated even if hinted_hand-offs=false
[ https://issues.apache.org/jira/browse/CASSANDRA-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189739#comment-15189739 ] Anubhav Kale commented on CASSANDRA-11168: -- I have attached for 2.2 as well. > Hint Metrics are updated even if hinted_hand-offs=false > --- > > Key: CASSANDRA-11168 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11168 > Project: Cassandra > Issue Type: Bug >Reporter: Anubhav Kale >Assignee: Anubhav Kale >Priority: Minor > Attachments: 0001-Hinted-Handoff-Fix.patch, > 0001-Hinted-Handoff-fix-2_2.patch, 0001-Hinted-handoff-metrics.patch > > > In our PROD logs, we noticed a lot of hint metrics even though we have > disabled hinted handoffs. > The reason is StorageProxy.ShouldHint has an inverted if condition. > We should also wrap the if (hintWindowExpired) block in if > (DatabaseDescriptor.hintedHandoffEnabled()). > The fix is easy, and I can provide a patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11168) Hint Metrics are updated even if hinted_hand-offs=false
[ https://issues.apache.org/jira/browse/CASSANDRA-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15188197#comment-15188197 ] Anubhav Kale commented on CASSANDRA-11168: -- Updated patch. Not really sure if its really necessary to be back-ported though. > Hint Metrics are updated even if hinted_hand-offs=false > --- > > Key: CASSANDRA-11168 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11168 > Project: Cassandra > Issue Type: Bug >Reporter: Anubhav Kale >Assignee: Anubhav Kale >Priority: Minor > Attachments: 0001-Hinted-Handoff-Fix.patch, > 0001-Hinted-handoff-metrics.patch > > > In our PROD logs, we noticed a lot of hint metrics even though we have > disabled hinted handoffs. > The reason is StorageProxy.ShouldHint has an inverted if condition. > We should also wrap the if (hintWindowExpired) block in if > (DatabaseDescriptor.hintedHandoffEnabled()). > The fix is easy, and I can provide a patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11168) Hint Metrics are updated even if hinted_hand-offs=false
[ https://issues.apache.org/jira/browse/CASSANDRA-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15183387#comment-15183387 ] Joel Knighton commented on CASSANDRA-11168: --- Yes, this accomplishes the desired behavior with regard to the metric. It isn't clear to me why the changes to the scope of the if from the original code are needed. Is there a change in logical behavior or is it just a suggested code style change? I think a patch that simply removes the metric increment in the case of early returns due to hinted handoff being disabled would be sufficient. > Hint Metrics are updated even if hinted_hand-offs=false > --- > > Key: CASSANDRA-11168 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11168 > Project: Cassandra > Issue Type: Bug >Reporter: Anubhav Kale >Assignee: Anubhav Kale >Priority: Minor > Attachments: 0001-Hinted-Handoff-Fix.patch > > > In our PROD logs, we noticed a lot of hint metrics even though we have > disabled hinted handoffs. > The reason is StorageProxy.ShouldHint has an inverted if condition. > We should also wrap the if (hintWindowExpired) block in if > (DatabaseDescriptor.hintedHandoffEnabled()). > The fix is easy, and I can provide a patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11168) Hint Metrics are updated even if hinted_hand-offs=false
[
https://issues.apache.org/jira/browse/CASSANDRA-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15176707#comment-15176707
]
Anubhav Kale commented on CASSANDRA-11168:
--
So, making sure I understand Aleksey's thought this correctly, what we want is
below. Can you confirm (only increment if hint window expired) ?
if (DatabaseDescriptor.hintedHandoffEnabled())
{
Set disabledDCs =
DatabaseDescriptor.hintedHandoffDisabledDCs();
if (!disabledDCs.isEmpty())
{
final String dc =
DatabaseDescriptor.getEndpointSnitch().getDatacenter(ep);
if (disabledDCs.contains(dc))
{
Tracing.trace("Not hinting {} since its
data center {} has been disabled {}", ep, dc, disabledDCs);
return false;
}
}
boolean hintWindowExpired =
Gossiper.instance.getEndpointDowntime(ep) >
DatabaseDescriptor.getMaxHintWindow();
if (hintWindowExpired)
{
HintsService.instance.metrics.incrPastWindow(ep);
Tracing.trace("Not hinting {} which has been
down {} ms", ep, Gossiper.instance.getEndpointDowntime(ep));
}
return !hintWindowExpired;
}
else
{
return false;
}
> Hint Metrics are updated even if hinted_hand-offs=false
> ---
>
> Key: CASSANDRA-11168
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11168
> Project: Cassandra
> Issue Type: Bug
>Reporter: Anubhav Kale
>Assignee: Anubhav Kale
>Priority: Minor
> Attachments: 0001-Hinted-Handoff-Fix.patch
>
>
> In our PROD logs, we noticed a lot of hint metrics even though we have
> disabled hinted handoffs.
> The reason is StorageProxy.ShouldHint has an inverted if condition.
> We should also wrap the if (hintWindowExpired) block in if
> (DatabaseDescriptor.hintedHandoffEnabled()).
> The fix is easy, and I can provide a patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11168) Hint Metrics are updated even if hinted_hand-offs=false
[ https://issues.apache.org/jira/browse/CASSANDRA-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15174002#comment-15174002 ] Joel Knighton commented on CASSANDRA-11168: --- Works for me. A patch for both 2.2 and 3.0+ would be appreciated [~anubhavk]. > Hint Metrics are updated even if hinted_hand-offs=false > --- > > Key: CASSANDRA-11168 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11168 > Project: Cassandra > Issue Type: Bug >Reporter: Anubhav Kale >Assignee: Anubhav Kale >Priority: Minor > Attachments: 0001-Hinted-Handoff-Fix.patch > > > In our PROD logs, we noticed a lot of hint metrics even though we have > disabled hinted handoffs. > The reason is StorageProxy.ShouldHint has an inverted if condition. > We should also wrap the if (hintWindowExpired) block in if > (DatabaseDescriptor.hintedHandoffEnabled()). > The fix is easy, and I can provide a patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11168) Hint Metrics are updated even if hinted_hand-offs=false
[ https://issues.apache.org/jira/browse/CASSANDRA-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15173961#comment-15173961 ] Aleksey Yeschenko commented on CASSANDRA-11168: --- I don't think this really matters. That said, I'm slightly inclined towards making the suggested change (only increment if hints are both enabled and past window). > Hint Metrics are updated even if hinted_hand-offs=false > --- > > Key: CASSANDRA-11168 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11168 > Project: Cassandra > Issue Type: Bug >Reporter: Anubhav Kale >Assignee: Anubhav Kale >Priority: Minor > Attachments: 0001-Hinted-Handoff-Fix.patch > > > In our PROD logs, we noticed a lot of hint metrics even though we have > disabled hinted handoffs. > The reason is StorageProxy.ShouldHint has an inverted if condition. > We should also wrap the if (hintWindowExpired) block in if > (DatabaseDescriptor.hintedHandoffEnabled()). > The fix is easy, and I can provide a patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11168) Hint Metrics are updated even if hinted_hand-offs=false
[ https://issues.apache.org/jira/browse/CASSANDRA-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15154317#comment-15154317 ] Joshua McKenzie commented on CASSANDRA-11168: - A patch would be great. We'll get a reviewer for that when you have that available. Thanks. > Hint Metrics are updated even if hinted_hand-offs=false > --- > > Key: CASSANDRA-11168 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11168 > Project: Cassandra > Issue Type: Bug >Reporter: Anubhav Kale >Assignee: Anubhav Kale >Priority: Minor > > In our PROD logs, we noticed a lot of hint metrics even though we have > disabled hinted handoffs. > The reason is StorageProxy.ShouldHint has an inverted if condition. > We should also wrap the if (hintWindowExpired) block in if > (DatabaseDescriptor.hintedHandoffEnabled()). > The fix is easy, and I can provide a patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
