[jira] [Resolved] (STORM-2869) KafkaSpout discards all pending records when adjusting the consumer position after a commit
[ https://issues.apache.org/jira/browse/STORM-2869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved STORM-2869. - Resolution: Fixed Fix Version/s: 1.2.0 2.0.0 Thanks [~Srdo], I merged into master and 1.x-branch. > KafkaSpout discards all pending records when adjusting the consumer position > after a commit > --- > > Key: STORM-2869 > URL: https://issues.apache.org/jira/browse/STORM-2869 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.2.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing > Labels: pull-request-available > Fix For: 2.0.0, 1.2.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > As part of the STORM-2666 fix the spout clears out waitingToEmit when the > consumer position falls behind the committed offset during a commit. We only > need to do it for the affected partition, and then only for the records that > are behind the committed offset. > Also the validation check in emitOrRetryTuple is slightly too permissive, it > should check whether the current record is behind the committed offset, not > whether the consumer position is behind the committed offset. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2872) Memory availability and rebalance broken in GRAS
[ https://issues.apache.org/jira/browse/STORM-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated STORM-2872: -- Labels: pull-request-available (was: ) > Memory availability and rebalance broken in GRAS > > > Key: STORM-2872 > URL: https://issues.apache.org/jira/browse/STORM-2872 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-core >Reporter: Govind Menon >Assignee: Govind Menon > Labels: pull-request-available > Fix For: 2.0.0 > > > * The calculation for available memory resources was broken in the original > GRAS implementation due to incorrect string comparison > * Rebalance was also broken -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2872) Memory availability and rebalance broken in GRAS
[ https://issues.apache.org/jira/browse/STORM-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Govind Menon updated STORM-2872: Description: * The calculation for available memory resources was broken in the original GRAS implementation due to incorrect string comparison * Rebalance was also broken > Memory availability and rebalance broken in GRAS > > > Key: STORM-2872 > URL: https://issues.apache.org/jira/browse/STORM-2872 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-core >Reporter: Govind Menon >Assignee: Govind Menon > Labels: pull-request-available > Fix For: 2.0.0 > > > * The calculation for available memory resources was broken in the original > GRAS implementation due to incorrect string comparison > * Rebalance was also broken -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (STORM-2872) Memory availability and rebalance broken in GRAS
Govind Menon created STORM-2872: --- Summary: Memory availability and rebalance broken in GRAS Key: STORM-2872 URL: https://issues.apache.org/jira/browse/STORM-2872 Project: Apache Storm Issue Type: Sub-task Reporter: Govind Menon Assignee: Govind Menon -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2856) Make Storm build work on post 2017Q4 Travis Trusty image
[ https://issues.apache.org/jira/browse/STORM-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated STORM-2856: -- Labels: pull-request-available (was: ) > Make Storm build work on post 2017Q4 Travis Trusty image > > > Key: STORM-2856 > URL: https://issues.apache.org/jira/browse/STORM-2856 > Project: Apache Storm > Issue Type: Bug >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing > Labels: pull-request-available > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2859) NormalizedResources has some bugs in special cases where 0 of a resource is available.
[ https://issues.apache.org/jira/browse/STORM-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2859: -- Issue Type: Bug (was: Sub-task) Parent: (was: STORM-2856) > NormalizedResources has some bugs in special cases where 0 of a resource is > available. > -- > > Key: STORM-2859 > URL: https://issues.apache.org/jira/browse/STORM-2859 > Project: Apache Storm > Issue Type: Bug > Components: storm-server >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2859) NormalizedResources has some bugs in special cases where 0 of a resource is available.
[ https://issues.apache.org/jira/browse/STORM-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stig Rohde Døssing updated STORM-2859: -- Summary: NormalizedResources has some bugs in special cases where 0 of a resource is available. (was: NormalizedResources is leaking static state in tests, and has some other bugs in special cases where 0 of a resource is available) > NormalizedResources has some bugs in special cases where 0 of a resource is > available. > -- > > Key: STORM-2859 > URL: https://issues.apache.org/jira/browse/STORM-2859 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-server >Affects Versions: 2.0.0 >Reporter: Stig Rohde Døssing >Assignee: Stig Rohde Døssing > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2871) Performance optimizations for getOutgoingTasks
[ https://issues.apache.org/jira/browse/STORM-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-2871: --- Component/s: storm-client > Performance optimizations for getOutgoingTasks > --- > > Key: STORM-2871 > URL: https://issues.apache.org/jira/browse/STORM-2871 > Project: Apache Storm > Issue Type: Improvement > Components: storm-client >Affects Versions: 2.0.0 >Reporter: Roshan Naik > > Task.getOutgoingTasks() is in critical messaging path. Two observed > bottlenecks in it : > - Looking up HashMap 'streamToGroupers'. Need to look into converting HashMap > into Array lookup ? > - > [outTasks.addAll(compTasks)|https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/Task.java#L139] > seems to be impacting throughput as well. Identified by .. running > ConstSpoutNullBoltTopo with 1 spout & bolt paralllelism (no Ack) and > replacing this line with hard coded logic to add the single known bolt's > taskID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2871) Performance optimizations for getOutgoingTasks
[ https://issues.apache.org/jira/browse/STORM-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-2871: --- Affects Version/s: 2.0.0 > Performance optimizations for getOutgoingTasks > --- > > Key: STORM-2871 > URL: https://issues.apache.org/jira/browse/STORM-2871 > Project: Apache Storm > Issue Type: Improvement > Components: storm-client >Affects Versions: 2.0.0 >Reporter: Roshan Naik > > Task.getOutgoingTasks() is in critical messaging path. Two observed > bottlenecks in it : > - Looking up HashMap 'streamToGroupers'. Need to look into converting HashMap > into Array lookup ? > - > [outTasks.addAll(compTasks)|https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/Task.java#L139] > seems to be impacting throughput as well. Identified by .. running > ConstSpoutNullBoltTopo with 1 spout & bolt paralllelism (no Ack) and > replacing this line with hard coded logic to add the single known bolt's > taskID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2871) Performance optimizations for getOutgoingTasks
[ https://issues.apache.org/jira/browse/STORM-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-2871: --- Description: Task.getOutgoingTasks() is in critical messaging path. Two observed bottlenecks in it : - [Looking up HashMap|] 'streamToGroupers'. Need to look into converting HashMap into Array lookup ? - [outTasks.addAll(compTasks)|https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/Task.java#L139] seems to be impacting throughput as well. Identified by .. running ConstSpoutNullBoltTopo with 1 spout & bolt paralllelism (no Ack) and replacing this line with hard coded logic to add the single known bolt's taskID. was: Task.getOutgoingTasks() is in critical messaging path. Two observed bottlenecks in it : -[Looking up HashMap|] 'streamToGroupers'. Need to look into converting HashMap into Array lookup ? - [outTasks.addAll(compTasks)|https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/Task.java#L139] seems to be impacting throughput as well. Identified by .. running ConstSpoutNullBoltTopo with 1 spout & bolt paralllelism (no Ack) and replacing this line with hard coded logic to add the single known bolt's taskID. > Performance optimizations for getOutgoingTasks > --- > > Key: STORM-2871 > URL: https://issues.apache.org/jira/browse/STORM-2871 > Project: Apache Storm > Issue Type: Improvement >Reporter: Roshan Naik > > Task.getOutgoingTasks() is in critical messaging path. Two observed > bottlenecks in it : > - [Looking up HashMap|] 'streamToGroupers'. Need to look into converting > HashMap into Array lookup ? > - > [outTasks.addAll(compTasks)|https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/Task.java#L139] > seems to be impacting throughput as well. Identified by .. running > ConstSpoutNullBoltTopo with 1 spout & bolt paralllelism (no Ack) and > replacing this line with hard coded logic to add the single known bolt's > taskID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2871) Performance optimizations for getOutgoingTasks
[ https://issues.apache.org/jira/browse/STORM-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-2871: --- Description: Task.getOutgoingTasks() is in critical messaging path. Two observed bottlenecks in it : - Looking up HashMap 'streamToGroupers'. Need to look into converting HashMap into Array lookup ? - [outTasks.addAll(compTasks)|https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/Task.java#L139] seems to be impacting throughput as well. Identified by .. running ConstSpoutNullBoltTopo with 1 spout & bolt paralllelism (no Ack) and replacing this line with hard coded logic to add the single known bolt's taskID. was: Task.getOutgoingTasks() is in critical messaging path. Two observed bottlenecks in it : - [Looking up HashMap|] 'streamToGroupers'. Need to look into converting HashMap into Array lookup ? - [outTasks.addAll(compTasks)|https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/Task.java#L139] seems to be impacting throughput as well. Identified by .. running ConstSpoutNullBoltTopo with 1 spout & bolt paralllelism (no Ack) and replacing this line with hard coded logic to add the single known bolt's taskID. > Performance optimizations for getOutgoingTasks > --- > > Key: STORM-2871 > URL: https://issues.apache.org/jira/browse/STORM-2871 > Project: Apache Storm > Issue Type: Improvement >Reporter: Roshan Naik > > Task.getOutgoingTasks() is in critical messaging path. Two observed > bottlenecks in it : > - Looking up HashMap 'streamToGroupers'. Need to look into converting HashMap > into Array lookup ? > - > [outTasks.addAll(compTasks)|https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/Task.java#L139] > seems to be impacting throughput as well. Identified by .. running > ConstSpoutNullBoltTopo with 1 spout & bolt paralllelism (no Ack) and > replacing this line with hard coded logic to add the single known bolt's > taskID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (STORM-2871) Performance optimizations for getOutgoingTasks
Roshan Naik created STORM-2871: -- Summary: Performance optimizations for getOutgoingTasks Key: STORM-2871 URL: https://issues.apache.org/jira/browse/STORM-2871 Project: Apache Storm Issue Type: Improvement Reporter: Roshan Naik Task.getOutgoingTasks() is in critical messaging path. Two observed bottlenecks in it : -[Looking up HashMap|] 'streamToGroupers'. Need to look into converting HashMap into Array lookup ? - [outTasks.addAll(compTasks)|https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/Task.java#L139] seems to be impacting throughput as well. Identified by .. running ConstSpoutNullBoltTopo with 1 spout & bolt paralllelism (no Ack) and replacing this line with hard coded logic to add the single known bolt's taskID. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
