[jira] [Commented] (STORM-2844) KafkaSpout Throws IllegalStateException After Committing to Kafka When First Poll Strategy Set to EARLIEST
[ https://issues.apache.org/jira/browse/STORM-2844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280858#comment-16280858 ] Stig Rohde Døssing commented on STORM-2844: --- No, I don't know why it adds 1. It doesn't make sense for totalTaskCount to be 0 as far as I can tell. > KafkaSpout Throws IllegalStateException After Committing to Kafka When First > Poll Strategy Set to EARLIEST > -- > > Key: STORM-2844 > URL: https://issues.apache.org/jira/browse/STORM-2844 > Project: Apache Storm > Issue Type: Bug > Components: storm-kafka-client >Affects Versions: 2.0.0, 1.2.0 >Reporter: Hugo Louro >Assignee: Hugo Louro >Priority: Critical > > This > [code|https://github.com/apache/storm/blob/1.x-branch/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java#L407-L409], > which was committed to fix > [STORM-2666|https://issues.apache.org/jira/browse/STORM-2666] throws > IllegalStateException when the KafkaSpout commits to Kafka and is restarted > with the same consumer group id and first poll strategy is set to EARLIEST. > For example consider the following sequence: > # KafkaSpout with consumer_group_id=TEST polls and commits offsets 1-5 > # KafkaSpout with consumer_group_id=TEST is restarted with first poll > strategy set to EARLIEST > ==> IllegalStateException will be thrown > This bug could be a blocker. I am setting it to Critical because assigning a > different consumer id serves as a workaround to the problem. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (STORM-2306) Redeisgn Messaging Subsystem and switch to JCTools Queues
[ https://issues.apache.org/jira/browse/STORM-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280106#comment-16280106 ] Roshan Naik commented on STORM-2306: Since the PR webpage has become terribly slow due to the large number of comments, moving the non-code-review type conversation to the JIRA. The PR is ready to move forward with additional reviews and testing by anyone interested in test driving their topologies on this (much appreciated!). - I believe all the key issues raised so far should be addressed, please take a look. Also includes fixes to issues discovered during testing and perf runs. - Have added a new design doc detailing the BackPressure model. That is the part that has undergone the most change off late. - Based on my observations from perf runs and prior feedback on the PR, the new defaults have been tweaked to make it easy for existing workloads to transition to this with minimal or no tweaking but still get good perf. - My colleague is in the process of running some perf numbers comparing master vs the latest 2306. Will share them soon. cc: [~revans2] > Redeisgn Messaging Subsystem and switch to JCTools Queues > - > > Key: STORM-2306 > URL: https://issues.apache.org/jira/browse/STORM-2306 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Roshan Naik > Labels: pull-request-available > Time Spent: 33h > Remaining Estimate: 0h > > Details in these documents: > 1) *Redesign of the messaging subsystem* > https://docs.google.com/document/d/1NK1DJ3aAkta-Im0m-2FObQ4cSRp8xSa301y6zoqcBeE/edit?usp=sharing > This doc discusses the new design for the messaging system. Plus some of the > optimizations being made. > 2) *Choosing a high performance messaging queue:* > https://docs.google.com/document/d/1PpQaWVHg06-OqxTzYxQlzg1yEhzA4Y46_NC7HMO6tsI/edit?usp=sharing > This doc looks into how fast hardware can do inter-thread messaging and why > we chose the JCTools queues. > 3) *Backpressure Model* > https://docs.google.com/document/d/1Z9pRdI5wtnK-hVwE3Spe6VGCTsz9g8TkgxbTFcbL3jM/edit?usp=sharing > Describes the Backpressure model integrated into the new messaging subsystem. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2306) Redeisgn Messaging Subsystem and switch to JCTools Queues
[ https://issues.apache.org/jira/browse/STORM-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-2306: --- Description: Details in these documents: 1) *Redesign of the messaging subsystem* https://docs.google.com/document/d/1NK1DJ3aAkta-Im0m-2FObQ4cSRp8xSa301y6zoqcBeE/edit?usp=sharing This doc discusses the new design for the messaging system. Plus some of the optimizations being made. 2) *Choosing a high performance messaging queue:* https://docs.google.com/document/d/1PpQaWVHg06-OqxTzYxQlzg1yEhzA4Y46_NC7HMO6tsI/edit?usp=sharing This doc looks into how fast hardware can do inter-thread messaging and why we chose the JCTools queues. 3) *Backpressure Model* https://docs.google.com/document/d/1Z9pRdI5wtnK-hVwE3Spe6VGCTsz9g8TkgxbTFcbL3jM/edit?usp=sharing Describes the Backpressure model integrated into the new messaging subsystem. was: Details in these documents: 1) *Redesign of the messaging subsystem* https://docs.google.com/document/d/1NK1DJ3aAkta-Im0m-2FObQ4cSRp8xSa301y6zoqcBeE/edit?usp=sharing This doc discusses the new design for the messaging system. Plus some of the optimizations being made. 2) *Choosing a high performance messaging queue:* https://docs.google.com/document/d/1PpQaWVHg06-OqxTzYxQlzg1yEhzA4Y46_NC7HMO6tsI/edit?usp=sharing This doc looks into how fast hardware can do inter-thread messaging and why we chose the JCTools queues. 3) *BackPressure Model* https://docs.google.com/document/d/1Z9pRdI5wtnK-hVwE3Spe6VGCTsz9g8TkgxbTFcbL3jM/edit?usp=sharing > Redeisgn Messaging Subsystem and switch to JCTools Queues > - > > Key: STORM-2306 > URL: https://issues.apache.org/jira/browse/STORM-2306 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Roshan Naik > Labels: pull-request-available > Time Spent: 33h > Remaining Estimate: 0h > > Details in these documents: > 1) *Redesign of the messaging subsystem* > https://docs.google.com/document/d/1NK1DJ3aAkta-Im0m-2FObQ4cSRp8xSa301y6zoqcBeE/edit?usp=sharing > This doc discusses the new design for the messaging system. Plus some of the > optimizations being made. > 2) *Choosing a high performance messaging queue:* > https://docs.google.com/document/d/1PpQaWVHg06-OqxTzYxQlzg1yEhzA4Y46_NC7HMO6tsI/edit?usp=sharing > This doc looks into how fast hardware can do inter-thread messaging and why > we chose the JCTools queues. > 3) *Backpressure Model* > https://docs.google.com/document/d/1Z9pRdI5wtnK-hVwE3Spe6VGCTsz9g8TkgxbTFcbL3jM/edit?usp=sharing > Describes the Backpressure model integrated into the new messaging subsystem. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (STORM-2306) Redeisgn Messaging Subsystem and switch to JCTools Queues
[ https://issues.apache.org/jira/browse/STORM-2306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-2306: --- Description: Details in these documents: 1) *Redesign of the messaging subsystem* https://docs.google.com/document/d/1NK1DJ3aAkta-Im0m-2FObQ4cSRp8xSa301y6zoqcBeE/edit?usp=sharing This doc discusses the new design for the messaging system. Plus some of the optimizations being made. 2) *Choosing a high performance messaging queue:* https://docs.google.com/document/d/1PpQaWVHg06-OqxTzYxQlzg1yEhzA4Y46_NC7HMO6tsI/edit?usp=sharing This doc looks into how fast hardware can do inter-thread messaging and why we chose the JCTools queues. 3) *BackPressure Model* https://docs.google.com/document/d/1Z9pRdI5wtnK-hVwE3Spe6VGCTsz9g8TkgxbTFcbL3jM/edit?usp=sharing was: Details in these documents: 1) *Redesign of the messaging subsystem* https://docs.google.com/document/d/1NK1DJ3aAkta-Im0m-2FObQ4cSRp8xSa301y6zoqcBeE/edit?usp=sharing This doc discusses the new design for the messaging system. Plus some of the optimizations being made. 2) *Choosing a high performance messaging queue:* https://docs.google.com/document/d/1PpQaWVHg06-OqxTzYxQlzg1yEhzA4Y46_NC7HMO6tsI/edit?usp=sharing This doc looks into how fast hardware can do inter-thread messaging and why we chose the JCTools queues. > Redeisgn Messaging Subsystem and switch to JCTools Queues > - > > Key: STORM-2306 > URL: https://issues.apache.org/jira/browse/STORM-2306 > Project: Apache Storm > Issue Type: Sub-task > Components: storm-core >Affects Versions: 2.0.0 >Reporter: Roshan Naik >Assignee: Roshan Naik > Labels: pull-request-available > Time Spent: 33h > Remaining Estimate: 0h > > Details in these documents: > 1) *Redesign of the messaging subsystem* > https://docs.google.com/document/d/1NK1DJ3aAkta-Im0m-2FObQ4cSRp8xSa301y6zoqcBeE/edit?usp=sharing > This doc discusses the new design for the messaging system. Plus some of the > optimizations being made. > 2) *Choosing a high performance messaging queue:* > https://docs.google.com/document/d/1PpQaWVHg06-OqxTzYxQlzg1yEhzA4Y46_NC7HMO6tsI/edit?usp=sharing > This doc looks into how fast hardware can do inter-thread messaging and why > we chose the JCTools queues. > 3) *BackPressure Model* > https://docs.google.com/document/d/1Z9pRdI5wtnK-hVwE3Spe6VGCTsz9g8TkgxbTFcbL3jM/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.4.14#64029)
