[ 
https://issues.apache.org/jira/browse/STORM-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16352005#comment-16352005
 ] 

Jungtaek Lim commented on STORM-2913:
-------------------------------------

Thanks [~Srdo], merged into master.

Since there's some diverge between master and 1.x-branch for 
storm-kafka-client, I'd like to wait for [~Srdo] to submit pull request against 
1.x branch. We can merge it afterwards.

> STORM-2844 made autocommit and at-most-once storm-kafka-client spouts log 
> warnings on every emit, because those modes don't commit the right metadata 
> to Kafka
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: STORM-2913
>                 URL: https://issues.apache.org/jira/browse/STORM-2913
>             Project: Apache Storm
>          Issue Type: Bug
>          Components: storm-kafka-client
>    Affects Versions: 2.0.0, 1.2.0
>            Reporter: Stig Rohde Døssing
>            Assignee: Stig Rohde Døssing
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 2.0.0
>
>          Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> The mechanism added in https://issues.apache.org/jira/browse/STORM-2844 to 
> allow us to check whether a committed offset was committed by the currently 
> running topology requires that we commit some metadata along with the offset.
> We are using this metadata for two things: Only applying the 
> FirstPollOffsetStrategy when the topology is deployed, rather than when the 
> worker is restarted, and an (IMO fairly unimportant) runtime check that the 
> spout offset tracking is not in a bad state.
> Autocommit spouts don't include this metadata, and we also don't include it 
> when committing offsets in at-most-once mode. We can fix at-most-once by 
> switching to committing a custom OffsetAndMetadata, rather than using the 
> no-arg commitSync variant. 
> I'm not sure what we should do to fix the autocommit case. There doesn't seem 
> to be a way to include metadata in autocommits, so I don't think we can 
> support this mechanism for autocommits. 
> If we can't fix the autocommit case, I see two options for fixing this:
> * Make doSeek have the old behavior for autocommits only (i.e. apply the 
> FirstPollOffsetStrategy on every worker restart), and keep the new behavior 
> for at-least-once/at-most-once. I think this behavior could be a little 
> confusing.
> * Revert doSeek to the old behavior in all cases, and throw out the runtime 
> check that uses the metadata. This also isn't a great option, because the new 
> seek behavior is more useful than restarting on every worker reboot.
> What do you think [~hmclouro]? I'm leaning toward the first option.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to