GitHub user rick-kilgore opened a pull request:
https://github.com/apache/storm/pull/276
STORM-495: KafkaSpout retries with exponential backoff
This is an implementation of the improvement request STORM-495:
https://issues.apache.org/jira/browse/STORM-495
It was recreated from my fork's master branch after the PR reviews were
conducted here:
https://github.com/apache/storm/pull/254
The KafkaSpout keeps track of failed messages through multiple retries, and
delays their resubmission to the topology with an exponentially increasing
delay time. The spout does not terminate the retry cycle (it is my conviction
that it should not do so, because it cannot report context about the failure
that happened to abort the reqeust), it only handles delaying the retries. A
bolt in the topology is still expected to eventually call ack() instead of
fail() to stop the cycle.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/HBOCodeLabs/incubator-storm master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/276.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #276
----
commit cac70e77d3417e13082495aaac126f25a6d07eff
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-08-23T04:49:08Z
exponential backoff for failed messages
commit f9d2948a9e2dadbbb99128fd6570038e2096eabd
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-03T22:04:32Z
Merge branch 'master' into retryDelay
commit 6138e945894c2e0b95666909af51fd336862ce6d
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-06T06:57:17Z
Merge branch 'master' into retryDelay
commit 1c748a7fde11ebc7d209c16ad2b30bf19a316395
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-06T07:08:12Z
Merge pull request #1 from rick-kilgore/retryDelay
exponential backoff for failed messages
moving repo to HBOCodeLabs
commit 04b64bc66d5c29ba77c455f0125200ed8335376f
Author: rick-kilgore <rick.kilg...@gmail.com>
Date: 2014-09-08T20:49:18Z
add comment and different default for retry
commit 05336764408921f3ebcc4ac5b3e82da10e713bea
Author: rick-kilgore <rick.kilg...@gmail.com>
Date: 2014-09-08T20:58:01Z
default settings = no retry delay
default settings = no retry delay - i.e., preserve legacy behavior
commit cbbea7c5b2d014e0cb76b120223e6c257a96248b
Author: rick-kilgore <rick.kilg...@gmail.com>
Date: 2014-09-08T22:44:22Z
publish artifacts to HBO archiva repository
mvn deploy publishes artifacts to HBO archiva repository
commit b95b6d68c54b3e80431c73609870c6935b6d56ce
Author: rick-kilgore <rick.kilg...@gmail.com>
Date: 2014-09-09T00:17:29Z
change everything to com.hbo.hurley
change everything to com.hbo.hurley so relative links work
commit ae65a40473158aaaa546981943e922131aa28b36
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-09T18:45:04Z
add link to apache project in README.markdown file
commit 1e1e57289fe4d1317fcb4cc6e4819c18178b742b
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-09T18:51:18Z
add myself to list of contributors
commit 79e3efe2558d252f2122facdb0ce5e69c9c62104
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-09T21:52:37Z
make config names more consistent with each other
commit ea0989d8fb46ac213d813ec58d9c0ee07b75a6be
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-09T21:54:25Z
remove garbage - I don't know where it came from
commit 5c1029a921483bc86c0b01fec37bfdaba13a206d
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-09T21:56:00Z
Merge branch 'retries' into hboRepo
commit 49f91dc3e644643d774afa159cf521faa9062e42
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-09T22:32:39Z
remove garbage file
commit 2aa03468db3971a11d07048628908074f2a0d3b3
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-09T23:03:34Z
do the time calc only once
do the due time calculation for a MessageRetryRecord when one is
created, not every time isReadyForRetry() is calculated
commit 6f6ece94743eb5432f079db636f4c76dc880f076
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-09T23:04:30Z
Merge branch 'retries' into hboRepo
commit f4fa76c63efefa987ba124dd7dc8f895f9fe6dcc
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-09T23:06:56Z
silly fix
use member var, don't pass it
commit e54fae316d6fc97f0e35a442901adc5ce20166ab
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-09T23:07:15Z
Merge branch 'retries' into hboRepo
commit 26af8daa3e16e2199828c7760a68849dbbd27af1
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-10T23:05:36Z
fixes from review
replace new Date().getTime() with System.currentTimeMillis()
commit d010cb8f25b4a4e0a7882391fc74566c657dbcd1
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-11T00:18:53Z
Merge branch 'retries' into hboRepo
commit 36c5dc5169555325e9e7680939b2f87d6de44487
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-11T21:03:37Z
Merge pull request #2 from HBOCodeLabs/hboRepo
exponential backoff for failed messages
commit 8c78c13de01002fd39536430a32da5a0230184d6
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-11T21:05:37Z
add me
commit 40ab899fe3d67e77c9f1678dbc70d19dd2faf065
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-11T21:16:05Z
added to README the need to run mvn deploy
commit 77c2df28517f9b9acd87724116d9b7b3573be4fc
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-12T00:29:05Z
update storm-kafka README.md
Update storm-kafka README.md with info about retry settings.
Also update the sentence on what SpoutConfig provides over KafkaConfig.
commit ed593480ba543127c9491b5beab68ca49c6bd9a1
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-09-26T23:10:49Z
Merge remote-tracking branch 'apache/master'
Conflicts:
README.markdown
commit 3cda45d5045eafc35a9aaa4136681fe5628360de
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-10-01T20:29:19Z
Merge branch 'master' into retries
Conflicts:
README.markdown
commit 9faf431ff347549380e5a1cf34273018daa815ac
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-10-01T20:29:41Z
Merge branch 'master' into readme
commit d49d014c0c540cebd0a985ace52c57fc5ee88ec7
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-10-01T20:41:45Z
Merge branch 'master' into retries
commit fabd682a2880cc77ff025cb7c09ad19834d47539
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-10-01T21:16:26Z
back out deploy to HBO maven server
back out deploy to HBO maven server
commit 2bf7f6fa1321ca720f2f03d34b638fa502a7e61b
Author: Rick Kilgore <rick.kilg...@hbo.com>
Date: 2014-10-01T21:18:16Z
back out whitepsace change
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
