GitHub user srishtyagrawal opened a pull request:
https://github.com/apache/storm/pull/2637
Map of Spout configurations from `storm-kafka` to `storm-kafka-client`
As per @srdo and @ptgoetz's replies on the Storm Dev mailing list, I am
adding the spout configuration map in the `storm-kafka-client` document .
[The
gist](https://gist.github.com/srishtyagrawal/850b0c3f661cf3c620c27f314791224b),
with initial changes, had comments from @srdo and questions from me which I am
pasting here for convenience:
Last comment by @srdo:
Thanks, I think this is nearly there. The maxOffsetBehind section says that
"If a failing tuple's offset is less than maxOffsetBehind, the spout stops
retrying the tuple.". Shouldn't it be more than? i.e. if the latest offset is
100, and you set maxOffsetBehind to 50, and then offset 30 fails, 30 is more
than maxOffsetBehind behind the latest offset, so it is not retried.
Regarding the links, I think we should try to use links that automatically
point at the right release. There's some documentation about it here
https://github.com/apache/storm-site#how-release-specific-docs-work, and
example usage "The allowed values are listed in the FirstPollOffsetStrategy
javadocs" (from
https://github.com/apache/storm/blob/master/docs/storm-kafka-client.md). It
would be great if you fix any broken links you find, or any links that are hard
coded to point at a specific release.
My reply:
I copied the [maxOffsetBehind
documentation](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_storm-component-guide/content/storm-kafkaspout-config-core.html)
from here. It is confusing because from your earlier example the value 30
itself is lesser than 100-50, but I like the idea of adding behind to make it
more clear. As there are more than 1 scenarios where maxOffsetBehind is used, I
have modified the documentation to specify the fail scenario as an example.
Thanks for the documentation on links, I will fix all the existing links
and the ones which are currently broken in storm-kafka-client documentation.
Question:
Seems like all the release related links in
[storm-kafka-client.md](https://github.com/apache/storm/blob/master/docs/storm-kafka-client.md)
don't work. I looked at other docs as well, for example
[Hooks.md](https://github.com/apache/storm/blob/a4afacd9617d620f50cf026fc599821f7ac25c79/docs/Hooks.md),
[Concepts.md](https://github.com/apache/storm/blob/09e01231cc427004bab475c9c70f21fa79cfedef/docs/Concepts.md),
[Configuration.md](https://github.com/apache/storm/blob/a4afacd9617d620f50cf026fc599821f7ac25c79/docs/Configuration.md),
[Common-patterns.md](https://github.com/apache/storm/blob/a4afacd9617d620f50cf026fc599821f7ac25c79/docs/Common-patterns.md)
(the first 4 documents I looked into for relative links) where these links
gave a 404. Yet to figure out why these links don't work.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/srishtyagrawal/storm migrateSpoutConfigs
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/2637.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2637
----
commit 2ca4fc851c17e1cb8a4208fe5cb0c3916551080b
Author: Srishty Agrawal <sagrawal@...>
Date: 2018-04-19T00:13:57Z
Map of Spout configurations from storm-kafka to storm-kafka-client
----
---
