GitHub user jfenc91 opened a pull request:
https://github.com/apache/storm/pull/1679
storm-kafka-client tests: tuples not being properly replayed
I have been working with kafka 10 and storm 1.0.2 using the
storm-kafka-client in master and am having issues with topologies that fail
Github user jfenc91 commented on the issue:
https://github.com/apache/storm/pull/1679
A larger refactor here is probably needed to make this more performant.
These changes understandably seem to make the spout struggle to get the
processing tuple count anywhere near the max spout
Github user jfenc91 commented on the issue:
https://github.com/apache/storm/pull/1679
I am on version 0.10.0.1. Interesting, didn't realize unclean leader
election was a default... So ya I had that enabled. Turning that off now
though! Thanks @srdo for the tip and hopefully
Github user jfenc91 commented on the issue:
https://github.com/apache/storm/pull/1679
@srdo @hmcl I tracked down one source of the "double acking"
It looks like I am being switched to reading from an out of sync replica.
Looking at my offsets:
Github user jfenc91 commented on the issue:
https://github.com/apache/storm/pull/1679
I bumped the logging level for unexpected offsets to Warn. Anyway can
we get this and the other PR merged @srdo @hmcl? Admittedly there is still work
that needs to be done, but this PR is pretty
Github user jfenc91 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1679#discussion_r79540459
--- Diff:
external/storm-kafka-client/src/test/java/org/apache/storm/kafka/spout/SingleTopicKafkaSpoutTest.java
---
@@ -0,0 +1,248
Github user jfenc91 commented on the issue:
https://github.com/apache/storm/pull/1679
Thanks for the review here @srdo! I will have a few more PRs headed your
way in a week or two for this kafka client to make it fully usable.
---
If your project is set up for it, you can reply
Github user jfenc91 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1679#discussion_r79480763
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
---
@@ -479,16 +487,17 @@ public OffsetAndMetadata
Github user jfenc91 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1696#discussion_r79973663
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
---
@@ -145,6 +154,10 @@ private void initialize(Collection
Github user jfenc91 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1679#discussion_r79956690
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
---
@@ -266,26 +266,32 @@ private void
Github user jfenc91 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1679#discussion_r79956580
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
---
@@ -266,26 +266,32 @@ private void
Github user jfenc91 commented on the issue:
https://github.com/apache/storm/pull/1679
@srdo Alright I think everything has been addressed here. I have actually
been running this merged with your other PR for the last 12 hours processing
100M tuples and its looked pretty good. Only
Github user jfenc91 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1679#discussion_r79659246
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
---
@@ -266,26 +266,32 @@ private void
Github user jfenc91 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1679#discussion_r79662312
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
---
@@ -479,16 +487,17 @@ public OffsetAndMetadata
Github user jfenc91 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1679#discussion_r79657089
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
---
@@ -451,11 +454,11 @@ public int compare
Github user jfenc91 commented on the issue:
https://github.com/apache/storm/pull/1679
Thats fine @srdo . Thanks again for the reviews. I also created STORM-2106
to keep track of the consequences of this change.
---
If your project is set up for it, you can reply to this email
Github user jfenc91 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1679#discussion_r79480290
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
---
@@ -266,26 +266,32 @@ private void
Github user jfenc91 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1679#discussion_r79481211
--- Diff:
external/storm-kafka-client/src/test/java/org/apache/storm/kafka/spout/SingleTopicKafkaSpoutTest.java
---
@@ -0,0 +1,248
Github user jfenc91 commented on the issue:
https://github.com/apache/storm/pull/1679
Hey, trying not to let this thread die. Could we get some help here from a
committer? @HeartSaVioR @revans2 Thanks!!
---
If your project is set up for it, you can reply to this email and have
Github user jfenc91 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1679#discussion_r87907464
--- Diff:
external/storm-kafka-client/src/test/java/org/apache/storm/kafka/spout/SingleTopicKafkaSpoutTest.java
---
@@ -0,0 +1,250
Github user jfenc91 commented on the issue:
https://github.com/apache/storm/pull/1679
@hmcl thanks for the review. I believe I made all the requested changes
here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user jfenc91 commented on a diff in the pull request:
https://github.com/apache/storm/pull/1679#discussion_r82872195
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java
---
@@ -266,26 +266,31 @@ private void
Github user jfenc91 commented on the issue:
https://github.com/apache/storm/pull/1826
Thanks for picking this up @srdo!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
23 matches
Mail list logo
