[GitHub] storm pull request #1927: Update storm-kafka-client.md

GitHub user cutd opened a pull request: https://github.com/apache/storm/pull/1927 Update storm-kafka-client.md fix a word fault You can merge this pull request into a Git repository by running: $ git pull https://github.com/cutd/storm patch-2 Alternatively you can review and

[GitHub] storm pull request #1922: STORM-2344 Flux YAML File Viewer for Nimbus UI

Github user vesense commented on a diff in the pull request: https://github.com/apache/storm/pull/1922#discussion_r99776407 --- Diff: storm-core/src/ui/public/flux.html --- @@ -0,0 +1,139 @@ + --- End diff -- I have the same idea as @HeartSaVioR :smile: --- If

[GitHub] storm pull request #1928: Create ZippedTextFileReader.java

Github user megha10 closed the pull request at: https://github.com/apache/storm/pull/1928 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] storm pull request #1928: Create ZippedTextFileReader.java

GitHub user megha10 opened a pull request: https://github.com/apache/storm/pull/1928 Create ZippedTextFileReader.java This is to read from a zip file You can merge this pull request into a Git repository by running: $ git pull https://github.com/megha10/storm patch-1

[GitHub] storm pull request #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1919#discussion_r99868716 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -319,8 +320,10 @@ private boolean

[GitHub] storm pull request #1924: STORM-2343: New Kafka spout can stop emitting tupl...

Github user hmcl commented on a diff in the pull request: https://github.com/apache/storm/pull/1924#discussion_r99869121 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutRetryExponentialBackoff.java --- @@ -27,13 +27,15 @@ import

[GitHub] storm issue #1927: Update storm-kafka-client.md

Github user hmcl commented on the issue: https://github.com/apache/storm/pull/1927 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] storm pull request #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1919#discussion_r99882134 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -319,8 +320,10 @@ private boolean

[GitHub] storm issue #1902: STORM-2281: Running Multiple Kafka Spouts (Trident) Throw...

Github user hmcl commented on the issue: https://github.com/apache/storm/pull/1902 @harshach @HeartSaVioR can you please take a look. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] storm issue #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

Github user XuMingmin commented on the issue: https://github.com/apache/storm/pull/1919 remove at-most-once for better describe the changes; emit null msgId when AutoCommitMode to disable tuple track in storm; --- If your project is set up for it, you can reply to this email and

[GitHub] storm pull request #1832: STORM-2250: Kafka Spout Refactoring to Increase Mo...

Github user hmcl commented on a diff in the pull request: https://github.com/apache/storm/pull/1832#discussion_r99937519 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -58,27 +56,26 @@ public class KafkaSpout

[GitHub] storm pull request #1924: STORM-2343: New Kafka spout can stop emitting tupl...

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1924#discussion_r99941130 --- Diff: external/storm-kafka-client/src/test/java/org/apache/storm/kafka/spout/KafkaSpoutEmitTest.java --- @@ -0,0 +1,156 @@ +/* + * Copyright 2017

[GitHub] storm issue #1902: STORM-2281: Running Multiple Kafka Spouts (Trident) Throw...

Github user harshach commented on the issue: https://github.com/apache/storm/pull/1902 +1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] storm issue #1902: STORM-2281: Running Multiple Kafka Spouts (Trident) Throw...

Github user harshach commented on the issue: https://github.com/apache/storm/pull/1902 @hmcl yes we need one for master as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] storm issue #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

Github user srdo commented on the issue: https://github.com/apache/storm/pull/1919 Please see https://github.com/apache/storm/pull/1919#discussion_r99888764 and https://storm.apache.org/releases/1.0.0/javadocs/org/apache/storm/Config.html#TOPOLOGY_ACKER_EXECUTORS. If ack=0 Storm

[GitHub] storm pull request #1924: STORM-2343: New Kafka spout can stop emitting tupl...

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1924#discussion_r99940042 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutRetryExponentialBackoff.java --- @@ -54,59 +56,92 @@

[GitHub] storm pull request #1924: STORM-2343: New Kafka spout can stop emitting tupl...

Github user hmcl commented on a diff in the pull request: https://github.com/apache/storm/pull/1924#discussion_r99881564 --- Diff: external/storm-kafka-client/src/test/java/org/apache/storm/kafka/spout/KafkaSpoutEmitTest.java --- @@ -0,0 +1,156 @@ +/* + * Copyright 2017

[GitHub] storm pull request #1924: STORM-2343: New Kafka spout can stop emitting tupl...

Github user hmcl commented on a diff in the pull request: https://github.com/apache/storm/pull/1924#discussion_r99869535 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutRetryExponentialBackoff.java --- @@ -54,59 +56,92 @@

[GitHub] storm pull request #1924: STORM-2343: New Kafka spout can stop emitting tupl...

Github user hmcl commented on a diff in the pull request: https://github.com/apache/storm/pull/1924#discussion_r99876099 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -289,6 +293,13 @@ private void

[GitHub] storm pull request #1924: STORM-2343: New Kafka spout can stop emitting tupl...

Github user hmcl commented on a diff in the pull request: https://github.com/apache/storm/pull/1924#discussion_r99869978 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutRetryExponentialBackoff.java --- @@ -54,59 +56,92 @@

[GitHub] storm pull request #1832: STORM-2250: Kafka Spout Refactoring to Increase Mo...

Github user hmcl commented on a diff in the pull request: https://github.com/apache/storm/pull/1832#discussion_r99934350 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -347,9 +345,15 @@ private void

[VOTE] Release Apache Storm 1.0.3 (RC2)

This is a call to vote on releasing Apache Storm 1.0.3 (rc2) Full list of changes in this release: https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;h=50878fab679973a1230466920006dc0746ffddd5;hb=eac433b0beb3798c4723deb39b3c4fad446378f4 The tag/commit to be voted

[GitHub] storm pull request #1924: STORM-2343: New Kafka spout can stop emitting tupl...

Github user hmcl commented on a diff in the pull request: https://github.com/apache/storm/pull/1924#discussion_r99932405 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java --- @@ -329,11 +328,13 @@ private Builder(Builder

[GitHub] storm pull request #1924: STORM-2343: New Kafka spout can stop emitting tupl...

Github user hmcl commented on a diff in the pull request: https://github.com/apache/storm/pull/1924#discussion_r99931862 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -289,6 +293,13 @@ private void

[GitHub] storm pull request #1924: STORM-2343: New Kafka spout can stop emitting tupl...

Github user hmcl commented on a diff in the pull request: https://github.com/apache/storm/pull/1924#discussion_r99869874 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutRetryExponentialBackoff.java --- @@ -54,59 +56,92 @@

[GitHub] storm issue #1832: STORM-2250: Kafka Spout Refactoring to Increase Modularit...

Github user hmcl commented on the issue: https://github.com/apache/storm/pull/1832 @srdo everything looks OK for me. I just would rather leave the `Timer` in ns than in ms, and change the `SimulatedTime` class to support nanos. I think that moving forward that will be useful for

Seeing some spurious entries in Storm UI

Hi, Please see the screenshot below: ​I am seeing 6 hosts in nimbus summary where as I have only 3. I have checked the storm.yaml for each host too and it has only 3 entries for nimbus.seeds with "-ha" names. Can someone suggest please how I can get rid of the spurious hosts? Note: I changed

[GitHub] storm issue #1929: STORM-2281: Running Multiple Kafka Spouts (Trident) Throw...

Github user hmcl commented on the issue: https://github.com/apache/storm/pull/1929 @harshach @HeartSaVioR this is the cherry pick from [1.x-branch PR](https://github.com/apache/storm/pull/1902) --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] storm pull request #1922: STORM-2344 Flux YAML File Viewer for Nimbus UI

Github user ambud commented on a diff in the pull request: https://github.com/apache/storm/pull/1922#discussion_r99966356 --- Diff: storm-core/src/ui/public/flux.html --- @@ -0,0 +1,139 @@ + --- End diff --

[GitHub] storm pull request #1924: STORM-2343: New Kafka spout can stop emitting tupl...

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1924#discussion_r99950879 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -289,6 +293,13 @@ private void

[GitHub] storm pull request #1914: STORM-2334 - Join Bolt implementation with unit te...

Github user roshannaik commented on a diff in the pull request: https://github.com/apache/storm/pull/1914#discussion_r99945134 --- Diff: docs/Joins.md --- @@ -0,0 +1,102 @@ +--- +title: Joining Streams in Storm Core +layout: documentation +documentation: true

[GitHub] storm issue #1902: STORM-2281: Running Multiple Kafka Spouts (Trident) Throw...

Github user hmcl commented on the issue: https://github.com/apache/storm/pull/1902 @harshach @HeartSaVioR thanks for the review. I have squashed the commits. It should be good to go. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] storm pull request #1914: STORM-2334 - Join Bolt implementation with unit te...

Github user roshannaik commented on a diff in the pull request: https://github.com/apache/storm/pull/1914#discussion_r99951845 --- Diff: docs/Joins.md --- @@ -0,0 +1,102 @@ +--- +title: Joining Streams in Storm Core +layout: documentation +documentation: true

[GitHub] storm pull request #1922: STORM-2344 Flux YAML File Viewer for Nimbus UI

Github user ambud commented on a diff in the pull request: https://github.com/apache/storm/pull/1922#discussion_r99961278 --- Diff: storm-core/src/ui/public/flux.html --- @@ -0,0 +1,139 @@ + --- End diff -- Added a link on the top right hand side after the username

[GitHub] storm pull request #1929: STORM-2281: Running Multiple Kafka Spouts (Trident...

GitHub user hmcl opened a pull request: https://github.com/apache/storm/pull/1929 STORM-2281: Running Multiple Kafka Spouts (Trident) Throws Illegal State Exception - Assign topic partitions to tasks running the instance of Kafka consumer that has assigned the same list of

[GitHub] storm pull request #1914: STORM-2334 - Join Bolt implementation with unit te...

Github user roshannaik commented on a diff in the pull request: https://github.com/apache/storm/pull/1914#discussion_r99945247 --- Diff: docs/Joins.md --- @@ -0,0 +1,102 @@ +--- +title: Joining Streams in Storm Core +layout: documentation +documentation: true

[GitHub] storm pull request #1922: STORM-2344 Flux YAML File Viewer for Nimbus UI

Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/storm/pull/1922#discussion_r99963942 --- Diff: storm-core/src/ui/public/flux.html --- @@ -0,0 +1,139 @@ + --- End diff -- Not sure which is better between F (for Flux) and Y

[GitHub] storm pull request #1922: STORM-2344 Flux YAML File Viewer for Nimbus UI

Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/storm/pull/1922#discussion_r99964052 --- Diff: storm-core/src/ui/public/flux.html --- @@ -0,0 +1,139 @@ + --- End diff -- And posting screenshot to the comment would help us

[GitHub] storm issue #1831: STORM-2249: Make Distribution Scripts Put Examples to the...

Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1831 @hmcl Right. But I changed my mind that all examples should contain the code, like storm-starter in binary dist. We're extracting examples to show example **codes** for each modules, so the

[GitHub] storm issue #1902: STORM-2281: Running Multiple Kafka Spouts (Trident) Throw...

Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1902 @hmcl Thanks for the update. +1 from me. There's a conflict while cherry-picking into master, so we might need to have following PR for master. --- If your project is set up for it, you can

[GitHub] storm issue #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

Github user XuMingmin commented on the issue: https://github.com/apache/storm/pull/1919 when ack=0, explicit ack()/fail() should not take affects as no AckExecutor is running. Regarding to spout crash, possiblely some tuples are replayed in a periodically commit manner,

[GitHub] storm pull request #1931: Convert NoNodeException to KeyNotFoundException in...

GitHub user HeartSaVioR opened a pull request: https://github.com/apache/storm/pull/1931 Convert NoNodeException to KeyNotFoundException in getNimbodesWithLatestSequenceNumberOfBlob * since callers are able to handle KeyNotFoundException but not NoNodeException This patch

[GitHub] storm issue #1920: STORM-2320: DRPC client printer class reusable for local ...

Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1920 +1 Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] storm issue #1926: STORM-2350: Storm-HDFS's listFilesByModificationTime is b...

Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1926 +1 Nice catch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

Re: [VOTE] Release Apache Storm 1.0.3 (RC2)

+1 (binding) * signature, MD5, SHA: OK * extracting files: OK * diff-ing targz vs zip: OK * launch daemons: OK * check topology running (remote): OK, topology visualization issue fixed (STORM-2335) * check topology running (local): OK, shutdown issue fixed (STORM-2336) Thanks, Jungtaek Lim

[GitHub] storm issue #1902: STORM-2281: Running Multiple Kafka Spouts (Trident) Throw...

Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1902 +1 again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] storm pull request #1930: replace kafka-unit by kafka_2.11 and kafka-clients...

GitHub user lukess opened a pull request: https://github.com/apache/storm/pull/1930 replace kafka-unit by kafka_2.11 and kafka-clients for test Hi: This pr is for testing kafka-clients:0.10.1.1 on storm-kafka-client. I replace the kafka-unit by kafka_2.11 and kafka-clients.

[GitHub] storm issue #1929: STORM-2281: Running Multiple Kafka Spouts (Trident) Throw...

Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1929 +1, and this can be merged with 24hr policy since it's just for forward porting. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] storm issue #1920: STORM-2320: DRPC client printer class reusable for local ...

Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/1920 @hmcl Could you rebase this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] storm pull request #1929: STORM-2281: Running Multiple Kafka Spouts (Trident...

Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1929 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] storm pull request #1902: STORM-2281: Running Multiple Kafka Spouts (Trident...

Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/1902 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] storm issue #1920: STORM-2320: DRPC client printer class reusable for local ...

Github user hmcl commented on the issue: https://github.com/apache/storm/pull/1920 @HeartSaVioR done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] storm pull request #1914: STORM-2334 - Join Bolt implementation with unit te...

Github user arunmahadevan commented on a diff in the pull request: https://github.com/apache/storm/pull/1914#discussion_r99983167 --- Diff: docs/Joins.md --- @@ -0,0 +1,102 @@ +--- +title: Joining Streams in Storm Core +layout: documentation +documentation: true

[GitHub] storm issue #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

Github user XuMingmin commented on the issue: https://github.com/apache/storm/pull/1919 The situations in my mind: 1. autocommit=true + ack=0, --> work, at-most-once 2. autocommit=true + ack>0 --> work, at-most-once ( acker be idle ? ) 3. autocommit=fale + ack=0 --> not

[GitHub] storm issue #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

Github user srdo commented on the issue: https://github.com/apache/storm/pull/1919 Autocommit=true + ack=0 means the consumer will commit periodically. Tuples cannot fail. Messages may be replayed if the spout crashes (it may not have committed them because it only happens

[GitHub] storm issue #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

Github user XuMingmin commented on the issue: https://github.com/apache/storm/pull/1919 Good point to document the impact, is preparing an update PR. According to my understand, there's no replay for AT-MOST-ONCE mode in storm. We use this mode when data missing is not a

[GitHub] storm pull request #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1919#discussion_r99888764 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -319,8 +320,10 @@ private boolean

[GitHub] storm issue #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

Github user revans2 commented on the issue: https://github.com/apache/storm/pull/1919 This is the thing with auto-commit. auto-commit removes some of the overhead of tracking outstanding messages, but also violates a lot of the guarantees that storm has in place. To truly

[GitHub] storm pull request #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1919#discussion_r99891846 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -107,7 +107,8 @@ public void open(Map conf,

[GitHub] storm pull request #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

Github user XuMingmin commented on a diff in the pull request: https://github.com/apache/storm/pull/1919#discussion_r99892349 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -107,7 +107,8 @@ public void open(Map conf,

[GitHub] storm pull request #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1919#discussion_r99889319 --- Diff: external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java --- @@ -319,8 +320,10 @@ private boolean

[GitHub] storm issue #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

Github user srdo commented on the issue: https://github.com/apache/storm/pull/1919 There are two different settings influencing retry behavior here. Auto commit on the spout, and number of ackers configured for Storm

[GitHub] storm issue #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

Github user XuMingmin commented on the issue: https://github.com/apache/storm/pull/1919 it makes sense for me to totally rely on kafka autocommit in at-most-once semantic, and leverages manually offset management for at-least-once, not mix the two offset options. --- If your

[GitHub] storm issue #1919: STORM-2340: fix At-Most-Once issue in KafkaSpout

Github user srdo commented on the issue: https://github.com/apache/storm/pull/1919 The spout as is doesn't support true at-most-once. When auto commit is on, the consumer periodically commits offsets. When auto commit is off and ackers is 0, the spout periodically commits offsets