Re: Few observations related to KafkaSpout implementation (1.1.0)

Thanks @Stig for the detailed explanation. I have not yet used the KafkaSpout; was just going through the code to understand it. I will try the fixes too. In the topologies I run, I hardly create a scenario where the Spout is limiting the throughput. Still, I will try to do some bench marking by

[GitHub] storm pull request #2196: STORM-2619: Correct the JDK version in Setting-up-...

Github user vesense closed the pull request at: https://github.com/apache/storm/pull/2196 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] storm issue #2196: STORM-2619: Correct the JDK version in Setting-up-a-Storm...

Github user vesense commented on the issue: https://github.com/apache/storm/pull/2196 OK. No problem. I will close this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] storm pull request #2197: STORM-2620: Updated docs for JDK needed

Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2197 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] storm pull request #2198: STORM-2620: Updated docs for JDK needed (1.x)

Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2198 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] storm issue #2191: Fix typo in storm.py file

Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2191 Thanks @Ethanlm merged into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] storm pull request #2191: Fix typo in storm.py file

Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2191 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] storm pull request #2192: STORM-2618: Add TridentKafkaStateUpdater for storm...

Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2192 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] storm pull request #2193: STORM-2589: Apply new code style to storm-mongodb

Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2193 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] storm pull request #2194: STORM-2594: Apply new code style to storm-rocketmq

Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2194 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] storm pull request #2195: STORM-2608 Remove any pending offsets that are no ...

Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2195 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] storm pull request #2190: STORM-2608 Remove any pending offsets that are no ...

Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2190 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] storm issue #2189: [STORM-2615] Add topology readonly user configuration

Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2189 Merged via 2f2e9e11d0bb7c8d246517320a47337ff8b5bd19 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] storm pull request #2189: [STORM-2615] Add topology readonly user configurat...

Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2189 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] storm pull request #2086: STORM-2491: Adding extra Cassandra configuration p...

Github user asfgit closed the pull request at: https://github.com/apache/storm/pull/2086 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: Few observations related to KafkaSpout implementation (1.1.0)

> It is the spout thread which > is polling (consuming) and iterating (processing) over the polled messages. > If we separate the consumption in another thread and push messages in a > queue, iterating (processing) is now concurrent and decoupled Sure, but the bookkeeping done by the spout before

[GitHub] storm issue #2189: [STORM-2615] Add topology readonly user configuration

Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2189 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] storm issue #2195: STORM-2608 Remove any pending offsets that are no longer ...

Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2195 +1 It's forward port to #2190 so I think I don't need to follow 24hr rule. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] storm issue #2196: STORM-2619: Correct the JDK version in Setting-up-a-Storm...

Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2196 #2197 covers this and that's more explanatory so would like to pick that. Are you OK with this? --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126551738 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126551578 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm issue #2198: STORM-2620: Updated docs for JDK needed (1.x)

Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2198 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] storm issue #2197: STORM-2620: Updated docs for JDK needed

Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2197 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126548655 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user HeartSaVioR commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126547602 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm issue #2201: STORM-2557: add tuple_population metric

Github user HeartSaVioR commented on the issue: https://github.com/apache/storm/pull/2201 +1 In master branch we could try replacing AtomicLong with LongAdder for metrics, but I'd rather see Metrics V2 first and do the change after that. --- If your project is set up for

[GitHub] storm pull request #2202: STORM-2623: Add in a whitelist for scheduler strat...

GitHub user knusbaum opened a pull request: https://github.com/apache/storm/pull/2202 STORM-2623: Add in a whitelist for scheduler strategies You can merge this pull request into a Git repository by running: $ git pull https://github.com/knusbaum/incubator-storm STORM-2623

[GitHub] storm issue #2201: STORM-2557: add tuple_population metric

Github user revans2 commented on the issue: https://github.com/apache/storm/pull/2201 I ran some performance tests and the extra overhead is minimal. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

Re: Few observations related to KafkaSpout implementation (1.1.0)

Hi Hugo Hope I do not come across as arguing for its own sake. 1.1 Agreed but I was not suggesting to use KafkaConsumer from multiple threads. We can use a single thread which is different from spout thread. That thread and spout will share a queue of ConsumerRecords. 1.2 Agreed again. I had

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126529184 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126515387 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126515845 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126529945 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126519199 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126514663 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126533215 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126529604 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126518695 --- Diff: docs/Metrics.md --- @@ -44,7 +45,7 @@ topology.metrics.consumer.register: argument: "http://example.com:8080/metrics/my-topology/; ```

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126516022 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126526874 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126520704 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126532926 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126528804 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126516261 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126519537 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126522426 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/2200#discussion_r126515274 --- Diff: docs/Metrics.md --- @@ -125,3 +126,193 @@ The [builtin metrics]({{page.git-blob-base}}/storm-client/src/jvm/org/apache/sto

Re: Few observations related to KafkaSpout implementation (1.1.0)

1) Agreed that short duration blocks are less likely to cause an issue but it will be much better to avoid them. Anyway, a conclusion is not easy without some benchmark. I will let You know if I am able to do some volume testing on both options and observe significant difference. The auto spout

Re: Few observations related to KafkaSpout implementation (1.1.0)

I believe that "better" is relative most of the time.  Optimizing a failure case costs something, unless of course the code is doing something horrible and there truly is a better way to do it that uses less memory, cpu, etc.  If so please put up a pull request and lets fix the thing.  However

Re: Few observations related to KafkaSpout implementation (1.1.0)

@chandan #1 was designed to be single threaded. There are several reasons for that. 1.1 KafkaConsumer is single threaded. If you have multiple threads attempting to poll from it, you will get a ConcurrentModificationException("KafkaConsumer is not safe for multi-threaded access”); 1.2

Re: Few observations related to KafkaSpout implementation (1.1.0)

Sorry about being cryptic there. What I meant is that it will be much better if we don't make assumptions about frequency of failure rates in topologies. I know it is more of a commonsense but out of curiosity, can you point me to any Storm documentation which makes a comment on preferable failure

Re: Few observations related to KafkaSpout implementation (1.1.0)

1) It is true that nextTuple implementations should try to avoid blocking as much as possible, but the time nextTuple may block in poll is capped by a configuration parameter

Re: Few observations related to KafkaSpout implementation (1.1.0)

I'm not sure what assumptions you want to make that this is preventing, or why they would be helpful. - Bobby On Monday, July 10, 2017, 12:14:53 PM CDT, chandan singh wrote: Hi Stig & Bobby Thanks for confirming my understanding. 1) Ensuring that calls to nexTuple(),

[GitHub] storm pull request #2200: STORM-2616: Documentation for built in metrics

GitHub user revans2 opened a pull request: https://github.com/apache/storm/pull/2200 STORM-2616: Documentation for built in metrics I am happy to pull this back into 1.x branches too. I am not sure exactly how the metrics might have changed though, so we might want to be careful

[GitHub] storm issue #2189: [STORM-2615] Add topology readonly user configuration

Github user Ethanlm commented on the issue: https://github.com/apache/storm/pull/2189 Squashed two commits. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] storm issue #1785: [STORM-2201] Add dynamic scheduler configuration loading.

Github user Ethanlm commented on the issue: https://github.com/apache/storm/pull/1785 I copied the code and submitted a new pull request. Also fixed some checkstyle issues. https://github.com/apache/storm/pull/2199 --- If your project is set up for it, you can reply to this email

[GitHub] storm pull request #2199: [STORM-2201] Add dynamic scheduler configuration l...

GitHub user Ethanlm opened a pull request: https://github.com/apache/storm/pull/2199 [STORM-2201] Add dynamic scheduler configuration loading This adds an interface and two implementations, one that will load from a local file, and another that will load a config from

[GitHub] storm issue #2164: [STORM-2557]: A bug in DisruptorQueue causing severe unde...

Github user revans2 commented on the issue: https://github.com/apache/storm/pull/2164 I just realized that because the `sojourn_time_ms` is calculated based off of the request rate, that this updated, and the population (not updated and still based off of slots and not tuples), that

Re: Few observations related to KafkaSpout implementation (1.1.0)

Hi Stig & Bobby Thanks for confirming my understanding. 1) Ensuring that calls to nexTuple(), ack() and fail() are non-blocking has been a guideline on http://storm.apache.org/releases/1.1.0/Concepts.html for long. Copying verbatim here : "The main method on spouts is nextTuple. nextTuple

[GitHub] storm issue #2174: STORM-2554: Trident Kafka Spout Refactoring to Include Ma...

Github user kristopherkane commented on the issue: https://github.com/apache/storm/pull/2174 I had a parallel effort which arrived at most of the same conclusions before discovering this JIRA. There is a bug with activate/deactivate where the spout continues to call

Re: Few observations related to KafkaSpout implementation (1.1.0)

For #2 it is a question of what do you optimize for.  Storm typically assumes that failures should be rare so we optimize for that.  We keep the minimal information around to be able to replay the message, but not a lot more.  If you are getting lots of failures, you really should be more

Re: Few observations related to KafkaSpout implementation (1.1.0)

Hi Chandan, I'm going to assume we're talking about the storm-kafka-client spout, and not the storm-kafka spout. 1) Yes, we do polling and committing in the spout thread. I'm not aware of why that would be against spout best practices? Simplicity is definitely a reason, but I don't think anyone

[GitHub] storm issue #2198: STORM-2620: Updated docs for JDK needed (1.x)

Github user revans2 commented on the issue: https://github.com/apache/storm/pull/2198 This is the 1.x pull request for #2197 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] storm pull request #2198: STORM-2620: Updated docs for JDK needed

GitHub user revans2 opened a pull request: https://github.com/apache/storm/pull/2198 STORM-2620: Updated docs for JDK needed This should be able to be pulled into all of the 1.x release branches without any issues. You can merge this pull request into a Git repository by running:

[GitHub] storm pull request #2197: STORM-2620: Updated docs for JDK needed

GitHub user revans2 opened a pull request: https://github.com/apache/storm/pull/2197 STORM-2620: Updated docs for JDK needed I plan to do a separate pull request to the 1.x branch(s) too. You can merge this pull request into a Git repository by running: $ git pull

[GitHub] storm pull request #2196: STORM-2619: Correct the JDK version in Setting-up-...

GitHub user vesense opened a pull request: https://github.com/apache/storm/pull/2196 STORM-2619: Correct the JDK version in Setting-up-a-Storm-cluster.md The Storm 2.0 needs JDK 1.8+ You can merge this pull request into a Git repository by running: $ git pull

Few observations related to KafkaSpout implementation (1.1.0)

Hi I hope I am using the right mailing list. Please advice if I am wrong. I have few observations about the KafkaSpout and feel that some of these lead to inefficiencies. It will be of great help if someone can throw some light on the rationale behind the implementation. 1) Kafka polling and

Re: storm versions with jdk versions

Hi sam, Storm 0.9.x -> JDK1.6+ Storm 0.10.x -> JDK1.7+ Storm 1.x -> JDK1.7+ Storm 2.x -> JDK1.8+ IMHO, I recommend you to use the latest 1.1.x release. Thanks, Xin 2017-07-10 21:27 GMT+08:00 sam mohel : > Is there any help , please? > > On Monday, July 10, 2017, sam

[GitHub] storm issue #1785: [STORM-2201] Add dynamic scheduler configuration loading.

Github user Ethanlm commented on the issue: https://github.com/apache/storm/pull/1785 @HeartSaVioR I asked @revans2 . I will start to do it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

Re: [DISCUSS] Storm 2.0 Roadmap

Do we have a clear date to release Storm 2.0 beta version? I saw some users expecting to use Java8. BTW. Could somebody help to update the Storm site? The 2.0.0-SNAPSHOT documents should be updated. Thanks. - Xin 2017-06-29 23:27 GMT+08:00 Jungtaek Lim : > FYI I just gave

Re: storm versions with jdk versions

Is there any help , please? On Monday, July 10, 2017, sam mohel wrote: > I'm sorry for my question if it is silly , i've apache-storm version -0.9.6 . is that working with jdk1.8 ?? or it's not depending on JDK versions

[GitHub] storm issue #2189: [STORM-2615] Add topology readonly user configuration

Github user Ethanlm commented on the issue: https://github.com/apache/storm/pull/2189 I don't know why the 'INTERATION-TEST' failed. I tested it in my computer using "mvn -P integration-tests-only verify" command and the build succeeded. --- If your project is set up for it, you

[GitHub] storm pull request #2195: STORM-2608 Remove any pending offsets that are no ...

GitHub user kevinconaway opened a pull request: https://github.com/apache/storm/pull/2195 STORM-2608 Remove any pending offsets that are no longer valid (master) @HeartSaVioR this PR includes fixes for https://issues.apache.org/jira/browse/STORM-2608 targeting the master branch