[jira] [Updated] (STORM-1632) Disable event logging by default

[ https://issues.apache.org/jira/browse/STORM-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1632: --- Description: EventLogging has performance penalty. For a simple speed of light topology

[jira] [Updated] (STORM-1632) Disable event logging by default

[ https://issues.apache.org/jira/browse/STORM-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1632: --- Description: EventLogging has performance penalty. For a simple speed of light topology

[jira] [Updated] (STORM-1632) Disable event logging by default

[ https://issues.apache.org/jira/browse/STORM-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1632: --- Description: EventLogging has performance penalty. For a simple speed of light topology

[jira] [Updated] (STORM-1643) Performance Fix: Optimize clojure lookups related to throttling and stats

[ https://issues.apache.org/jira/browse/STORM-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1643: --- Attachment: before2.png after2.png Attaching another pair of before2/after2 profiler

[jira] [Updated] (STORM-1643) Performance Fix: Optimize clojure lookups related to throttling and stats

[ https://issues.apache.org/jira/browse/STORM-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1643: --- Attachment: after.png before.png > Performance Fix: Optimize clojure lookups rela

[jira] [Updated] (STORM-1643) Performance Fix: Optimize clojure lookups related to throttling and stats

[ https://issues.apache.org/jira/browse/STORM-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1643: --- Description: :key lookups in Clojure are expensive. And some of keys like :storm-conf are being

[jira] [Issue Comment Deleted] (STORM-1643) Performance Fix: Optimize clojure lookups related to throttling and stats

[ https://issues.apache.org/jira/browse/STORM-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1643: --- Comment: was deleted (was: :key lookups in Clojure are expensive. And some of keys like :storm-conf

[jira] [Commented] (STORM-1643) Performance Fix: Optimize clojure lookups related to throttling and stats

[ https://issues.apache.org/jira/browse/STORM-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204576#comment-15204576 ] Roshan Naik commented on STORM-1643: [~revans2] This is for 1.x. Wrt 2.x... the fixes are small

[jira] [Commented] (STORM-1643) Performance Fix: Optimize clojure lookups related to throttling and stats

[ https://issues.apache.org/jira/browse/STORM-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204568#comment-15204568 ] Roshan Naik commented on STORM-1643: :key lookups in Clojure are expensive. And some of keys like

[jira] [Updated] (STORM-1643) Performance Fix: Optimize clojure lookups related to throttling and stats

[ https://issues.apache.org/jira/browse/STORM-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1643: --- Summary: Performance Fix: Optimize clojure lookups related to throttling and stats (was: Performance

[jira] [Created] (STORM-1643) Performance Fix: Optimize clojure lookups related throttling and stats

Roshan Naik created STORM-1643: -- Summary: Performance Fix: Optimize clojure lookups related throttling and stats Key: STORM-1643 URL: https://issues.apache.org/jira/browse/STORM-1643 Project: Apache

[jira] [Updated] (STORM-1632) Disable event logging by default

[ https://issues.apache.org/jira/browse/STORM-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1632: --- Priority: Blocker (was: Major) > Disable event logging by defa

Re: Problem about secure hdfs spout

Looks like some bug there .. Can you file a jira ? -roshan On 2/25/16, 7:39 PM, "ght230" wrote: >Hi: > >Recently I was trying secure hdfs spout about ³STORM-1199: HDFS Spout >Implementation³, but some error occured as following: > >"Login successful for user t...@example.com

Re: Any guidance around disruptor batch size ?

ize of the queue. >I have been talking to a few people in academia that are working on >auto-tuning of the batch size to optimize throughput for a given SLA, but >that is still a ways off. > - Bobby > >On Wednesday, February 10, 2016 8:59 PM, Roshan Naik ><ros...@horto

[jira] [Commented] (STORM-1539) Improve Storm ACK-ing performance

[ https://issues.apache.org/jira/browse/STORM-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15143663#comment-15143663 ] Roshan Naik commented on STORM-1539: The attached profiler info related to the amount of time taken

[jira] [Updated] (STORM-1539) Improve Storm ACK-ing performance

[ https://issues.apache.org/jira/browse/STORM-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1539: --- Attachment: after.png before.png Attaching before/after screenshots of profiler

Any guidance around disruptor batch size ?

Wondering if there is any rule of thumb guidance around setting topology.disruptor.batch.size, for increasing throughput ? For example any correlation with topology.max.spout.pending, bolt executor count, spout count, etc ? -roshan

[jira] [Commented] (STORM-1526) Improve Storm core performance

[ https://issues.apache.org/jira/browse/STORM-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15139711#comment-15139711 ] Roshan Naik commented on STORM-1526: Thanks [~kabhwan] for merging to 1.x also. > Improve Storm c

[jira] [Commented] (STORM-1526) Improve Storm core performance

[ https://issues.apache.org/jira/browse/STORM-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15134640#comment-15134640 ] Roshan Naik commented on STORM-1526: [~dossett] i updated the PR title. > Improve Storm c

Re: Performance Profiling - Storm core

>> >> Does the above work? You can also give the type hint for target tasks >> (^List target-tasks) >> >> It is not clear to me why this code path will be even invoked, since you >> are not using LoadAwareCustomStreamGrouping in your topology. &

[jira] [Assigned] (STORM-1526) Improve Storm core performance

[ https://issues.apache.org/jira/browse/STORM-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik reassigned STORM-1526: -- Assignee: Roshan Naik > Improve Storm core performa

[jira] [Created] (STORM-1526) Improve Storm core performance

Roshan Naik created STORM-1526: -- Summary: Improve Storm core performance Key: STORM-1526 URL: https://issues.apache.org/jira/browse/STORM-1526 Project: Apache Storm Issue Type: Bug

Re: Performance Profiling - Storm core

d what is the size of disruptor queue? Also If your >spout is slower compared to the bolts, this behavior is very much >expected. >Isn't it? > >On Wed, Feb 3, 2016 at 11:54 AM, Roshan Naik <ros...@hortonworks.com> >wrote: > >> Looks like the attachments were stripped off

Re: Performance Profiling - Storm core

/roshannaik/storm-benchmark-sol/blob/master/profiler/storm%20core%20-%20sol%20-%200%20acker/storm%20core%20-%20method%20stats.png -roshan From: Roshan Naik <ros...@hortonworks.com<mailto:ros...@hortonworks.com>> Reply-To: "dev@storm.apache.org<mailto:dev@storm.apache.org>&

Performance Profiling - Storm core

Attached is a screenshots of the performance profile for Storm core using a Speed of Light topology. Topology info: - 1 bolt instance, 1 spout instance, 1 worker. - ACKer count = 0 - Spout generates precomputes a random list of tuples, then keeps emitting them endlessly - Bolt just remits the

Re: HdfsSpout

It has now been committed. -roshan On 1/13/16, 4:17 PM, "Jungtaek Lim" wrote: >Hi Matthias, > >AFAIK, Roshan is working on providing HdfsSpout - >https://issues.apache.org/jira/browse/STORM-1199. >It's in reviewing so you can see the pull request and start reviewing, >too. >

[jira] [Updated] (STORM-1474) Address remaining minor review comments for STORM-1199

[ https://issues.apache.org/jira/browse/STORM-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1474: --- Issue Type: Sub-task (was: Bug) Parent: STORM-1199 > Address remaining minor review comme

[jira] [Created] (STORM-1474) Address remaining minor review comments for STORM-1199

Roshan Naik created STORM-1474: -- Summary: Address remaining minor review comments for STORM-1199 Key: STORM-1474 URL: https://issues.apache.org/jira/browse/STORM-1474 Project: Apache Storm

[jira] [Commented] (STORM-1199) Create HDFS Spout

[ https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098948#comment-15098948 ] Roshan Naik commented on STORM-1199: Thanks all for your input/feedback/reviews .. they were very

[jira] [Updated] (STORM-1199) Create HDFS Spout

[ https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1199: --- Attachment: hdfs-spout.1.patch Uploading initial implementation with a few UT for review. Includes

[jira] [Commented] (STORM-1351) Storm spouts and bolts need a way to communicate problems back to toplogy runner

[ https://issues.apache.org/jira/browse/STORM-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037197#comment-15037197 ] Roshan Naik commented on STORM-1351: My observation here is ... it is preferable to externalize

[jira] [Commented] (STORM-1351) Storm spouts and bolts need a way to communicate problems back to toplogy runner

[ https://issues.apache.org/jira/browse/STORM-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034553#comment-15034553 ] Roshan Naik commented on STORM-1351: The bolt will need backoff in case of certain error conditions

[jira] [Commented] (STORM-1351) Storm spouts and bolts need a way to communicate problems back to toplogy runner

[ https://issues.apache.org/jira/browse/STORM-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15032431#comment-15032431 ] Roshan Naik commented on STORM-1351: Thanks [~basti.lj] for the insight on CPU usage drop

[jira] [Commented] (STORM-1351) Storm spouts and bolts need a way to communicate problems back to toplogy runner

[ https://issues.apache.org/jira/browse/STORM-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027632#comment-15027632 ] Roshan Naik commented on STORM-1351: I dont think there is a need for a different backoff logic

[jira] [Updated] (STORM-1351) Storm spouts and bolts need a way to communicate problems back to toplogy runner

[ https://issues.apache.org/jira/browse/STORM-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1351: --- Description: A spout can be having a problem generating a tuple in nextTuple() because

[jira] [Commented] (STORM-1351) Storm spouts and bolts need a way to communicate problems back to toplogy runner

[ https://issues.apache.org/jira/browse/STORM-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025659#comment-15025659 ] Roshan Naik commented on STORM-1351: Similar problem exists with Bolts too .. for example a Hive Bolt

[jira] [Assigned] (STORM-1351) Storm spouts and bolts need a way to communicate problems back to toplogy runner

[ https://issues.apache.org/jira/browse/STORM-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik reassigned STORM-1351: -- Assignee: Roshan Naik > Storm spouts and bolts need a way to communicate problems b

[jira] [Created] (STORM-1351) Storm spouts and bolts need a way to communicate problems back to toplogy runner

Roshan Naik created STORM-1351: -- Summary: Storm spouts and bolts need a way to communicate problems back to toplogy runner Key: STORM-1351 URL: https://issues.apache.org/jira/browse/STORM-1351 Project

[jira] [Assigned] (STORM-1199) Create HDFS Spout

[ https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik reassigned STORM-1199: -- Assignee: Roshan Naik > Create HDFS Spout > - > >

[jira] [Updated] (STORM-1199) Create HDFS Spout

[ https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1199: --- Attachment: HDFSSpoutforStorm v2.pdf Uploading the v2 of the design and requirements doc

[jira] [Updated] (STORM-1199) Create HDFS Spout

[ https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1199: --- Attachment: HDFSSpoutforStorm.pdf Uploading requirements and design notes document > Create H

[jira] [Updated] (STORM-1199) Create HDFS Spout

[ https://issues.apache.org/jira/browse/STORM-1199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roshan Naik updated STORM-1199: --- Issue Type: New Feature (was: Bug) > Create HDFS Spout > - > >

[jira] [Created] (STORM-1199) Create HDFS Spout

Roshan Naik created STORM-1199: -- Summary: Create HDFS Spout Key: STORM-1199 URL: https://issues.apache.org/jira/browse/STORM-1199 Project: Apache Storm Issue Type: Bug Reporter

<    1   2