Re: [DISCUSS] Feature Branch for Apache Beam Runner

Taylor, I am interested in contributing to this effort. Gone through Beam APIs earlier and had some initial thoughts on Storm runner. We can start with existing core storm constructs but it is better to design in such a way that these can be replaced with new APIs. Thanks, Satish. On Fri, Sep

Re: [DISCUSS] Feature Branch for Apache Beam Runner

I'm open to change, but yes, I started with core storm since it offers the most flexibility wrt how Beam constructs are translated. -Taylor > On Sep 15, 2016, at 5:51 PM, Roshan Naik wrote: > > Good idea. Will the Beam API be implemented to run on top Storm Core >

Re: [DISCUSS] Feature Branch for Apache Beam Runner

Good idea. Will the Beam API be implemented to run on top Storm Core primitives ? -roshan On 9/15/16, 2:00 PM, "P. Taylor Goetz" wrote: >I¹ve been tinkering with implementing an Apache Beam runner on top of >Storm and would like to open it up so others in the community can

[GitHub] storm pull request #1685: STORM-2097: Improve logging in trident core and ex...

GitHub user hmcl opened a pull request: https://github.com/apache/storm/pull/1685 STORM-2097: Improve logging in trident core and examples - Improve logging in trident core, MasterBatchCoordinator, and examples - Added DebugMemoryMapState You can merge this pull request into

[DISCUSS] Feature Branch for Apache Beam Runner

I’ve been tinkering with implementing an Apache Beam runner on top of Storm and would like to open it up so others in the community can contribute. To that end I’d like to propose creating a feature branch for that work if there are others who are interested in getting involved. We did that a

[GitHub] storm issue #1642: STORM-2018: Supervisor V2.

Github user srdo commented on the issue: https://github.com/apache/storm/pull/1642 @revans2 Thanks, looks great. +1 for merging. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] storm issue #1642: STORM-2018: Supervisor V2.

Github user ptgoetz commented on the issue: https://github.com/apache/storm/pull/1642 I have a few style-related nits, but I've refrained from pointing them out because the style(s) in the codebase are all over the place, and the style is inherited in some places. If we want to get

[GitHub] storm issue #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on the issue: https://github.com/apache/storm/pull/1642 Not sure why travis has so many issues with the build and maven but here is my travis setup on the same branch with the build passing.

[GitHub] storm issue #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on the issue: https://github.com/apache/storm/pull/1642 @srdo Pushed a new version that hopefully addresses your review comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79042371 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,776 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79041849 --- Diff: storm-core/src/jvm/org/apache/storm/localizer/AsyncLocalizer.java --- @@ -0,0 +1,422 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79040636 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,776 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79038947 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,776 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79039785 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/ReadClusterState.java --- @@ -0,0 +1,328 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79037939 --- Diff: storm-core/src/jvm/org/apache/storm/localizer/AsyncLocalizer.java --- @@ -0,0 +1,422 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79036299 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,776 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79036111 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,776 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79035641 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/ReadClusterState.java --- @@ -0,0 +1,328 @@ +/** + * Licensed to the Apache Software

[GitHub] storm issue #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on the issue: https://github.com/apache/storm/pull/1642 @srdo on the `newAssignment.compareAndSet` you are 100% correct. I originally had it in there to clear out the assignment when it is processed, but then realized that each time a new assignment is

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79034054 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java --- @@ -0,0 +1,644 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79034214 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java --- @@ -0,0 +1,644 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79034020 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java --- @@ -0,0 +1,644 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79033354 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,776 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79033063 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,776 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79032969 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,776 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79032466 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,776 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79031670 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/ReadClusterState.java --- @@ -0,0 +1,328 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79031226 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/ReadClusterState.java --- @@ -0,0 +1,328 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79030835 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java --- @@ -0,0 +1,644 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79030558 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java --- @@ -0,0 +1,644 @@ +/** + * Licensed to the Apache Software

[GitHub] storm issue #1642: STORM-2018: Supervisor V2.

Github user ptgoetz commented on the issue: https://github.com/apache/storm/pull/1642 @revans2 I'm also in the process of reviewing this. I hope to finish by the end of the day. So far everything looks good to me. --- If your project is set up for it, you can reply to this email and

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79029515 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java --- @@ -0,0 +1,644 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79027965 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java --- @@ -0,0 +1,644 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79027439 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java --- @@ -0,0 +1,644 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r79026674 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/ContainerLauncher.java --- @@ -0,0 +1,103 @@ +/** + * Licensed to the Apache Software

[GitHub] storm issue #1677: STORM-1985: Admin Commands

Github user kamleshbhatt commented on the issue: https://github.com/apache/storm/pull/1677 Applied the review comments. Please take a look. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

Re: Get the Executor-Id to Slot-Id mapping in a file

The UI pulls it from nimbus using the thrift API. https://github.com/apache/storm/blob/master/storm-core/src/storm.thrift#L611 is the API definition, although it may be different depending on which version of storm you are using.  getTopologyInfo has executors in it, and each of them has a

Get the Executor-Id to Slot-Id mapping in a file

Greetings, I am running storm on two VMs(ubuntu 16.04, 2GB Ram) on my laptop(in distributed mode). The issue is that the VM runs out of RAM as soon as I start a simple(but with a large number of workers) topology. Actually I am interested only in the executor-Id to Slot-Id mapping from the

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78997835 --- Diff: storm-core/src/jvm/org/apache/storm/localizer/AsyncLocalizer.java --- @@ -0,0 +1,422 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78971109 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,776 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78960818 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/ReadClusterState.java --- @@ -0,0 +1,328 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78952788 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java --- @@ -0,0 +1,644 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78993658 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Supervisor.java --- @@ -163,15 +259,94 @@ public Integer call() throws Exception {

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78956720 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java --- @@ -0,0 +1,644 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78950399 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java --- @@ -0,0 +1,644 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78974501 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,776 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78965321 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/RunAsUserContainer.java --- @@ -0,0 +1,94 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78975015 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,776 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78949081 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/ContainerLauncher.java --- @@ -0,0 +1,103 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78971801 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,776 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78963666 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/ReadClusterState.java --- @@ -0,0 +1,328 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78991476 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,769 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78999636 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java --- @@ -0,0 +1,644 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78981043 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,776 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78963034 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/ReadClusterState.java --- @@ -0,0 +1,328 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78972625 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,776 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78993008 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Supervisor.java --- @@ -17,135 +17,231 @@ */ package

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78942716 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/AdvancedFSOps.java --- @@ -0,0 +1,335 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78961556 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/ReadClusterState.java --- @@ -0,0 +1,328 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78959596 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/ReadClusterState.java --- @@ -0,0 +1,328 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78973993 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,776 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78969293 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/Slot.java --- @@ -0,0 +1,776 @@ +/** + * Licensed to the Apache Software Foundation

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78957510 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java --- @@ -0,0 +1,644 @@ +/** + * Licensed to the Apache Software

[GitHub] storm pull request #1642: STORM-2018: Supervisor V2.

Github user srdo commented on a diff in the pull request: https://github.com/apache/storm/pull/1642#discussion_r78941512 --- Diff: storm-core/src/jvm/org/apache/storm/daemon/supervisor/AdvancedFSOps.java --- @@ -0,0 +1,335 @@ +/** + * Licensed to the Apache Software

Re: How does Storm's OutputCollector work?

Sorry I have been a bit slow in responding to e-mail lately.  The master branch is in the process of moving the code to java.  The ultimate plan is to only leave a storm-clojure package in clojure that provides the current clojure client/worker API for storm.  When will it be released, well

Re: [DISCUSS] ElasticSearch 2.x support

I personally don't use ES as part of my storm work, so I don't necessarily feel qualified to answer this.  In general though I really do like to see storm come with batteries included.  If ES1 is not end of life, and there is a community of people who want to continue using it/supporting it, I

[GitHub] storm issue #1642: STORM-2018: Supervisor V2.

Github user revans2 commented on the issue: https://github.com/apache/storm/pull/1642 @harshach did you get a chance to take a look yet? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this