Re: Storm 2.0 blogs ?

[Bobby Evans]

I totally agree, especially with the performance improvements in it.

Thanks,

Bobby

On Tue, Jan 22, 2019 at 12:40 AM Roshan Naik 
wrote:

> Now that  2.0 has all the votes it needs to move forward, maybe a good
> time to think of some blogs to go with this long awaited release.
>  Some potential topics that come to mind are:
> 1- Overview of new features and major changes since 1.x2- Re-architecture
> (messaging, threading, back pressure)3- Micro benchmarks4- Revisit the
> famous Yahoo benchmark5- Window state persistence6- SQL enhancements7- new
> Metrics stuff8- Kafka related changes9- Security10- An area you have worked
> on ?11- Other ideas ?
> Anyone interested in contributing blogs ?
> FYI: I am working on content for topics 2 & 3.
>
> -roshan

Me going forward

[Bobby Evans]

I wanted to give everyone a heads up that I have taken a new job at Nvidia
and my day to day work, though still a bit up in the air, is not likely to
concentrate on Storm as much as it did before.  I am still around and will
still try to participate as much as I can, but you may need to explicitly
call me out if you want me to look at something.

Thanks,

Bobby

Re: [VOTE] Release Apache Storm 1.1.4 (rc1)

[Bobby Evans]

+1 I built the code from the git tag and all the unit tests passed.  I also
ran some manual tests.

Thanks,

Bobby

On Wed, Jan 9, 2019 at 11:01 AM P. Taylor Goetz  wrote:

> This is a call to vote on releasing Apache Storm 1.1.4 (rc1)
>
> Full list of changes in this release:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.4-rc1/RELEASE_NOTES.html
>
> The tag/commit to be voted upon is v1.1.4:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=dfb1f971c35ca985a35692aed6d28be3d7e707de;hb=babab9648a6deb3d444c45bdd6d0ec30804e91f1
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.4-rc1/apache-storm-1.1.4-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.4-rc1/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1075
>
> Please vote on releasing this package as Apache Storm 1.1.4.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 1.1.4
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor

New Storm Committer and PMC Member

[Bobby Evans]

I am happy to announce that Govind Menon has just been added as the latest
Committer and PMC member to the Apache Storm Project.  Please join me in
congratulating him on this and thanking him for his contributions so far.

Thanks,

Bobby Evans

Re: [VOTE] Release Apache Storm 1.2.3 (rc1)

[Bobby Evans]

+1 built from the tag.  All the unit tests passed and I ran some manual
tests as well.

Thanks,

Bobby

On Wed, Jan 9, 2019 at 1:16 AM Julien Nioche 
wrote:

> Non-binding +1. Launched a Storm cluster in pseudo distributed mode and
> successfully ran a StormCrawler topology.
>
>  Thanks
>
> Julien
>
> On Tue, 8 Jan 2019 at 20:20, P. Taylor Goetz  wrote:
>
> > This is a call to vote on releasing Apache Storm 1.2.3 (rc1)
> >
> > Full list of changes in this release:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.3-rc1/RELEASE_NOTES.html
> >
> > The tag/commit to be voted upon is v1.2.3:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=5eaaa51591f2e4dc3e31f22bcc581af4a1f39c03;hb=6ba98b215857656e0186887b5d1a6a5aceee10c4
> >
> > The source archive being voted upon can be found here:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.3-rc1/apache-storm-1.2.3-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.3-rc1/
> >
> > The release artifacts are signed with the following key:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1074
> >
> > Please vote on releasing this package as Apache Storm 1.2.3.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 1.2.3
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
>
>
>
> --
>
> *Open Source Solutions for Text Engineering*
>
> http://www.digitalpebble.com
> http://digitalpebble.blogspot.com/
> #digitalpebble 
>

Re: [VOTE] Release Apache Storm 2.0.0 (rc4)

[Bobby Evans]

+1 built from the git tag.  Ran all of the unit tests and ran some manual
tests they all passed.

Thanks,

Bobby

On Tue, Jan 8, 2019 at 6:30 PM Xin Wang  wrote:

> +1
>
> Built it and ran all of the tests.  Everything passed.
>
> -Xin
>
> Kishorkumar Patil  于2019年1月9日周三 上午5:08写道：
>
> > +1
> >
> >  - built from source code and deployment works.
> >  -  Ran some of the tests for UI, DRPC, ThroughputVsLatency
> >  -  Validated UI bugs reported in the recent past are fixed in this
> version
> >
> > -Kishor
> >
> >
> > On Tue, Jan 8, 2019 at 2:29 PM Arun Mahadevan  wrote:
> >
> > > +1
> > >
> > > - Downloaded the binaries and validated signatures.
> > > - Deployed the binaries, ran some sample topologies and checked the UI.
> > > - Ran top level build using the source zip.
> > >
> > > Thanks,
> > > Arun
> > >
> > >
> > > On Tue, 8 Jan 2019 at 11:03, P. Taylor Goetz 
> wrote:
> > >
> > > > This is a call to vote on releasing Apache Storm 2.0.0 (rc4)
> > > >
> > > > Full list of changes in this release:
> > > >
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc4/RELEASE_NOTES.html
> > > >
> > > > The tag/commit to be voted upon is v2.0.0:
> > > >
> > > >
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=1eece73e8c9ed7f41d2f20f727bc7f644c499360;hb=ddee8decac57d1a4a0aa23cc76066609a2abc8d2
> > > >
> > > > The source archive being voted upon can be found here:
> > > >
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc4/apache-storm-2.0.0-src.tar.gz
> > > >
> > > > Other release files, signatures and digests can be found here:
> > > >
> > > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc4/
> > > >
> > > > The release artifacts are signed with the following key:
> > > >
> > > >
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > > >
> > > > The Nexus staging repository for this release is:
> > > >
> > > >
> https://repository.apache.org/content/repositories/orgapachestorm-1073
> > > >
> > > > Please vote on releasing this package as Apache Storm 2.0.0.
> > > >
> > > > When voting, please list the actions taken to verify the release.
> > > >
> > > > This vote will be open for at least 72 hours.
> > > >
> > > > [ ] +1 Release this package as Apache Storm 2.0.0
> > > > [ ]  0 No opinion
> > > > [ ] -1 Do not release this package because...
> > > >
> > > > Thanks to everyone who contributed to this release.
> > > >
> > > > -Taylor
> > >
> >
>
>
> --
> Thanks,
> Xin
>

Re: Storm 2.0.0 release?

[Bobby Evans]

I think all of the blockers are in now.  Please take a look and hopefully,
we can get a release out soon.

Thanks,

Bobby


On Fri, Dec 14, 2018 at 9:02 AM Bobby Evans  wrote:

> Sorry I was out at a conference for the past week, and have been heads down
> on a different project for a while before that.  I'll respond to the JIRA.
> I am happy to let it go in.
>
> Thanks,
>
> Bobby
>
> On Thu, Dec 13, 2018 at 1:07 PM Stig Rohde Døssing  >
> wrote:
>
> > I think STORM-2990/3279 is ready. Bobby had a question (
> > https://github.com/apache/storm/pull/2907#discussion_r234329136)
> regarding
> > whether Kafka offsets loop, but I wasn't sure where he was going with it,
> > so I didn't want to merge prematurely.
> >
> > I agree that we can postpone STORM-2720. As far as I know it's waiting
> for
> > STORM-2990 to go in, since it's going to be touching the same code.
> >
> > Den tor. 13. dec. 2018 kl. 19.54 skrev Roshan Naik
> > :
> >
> > >  Sounds like -  https://github.com/apache/storm/pull/2913 (STORM-3290)
> > is
> > > merged -  https://github.com/apache/storm/pull/2908 (STORM-3276) is
> > > nearly complete and may need some small tweaks. -
> > > https://github.com/apache/storm/pull/2907  (STORM-2990, STORM-3279)
> > > appears ready to be committed ?
> > >
> > > and- https://github.com/apache/storm/pull/2911   - (STORM-2720)
> seems a
> > > bit inactive and may not be critical enough to wait on.
> > >
> > > -roshan
> > > On Monday, November 26, 2018, 9:49:30 AM PST, Stig Rohde Døssing <
> > > stigdoess...@gmail.com> wrote:
> > >
> > >  I would like to get at least
> https://github.com/apache/storm/pull/2913
> > > (breaking changes) and https://github.com/apache/storm/pull/2908
> > > (regression) in.
> > >
> > > I think it would be nice to also get
> > > https://github.com/apache/storm/pull/2907 and
> > > https://github.com/apache/storm/pull/2911 in, but if we're in a hurry
> > they
> > > could go in the next release.
> > >
> > > Den man. 26. nov. 2018 kl. 17.37 skrev Julien Nioche <
> > > lists.digitalpeb...@gmail.com>:
> > >
> > > > Hi devs,
> > > >
> > > > Is there anything blocking the release of Storm 2.0? Any idea of when
> > it
> > > > could happen?
> > > >
> > > > Thanks
> > > >
> > > > Julien
> > > >
> > > > --
> > > >
> > > > *Open Source Solutions for Text Engineering*
> > > >
> > > > http://www.digitalpebble.com
> > > > http://digitalpebble.blogspot.com/
> > > > #digitalpebble <http://twitter.com/digitalpebble>
> > > >
> > >
> >
>

Re: Storm 2.0.0 release?

[Bobby Evans]

Sorry I was out at a conference for the past week, and have been heads down
on a different project for a while before that.  I'll respond to the JIRA.
I am happy to let it go in.

Thanks,

Bobby

On Thu, Dec 13, 2018 at 1:07 PM Stig Rohde Døssing 
wrote:

> I think STORM-2990/3279 is ready. Bobby had a question (
> https://github.com/apache/storm/pull/2907#discussion_r234329136) regarding
> whether Kafka offsets loop, but I wasn't sure where he was going with it,
> so I didn't want to merge prematurely.
>
> I agree that we can postpone STORM-2720. As far as I know it's waiting for
> STORM-2990 to go in, since it's going to be touching the same code.
>
> Den tor. 13. dec. 2018 kl. 19.54 skrev Roshan Naik
> :
>
> >  Sounds like -  https://github.com/apache/storm/pull/2913 (STORM-3290)
> is
> > merged -  https://github.com/apache/storm/pull/2908 (STORM-3276) is
> > nearly complete and may need some small tweaks. -
> > https://github.com/apache/storm/pull/2907  (STORM-2990, STORM-3279)
> > appears ready to be committed ?
> >
> > and- https://github.com/apache/storm/pull/2911   - (STORM-2720)  seems a
> > bit inactive and may not be critical enough to wait on.
> >
> > -roshan
> > On Monday, November 26, 2018, 9:49:30 AM PST, Stig Rohde Døssing <
> > stigdoess...@gmail.com> wrote:
> >
> >  I would like to get at least https://github.com/apache/storm/pull/2913
> > (breaking changes) and https://github.com/apache/storm/pull/2908
> > (regression) in.
> >
> > I think it would be nice to also get
> > https://github.com/apache/storm/pull/2907 and
> > https://github.com/apache/storm/pull/2911 in, but if we're in a hurry
> they
> > could go in the next release.
> >
> > Den man. 26. nov. 2018 kl. 17.37 skrev Julien Nioche <
> > lists.digitalpeb...@gmail.com>:
> >
> > > Hi devs,
> > >
> > > Is there anything blocking the release of Storm 2.0? Any idea of when
> it
> > > could happen?
> > >
> > > Thanks
> > >
> > > Julien
> > >
> > > --
> > >
> > > *Open Source Solutions for Text Engineering*
> > >
> > > http://www.digitalpebble.com
> > > http://digitalpebble.blogspot.com/
> > > #digitalpebble 
> > >
> >
>

Re: Storm Developer

[Bobby Evans]

Ashwin,

Attachments don't come through on the mailing lists.  Could you try putting
them in gist or something like that?

Thanks,

Bobby

On Fri, Nov 16, 2018 at 12:36 PM Ashwin Siddharth 
wrote:

> Hi ,
>
> Regarding Storm Build,
>
> 1)Please refer to the attachment below , I am getting a Failure for
> Storm-core, is this a successful build , if not why is the tar- file
> created in the target directory.Please help me to fix the Issue.
>
> 2)In the screenshot provided below, the Error part highlighted , is there
> a problem with Clojure, Is that something to be concerned about.If so how
> do debug it , how to go about it.
>
> 3)Also, for a successful storm build, which is the best version of the
> apache storm and maven .
>
> 4)Is git checkout an integral step for a successful build, if so what is
> the ideal version/tags , and the appropriate maven version.
>
> Please help me out here.
>
> Regards,
> Ashwin
>

Re: [VOTE] Release Apache Storm 2.0.0 (rc3)

[Bobby Evans]

Julien,

I have put up pull requests for the docs and for fixing some of the issues
with LocalCluster that you found.

https://github.com/apache/storm/pull/2891

https://github.com/apache/storm/pull/2892

The VersionInfo change is a blocker and we should fix it before releasing
(Sorry Taylor).

For the other stuff if you find more issues we can move it to a different
thread and work through them.

Thanks,

Bobby

On Mon, Oct 22, 2018 at 9:23 AM Bobby Evans  wrote:

> I'll look at upgrading that version of http client too.
>
> On Mon, Oct 22, 2018 at 9:15 AM Julien Nioche <
> lists.digitalpeb...@gmail.com> wrote:
>
>> Hi,
>>
>> I've looked into it a bit more and found that SC had a dependency on
>> storm-core and not storm-client; I've fixed this in 40612a3...
>> <
>> https://github.com/DigitalPebble/storm-crawler/commit/40612a3588d66e1d410a70b1c7e5db58d5c2ba4d
>> >
>> however
>> this doesn't affect the issues I had last week.
>>
>> *httpclient dependency conflict*
>> As seen last week, this is not shaded by Storm and the version used (4.3.3
>> <
>> https://github.com/apache/storm/blob/ce984cd31a16e7fe4b983659005f1f7648455404/pom.xml#L266
>> >)
>> is quite old. Even within Storm, the Storm-SOLR module uses a more recent
>> one (4.5
>> <
>> https://github.com/apache/storm/blob/master/external/storm-solr/pom.xml#L64
>> >).
>> StormCrawler needs at least 4.5.5
>> <
>> https://github.com/DigitalPebble/storm-crawler/blob/master/core/pom.xml#L26
>> >.
>> I expect other Storm users would use *httpclient* and have a similar
>> problem. Unless I am missing something, I can see the following solutions
>> sorted by how convenient they are to me as a user:
>>
>>1. the dependency is shaded by Storm
>>2. the dependency is upgraded to 4.5.5 by Storm
>>3. the dependency is shaded by StormCrawler
>>
>> Obviously, I'd rather not have to deal with (3) and anyone using
>> httpclient with Storm would have to do the same.
>>
>> Note: I can get my topology to work by specifying a protocol
>> implementation
>> based on OKHttp
>> *  http.protocol.implementation:
>> "com.digitalpebble.stormcrawler.protocol.okhttp.HttpProtocol"*
>> *  https.protocol.implementation:
>> "com.digitalpebble.stormcrawler.protocol.okhttp.HttpProtocol"*
>>
>> *LocalCluster*
>> Since removing the dependency on storm-core, I can't use LocalCluster
>> directly. I'll create a separate branch on my test repo to try to
>> reproduce
>> the issue.
>>
>> *Documentation for Local mode*
>> http://storm.apache.org/releases/2.0.0-SNAPSHOT/Local-mode.html
>> does not mention *--local-ttl *would be good to document it and indicate
>> what the default value is otherwise users might wonder why their
>> topologies
>> run for 20 secs only.  Personally, I'd rather be able to have a default
>> behaviour where the topology runs forever or at least be able to
>> deactivate
>> the TTL completely by setting it to -1.
>>
>> *ConfigurableTopology*
>> I am getting a different behavior between the original
>> ConfigurableTopology from
>> StormCrawler
>> <
>> https://github.com/DigitalPebble/storm-crawler/blob/master/core/src/main/java/com/digitalpebble/stormcrawler/ConfigurableTopology.java
>> >
>> and when I extend the one in Storm
>> <
>> https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/topology/ConfigurableTopology.java
>> >;
>> with the latter, any configuration found in the conf files passed in args
>> to the command line are added to the default values I provide instead of
>> replacing them. I'll investigate that further and open an issue if I find
>> a
>> bug.
>>
>> *Distributed mode*
>> I managed to launch the various services and run my test topology in
>> remote
>> mode (by changing the protocol implementation as explained above)
>>
>> *Flux*
>> http://storm.apache.org/releases/2.0.0-SNAPSHOT/flux.html tells me to run
>>
>> storm jar myTopology-0.1.0-SNAPSHOT.jar org.apache.storm.flux.Flux
>> --local my_config.yaml
>>
>>
>>
>> *apache-storm-2.0.0/bin/storm jar target/2-1.0-SNAPSHOT.jar
>> org.apache.storm.flux.Flux --local crawler.flux*
>>
>> but am getting
>>
>> *15:07:26.206 [main] ERROR o.a.s.f.Flux - To run in local mode run with
>> 'storm local' instead of 'storm jar'*
>>
>> *so *I tried both
>>
>> apache-storm-2.0.0/bin/storm local target/2-1.0-SNAP

Re: [VOTE] Release Apache Storm 2.0.0 (rc3)

[Bobby Evans]

Yup that is a bug.  I'll file a JIRA and work on a fix for it ASAP.

On Fri, Oct 19, 2018 at 1:33 PM Julien Nioche 
wrote:

> Hi Bobby
>
> The dependency issue happens when I have only storm-client as a dependency
> and not server.
>
> When trying to run it from Eclipse I had to add server to the pom, as
> expected but also client as I was getting
>
> 19:22:13.044 [main] ERROR o.a.s.u.VersionInfo - Could not load
> storm-core-version-info.properties
> java.io.IOException: Resource not found
> at
>
> org.apache.storm.utils.VersionInfo$VersionInfoImpl.(VersionInfo.java:53)
> [storm-client-2.0.0.jar:2.0.0]
> at org.apache.storm.utils.VersionInfo.(VersionInfo.java:41)
> [storm-client-2.0.0.jar:2.0.0]
> at org.apache.storm.daemon.nimbus.Nimbus.(Nimbus.java:281)
> [storm-server-2.0.0.jar:2.0.0]
> at org.apache.storm.LocalCluster.(LocalCluster.java:235)
> [storm-server-2.0.0.jar:2.0.0]
> at org.apache.storm.LocalCluster.(LocalCluster.java:156)
> [storm-server-2.0.0.jar:2.0.0]
> at
>
> com.digitalpebble.stormcrawler.ConfigurableTopology.submit(ConfigurableTopology.java:74)
> [classes/:?]
> at com.dipe.sc.CrawlTopology.run(CrawlTopology.java:80) [classes/:?]
> at
>
> com.digitalpebble.stormcrawler.ConfigurableTopology.start(ConfigurableTopology.java:49)
> [classes/:?]
> at com.dipe.sc.CrawlTopology.main(CrawlTopology.java:39) [classes/:?]
>
> I've put the code in https://github.com/DigitalPebble/storm2  if you want
> to have a look. You'll need to compile the branch 2.x of SC first
> https://github.com/DigitalPebble/storm-crawler/tree/2.x
>
> To reproduce the ZK issue, open the project in Eclipse and run the
> CrawlTopology class with "-local -conf crawler-conf.yaml" in arguments.
>
> For the dependency problem, mvn clean package followed by
> /data/apache-storm-2.0.0/bin/storm local target/2-1.0-SNAPSHOT.jar
> com.dipe.sc.CrawlTopology -conf crawler-conf.yaml
> should give java.lang.NoSuchMethodError:
>
> org.apache.http.impl.client.HttpClientBuilder.setConnectionManagerShared(Z)Lorg/apache/http/impl/client/HttpClientBuilder;
>
> Thanks
>
> Julien
>
> On Fri, 19 Oct 2018 at 17:26, Bobby Evans  wrote:
>
> > Sorry I should clarify a bit.
> >
> > `storm local` will run things in local mode, but the classpath will
> include
> > things that are not shaded.
> >
> > This is also true for trying to run tests from eclipse.  LocalCluster is
> a
> > part of storm-server so you will need to pull that in just for testing.
> > storm-client is what you want to depend on for the majority of your
> > topology.
> >
> > The ZK issue is new to me  We have done a lot in local mode and not seen
> > that as an issue.  If you can help me reproduce it I am happy to try and
> > debug it to see what is happening.
> >
> > Thanks,
> >
> > Bobby
> >
> > On Fri, Oct 19, 2018 at 11:21 AM Bobby Evans  wrote:
> >
> > > It is shaded in storm 2.x, but we split the classpath up, so what you
> > want
> > > to depend on is storm-client only.  I see you are pulling in storm-core
> > and
> > > a few other things that are not shaded, because they are only used by
> the
> > > daemons, not the clients.
> > >
> > > On Fri, Oct 19, 2018 at 10:55 AM Julien Nioche <
> > > lists.digitalpeb...@gmail.com> wrote:
> > >
> > >> Sorry, hit Return too quickly
> > >>
> > >> I am testing Storm 2.0.0 with StormCrawler, not very successfully. One
> > >> immediate issue is that I am getting a version conflict on httpclient
> as
> > >> the version set by Storm is older than the one I need.
> > >>
> > >> java.lang.NoSuchMethodError:
> > >>
> > >>
> >
> org.apache.http.impl.client.HttpClientBuilder.setConnectionManagerShared(Z)Lorg/apache/http/impl/client/HttpClientBuilder;
> > >> at
> > >>
> > >>
> >
> com.digitalpebble.stormcrawler.protocol.httpclient.HttpProtocol.configure(HttpProtocol.java:141)
> > >> ~[2-1.0-SNAPSHOT.jar:?]
> > >> at
> > >>
> > >>
> >
> com.digitalpebble.stormcrawler.protocol.ProtocolFactory.(ProtocolFactory.java:69)
> > >> ~[2-1.0-SNAPSHOT.jar:?]
> > >> at
> > >>
> > >>
> >
> com.digitalpebble.stormcrawler.bolt.FetcherBolt.prepare(FetcherBolt.java:760)
> > >> ~[2-1.0-SNAPSHOT.jar:?]
> > >> at
> > org.apache.storm.executor.bolt.BoltExecutor.init(BoltExecutor.java:144)
> > >> ~[storm-client-2.0.0.jar:2.0.0]
> > >> at
> > org.apache.storm.e

Re: [VOTE] Release Apache Storm 2.0.0 (rc3)

[Bobby Evans]

Sorry I should clarify a bit.

`storm local` will run things in local mode, but the classpath will include
things that are not shaded.

This is also true for trying to run tests from eclipse.  LocalCluster is a
part of storm-server so you will need to pull that in just for testing.
storm-client is what you want to depend on for the majority of your
topology.

The ZK issue is new to me  We have done a lot in local mode and not seen
that as an issue.  If you can help me reproduce it I am happy to try and
debug it to see what is happening.

Thanks,

Bobby

On Fri, Oct 19, 2018 at 11:21 AM Bobby Evans  wrote:

> It is shaded in storm 2.x, but we split the classpath up, so what you want
> to depend on is storm-client only.  I see you are pulling in storm-core and
> a few other things that are not shaded, because they are only used by the
> daemons, not the clients.
>
> On Fri, Oct 19, 2018 at 10:55 AM Julien Nioche <
> lists.digitalpeb...@gmail.com> wrote:
>
>> Sorry, hit Return too quickly
>>
>> I am testing Storm 2.0.0 with StormCrawler, not very successfully. One
>> immediate issue is that I am getting a version conflict on httpclient as
>> the version set by Storm is older than the one I need.
>>
>> java.lang.NoSuchMethodError:
>>
>> org.apache.http.impl.client.HttpClientBuilder.setConnectionManagerShared(Z)Lorg/apache/http/impl/client/HttpClientBuilder;
>> at
>>
>> com.digitalpebble.stormcrawler.protocol.httpclient.HttpProtocol.configure(HttpProtocol.java:141)
>> ~[2-1.0-SNAPSHOT.jar:?]
>> at
>>
>> com.digitalpebble.stormcrawler.protocol.ProtocolFactory.(ProtocolFactory.java:69)
>> ~[2-1.0-SNAPSHOT.jar:?]
>> at
>>
>> com.digitalpebble.stormcrawler.bolt.FetcherBolt.prepare(FetcherBolt.java:760)
>> ~[2-1.0-SNAPSHOT.jar:?]
>> at org.apache.storm.executor.bolt.BoltExecutor.init(BoltExecutor.java:144)
>> ~[storm-client-2.0.0.jar:2.0.0]
>> at org.apache.storm.executor.bolt.BoltExecutor.call(BoltExecutor.java:154)
>> ~[storm-client-2.0.0.jar:2.0.0]
>> at org.apache.storm.executor.bolt.BoltExecutor.call(BoltExecutor.java:58)
>> ~[storm-client-2.0.0.jar:2.0.0]
>> at org.apache.storm.utils.Utils$1.run(Utils.java:353)
>> [storm-client-2.0.0.jar:2.0.0]
>> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
>>
>> Here is the classpath when calling *storm local *
>>
>> *16:38:03.445 [main] INFO  o.a.s.s.o.a.z.ZooKeeper - Client
>>
>> environment:java.class.path=/data/apache-storm-2.0.0/*:/data/apache-storm-2.0.0/lib/log4j-over-slf4j-1.6.6.jar:/data/apache-storm-2.0.0/lib/hadoop-auth-2.6.1.jar:/data/apache-storm-2.0.0/lib/jaxb-api-2.3.0.jar:/data/apache-storm-2.0.0/lib/kryo-shaded-3.0.3.jar:/data/apache-storm-2.0.0/lib/kryo-3.0.3.jar:/data/apache-storm-2.0.0/lib/commons-cli-1.4.jar:/data/apache-storm-2.0.0/lib/log4j-slf4j-impl-2.11.1.jar:/data/apache-storm-2.0.0/lib/jetty-continuation-9.4.7.v20170914.jar:/data/apache-storm-2.0.0/lib/httpclient-4.3.3.jar:/data/apache-storm-2.0.0/lib/commons-io-2.6.jar:/data/apache-storm-2.0.0/lib/commons-collections-3.2.2.jar:/data/apache-storm-2.0.0/lib/guava-16.0.1.jar:/data/apache-storm-2.0.0/lib/metrics-graphite-3.2.6.jar:/data/apache-storm-2.0.0/lib/jetty-http-9.4.7.v20170914.jar:/data/apache-storm-2.0.0/lib/tools.logging-0.2.3.jar:/data/apache-storm-2.0.0/lib/jetty-util-9.4.7.v20170914.jar:/data/apache-storm-2.0.0/lib/rocksdbjni-5.8.6.jar:/data/apache-storm-2.0.0/lib/commons-fileupload-1.3.3.jar:/data/apache-storm-2.0.0/lib/curator-framework-4.0.1.jar:/data/apache-storm-2.0.0/lib/jackson-dataformat-smile-2.9.4.jar:/data/apache-storm-2.0.0/lib/asm-5.0.3.jar:/data/apache-storm-2.0.0/lib/jetty-io-9.4.7.v20170914.jar:/data/apache-storm-2.0.0/lib/chill-java-0.8.0.jar:/data/apache-storm-2.0.0/lib/curator-client-4.0.1.jar:/data/apache-storm-2.0.0/lib/httpcore-4.3.2.jar:/data/apache-storm-2.0.0/lib/log4j-api-2.11.1.jar:/data/apache-storm-2.0.0/lib/jetty-security-9.4.7.v20170914.jar:/data/apache-storm-2.0.0/lib/storm-clojure-2.0.0.jar:/data/apache-storm-2.0.0/lib/commons-compress-1.16.1.jar:/data/apache-storm-2.0.0/lib/jetty-server-9.4.7.v20170914.jar:/data/apache-storm-2.0.0/lib/netty-3.7.0.Final.jar:/data/apache-storm-2.0.0/lib/json-simple-1.1.jar:/data/apache-storm-2.0.0/lib/junit-4.12.jar:/data/apache-storm-2.0.0/lib/jetty-servlet-9.4.7.v20170914.jar:/data/apache-storm-2.0.0/lib/objenesis-2.6.jar:/data/apache-storm-2.0.0/lib/jetty-servlets-9.4.7.v20170914.jar:/data/apache-storm-2.0.0/lib/carbonite-1.5.0.jar:/data/apache-storm-2.0.0/lib/storm-server-2.0.0.jar:/data/apache-storm-2.0.0/lib/shaded-deps-2.0.0.jar:/data/apache-storm-2.0.0/lib/javax.servlet-api-3.1.0.jar:/data/apache-storm-2.0.0/lib/commons-logging-1.1.3.jar:/data/apache-storm-2.0.0/lib/jline-0.9.94.jar:/data/apache-storm-2.0.0/lib/storm-client-2.0.0.jar:/data/apache-storm-2.0.0

Re: [VOTE] Release Apache Storm 2.0.0 (rc3)

[Bobby Evans]

It is shaded in storm 2.x, but we split the classpath up, so what you want
to depend on is storm-client only.  I see you are pulling in storm-core and
a few other things that are not shaded, because they are only used by the
daemons, not the clients.

On Fri, Oct 19, 2018 at 10:55 AM Julien Nioche <
lists.digitalpeb...@gmail.com> wrote:

> Sorry, hit Return too quickly
>
> I am testing Storm 2.0.0 with StormCrawler, not very successfully. One
> immediate issue is that I am getting a version conflict on httpclient as
> the version set by Storm is older than the one I need.
>
> java.lang.NoSuchMethodError:
>
> org.apache.http.impl.client.HttpClientBuilder.setConnectionManagerShared(Z)Lorg/apache/http/impl/client/HttpClientBuilder;
> at
>
> com.digitalpebble.stormcrawler.protocol.httpclient.HttpProtocol.configure(HttpProtocol.java:141)
> ~[2-1.0-SNAPSHOT.jar:?]
> at
>
> com.digitalpebble.stormcrawler.protocol.ProtocolFactory.(ProtocolFactory.java:69)
> ~[2-1.0-SNAPSHOT.jar:?]
> at
>
> com.digitalpebble.stormcrawler.bolt.FetcherBolt.prepare(FetcherBolt.java:760)
> ~[2-1.0-SNAPSHOT.jar:?]
> at org.apache.storm.executor.bolt.BoltExecutor.init(BoltExecutor.java:144)
> ~[storm-client-2.0.0.jar:2.0.0]
> at org.apache.storm.executor.bolt.BoltExecutor.call(BoltExecutor.java:154)
> ~[storm-client-2.0.0.jar:2.0.0]
> at org.apache.storm.executor.bolt.BoltExecutor.call(BoltExecutor.java:58)
> ~[storm-client-2.0.0.jar:2.0.0]
> at org.apache.storm.utils.Utils$1.run(Utils.java:353)
> [storm-client-2.0.0.jar:2.0.0]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
>
> Here is the classpath when calling *storm local *
>
> *16:38:03.445 [main] INFO  o.a.s.s.o.a.z.ZooKeeper - Client
>
> environment:java.class.path=/data/apache-storm-2.0.0/*:/data/apache-storm-2.0.0/lib/log4j-over-slf4j-1.6.6.jar:/data/apache-storm-2.0.0/lib/hadoop-auth-2.6.1.jar:/data/apache-storm-2.0.0/lib/jaxb-api-2.3.0.jar:/data/apache-storm-2.0.0/lib/kryo-shaded-3.0.3.jar:/data/apache-storm-2.0.0/lib/kryo-3.0.3.jar:/data/apache-storm-2.0.0/lib/commons-cli-1.4.jar:/data/apache-storm-2.0.0/lib/log4j-slf4j-impl-2.11.1.jar:/data/apache-storm-2.0.0/lib/jetty-continuation-9.4.7.v20170914.jar:/data/apache-storm-2.0.0/lib/httpclient-4.3.3.jar:/data/apache-storm-2.0.0/lib/commons-io-2.6.jar:/data/apache-storm-2.0.0/lib/commons-collections-3.2.2.jar:/data/apache-storm-2.0.0/lib/guava-16.0.1.jar:/data/apache-storm-2.0.0/lib/metrics-graphite-3.2.6.jar:/data/apache-storm-2.0.0/lib/jetty-http-9.4.7.v20170914.jar:/data/apache-storm-2.0.0/lib/tools.logging-0.2.3.jar:/data/apache-storm-2.0.0/lib/jetty-util-9.4.7.v20170914.jar:/data/apache-storm-2.0.0/lib/rocksdbjni-5.8.6.jar:/data/apache-storm-2.0.0/lib/commons-fileupload-1.3.3.jar:/data/apache-storm-2.0.0/lib/curator-framework-4.0.1.jar:/data/apache-storm-2.0.0/lib/jackson-dataformat-smile-2.9.4.jar:/data/apache-storm-2.0.0/lib/asm-5.0.3.jar:/data/apache-storm-2.0.0/lib/jetty-io-9.4.7.v20170914.jar:/data/apache-storm-2.0.0/lib/chill-java-0.8.0.jar:/data/apache-storm-2.0.0/lib/curator-client-4.0.1.jar:/data/apache-storm-2.0.0/lib/httpcore-4.3.2.jar:/data/apache-storm-2.0.0/lib/log4j-api-2.11.1.jar:/data/apache-storm-2.0.0/lib/jetty-security-9.4.7.v20170914.jar:/data/apache-storm-2.0.0/lib/storm-clojure-2.0.0.jar:/data/apache-storm-2.0.0/lib/commons-compress-1.16.1.jar:/data/apache-storm-2.0.0/lib/jetty-server-9.4.7.v20170914.jar:/data/apache-storm-2.0.0/lib/netty-3.7.0.Final.jar:/data/apache-storm-2.0.0/lib/json-simple-1.1.jar:/data/apache-storm-2.0.0/lib/junit-4.12.jar:/data/apache-storm-2.0.0/lib/jetty-servlet-9.4.7.v20170914.jar:/data/apache-storm-2.0.0/lib/objenesis-2.6.jar:/data/apache-storm-2.0.0/lib/jetty-servlets-9.4.7.v20170914.jar:/data/apache-storm-2.0.0/lib/carbonite-1.5.0.jar:/data/apache-storm-2.0.0/lib/storm-server-2.0.0.jar:/data/apache-storm-2.0.0/lib/shaded-deps-2.0.0.jar:/data/apache-storm-2.0.0/lib/javax.servlet-api-3.1.0.jar:/data/apache-storm-2.0.0/lib/commons-logging-1.1.3.jar:/data/apache-storm-2.0.0/lib/jline-0.9.94.jar:/data/apache-storm-2.0.0/lib/storm-client-2.0.0.jar:/data/apache-storm-2.0.0/lib/snakeyaml-1.11.jar:/data/apache-storm-2.0.0/lib/hamcrest-core-1.3.jar:/data/apache-storm-2.0.0/lib/minlog-1.3.0.jar:/data/apache-storm-2.0.0/lib/slf4j-api-1.7.21.jar:/data/apache-storm-2.0.0/lib/log4j-core-2.11.1.jar:/data/apache-storm-2.0.0/lib/commons-exec-1.3.jar:/data/apache-storm-2.0.0/lib/storm-core-2.0.0.jar:/data/apache-storm-2.0.0/lib/jackson-core-2.9.4.jar:/data/apache-storm-2.0.0/lib/zookeeper-3.4.6.jar:/data/apache-storm-2.0.0/lib/commons-lang-2.6.jar:/data/apache-storm-2.0.0/lib/clojure-1.7.0.jar:/data/apache-storm-2.0.0/lib/metrics-core-3.2.6.jar:/data/apache-storm-2.0.0/lib/reflectasm-1.10.1.jar:/data/apache-storm-2.0.0/lib/commons-codec-1.11.jar:/data/apache-storm-2.0.0/lib/joda-time-2.3.jar:/data/apache-storm-2.0.0/extlib/*:target/2-1.0-SNAPSHOT.jar:/data/apache-storm-2.0.0/conf:/data/apache-storm-2.0.0/bin*
>
> This doesn't happen with Storm 1.2.2. Aren't these libs supposed 

Re: [VOTE] Release Apache Storm 2.0.0 (rc3)

[Bobby Evans]

+1 I checked out the code form the label.

Built it and ran all of the tests.  Everything passed.

I ran some small tests on a single node cluster and they all passed.  Also
did some performance tests and they look good too.

Thanks,

Bobby

On Tue, Oct 16, 2018 at 2:48 PM P. Taylor Goetz  wrote:

> This is a call to vote on releasing Apache Storm 2.0.0 (rc3)
>
> Full list of changes in this release:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc3/RELEASE_NOTES.html
>
> The tag/commit to be voted upon is v2.0.0:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=commit;h=d2d6f40344e6cc92ab07f3a462d577ef6b61f8b1
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc3/apache-storm-2.0.0-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc3/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1072
>
> Please vote on releasing this package as Apache Storm 2.0.0.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 2.0.0
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
>

Ready for another 2.0.0 RC

[Bobby Evans]

Taylor,

I think we have everything merged that was blocking the last 2.0.0
release.  If you have time please take a look at doing another one.

Thanks,

Bobby

Re: [VOTE] Release Apache Storm 2.0.0 (rc2)

[Bobby Evans]

Sorry everyone.

I forgot that I had not reverted the 3.4.13 ZK upmerge.  We need to do that
before a 2.0 is released.  I'll do it right now.

On Wed, Oct 10, 2018 at 6:47 PM Jungtaek Lim  wrote:

> Thanks all for the quick turnaround! Here's my +1 (binding).
>
> > source
>
> - verify file (signature, MD5, SHA)
> -- source, tar.gz : OK
> -- source, zip : OK
>
> - extract file
> -- source, tar.gz : OK
> -- source, zip : OK
>
> - diff-ing extracted files between tar.gz and zip : OK
>
> - build source with JDK 8 (-Pall-tests && -Pexternals)
> -- source, tar.gz : OK
>
> - build source dist
> -- source, tar.gz : OK
>
> - build binary dist
> -- source, tar.gz : OK
>
> > binary
>
> - verify file (signature, MD5, SHA)
> -- binary, tar.gz : OK
> -- binary, zip : OK
>
> - extract file
> -- binary, tar.gz : OK
> -- binary, zip : OK
>
> - diff-ing extracted files between tar.gz and zip : OK
>
> - launch daemons : OK
>
> - run RollingTopWords (local) : OK
>
> - run RollingTopWords (remote) : OK
>   - activate / deactivate / rebalance / kill : OK
>   - logviewer (worker dir, daemon dir) :OK
>   - change log level : OK
>   - thread dump, heap dump, restart worker : OK
>   - log search :OK
>
> Note that "profiling worker" and "topology log search" works now which were
> failing in RC1.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2018년 10월 11일 (목) 오전 3:02, Stig Rohde Døssing 님이
> 작성:
>
> > +1
> >
> > Built and ran unit tests from the tag.
> > Ran ExclamationTopology locally using the Storm tar, verified that UI
> looks
> > as expected, that logviewer works, and that there were no errors in the
> > logs.
> > Verified the signature and SHA512 for the source and binary tars.
> >
> > We should consider deleting the md5 files, Apache's release policy
> > recommends against including them in a release
> > https://www.apache.org/dev/release-distribution#sigs-and-sums.
> >
> >
> > Den ons. 10. okt. 2018 kl. 17.02 skrev Bobby Evans :
> >
> > > +1
> > >
> > > built and ran all of the unit tests from the tag.
> > > Ran some small perf tests on a single node cluster.  Things look really
> > > good there.
> > >
> > >
> > > On a side note our CI pipeline has been running and passing builds very
> > > close to this release too.  (we are following master currently) and it
> is
> > > looking good.
> > >
> > > Thanks,
> > >
> > > Bobby
> > >
> > > On Tue, Oct 9, 2018 at 4:02 PM Kishorkumar Patil
>  > >
> > > wrote:
> > >
> > > > +1  to release this package.
> > > >
> > > > I ran basic tests, and fucntionality tested manually some of the UI
> > > > features and profiling issues reported as part of the blockers. I did
> > not
> > > > notice any silent failures either - or any failures/exception in the
> > > logs.
> > > >
> > > > Regards
> > > > -Kishor
> > > >
> > > >
> > > > On Tue, Oct 9, 2018 at 4:05 PM P. Taylor Goetz 
> > > wrote:
> > > >
> > > > > This is a call to vote on releasing Apache Storm 2.0.0 (rc2)
> > > > >
> > > > > Full list of changes in this release:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc2/RELEASE_NOTES.html
> > > > >
> > > > > The tag/commit to be voted upon is v2.0.0:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=f8d04910dc3fd14534c186232ecf7882d8916f67;hb=f8d04910dc3fd14534c186232ecf7882d8916f67
> > > > >
> > > > > The source archive being voted upon can be found here:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc2/apache-storm-2.0.0-src.tar.gz
> > > > >
> > > > > Other release files, signatures and digests can be found here:
> > > > >
> > > > >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc2/
> > > > >
> > > > > The release artifacts are signed with the following key:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > > > >
> > > > > The Nexus staging repository for this release is:
> > > > >
> > > > >
> > https://repository.apache.org/content/repositories/orgapachestorm-1071
> > > > >
> > > > > Please vote on releasing this package as Apache Storm 2.0.0.
> > > > >
> > > > > When voting, please list the actions taken to verify the release.
> > > > >
> > > > > This vote will be open for at least 72 hours.
> > > > >
> > > > > [ ] +1 Release this package as Apache Storm 2.0.0
> > > > > [ ]  0 No opinion
> > > > > [ ] -1 Do not release this package because...
> > > > >
> > > > > Thanks to everyone who contributed to this release.
> > > > >
> > > > > -Taylor
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Release Apache Storm 2.0.0 (rc2)

[Bobby Evans]

+1

built and ran all of the unit tests from the tag.
Ran some small perf tests on a single node cluster.  Things look really
good there.


On a side note our CI pipeline has been running and passing builds very
close to this release too.  (we are following master currently) and it is
looking good.

Thanks,

Bobby

On Tue, Oct 9, 2018 at 4:02 PM Kishorkumar Patil 
wrote:

> +1  to release this package.
>
> I ran basic tests, and fucntionality tested manually some of the UI
> features and profiling issues reported as part of the blockers. I did not
> notice any silent failures either - or any failures/exception in the logs.
>
> Regards
> -Kishor
>
>
> On Tue, Oct 9, 2018 at 4:05 PM P. Taylor Goetz  wrote:
>
> > This is a call to vote on releasing Apache Storm 2.0.0 (rc2)
> >
> > Full list of changes in this release:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc2/RELEASE_NOTES.html
> >
> > The tag/commit to be voted upon is v2.0.0:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=f8d04910dc3fd14534c186232ecf7882d8916f67;hb=f8d04910dc3fd14534c186232ecf7882d8916f67
> >
> > The source archive being voted upon can be found here:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc2/apache-storm-2.0.0-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc2/
> >
> > The release artifacts are signed with the following key:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1071
> >
> > Please vote on releasing this package as Apache Storm 2.0.0.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 2.0.0
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
> >
>

Re: Cleaning Up Old Pull Requests STORM-3250

[Bobby Evans]

+1

Sounds good to me

On Mon, Oct 8, 2018 at 2:50 PM Derek Dagit  wrote:

> Currently there are over 150 open pull requests on the Apache Storm GitHub
> project. Over 100 of these have not been modified in 2018.
>
> It seems we are unlikely to handle each one of these without significant
> effort and time. Looking at many of them, they seem to be abandoned by the
> requester.
>
> I propose in STORM-3250 that we close all pull requests that have not been
> updated in 2018 and leave any corresponding Jira issues as they are. If
> there are any pull requests among these should remain open, please let me
> know. I plan to wait at least a week before requesting any changes.
>
>
>
> https://github.com/apache/storm/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+updated%3A%3C2018-01-01
>
> --
> Derek
>

Re: [VOTE] Release Apache Storm 2.0.0 (rc1)

[Bobby Evans]

Sounds good, I'll move them all over to 2.0.0 for this.

On Mon, Oct 8, 2018 at 2:36 PM P. Taylor Goetz  wrote:

> If we can keep the JIRA fix version for this release 2.0.0 it will keep
> the release notes cleaner.
>
> I can put together a release candidate tomorrow.
>
> -Taylor
>
> > On Oct 8, 2018, at 9:44 AM, Bobby Evans  wrote:
> >
> > Taylor,
> >
> > All of the blockers are now merged into master.  I think we are ready to
> do
> > another release.  Just so you know I marked all of the JIRAs that were
> > merged into master since this vote with 2.0.1 release version.  Not sure
> if
> > we just want to fast forward to 2.0.1, or if you want it to stay as
> 2.0.0.
> > If it is the later let me know and I am happy to move all of the JIRA to
> > 2.0.0 release version.
> >
> > Thanks,
> >
> > Bobby
> >
> > On Sun, Sep 30, 2018 at 2:55 AM Jungtaek Lim  wrote:
> >
> >> Just filed STORM-3238 [1] for topology log search error, and STORM-3239
> [2]
> >> for worker profile.
> >>
> >> - Jungtaek Lim (HeartSaVioR)
> >>
> >> 1. https://issues.apache.org/jira/browse/STORM-3238
> >> 2. https://issues.apache.org/jira/browse/STORM-3239
> >>
> >> 2018년 9월 28일 (금) 오전 12:13, Govind Menon 님이 작성:
> >>
> >>> I have a fix for the worker profiling that I'm testing now - I should
> >> have
> >>> a PR up today.
> >>>
> >>> On Thu, Sep 27, 2018 at 3:50 AM Jungtaek Lim 
> wrote:
> >>>
> >>>> Sorry I was in a long holidays in South Korea and am just back today.
> >>> I'll
> >>>> reproduce them and file JIRAs but it is pretty simple to reproduce,
> >> basic
> >>>> search in topology didn't work as well as operations in worker
> >> profiling
> >>>> didn't work. So if someone would like to tackle them it should be
> >> pretty
> >>>> easy to reproduce for someone as well.
> >>>>
> >>>> -Jungtaek Lim (HeartSaVioR)
> >>>>
> >>>> 2018년 9월 26일 (수) 오후 9:02, Kishorkumar Patil  >>> 님이
> >>>> 작성:
> >>>>
> >>>>> I have patch up for  WorkerToken Auto renewal fix. I think this
> >> should
> >>> be
> >>>>> added to 2.x RC.
> >>>>>
> >>>>> https://issues.apache.org/jira/browse/STORM-3235
> >>>>> https://github.com/apache/storm/pull/2850
> >>>>>
> >>>>> -Kishor
> >>>>>
> >>>>> On Tue, Sep 25, 2018 at 8:04 PM P. Taylor Goetz 
> >>>> wrote:
> >>>>>
> >>>>>> Jungtaek, can you elaborate on the issues you ran into when testing
> >>> the
> >>>>> rc
> >>>>>> and/or file JIRAs?
> >>>>>>
> >>>>>> -Taylor
> >>>>>>
> >>>>>>> On Sep 25, 2018, at 4:02 PM, Bobby Evans 
> >> wrote:
> >>>>>>>
> >>>>>>> Nope.  I plan to start looking into them, unless someone else is
> >>>>> already
> >>>>>>> doing so.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> Bobby
> >>>>>>>
> >>>>>>> On Tue, Sep 25, 2018 at 2:48 PM P. Taylor Goetz <
> >> ptgo...@gmail.com
> >>>>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> Thanks for the update. I will cancel the vote.
> >>>>>>>>
> >>>>>>>> Do the fixes you merged cover the bugs that Jungtaek pointed
> >> out?
> >>>>>>>>
> >>>>>>>> -Taylor
> >>>>>>>>
> >>>>>>>>> On Sep 25, 2018, at 1:56 PM, Bobby Evans 
> >>> wrote:
> >>>>>>>>>
> >>>>>>>>> Just FYI Taylor I just merged in 3 JIRA to the master branch
> >> and
> >>>> will
> >>>>>>>>> probably merge in the fix for the UI that Govind mentioned, but
> >>> all
> >>>>> of
> >>>>>>>> them
> >>>>>>>>> would be okay to go into 2.0.0 as well.  Since this RC looks
> >> like
&

Re: [VOTE] Release Apache Storm 2.0.0 (rc1)

[Bobby Evans]

Taylor,

All of the blockers are now merged into master.  I think we are ready to do
another release.  Just so you know I marked all of the JIRAs that were
merged into master since this vote with 2.0.1 release version.  Not sure if
we just want to fast forward to 2.0.1, or if you want it to stay as 2.0.0.
If it is the later let me know and I am happy to move all of the JIRA to
2.0.0 release version.

Thanks,

Bobby

On Sun, Sep 30, 2018 at 2:55 AM Jungtaek Lim  wrote:

> Just filed STORM-3238 [1] for topology log search error, and STORM-3239 [2]
> for worker profile.
>
> - Jungtaek Lim (HeartSaVioR)
>
> 1. https://issues.apache.org/jira/browse/STORM-3238
> 2. https://issues.apache.org/jira/browse/STORM-3239
>
> 2018년 9월 28일 (금) 오전 12:13, Govind Menon 님이 작성:
>
> > I have a fix for the worker profiling that I'm testing now - I should
> have
> > a PR up today.
> >
> > On Thu, Sep 27, 2018 at 3:50 AM Jungtaek Lim  wrote:
> >
> > > Sorry I was in a long holidays in South Korea and am just back today.
> > I'll
> > > reproduce them and file JIRAs but it is pretty simple to reproduce,
> basic
> > > search in topology didn't work as well as operations in worker
> profiling
> > > didn't work. So if someone would like to tackle them it should be
> pretty
> > > easy to reproduce for someone as well.
> > >
> > > -Jungtaek Lim (HeartSaVioR)
> > >
> > > 2018년 9월 26일 (수) 오후 9:02, Kishorkumar Patil  >님이
> > > 작성:
> > >
> > > > I have patch up for  WorkerToken Auto renewal fix. I think this
> should
> > be
> > > > added to 2.x RC.
> > > >
> > > > https://issues.apache.org/jira/browse/STORM-3235
> > > > https://github.com/apache/storm/pull/2850
> > > >
> > > > -Kishor
> > > >
> > > > On Tue, Sep 25, 2018 at 8:04 PM P. Taylor Goetz 
> > > wrote:
> > > >
> > > > > Jungtaek, can you elaborate on the issues you ran into when testing
> > the
> > > > rc
> > > > > and/or file JIRAs?
> > > > >
> > > > > -Taylor
> > > > >
> > > > > > On Sep 25, 2018, at 4:02 PM, Bobby Evans 
> wrote:
> > > > > >
> > > > > > Nope.  I plan to start looking into them, unless someone else is
> > > > already
> > > > > > doing so.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Bobby
> > > > > >
> > > > > > On Tue, Sep 25, 2018 at 2:48 PM P. Taylor Goetz <
> ptgo...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > >> Thanks for the update. I will cancel the vote.
> > > > > >>
> > > > > >> Do the fixes you merged cover the bugs that Jungtaek pointed
> out?
> > > > > >>
> > > > > >> -Taylor
> > > > > >>
> > > > > >>> On Sep 25, 2018, at 1:56 PM, Bobby Evans 
> > wrote:
> > > > > >>>
> > > > > >>> Just FYI Taylor I just merged in 3 JIRA to the master branch
> and
> > > will
> > > > > >>> probably merge in the fix for the UI that Govind mentioned, but
> > all
> > > > of
> > > > > >> them
> > > > > >>> would be okay to go into 2.0.0 as well.  Since this RC looks
> like
> > > it
> > > > is
> > > > > >> not
> > > > > >>> going to pass if you want to move the tag up on the master
> > branch,
> > > > that
> > > > > >>> would be fine, once we have the fixes in.
> > > > > >>>
> > > > > >>> Thanks,
> > > > > >>>
> > > > > >>> Bobby
> > > > > >>>
> > > > > >>> On Tue, Sep 25, 2018 at 9:31 AM Govind Menon
> > >  > > > >
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> Hi all,
> > > > > >>>>
> > > > > >>>> I introduced a bug with the visualization as part of the UI
> > > > migration
> > > > > >> with
> > > > > >>>> the following fix - https://github.com/apache/storm/pull/2849
> .
> > > It's
> > > > > >> not a
> > > 

Re: [VOTE] Release Apache Storm 2.0.0 (rc1)

[Bobby Evans]

Nope.  I plan to start looking into them, unless someone else is already
doing so.

Thanks,

Bobby

On Tue, Sep 25, 2018 at 2:48 PM P. Taylor Goetz  wrote:

> Thanks for the update. I will cancel the vote.
>
> Do the fixes you merged cover the bugs that Jungtaek pointed out?
>
> -Taylor
>
> > On Sep 25, 2018, at 1:56 PM, Bobby Evans  wrote:
> >
> > Just FYI Taylor I just merged in 3 JIRA to the master branch and will
> > probably merge in the fix for the UI that Govind mentioned, but all of
> them
> > would be okay to go into 2.0.0 as well.  Since this RC looks like it is
> not
> > going to pass if you want to move the tag up on the master branch, that
> > would be fine, once we have the fixes in.
> >
> > Thanks,
> >
> > Bobby
> >
> > On Tue, Sep 25, 2018 at 9:31 AM Govind Menon 
> > wrote:
> >
> >> Hi all,
> >>
> >> I introduced a bug with the visualization as part of the UI migration
> with
> >> the following fix - https://github.com/apache/storm/pull/2849. It's
> not a
> >> show stopper by any means and there's a workaround (specifically hiding
> >> system stats on the UI). I think we should include this in the RC.
> >>
> >> Thanks,
> >> Govind.
> >>
> >>
> >> On Mon, Sep 24, 2018 at 3:09 PM Alexandre Vermeerbergen <
> >> avermeerber...@gmail.com> wrote:
> >>
> >>> Hello All,
> >>>
> >>> I will need a little bit time to test this new release with our 15+
> >>> topologies, but it's definitely worth and exiting !
> >>> "stay tuned" :)
> >>>
> >>> Best regards,
> >>> Alexandre Vermeerbergen
> >>>
> >>> Le ven. 21 sept. 2018 à 20:23, P. Taylor Goetz  a
> >>> écrit :
> >>>>
> >>>> This is a call to vote on releasing Apache Storm 2.0.0 (rc1)
> >>>>
> >>>> Full list of changes in this release:
> >>>>
> >>>>
> >>>
> >>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc1/RELEASE_NOTES.html
> >>>>
> >>>> The tag/commit to be voted upon is v2.0.0:
> >>>>
> >>>>
> >>>
> >>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=062bb2f158a66b802a3ad7bdda14fb8f70cf9b88;hb=062bb2f158a66b802a3ad7bdda14fb8f70cf9b88
> >>>>
> >>>> The source archive being voted upon can be found here:
> >>>>
> >>>>
> >>>
> >>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc1/apache-storm-2.0.0-src.tar.gz
> >>>>
> >>>> Other release files, signatures and digests can be found here:
> >>>>
> >>>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc1/
> >>>>
> >>>> The release artifacts are signed with the following key:
> >>>>
> >>>>
> >>>
> >>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >>>>
> >>>> The Nexus staging repository for this release is:
> >>>>
> >>>>
> https://repository.apache.org/content/repositories/orgapachestorm-1070
> >>>>
> >>>> Please vote on releasing this package as Apache Storm 2.0.0.
> >>>>
> >>>> When voting, please list the actions taken to verify the release.
> >>>>
> >>>> This vote will be open for at least 72 hours.
> >>>>
> >>>> [ ] +1 Release this package as Apache Storm 2.0.0
> >>>> [ ]  0 No opinion
> >>>> [ ] -1 Do not release this package because...
> >>>>
> >>>> Thanks to everyone who contributed to this release.
> >>>>
> >>>> -Taylor
> >>>
> >>
>
>

Re: [VOTE] Release Apache Storm 2.0.0 (rc1)

[Bobby Evans]

Just FYI Taylor I just merged in 3 JIRA to the master branch and will
probably merge in the fix for the UI that Govind mentioned, but all of them
would be okay to go into 2.0.0 as well.  Since this RC looks like it is not
going to pass if you want to move the tag up on the master branch, that
would be fine, once we have the fixes in.

Thanks,

Bobby

On Tue, Sep 25, 2018 at 9:31 AM Govind Menon 
wrote:

> Hi all,
>
> I introduced a bug with the visualization as part of the UI migration with
> the following fix - https://github.com/apache/storm/pull/2849. It's not a
> show stopper by any means and there's a workaround (specifically hiding
> system stats on the UI). I think we should include this in the RC.
>
> Thanks,
> Govind.
>
>
> On Mon, Sep 24, 2018 at 3:09 PM Alexandre Vermeerbergen <
> avermeerber...@gmail.com> wrote:
>
> > Hello All,
> >
> > I will need a little bit time to test this new release with our 15+
> > topologies, but it's definitely worth and exiting !
> > "stay tuned" :)
> >
> > Best regards,
> > Alexandre Vermeerbergen
> >
> > Le ven. 21 sept. 2018 à 20:23, P. Taylor Goetz  a
> > écrit :
> > >
> > > This is a call to vote on releasing Apache Storm 2.0.0 (rc1)
> > >
> > > Full list of changes in this release:
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc1/RELEASE_NOTES.html
> > >
> > > The tag/commit to be voted upon is v2.0.0:
> > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=062bb2f158a66b802a3ad7bdda14fb8f70cf9b88;hb=062bb2f158a66b802a3ad7bdda14fb8f70cf9b88
> > >
> > > The source archive being voted upon can be found here:
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc1/apache-storm-2.0.0-src.tar.gz
> > >
> > > Other release files, signatures and digests can be found here:
> > >
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc1/
> > >
> > > The release artifacts are signed with the following key:
> > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > >
> > > The Nexus staging repository for this release is:
> > >
> > > https://repository.apache.org/content/repositories/orgapachestorm-1070
> > >
> > > Please vote on releasing this package as Apache Storm 2.0.0.
> > >
> > > When voting, please list the actions taken to verify the release.
> > >
> > > This vote will be open for at least 72 hours.
> > >
> > > [ ] +1 Release this package as Apache Storm 2.0.0
> > > [ ]  0 No opinion
> > > [ ] -1 Do not release this package because...
> > >
> > > Thanks to everyone who contributed to this release.
> > >
> > > -Taylor
> >
>

Re: [VOTE] Release Apache Storm 2.0.0 (rc1)

[Bobby Evans]

+1 I rebuilt the code from the git tag and all of the unit/integration
tests passed.  I also ran a single node cluster with both some simple
functionality and performance tests and everything looked good.

On a side note we just deployed a version based off of the same code as
2.0.0 to staging (think just some packaging differences and a few plugins),
but with the compatibility layer to run older topologies and everything is
looking good so far.

Thanks,

Bobby

On Fri, Sep 21, 2018 at 1:25 PM P. Taylor Goetz  wrote:

> This is a call to vote on releasing Apache Storm 2.0.0 (rc1)
>
> Full list of changes in this release:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc1/RELEASE_NOTES.html
>
> The tag/commit to be voted upon is v2.0.0:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=062bb2f158a66b802a3ad7bdda14fb8f70cf9b88;hb=062bb2f158a66b802a3ad7bdda14fb8f70cf9b88
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc1/apache-storm-2.0.0-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc1/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1070
>
> Please vote on releasing this package as Apache Storm 2.0.0.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 2.0.0
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
>

Hadoop Contributor Meetup

[Bobby Evans]

There is a Hadoop Contributor Meetup at Oath/Yahoo on Monday in Sunnyvale CA

https://www.meetup.com/Hadoop-Contributors/events/254012512/

This is short notice and we tend to be distributed very geographically as
an open source project, but I am going to be there and if anyone wants to
show up to talk about storm/stream processing that would be great.

Also happy to talk after the event.  just send me an email at
bo...@apache.org and we can schedule something.

Thanks,

Bobby

Re: Regarding releasing Apache Storm 2.0.0

[Bobby Evans]

I just merged in the last outstanding JIRA/pull request.  I think we are
good for a 2.0.0 RC.

Thanks,

Bobby

On Wed, Sep 19, 2018 at 12:33 PM Bobby Evans  wrote:

> Sounds good.
>
> I just filed https://issues.apache.org/jira/browse/STORM-3230 and I'll be
> putting up a pull request shortly. I would like to see it in before a 2.x
> release, but it is kind of minor because ZK has to really be overloaded to
> hit this, and we tend to recover after a while.
>
> I'll look at getting the rest in ASAP.
>
> Thanks,
>
> Bobby
>
>
> On Tue, Sep 18, 2018 at 3:54 PM P. Taylor Goetz  wrote:
>
>> I’m ready to release when everything is ready to go. Since we haven’t
>> released from the 2.0-based master branch, I wouldn’t be surprised if I run
>> into release issues, but I’ll slog through it.
>>
>> -Taylor
>>
>> > On Sep 18, 2018, at 10:46 AM, Bobby Evans  wrote:
>> >
>> > Great work everyone.  We are really close on this.  We have everything
>> in
>> > except for https://github.com/apache/storm/pull/2719, but there has
>> been no
>> > movement there, so I will try and put up an alternative pull request.
>> >
>> > Also We noticed that a recent merge broke some things fairly badly so we
>> > need to get https://github.com/apache/storm/pull/2839 in, but that is
>> just
>> > a matter of waiting a few more hours for the 24 hours to be up.
>> >
>> > Great work everyone, hopefully we will have an RC up for a vote a little
>> > over a day from now.
>> >
>> > Thanks,
>> >
>> > Bobby
>> >
>> > P.S. Taylor,  You have put up all of the release candidates in the past
>> and
>> > done all of the votes for them.  If you want to continue the trend that
>> is
>> > fine with me, but if not I am happy to do it, but I might have to bug
>> you
>> > to be sure I do it all correctly.
>> >
>> > On Mon, Sep 17, 2018 at 9:13 AM Bobby Evans  wrote:
>> >
>> >> I think we are really close on this and I would love to see us get an
>> RC
>> >> out ASAP.
>> >>
>> >> We are still missing some things that Stig called out.
>> >>
>> >> https://github.com/apache/storm/pull/2719 has a build issue, not sure
>> if
>> >> we need to make an alternative patch or not.
>> >> https://github.com/apache/storm/pull/2800  has a newer alternative
>> patch
>> >> https://github.com/apache/storm/pull/2836 please take a look.
>> >> https://github.com/apache/storm/pull/2805 has some merge conflicts
>> >> currently, but everyone please take a chance to review it.
>> >>
>> >> Thanks,
>> >>
>> >> Bobby
>> >>
>> >> On Fri, Sep 14, 2018 at 2:57 AM Jungtaek Lim 
>> wrote:
>> >>
>> >>> I have sought the name of client artifact from some of streaming
>> >>> frameworks. Please refer below:
>> >>>
>> >>> Spark: spark-core
>> >>> Kafka: kafka-clients
>> >>> Flink: flink-clients
>> >>> Heron: heron-api
>> >>>
>> >>> Based on divergence, I don't see the reason "storm-core" is the only
>> name
>> >>> which avoid confusion. Actually, if my understanding is right, we
>> need to
>> >>> let end users including "storm-server" when running local cluster,
>> then
>> >>> "storm-core" vs "storm-server" would give real confusion. I guess we
>> >>> already discussed about the naming, and given that we don't rename it
>> we
>> >>> are OK with renamed artifacts.
>> >>>
>> >>> 2018년 9월 14일 (금) 오후 4:07, Roshan Naik > >님이
>> >>> 작성:
>> >>>
>> >>>> Happy to see consensus in moving fwd with 2.0 soon.
>> >>>> I will try to get a minor patch (STORM-3205) within 24 hours ... as
>> it
>> >>>> seems like it has potential to deliver a decent perf boost and energy
>> >>>> savings.
>> >>>> One thing I am hoping we can address before releasing Storm 2 is...
>> to
>> >>> fix
>> >>>> the naming of the storm-client.jar.  Its such a core jar really, it
>> >>> should
>> >>>> have been really called storm-core or something like that... but
>> >>>> unfortunately we already have another jar with that name.  Retaining
>> the
>&g

Re: Regarding releasing Apache Storm 2.0.0

[Bobby Evans]

Sounds good.

I just filed https://issues.apache.org/jira/browse/STORM-3230 and I'll be
putting up a pull request shortly. I would like to see it in before a 2.x
release, but it is kind of minor because ZK has to really be overloaded to
hit this, and we tend to recover after a while.

I'll look at getting the rest in ASAP.

Thanks,

Bobby


On Tue, Sep 18, 2018 at 3:54 PM P. Taylor Goetz  wrote:

> I’m ready to release when everything is ready to go. Since we haven’t
> released from the 2.0-based master branch, I wouldn’t be surprised if I run
> into release issues, but I’ll slog through it.
>
> -Taylor
>
> > On Sep 18, 2018, at 10:46 AM, Bobby Evans  wrote:
> >
> > Great work everyone.  We are really close on this.  We have everything in
> > except for https://github.com/apache/storm/pull/2719, but there has
> been no
> > movement there, so I will try and put up an alternative pull request.
> >
> > Also We noticed that a recent merge broke some things fairly badly so we
> > need to get https://github.com/apache/storm/pull/2839 in, but that is
> just
> > a matter of waiting a few more hours for the 24 hours to be up.
> >
> > Great work everyone, hopefully we will have an RC up for a vote a little
> > over a day from now.
> >
> > Thanks,
> >
> > Bobby
> >
> > P.S. Taylor,  You have put up all of the release candidates in the past
> and
> > done all of the votes for them.  If you want to continue the trend that
> is
> > fine with me, but if not I am happy to do it, but I might have to bug you
> > to be sure I do it all correctly.
> >
> > On Mon, Sep 17, 2018 at 9:13 AM Bobby Evans  wrote:
> >
> >> I think we are really close on this and I would love to see us get an RC
> >> out ASAP.
> >>
> >> We are still missing some things that Stig called out.
> >>
> >> https://github.com/apache/storm/pull/2719 has a build issue, not sure
> if
> >> we need to make an alternative patch or not.
> >> https://github.com/apache/storm/pull/2800  has a newer alternative
> patch
> >> https://github.com/apache/storm/pull/2836 please take a look.
> >> https://github.com/apache/storm/pull/2805 has some merge conflicts
> >> currently, but everyone please take a chance to review it.
> >>
> >> Thanks,
> >>
> >> Bobby
> >>
> >> On Fri, Sep 14, 2018 at 2:57 AM Jungtaek Lim  wrote:
> >>
> >>> I have sought the name of client artifact from some of streaming
> >>> frameworks. Please refer below:
> >>>
> >>> Spark: spark-core
> >>> Kafka: kafka-clients
> >>> Flink: flink-clients
> >>> Heron: heron-api
> >>>
> >>> Based on divergence, I don't see the reason "storm-core" is the only
> name
> >>> which avoid confusion. Actually, if my understanding is right, we need
> to
> >>> let end users including "storm-server" when running local cluster, then
> >>> "storm-core" vs "storm-server" would give real confusion. I guess we
> >>> already discussed about the naming, and given that we don't rename it
> we
> >>> are OK with renamed artifacts.
> >>>
> >>> 2018년 9월 14일 (금) 오후 4:07, Roshan Naik  >님이
> >>> 작성:
> >>>
> >>>> Happy to see consensus in moving fwd with 2.0 soon.
> >>>> I will try to get a minor patch (STORM-3205) within 24 hours ... as it
> >>>> seems like it has potential to deliver a decent perf boost and energy
> >>>> savings.
> >>>> One thing I am hoping we can address before releasing Storm 2 is... to
> >>> fix
> >>>> the naming of the storm-client.jar.  Its such a core jar really, it
> >>> should
> >>>> have been really called storm-core or something like that... but
> >>>> unfortunately we already have another jar with that name.  Retaining
> the
> >>>> 'client' name for this new jar would be confusing and give wrong
> >>>> impressions to users and any new devs IMO.
> >>>> -roshan
> >>>>
> >>>>On Thursday, September 13, 2018, 2:12:40 PM PDT, Govind Menon
> >>>>  wrote:
> >>>>
> >>>> STORM-3217 and STORM-3221 have been fixed - +1 from me for 2.0 RC.
> >>>>
> >>>> On Wed, Sep 12, 2018 at 10:01 AM Govind Menon 
> wrote:
> >>>>
> >>>>> Hi all,
> >>>>>
> >>>>> 

Re: Regarding releasing Apache Storm 2.0.0

[Bobby Evans]

Great work everyone.  We are really close on this.  We have everything in
except for https://github.com/apache/storm/pull/2719, but there has been no
movement there, so I will try and put up an alternative pull request.

Also We noticed that a recent merge broke some things fairly badly so we
need to get https://github.com/apache/storm/pull/2839 in, but that is just
a matter of waiting a few more hours for the 24 hours to be up.

Great work everyone, hopefully we will have an RC up for a vote a little
over a day from now.

Thanks,

Bobby

P.S. Taylor,  You have put up all of the release candidates in the past and
done all of the votes for them.  If you want to continue the trend that is
fine with me, but if not I am happy to do it, but I might have to bug you
to be sure I do it all correctly.

On Mon, Sep 17, 2018 at 9:13 AM Bobby Evans  wrote:

> I think we are really close on this and I would love to see us get an RC
> out ASAP.
>
> We are still missing some things that Stig called out.
>
> https://github.com/apache/storm/pull/2719 has a build issue, not sure if
> we need to make an alternative patch or not.
> https://github.com/apache/storm/pull/2800  has a newer alternative patch
> https://github.com/apache/storm/pull/2836 please take a look.
> https://github.com/apache/storm/pull/2805 has some merge conflicts
> currently, but everyone please take a chance to review it.
>
> Thanks,
>
> Bobby
>
> On Fri, Sep 14, 2018 at 2:57 AM Jungtaek Lim  wrote:
>
>> I have sought the name of client artifact from some of streaming
>> frameworks. Please refer below:
>>
>> Spark: spark-core
>> Kafka: kafka-clients
>> Flink: flink-clients
>> Heron: heron-api
>>
>> Based on divergence, I don't see the reason "storm-core" is the only name
>> which avoid confusion. Actually, if my understanding is right, we need to
>> let end users including "storm-server" when running local cluster, then
>> "storm-core" vs "storm-server" would give real confusion. I guess we
>> already discussed about the naming, and given that we don't rename it we
>> are OK with renamed artifacts.
>>
>> 2018년 9월 14일 (금) 오후 4:07, Roshan Naik 님이
>> 작성:
>>
>> >  Happy to see consensus in moving fwd with 2.0 soon.
>> > I will try to get a minor patch (STORM-3205) within 24 hours ... as it
>> > seems like it has potential to deliver a decent perf boost and energy
>> > savings.
>> > One thing I am hoping we can address before releasing Storm 2 is... to
>> fix
>> > the naming of the storm-client.jar.  Its such a core jar really, it
>> should
>> > have been really called storm-core or something like that... but
>> > unfortunately we already have another jar with that name.  Retaining the
>> > 'client' name for this new jar would be confusing and give wrong
>> > impressions to users and any new devs IMO.
>> > -roshan
>> >
>> > On Thursday, September 13, 2018, 2:12:40 PM PDT, Govind Menon
>> >  wrote:
>> >
>> >  STORM-3217 and STORM-3221 have been fixed - +1 from me for 2.0 RC.
>> >
>> > On Wed, Sep 12, 2018 at 10:01 AM Govind Menon  wrote:
>> >
>> > > Hi all,
>> > >
>> > > There are some regressions that I introduced as part of STORM-1311
>> which
>> > > I'm working on as part of
>> > https://issues.apache.org/jira/browse/STORM-3217
>> > > and https://issues.apache.org/jira/browse/STORM-3221. These should be
>> > > fixed before a 2.x release
>> > >
>> > > I have code working on the Yahoo internal branch and should have PRs
>> up
>> > > for them in community soon.
>> > >
>> > > I apologize for slowing things up.
>> > >
>> > > Thanks,
>> > > Govind.
>> > >
>> > > On Tue, Sep 11, 2018 at 3:31 PM Arun Mahadevan 
>> wrote:
>> > >
>> > >> +1 for releasing 2.0.
>> > >>
>> > >> May be the RC can be cut once critical patches are merged.
>> > >>
>> > >> On Tue, 11 Sep 2018 at 10:28, Stig Rohde Døssing <
>> > stigdoess...@gmail.com>
>> > >> wrote:
>> > >>
>> > >> > +1 to cut an RC.
>> > >> >
>> > >> > Here are a couple of PRs that could maybe go in
>> > >> >
>> > >> > https://github.com/apache/storm/pull/2719
>> > >> > https://github.com/apache/storm/pull/2800 (this one requires some
>> > >> changes,
>> > >>

Re: Regarding releasing Apache Storm 2.0.0

[Bobby Evans]

on't see any open issues/blockers at this point for going ahead
> > >> with
> > >> > 2.x
> > >> > > > release.
> > >> > > >
> > >> > > > I am +1 to 2.0 release.
> > >> > > >
> > >> > > > Regards,
> > >> > > > -Kishor
> > >> > > >
> > >> > > > On Mon, Sep 10, 2018 at 2:24 PM, P. Taylor Goetz <
> > ptgo...@gmail.com
> > >> >
> > >> > > wrote:
> > >> > > >
> > >> > > > > I agree, and looking through the JIRAs against 2.0, I would
> say
> > a
> > >> > > majority
> > >> > > > > of the ones marked critical are not critical.
> > >> > > > >
> > >> > > > > I’m +1 on moving forward with a 2.0 release, but will give
> > others
> > >> > time
> > >> > > to
> > >> > > > > respond with any JIRAs they think should be included.
> > >> > > > >
> > >> > > > > > p.s. I don't want to create branch-2.x or branch-2.0.x until
> > >> > > absolutely
> > >> > > > > > necessary, I don't see any major features with pull requests
> > up
> > >> but
> > >> > > if
> > >> > > > > you
> > >> > > > > > do run across one please send something out before merging
> it
> > >> in,
> > >> > so
> > >> > > we
> > >> > > > > can
> > >> > > > > > set up the branches properly at that time.
> > >> > > > >
> > >> > > > >
> > >> > > > > Agree. We can always branch off the release tag/commit.
> > >> > > > >
> > >> > > > > -Taylor
> > >> > > > >
> > >> > > > >
> > >> > > > > > On Sep 10, 2018, at 12:25 PM, Bobby Evans  >
> > >> > wrote:
> > >> > > > > >
> > >> > > > > > It has been nearly a month since this was originally sent
> out,
> > >> and
> > >> > > this
> > >> > > > > is
> > >> > > > > > not the first of these kinds of emails to go out about a
> 2.0.0
> > >> > > release.
> > >> > > > > I
> > >> > > > > > think we have made a lot of really good progress on getting
> > >> ready
> > >> > > for a
> > >> > > > > 2.0
> > >> > > > > > release, and I really would like to see it happen before
> > another
> > >> > > month
> > >> > > > > > passes.
> > >> > > > > >
> > >> > > > > > We have a 2.0 based deploy in some of our staging clusters,
> > >> > currently
> > >> > > > > > following the master branch with a little that is Yahoo
> > >> specific on
> > >> > > top.
> > >> > > > > We
> > >> > > > > > would like to start pushing towards production with it soon.
> > >> > > > > >
> > >> > > > > > There are a few issues that we are aware of.
> > >> > > > > >
> > >> > > > > >
> > >> > >
> > >>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20STORM%20AND%
> > >> > > > > 20affectedVersion%20in%20(2.0.0)%20AND%20resolution%20%3D%
> > >> > > > > 20Unresolved%20ORDER%20BY%20priority%20DESC
> > >> > > > > >
> > >> > > > > > There are no blockers still open, and only 4 issues listed
> as
> > >> > > critical.
> > >> > > > > >
> > >> > > > > > If others have any open issues that feel need to be
> addressed
> > >> prior
> > >> > > to a
> > >> > > > > > 2.0.0 release please respond to this with the JIRA number.
> I
> > >> would
> > >> > > like
> > >> > > > > to
> > >> > > > > > set a goal/tentative date of Sep 17th (one week from today)
> to
> > >>

Re: Regarding releasing Apache Storm 2.0.0

[Bobby Evans]

It has been nearly a month since this was originally sent out, and this is
not the first of these kinds of emails to go out about a 2.0.0 release.  I
think we have made a lot of really good progress on getting ready for a 2.0
release, and I really would like to see it happen before another month
passes.

We have a 2.0 based deploy in some of our staging clusters, currently
following the master branch with a little that is Yahoo specific on top. We
would like to start pushing towards production with it soon.

There are a few issues that we are aware of.

https://issues.apache.org/jira/issues/?jql=project%20%3D%20STORM%20AND%20affectedVersion%20in%20(2.0.0)%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20priority%20DESC

There are no blockers still open, and only 4 issues listed as critical.

If others have any open issues that feel need to be addressed prior to a
2.0.0 release please respond to this with the JIRA number.  I would like to
set a goal/tentative date of Sep 17th (one week from today) to put together
a release candidate for a 2.0.0 release, and unless there are major
blockers that show up I think we can do it.

Thanks,

Bobby Evans

p.s. I don't want to create branch-2.x or branch-2.0.x until absolutely
necessary, I don't see any major features with pull requests up but if you
do run across one please send something out before merging it in, so we can
set up the branches properly at that time.


On Wed, Jul 18, 2018 at 10:47 PM Jungtaek Lim  wrote:

> I'd like to say first, thanks Stig to take up remaining issues. Thanks to
> his efforts, according to the epic, we have only one major issue left:
> porting UI to Java [1], and pull request [2] is available for that.
> There're another issues [3] [4] targeting 2.0.0 (since it is backward
> incompatible) but they are all about removing deprecated things, so easier
> to be reviewed and make decisions.
>
> Once we have a patch for that now, IMHO it would be good to review and ship
> in 2.0.0 if it wouldn't take a month or so. We could do some sanity tests
> in parallel, so waiting for UI port would not block much time on releasing
> Storm 2.0.0.
>
> - Jungtaek Lim (HeartSaVioR)
>
> 1. https://issues.apache.org/jira/browse/STORM-1311
> 2. https://github.com/apache/storm/pull/2752
> 3. https://issues.apache.org/jira/browse/STORM-2947
> 4. https://issues.apache.org/jira/browse/STORM-3156
>
>
> 2018년 7월 11일 (수) 오전 5:12, Alexandre Vermeerbergen <
> avermeerber...@gmail.com>님이
> 작성:
>
> > +1 would love to try it when an RC is avail!
> >
> > Alexandre Vermeerbergen
> >
> > 2018-07-10 21:15 GMT+02:00 Arun Mahadevan :
> > > +1 to get it out soon.
> > >
> > >
> > >
> > >
> > > On 7/10/18, 11:52 AM, "P. Taylor Goetz"  wrote:
> > >
> > >>+1 Sounds good to me.
> > >>
> > >>-Taylor
> > >>
> > >>> On Jul 10, 2018, at 2:18 AM, Jungtaek Lim  wrote:
> > >>>
> > >>> Hi devs,
> > >>>
> > >>> I hopefully have a time to sort out issues regarding Storm 2.0.0 and
> > link
> > >>> to epic issue.
> > >>>
> > >>> https://issues.apache.org/jira/browse/STORM-2714
> > >>> (require login to Apache JIRA to see issues in epic)
> > >>>
> > >>> I guess we are close to the release, mostly left reviewing some
> pending
> > >>> pull requests, and some manual sanity tests.
> > >>>
> > >>> Given that master branch is relatively stabilized for Travis CI
> build,
> > as
> > >>> well as style check and Java port make codebase better (at least for
> > me), I
> > >>> would really want to make Storm 2.0.0 released sooner than later, and
> > rely
> > >>> majorly on 2.x version line.
> > >>>
> > >>> So I would propose dev folks to concentrate on remaining tasks for
> > Storm
> > >>> 2.0.0 till we announce release. WDYT?
> > >>>
> > >>> Thanks,
> > >>> Jungtaek Lim (HeartSaVioR)
> > >>
> > >
> >
>

Re: [VOTE] Release Apache Storm 1.2.2 (rc3)

[Bobby Evans]

+1 I built from the git tag, all the unit tests passed.  I ran a single
node cluster and everything worked for my small unit tests.

Thanks,

bobby

On Wed, May 23, 2018 at 3:54 PM Jungtaek Lim  wrote:

> REMINDER: The vote has been opened from 6 days ago, and still has only two
> binding +1s. Please verify the release candidate and vote.
>
> 2018년 5월 21일 (월) 오후 11:16, Julien Nioche 님이
> 작성:
>
> > +1 non binding
> >
> > compiled + ran StormCrawler topology without problems
> >
> > Thanks
> >
> > Julien
> >
> >
> >
> > On 17 May 2018 at 20:59, P. Taylor Goetz  wrote:
> >
> > > This is a call to vote on releasing Apache Storm 1.2.2 (rc3)
> > >
> > > Full list of changes in this release:
> > >
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > > 2.2-rc3/RELEASE_NOTES.html
> > >
> > > The tag/commit to be voted upon is v1.2.2:
> > >
> > > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=
> > >
> >
> d29eaa70280dd0ac4b7d393387fab4a91eee9e9d;hb=d2d6f40344e6cc92ab07f3a462d577
> > > ef6b61f8b1
> > >
> > > The source archive being voted upon can be found here:
> > >
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > > 2.2-rc3/apache-storm-1.2.2-src.tar.gz
> > >
> > > Other release files, signatures and digests can be found here:
> > >
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.2-rc3/
> > >
> > > The release artifacts are signed with the following key:
> > >
> > > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> > > plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > >
> > > The Nexus staging repository for this release is:
> > >
> > > https://repository.apache.org/content/repositories/orgapachestorm-1068
> > >
> > > Please vote on releasing this package as Apache Storm 1.2.2.
> > >
> > > When voting, please list the actions taken to verify the release.
> > >
> > > This vote will be open for 72 hours or until at least 3 PMC members
> vote
> > > +1.
> > >
> > > [ ] +1 Release this package as Apache Storm 1.2.2
> > > [ ]  0 No opinion
> > > [ ] -1 Do not release this package because...
> > >
> > > Thanks to everyone who contributed to this release.
> > >
> > > -Taylor
> > >
> >
> >
> >
> > --
> >
> > *Open Source Solutions for Text Engineering*
> >
> > http://www.digitalpebble.com
> > http://digitalpebble.blogspot.com/
> > #digitalpebble 
> >
>

Re: [VOTE] Release Apache Storm 1.2.2 (rc3)

[Bobby Evans]

+1

I built it from source.  All the unit tests passed and I ran a few manual
tests on a single node cluster.

Thanks,

Bobby

On Thu, May 17, 2018 at 3:00 PM P. Taylor Goetz  wrote:

> This is a call to vote on releasing Apache Storm 1.2.2 (rc3)
>
> Full list of changes in this release:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.2-rc3/RELEASE_NOTES.html
>
> The tag/commit to be voted upon is v1.2.2:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=d29eaa70280dd0ac4b7d393387fab4a91eee9e9d;hb=d2d6f40344e6cc92ab07f3a462d577ef6b61f8b1
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.2-rc3/apache-storm-1.2.2-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.2-rc3/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1068
>
> Please vote on releasing this package as Apache Storm 1.2.2.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for 72 hours or until at least 3 PMC members vote
> +1.
>
> [ ] +1 Release this package as Apache Storm 1.2.2
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
>

Re: [VOTE] Release Apache Storm 1.1.3 (rc1)

[Bobby Evans]

+1

I built from the git tag, ran all of the unit tests launched a single node
cluster and ran a few simple tests on it.

On Thu, May 3, 2018 at 12:21 PM P. Taylor Goetz  wrote:

> This is a call to vote on releasing Apache Storm 1.1.3 (rc1)
>
> Full list of changes in this release:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.3-rc1/RELEASE_NOTES.html
>
> The tag/commit to be voted upon is v1.1.3:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=caffcf810cf28b8b8c7ea6e15a555a7160de1384;hb=e590a3289a9f7b9806a4c5cb78a2c06b4fbf019f
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.3-rc1/apache-storm-1.1.3-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.3-rc1/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1065
>
> Please vote on releasing this package as Apache Storm 1.1.3.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for 72 hours or until at least 3 PMC members vote
> +1.
>
> [ ] +1 Release this package as Apache Storm 1.1.3
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
>

Re: [VOTE] Release Apache Storm 1.2.2 (rc2)

[Bobby Evans]

+1 I built it from the tag (e001672cf0ea59fe6989b563fb6bbb450fe8e7e5) since
the tag did move recently.

I ran all the unit tests, I started a single node cluster and ran some
manual tests.

Thanks,

Bobby

On Thu, May 3, 2018 at 12:18 PM P. Taylor Goetz  wrote:

> CORRECTION: The Nexus staging repository for this rc is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1064
>
>
> On May 3, 2018, at 11:42 AM, P. Taylor Goetz  wrote:
>
> This is a call to vote on releasing Apache Storm 1.2.2 (rc2)
>
> Full list of changes in this release:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.2-rc2/RELEASE_NOTES.html
>
> The tag/commit to be voted upon is v1.2.2:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=7cb19fb3befa65e5ff9e5e02f38e16de865982a9;hb=e001672cf0ea59fe6989b563fb6bbb450fe8e7e5
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.2-rc2/apache-storm-1.2.2-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.2-rc2/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1062
>
> Please vote on releasing this package as Apache Storm 1.2.2.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for 72 hours or until at least 3 PMC members vote
> +1.
>
> [ ] +1 Release this package as Apache Storm 1.2.2
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
>
>
>

New Committer/PMC Member: Ethan Li

[Bobby Evans]

Please Join with me in welcoming Ethan Li as the newest Apache Storm
committer and PMC member.

Great work!

Re: New Committer/PMC Member: Erik Weathers

[Bobby Evans]

Sorry for the late reply.  Congrats and thanks for all of your great work
for the community Erik.

Thanks again,

Bobby

On Sun, Feb 25, 2018 at 10:45 PM Aaron Niskodé-Dossett 
wrote:

> Congrats, Erik, well deserved!
> On Sun, Feb 25, 2018 at 10:27 PM Satish Duggana 
> wrote:
>
> > Congrats and Welcome Erik!
> >
> >
> >
> > On Sat, Feb 24, 2018 at 9:25 PM, Harsha  wrote:
> >
> > > Congrats Erik.
> > > -Harsha
> > >
> > > On Fri, Feb 23, 2018, at 11:15 AM, Karthick Duraisamy Soundararaj
> wrote:
> > > > Whoa! Congratulations, Erik. Well deserved!
> > > >
> > > > On Fri, Feb 23, 2018 at 10:37 AM, Erik Weathers <
> > > > eweath...@groupon.com.invalid> wrote:
> > > >
> > > > > Thanks everyone!  And yes Alexandre, the relation of my surname to
> > the
> > > > > project name was not lost on me. ;-)(Also, my Grandpa went by
> > > "Stormy"
> > > > > too!)
> > > > >
> > > > > Also, I must thank my work teammates Srishty Agrawal and Jessica
> > > Hartog who
> > > > > have contributed greatly to any of the work that I've pushed.
> > > > >
> > > > > - Erik
> > > > >
> > > > > On Fri, Feb 23, 2018 at 8:28 AM, Ethan Li <
> ethanopensou...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Congratulations! Erik
> > > > > >
> > > > > > - Ethan
> > > > > >
> > > > > > > On Feb 22, 2018, at 7:42 PM, Xin Wang 
> > > wrote:
> > > > > > >
> > > > > > > Congrats!
> > > > > > >
> > > > > > > 2018-02-23 9:41 GMT+08:00 Hugo Da Cruz Louro <
> > > hlo...@hortonworks.com>:
> > > > > > >
> > > > > > >> Congrats & Welcome!
> > > > > > >>
> > > > > > >>> On Feb 22, 2018, at 2:45 PM, Jungtaek Lim  >
> > > wrote:
> > > > > > >>>
> > > > > > >>> Welcome Erik! Congrats!
> > > > > > >>>
> > > > > > >>> -Jungtaek Lim (HeartSaVioR)
> > > > > > >>>
> > > > > > >>> 2018년 2월 23일 (금) 오전 7:05, Stig Rohde Døssing <
> > > stigdoess...@gmail.com
> > > > > > >님이
> > > > > > >> 작성:
> > > > > > >>>
> > > > > >  Congratulations Erik. Happy to see you join.
> > > > > > 
> > > > > >  2018-02-22 20:53 GMT+01:00 Alexandre Vermeerbergen <
> > > > > >  avermeerber...@gmail.com
> > > > > > > :
> > > > > > 
> > > > > > > Hi,
> > > > > > >
> > > > > > > Welcome to Erik...
> > > > > > > ... a Stormy Weather(s) sounds like a fantastic match
> indeed!
> > > > > > >
> > > > > > > Alexandre Vermeerbergen (Storm addict)
> > > > > > >
> > > > > > > 2018-02-22 20:49 GMT+01:00 P. Taylor Goetz <
> > ptgo...@apache.org
> > > >:
> > > > > > >> The Apache Storm PMC has voted to add Erik Weathers as a
> > > Committer
> > > > > > and
> > > > > > > PMC Member.
> > > > > > >>
> > > > > > >> Please join me in congratulating Erik on his new role!
> > > > > > >>
> > > > > > >> -Taylor
> > > > > > >
> > > > > > 
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Thanks,
> > > > > > > Xin
> > > > > >
> > > > > >
> > > > >
> > >
> >
>

Re: [DISCUSS] Handling dependencies for Storm 2.0.0 (STORM-2882)

[Bobby Evans]

Sorry for the late reply.

I am 100% in favor of shading again for very common dependencies.  The big
issue is around how do we do this cleanly?  I like the way flink does it.
That can help with some of the issues we see with IDEs, but we just need to
be very careful with transitive dependencies when we do this.  I am not
aware of any issues like this, but if we have something like storm-client
depends on A and B, and B also depends on A.  If we shade A by itself, the
original A would still be needed on the classpath for B to work properly,
or we would have to do some kind of shading for B too, which if we don't do
it carefully might result in two separate copies of a shaded A on the
classpath.

Thanks,

Bobby

On Wed, Feb 21, 2018 at 1:20 AM Jungtaek Lim  wrote:

> Hi devs,
>
> I'm initiating this one for part of Storm 2.0.0. Actually we had some
> discussion from earlier thread [1] but it was originated for IDE
> complaining and we completely remove shading/relocating in Storm 2.0.0 at
> that time, ends up with exposing all the transitive dependencies in
> 'storm-client' to user topology code. We've reduced lots of dependencies
> while breaking down storm-core, but troublesome dependencies (for example,
> Guava, Jackson, Netty, and so on) are still left.
>
> There're two questions we should address:
>
> 1. Do we want to (and need to) shade/relocate the dependencies?
> (We may also want to see it as regression issue, since end users should
> tackle with dependencies if we don't shade/relocate.)
>
> 2. If we want to relocate the dependencies, how?
>
> If we would want to relocate them, there's a good reference on Flink: Flink
> creates separate repository [2] which contains pom for relocation for each
> target artifact. Links for related discussion [3] and issue [4] are
> available, so if you are interested on Flink's approach, please refer the
> links. I guess it may require considerable efforts, so if someone finds
> simpler and easier way that should be great.
>
> Please share your voices regarding two questions, and idea/knowledge of
> how. Thanks in advance.
>
> Jungtaek Lim (HeartSaVioR)
>
> 1.
>
> https://mail-archives.apache.org/mod_mbox/incubator-storm-dev/201703.mbox/%3ccaf5108gjjqzsyywcp99bgpgec7tufe-tbt9fi0m78ah8rkm...@mail.gmail.com%3E
> 2. https://github.com/apache/flink-shaded
> 3.
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Changing-Flink-s-shading-model-td17419.html
> 4. https://issues.apache.org/jira/browse/FLINK-6529
>

Re: Storm autoscaling

[Bobby Evans]

Yes that is something that we are working on, especially around the
resource aware scheduler.  We have added in a new metrics path to have CPU
and Memory usage reported to nimbus.  We plan to extend this to other
metrics in the future and then start working on how to properly do
elasticity.

Thanks,

Bobby

On Sun, Feb 18, 2018 at 11:52 PM Ali Nazemian  wrote:

> Hi All,
>
> Is there any plan to add autoscaling to Apache Storm? We are using Storm in
> production heavily. However, one of the biggest challenges we have faced is
> the lack of auto-scaling in Storm. Most of our Storm clusters are very
> under-utilised because Storm relies on saving a slot for topologies
> regardless of what the current load is. The Resource-Aware scheduler
> may help us a little, but it doesn't scale well from the operational
> perspective if you have tons of topologies to tune, so it is not what
> exactly we need. What would be really nice to have is an auto-scaling
> feature similar to Storm-594. Is there any plan to introduce this
> functionality in future?
>
> Regards,
> Ali
>

Re: [VOTE] Release Apache Storm 1.2.0 (rc4)

[Bobby Evans]

+1

I built from source all of the unit tests passed and some basic integration
tests passed too.

On Sat, Feb 10, 2018 at 12:00 PM Alexandre Vermeerbergen <
avermeerber...@gmail.com> wrote:

> Hello,
>
> +1 (non binding)
>
> Details: I almost got nuts when applying storm-kafka-client-1.2.0 RC4
> to our preproduction Storm cluster whose performances dropped quite a
> lot (at least x2 less throughput) ... but then I realized that I made
> the classic mistake of mixing two changes  : I had also upgraded our
> Kafka dependencies from Kafka Client 0.10.20.0 to Kafka  Client
> 1.0.0.0.
>
> Then I rollbacked our Kafka dependencies to Kafka 0.10.2.0 but I kept
> Storm 1.2.0 RC4 : since then (~10 hours ago) it's working smoothly.
>
> In particular, performances of Storm Kafka Client 1.2.0 RC4 having
> autocommit=true setting are equivalent to the ones of Storm Kafka
> Client 1.2.0 with
> .setProcessingGuarantee(ProcessingGuarantee.NO_GUARANTEE).
>
> Note : our preproduction Storm cluster consumes & produces data in a
> Kafka Cluster based on Kafka 1.0.0. I guess there's some details we
> missed when using Kafka Client 1.0.0 libraries instead of Kafka Client
> 0.10.2.0, but I think this is something we can study outside this
> 1.2.0 voting process, because consuming & producing Kafka messages
> against a Kafka cluster based on Kafka 1.0.0 Brokers is considered as
> compatible by Kakfa documentation.
>
> Since we want to move to Kafka Client 1.0.0 at some time (in
> particular, to take advantage of official JRE 9 support of this Kafka
> version), we will keep on testing our system with this newer Kafka
> version and we will feeback to Storm dev list whatever the outcome is
> (I'm guessing some property changes from 0.10.2 to 1.0.0)
>
> Thank you very much to all Storm developers for bringing this great
> Storm 1.2.0 to us, and for your taking into account my feedbacks: I
> have no more bad news for 1.2.0 :)
>
>
> Best regards,
> Alexandre Vermeerbergen
>
> PS: just one question: is there a way to access 1.2.0 RC4 Storm
> documentation before it's switched on public Storm site?
> PPS: I saw Stig's feedback about our deprecated usage of
> auto.commit.internal.ms, but I haven't yet tried the "new way",
> meaning that it has no impact on our big scale test. But I will try it
> later.
>
> 2018-02-09 9:18 GMT+01:00 Alexandre Vermeerbergen <
> avermeerber...@gmail.com>:
> > Hello,
> >
> > For your information, yesterday I upgraded our preproduction cluster
> > from Storm 1.2.0 SNAPSHOT of December 2017 to 1.2.0 Release Candidate
> > 4.
> > Our alerting system based on Storm behaved quite bad with 1.2.0 RC4,
> > and this morning, one of our team member noticed this message in our
> > topologie's startup logs:
> >
> > 1279 [main] WARN  o.a.s.k.s.KafkaSpoutConfig - The KafkaConsumer
> > enable.auto.commit setting is not supported. You can configure similar
> > behavior through KafkaSpoutConfig.Builder.setProcessingGuarantee.This
> > will be treated as an error in the next major release.
> >
> > 1279 [main] INFO  o.a.s.k.s.KafkaSpoutConfig - Setting Kafka consumer
> > property 'enable.auto.commit' to 'false', because the spout does not
> > support auto-commit
> >
> >
> > So I will resume our tests with this new setting and I will feedback
> > once I have enought uptime on our "large preproduction cluster" with
> > 1.2.0 RC4.
> >
> > Note: may I suggest to make this breaking change visible in Storm
> > 1.2.0 releases notes? this is quite impacting. Or event better: make
> > the topologies unable to start when they use such a removed property,
> > so that at least people aren't fooled until they wonder why their
> > Kafka spouts aren't anymore behaving like before and check logs?
> >
> > More to come when my preproduction tests will have been completed (1or
> > 2 days needed).
> >
> > Best regards,
> > Alexandre
> >
> > 2018-02-07 23:24 GMT+01:00 P. Taylor Goetz :
> >> This is a call to vote on releasing Apache Storm 1.2.0 (rc4)
> >>
> >> Note that the only difference between rc4 and rc3 is the fix for
> >> https://issues.apache.org/jira/browse/STORM-2942.
> >>
> >> Full list of changes in this release:
> >>
> >>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc4/RELEASE_NOTES.html
> >>
> >> The tag/commit to be voted upon is v1.2.0:
> >>
> >>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=cef4d49e222e53656f38c40d754d4f41799cd9a9;hb=2a0097f9a20b9df494caadb87c87d4e4db01a7ed
> >>
> >> The source archive being voted upon can be found here:
> >>
> >>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc4/apache-storm-1.2.0-src.tar.gz
> >>
> >> Other release files, signatures and digests can be found here:
> >>
> >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc4/
> >>
> >> The release artifacts are signed with the following key:
> >>
> >>
> 

Re: [VOTE] Release Apache Storm 1.0.6 (rc3)

[Bobby Evans]

Happy to do the check.  This is something we don't want to get wrong.

On Fri, Feb 9, 2018 at 5:38 PM P. Taylor Goetz <ptgo...@gmail.com> wrote:

> Doh! I didn’t even think consider that it was likely autogenerated by an
> IDE and likely identical to what the JDK would generate.
>
> I agree with Bobby, let’s leave it there and continue with the RC.
>
> Thanks for pointing that out Bobby.
>
> -Taylor
>
> > On Feb 9, 2018, at 4:07 PM, Bobby Evans <ev...@oath.com.INVALID> wrote:
> >
> > Actually it is set properly.  Do NOT change it.  It works just fine the
> way
> > it is.
> >
> > $ git checkout f6f35dd98d2492a38aa4d61da7f6caee4ec2f31a # The git version
> > right before this change on branch-1.x
> > $ mvn clean install -DskipTests
> > $ serialver -classpath ./storm-core/target/storm-core-1.1.0-SNAPSHOT.jar
> > org.apache.storm.tuple.Fields
> > org.apache.storm.tuple.Fields:private static final long
> > serialVersionUID = -3377931843059975424L;
> >
> > This is the same version that we set it to as a part of the change.
> >
> > - Bobby
> >
> > On Fri, Feb 9, 2018 at 3:00 PM Bobby Evans <ev...@oath.com> wrote:
> >
> >> Fields changing will cause problems if it is serialized as part of a
> >> bolt/spout or as part of a custom grouping.  I have not checked
> explicitly,
> >> but removing that line is the wrong thing to do.  By default the
> serialVersionUID
> >> is generated from the class itself, so removing it, but leaving in the
> >> modified code would still break backwards compatibility.
> >>
> >> I'll take a look and see what you need to set it to so you don't break
> >> backwards compatibility on 1.x
> >>
> >> On Fri, Feb 9, 2018 at 10:19 AM P. Taylor Goetz <ptgo...@gmail.com>
> wrote:
> >>
> >>> What are others’ opinions on removing the serialversionUid an moving
> >>> ahead with an RC4?
> >>>
> >>> -Taylor
> >>>
> >>>> On Feb 9, 2018, at 7:21 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
> >>>>
> >>>> I just went ahead verifying current RC except serialization UID issue
> in
> >>>> Fields. I could also vote for RC4 immediately if necessary.
> >>>>
> >>>> +1 (binding)
> >>>>
> >>>>> source
> >>>>
> >>>> - verify file (signature, MD5, SHA)
> >>>> -- source, tar.gz : OK
> >>>> -- source, zip : OK
> >>>>
> >>>> - extract file
> >>>> -- source, tar.gz : OK
> >>>> -- source, zip : OK
> >>>>
> >>>> - diff-ing extracted files between tar.gz and zip : OK
> >>>>
> >>>> - build source with JDK 7
> >>>> -- source, tar.gz : OK
> >>>>
> >>>> - build source dist
> >>>> -- source, tar.gz : OK
> >>>>
> >>>> - build binary dist
> >>>> -- source, tar.gz : OK
> >>>>
> >>>>> binary
> >>>>
> >>>> - verify file (signature, MD5, SHA)
> >>>> -- binary, tar.gz : OK
> >>>> -- binary, zip : OK
> >>>>
> >>>> - extract file
> >>>> -- binary, tar.gz : OK
> >>>> -- binary, zip : OK
> >>>>
> >>>> - diff-ing extracted files between tar.gz and zip : OK
> >>>>
> >>>> - launch daemons : OK
> >>>>
> >>>> - run RollingTopWords (local) : OK
> >>>>
> >>>> - run RollingTopWords (remote) : OK
> >>>> - activate / deactivate / rebalance / kill : OK
> >>>> - logviewer (worker dir, daemon dir) : OK
> >>>> - change log level : OK
> >>>> - thread dump, heap dump, restart worker : OK
> >>>> - log search : OK
> >>>>
> >>>> Thanks,
> >>>> Jungtaek Lim (HeartSaVioR)
> >>>>
> >>>> 2018년 2월 9일 (금) 오후 6:18, Erik Weathers <eweath...@groupon.com.invalid
> >님이
> >>> 작성:
> >>>>
> >>>>> I'm fine submitting a PR to back that line out (or any of you
> committer
> >>>>> folks could just rip it out).
> >>>>>
> >>>>> But I'd like to understand Storm a bit better as part of making this
> >>>>> decision. :-)  Am I correct in assuming it would only be a problem if
> &

Re: [VOTE] Release Apache Storm 1.0.6 (rc3)

[Bobby Evans]

Actually it is set properly.  Do NOT change it.  It works just fine the way
it is.

$ git checkout f6f35dd98d2492a38aa4d61da7f6caee4ec2f31a # The git version
right before this change on branch-1.x
$ mvn clean install -DskipTests
$ serialver -classpath ./storm-core/target/storm-core-1.1.0-SNAPSHOT.jar
org.apache.storm.tuple.Fields
org.apache.storm.tuple.Fields:private static final long
serialVersionUID = -3377931843059975424L;

This is the same version that we set it to as a part of the change.

- Bobby

On Fri, Feb 9, 2018 at 3:00 PM Bobby Evans <ev...@oath.com> wrote:

> Fields changing will cause problems if it is serialized as part of a
> bolt/spout or as part of a custom grouping.  I have not checked explicitly,
> but removing that line is the wrong thing to do.  By default the 
> serialVersionUID
> is generated from the class itself, so removing it, but leaving in the
> modified code would still break backwards compatibility.
>
> I'll take a look and see what you need to set it to so you don't break
> backwards compatibility on 1.x
>
> On Fri, Feb 9, 2018 at 10:19 AM P. Taylor Goetz <ptgo...@gmail.com> wrote:
>
>> What are others’ opinions on removing the serialversionUid an moving
>> ahead with an RC4?
>>
>> -Taylor
>>
>> > On Feb 9, 2018, at 7:21 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
>> >
>> > I just went ahead verifying current RC except serialization UID issue in
>> > Fields. I could also vote for RC4 immediately if necessary.
>> >
>> > +1 (binding)
>> >
>> >> source
>> >
>> > - verify file (signature, MD5, SHA)
>> > -- source, tar.gz : OK
>> > -- source, zip : OK
>> >
>> > - extract file
>> > -- source, tar.gz : OK
>> > -- source, zip : OK
>> >
>> > - diff-ing extracted files between tar.gz and zip : OK
>> >
>> > - build source with JDK 7
>> > -- source, tar.gz : OK
>> >
>> > - build source dist
>> > -- source, tar.gz : OK
>> >
>> > - build binary dist
>> > -- source, tar.gz : OK
>> >
>> >> binary
>> >
>> > - verify file (signature, MD5, SHA)
>> > -- binary, tar.gz : OK
>> > -- binary, zip : OK
>> >
>> > - extract file
>> > -- binary, tar.gz : OK
>> > -- binary, zip : OK
>> >
>> > - diff-ing extracted files between tar.gz and zip : OK
>> >
>> > - launch daemons : OK
>> >
>> > - run RollingTopWords (local) : OK
>> >
>> > - run RollingTopWords (remote) : OK
>> >  - activate / deactivate / rebalance / kill : OK
>> >  - logviewer (worker dir, daemon dir) : OK
>> >  - change log level : OK
>> >  - thread dump, heap dump, restart worker : OK
>> >  - log search : OK
>> >
>> > Thanks,
>> > Jungtaek Lim (HeartSaVioR)
>> >
>> > 2018년 2월 9일 (금) 오후 6:18, Erik Weathers <eweath...@groupon.com.invalid>님이
>> 작성:
>> >
>> >> I'm fine submitting a PR to back that line out (or any of you committer
>> >> folks could just rip it out).
>> >>
>> >> But I'd like to understand Storm a bit better as part of making this
>> >> decision. :-)  Am I correct in assuming it would only be a problem if
>> the
>> >> serialized Fields were stored somewhere (e.g., ZooKeeper, local
>> filesystem)
>> >> and then read back in after the Nimbus/Workers are brought back up
>> after
>> >> the upgrade?  Seems Fields is used in a *lot* of places, and I don't
>> know
>> >> precisely what is serialized for reused upon Storm Nimbus/Worker daemon
>> >> restarts.  I believe there are examples of Fields being used to create
>> >> Spout or Bolt objects that are used to create the StormTopology object,
>> >> which I believe is serialized into ZooKeeper.  But I'm not clear if
>> it's
>> >> directly the Fields object itself or some kind of translation from that
>> >> into the thrift objects that make up StormTopology.
>> >>
>> >> I also don't know exactly when kryo is applicable in Storm.  I've never
>> >> done anything with kryo directly.
>> >>
>> >> - Erik
>> >>
>> >> On Thu, Feb 8, 2018 at 10:00 PM, P. Taylor Goetz <ptgo...@gmail.com>
>> >> wrote:
>> >>
>> >>> *serialized* ;)
>> >>>
>> >>>> On Feb 9, 2018, at 12:48 AM, P. Taylor Goetz <ptgo...@gmail.com>
>> >> w

Re: [VOTE] Release Apache Storm 1.0.6 (rc3)

[Bobby Evans]

+1 I built the code from the git tag, ran all the unit tests (which passed
the first time), and ran some tests on a single node cluster.

It all looked good.

- Bobby

On Thu, Feb 8, 2018 at 1:22 PM P. Taylor Goetz  wrote:

> This is a call to vote on releasing Apache Storm 1.0.6 (rc3)
>
> Full list of changes in this release:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.6-rc3/RELEASE_NOTES.html
>
> The tag/commit to be voted upon is v1.0.6:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=e68365f9f947ddd1794b2edef2149fdfaa1590a2;hb=7993db01580ce62d44866dc00e0a7266984638d0
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.6-rc3/apache-storm-1.0.6-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.6-rc3/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1060
>
> Please vote on releasing this package as Apache Storm 1.0.6.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 1.0.6
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
>

Re: [VOTE] Release Apache Storm 1.1.2 (rc3)

[Bobby Evans]

+1 I built from the tag and did some basic tests on a single node cluster.

I am disappointed that it took me 7 times to get all of the unit tests to
pass for the build, but I don't think it should block a release.

- Bobby

On Thu, Feb 8, 2018 at 11:48 AM P. Taylor Goetz  wrote:

> This is a call to vote on releasing Apache Storm 1.1.2 (rc3)
>
> Full list of changes in this release:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.2-rc3/RELEASE_NOTES.html
>
> The tag/commit to be voted upon is v1.1.2:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=3aab00d053d3576d3f8a37bcc3c7e51072ead22e;hb=0bb7a66a9b26ad96afc81b27dd45f93ae8969c44
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.2-rc3/apache-storm-1.1.2-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.2-rc3/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1059
>
> Please vote on releasing this package as Apache Storm 1.1.2.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 1.1.2
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
>

Re: [CANCELED] [VOTE] Release Apache Storm 1.0.6 (rc1)

[Bobby Evans]

I agree about decoupling storm-kafka from storm core for releases.

We put them together initially because we wanted storm to come with
batteries included.  We have made some major changes to storm over the
years and it has been good to have kafka integration, along with a lot of
other integration, ready from day one each time we make a change.

But it is obvious that kafka is changing very quickly too and trying to
keep something in sync with both kafka and storm is likely going to require
a faster release cadence.  Plus it looks like there is enough of a
community around storm kafka integration that it can survive without the
storm project attached directly to it.

To make this happen though someone is going to need to step up and take
lead on this.

We will need a plan on what to do (is it becoming a separate repo with a
separate release cycle but still a part of the storm project?)
Do we plan to spin it off to be an incubator project on its own?

Once we decide that we can then have a vote and get started with it.

On Thu, Jan 25, 2018 at 11:19 AM P. Taylor Goetz  wrote:

> Canceling in order to include a fix for
> https://issues.apache.org/jira/browse/STORM-2912.
>
> -Taylor
>
> On Jan 24, 2018, at 1:41 PM, P. Taylor Goetz  wrote:
>
> This is a call to vote on releasing Apache Storm 1.0.6 (rc1)
>
> Full list of changes in this release:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.6-rc1/RELEASE_NOTES.html
>
> The tag/commit to be voted upon is v1.0.6:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=24a421e34a71353dc6c750b1f026d06df8ead3f2;hb=bce45993f8622e4d3e9ccba96cc78e4ef76e48ae
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.6-rc1/apache-storm-1.0.6-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.6-rc1/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1054
>
> Please vote on releasing this package as Apache Storm 1.0.6.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 1.0.6
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
>
>
>

Re: Back-pressure Mechamism

[Bobby Evans]

Algoby,

When you say transfer queue, which queue do you mean exactly.  In storm
there are a lot of queues currently and they sometimes have confusing names.

There is the receive queue, which holds tuples to be processed by a
specific executor.  Then there is the send queue, or some times called the
batch transfer queue.  All of the emit calls from an executor go into this
and then a second thread handles batching the massages and routing them to
where they need to go.  The there is the transfer queue.  This queue gets
all of the tuples that need to be sent outside this worker.

We have looked at supporting all of these different queues for back
pressure.  The receive queue is the big one as it is where most of the user
code likely executes.  The send queue tends to back up if the time taken to
serialize an object is more then the processing needed to produce the
object.  This is not that common, but I have seen it where a single large
message gets split up into many messages, each that may be kind of
difficult to serialize.  I thought we had a patch to include the send queue
as part of back pressure, but I don't know what happened to it.

The transfer queue is much less likely to back up, but the consequences are
much worse when it does backup.  The thread that reads from the transfer
queue only routes messages to clients that are buffering the messages and
sending them to other workers.  There is not much work happening here.  The
clients themselves don't have any back pressure built in either. So if the
transfer queue is backing up then your worker likely is writing messages
into memory as fast as it can, and you are going to get an OOM some time
soon.  To really make this work you would need some kind of back pressure
in the netty clients that could also be involved with this.

A patch that we will likely merge into 2.x shortly
https://github.com/apache/storm/pull/2241 has done all of this and also
redesigned back pressure to not go off of high/low water marks with signals
through zookeeper, but instead to push back to the upstream component when
a queue is full.  The only downside there is that we will only be able to
support DAGs for processing.  No loops in user code, or you could deadlock.
Until we get 2.x out the door and stable (which I really want to do in Q1
2018) you are probably going to have to live with some of these issues.

- Bobby

On Thu, Dec 21, 2017 at 9:50 AM Waleed Al-Gobi 
wrote:

> Dear All,
>
> My concern is about on which queue Storm relies to for back-pressure.
>
> I did simple test for back-pressure supported by Storm.
> Each instance (executor) maintains incoming(receive) Q and
> outgoing(transfer) Q, and according to min and max threshold on these
> queues, a back-pressure works to slow down the spout in case of queue
> buildup.
>
> The purpose I wanted to make sure in case of link bottleneck whether
> back-pressure still helps or not.
> The conclusion, it helps only in case of queue buildup due to CPU
> bottleneck. I guess the reason for which why it could not make it for link
> bottleneck, because back-pessure relies only on the executor receive Q.
>
> Does this make sense? If so, could we anyway make the back-pressure also
> working if ececutor transfer Q is full in case of link bottleneck?
>
> Thanks!
>
> Best,
> Algoby
>

Re: Question: ways to handle inactive pull requests

[Bobby Evans]

Yes I agree it should be on a case by case basis, no need for anything
formal, it is just being a nice person and explaining to everyone why you
are doing something.

- Bobby

On Fri, Sep 15, 2017 at 4:19 AM Jungtaek Lim <kabh...@gmail.com> wrote:

> Thanks for all your voices.
>
> Regarding question, I'm not sure we can define the time and frequency for
> that. I don't mean to build a new item for bylaw. That actually depends on
> individual, and how much the issue is urgent. For example, we can't just
> wait author for inactive pull request addressing critical or even blocker
> issue, including 'effectively' blocker for release preparing phase.
>
> Personally no updates (not code update but also no comment) in two months
> feels me as inactive and high chance to become stale. Someone may feel it
> too short, so maybe leaving comment before taking over and wait some more
> for others' feedback might be better.
>
> I feel similar for no code update but just reacting as leaving a comment
> more than three months. Three months are enough long to forget about their
> context of PR.
>
> Please keep in mind that above things assume that we are reviewing pull
> requests not too late. Inactivity due to late reviewing is out of topic.
>
> Similar consideration applies to JIRA issues which is in-progress but no
> patch is available.
>
> Thanks,
> Jungtaek Lim (HeartSaVIoR)
>
> 2017년 9월 15일 (금) 오전 1:40, Hugo Da Cruz Louro <hlo...@hortonworks.com>님이
> 작성:
>
> > Agree. Question is what is the reasonable time frame for a response, and
> > how many times one should reach out to a person asking for a response.
> >
> > > On Sep 14, 2017, at 8:43 AM, Stig Rohde Døssing <
> stigdoess...@gmail.com>
> > wrote:
> > >
> > > I agree, if the original author is not responding it seems totally fine
> > to
> > > me for someone else to finish up a PR. If the new PR is based on the
> > > previous effort, I think we should be careful to always preserve
> > authorship
> > > information. The easiest way is probably to keep the original commits.
> > > Ideally inactive PRs that we want to keep are rare enough that we can
> > live
> > > with keeping the original commits without making the commit log too
> > noisy.
> > >
> > > Amateur license parsing, so buyer beware:
> > > The way I understand "You represent that each of Your Contributions is
> > Your
> > > original creation" from the ICLA (
> > https://www.apache.org/licenses/icla.pdf)
> > > is that it's probably not okay to take someone else's commits along
> with
> > > your own, squash them and submit the whole thing under your own name.
> > Point
> > > 7 mentions how to submit on behalf of others.
> > >
> > > The first comment here may also be relevant regarding license for an
> > > unmerged PR https://issues.apache.org/jira/browse/LEGAL-156
> > >
> > > 2017-09-14 16:03 GMT+02:00 Bobby Evans <ev...@oath.com.invalid>:
> > >
> > >> I totally agree.  If you have reached out to an author and there has
> > been
> > >> no response for either a bug fix or a feature that you want, then feel
> > free
> > >> to take it over.  Just be polite about it and make sure it is clear to
> > >> everyone what you are doing.
> > >>
> > >> -
> > >> Bobby
> > >>
> > >> On Wed, Sep 13, 2017 at 11:19 PM Jungtaek Lim <kabh...@gmail.com>
> > wrote:
> > >>
> > >>> Hi devs,
> > >>>
> > >>> I have seen some old pull requests for bugfix and new feature going
> to
> > be
> > >>> stale. Some of us tried to ping to author several times but not
> respond
> > >> in
> > >>> some months. For new feature we may have to wait for authors, but for
> > >>> bugfix waiting authors means we are aware of the bug but we don't fix
> > the
> > >>> bug because of credit which doesn't make sense to me if we should
> wait
> > >> for
> > >>> months.
> > >>>
> > >>> So IMHO at least we may want to handle inactive bugfix pull requests
> > not
> > >>> too late, Maybe creating new PR addressing same thing without
> retaining
> > >>> commits, or taking over PR via retaining commits. If possible it may
> be
> > >>> ideal to take over inactive but valuable pull requests with retaining
> > >>> commits.
> > >>>
> > >>> What do you think about it? And does some of us know about any issues
> > >>> including license, authorship, or so if someone takes over inactive
> > pull
> > >>> request with retaining their credit (commits)?
> > >>>
> > >>> Thanks,
> > >>> Jungtaek Lim (HeartSaVioR)
> > >>>
> > >>
> >
> >
>

Re: Question: ways to handle inactive pull requests

[Bobby Evans]

I totally agree.  If you have reached out to an author and there has been
no response for either a bug fix or a feature that you want, then feel free
to take it over.  Just be polite about it and make sure it is clear to
everyone what you are doing.

-
Bobby

On Wed, Sep 13, 2017 at 11:19 PM Jungtaek Lim  wrote:

> Hi devs,
>
> I have seen some old pull requests for bugfix and new feature going to be
> stale. Some of us tried to ping to author several times but not respond in
> some months. For new feature we may have to wait for authors, but for
> bugfix waiting authors means we are aware of the bug but we don't fix the
> bug because of credit which doesn't make sense to me if we should wait for
> months.
>
> So IMHO at least we may want to handle inactive bugfix pull requests not
> too late, Maybe creating new PR addressing same thing without retaining
> commits, or taking over PR via retaining commits. If possible it may be
> ideal to take over inactive but valuable pull requests with retaining
> commits.
>
> What do you think about it? And does some of us know about any issues
> including license, authorship, or so if someone takes over inactive pull
> request with retaining their credit (commits)?
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>

Re: Bare minimum requirements to run a Storm Worker

[Bobby Evans]

Sadly storm is a very talky so reducing the amount of network traffic is
going to be difficult.  I don't think anyone has really tried this before
so you are going to be in new territory.

There are a few things you might want to look at.
1) all of the different daemons in storm tend to poll periodically for
updates.  Almost all of the polling intervals are configurable so you would
want to drop them down.  They probably are not making that much network
traffic, but it could be an easy win.
2) acking does use up a lot of messages.  They are not large, but it is
still a lot of messages.  If you can live with acking off it should reduce
your network load a lot.
3) Try to use the localOrShuffle grouping where ever possible, or even
better just combine your bolts together into a bigger bolts and avoid the
transfer all together.  LocalOrShuffle is a kind a brittle feature when
scheduling does not go perfectly your way and we are working to make it
much better through  https://issues.apache.org/jira/browse/STORM-2684 but
that code is not done yet.

Beyond those I'm not sure what else you would want to do.  You would need
to do some profiling to see what is taking up the memory, network, and CPU.

- Bobby


On Wed, Sep 13, 2017 at 10:36 PM Gayashan Amarasinghe <
gayashan.amarasin...@gmail.com> wrote:

> Hi Bobby,
>
> Thank you very much for the details. I'm not fixed on a specific version
> yet, would prefer the latest. Is it possible to run with minimum network
> usage, apart from receiving and delivering tuples, such as shutting down
> ackers, running with no supervision, pros and cons of that, etc. Do you
> have any study or details on that or can you point me to where I should
> look at to get some idea?
>
> Thank you.
>
> Best regards,
> /Gayashan
>
> On Tue, Sep 12, 2017 at 12:26 AM, Bobby Evans <ev...@oath.com.invalid>
> wrote:
>
> > A lot of that depends on the version of storm you have, and it is not a
> use
> > case that we have played around with too much.
> >
> > Just looking at a few of our production clusters the CPU usage is very
> > small (but trying to determine what the CPU usage on a raspberry pi arm
> > chip when all you have is the usage on a server CPU is hard to do).  The
> > memory usage, however can be somewhat high, especially for a raspberry
> pi.
> > We are setting the heap size for the supervisor on our clusters to 256MB,
> > but the heap size on the logviewer is 768MB.  But we have not tuned these
> > at all, so it could easily be a lot smaller and still work.
> >
> > - Bobby
> >
> >
> >
> > On Sun, Sep 10, 2017 at 4:36 AM Gayashan Amarasinghe <
> > gayashan.amarasin...@gmail.com> wrote:
> >
> > > Hi devs,
> > >
> > > What are the bare minimum requirements to run a Storm worker on a low
> > > resource environment such as a raspberry pi node where bandwidth
> > > consumption and power consumption could be critical?
> > >
> > > Thank you.
> > >
> > > Best regards,
> > > /Gayashan
> > >
> >
>

Re: Bare minimum requirements to run a Storm Worker

[Bobby Evans]

A lot of that depends on the version of storm you have, and it is not a use
case that we have played around with too much.

Just looking at a few of our production clusters the CPU usage is very
small (but trying to determine what the CPU usage on a raspberry pi arm
chip when all you have is the usage on a server CPU is hard to do).  The
memory usage, however can be somewhat high, especially for a raspberry pi.
We are setting the heap size for the supervisor on our clusters to 256MB,
but the heap size on the logviewer is 768MB.  But we have not tuned these
at all, so it could easily be a lot smaller and still work.

- Bobby



On Sun, Sep 10, 2017 at 4:36 AM Gayashan Amarasinghe <
gayashan.amarasin...@gmail.com> wrote:

> Hi devs,
>
> What are the bare minimum requirements to run a Storm worker on a low
> resource environment such as a raspberry pi node where bandwidth
> consumption and power consumption could be critical?
>
> Thank you.
>
> Best regards,
> /Gayashan
>

Re: Running topologies in LocalCluster mode leaves junk in ${user.dir}/logs

[Bobby Evans]

Under 2.x LocalCluster is much more fully featured and you can set daemon
configs using the LocalCluster.Builder.withDaemonConf.  But for all other
versions of storm the only way to change things like the daemon conf are
through the clojure API, which I assume you don't want to use.

So sorry but no there is no better way right now unless you want to move to
an unreleased version of Storm.

If you do want to clean it up and make support for something like this more
official I would love to see it happen, but I don't really have a lot of
time to make the changes myself right now.

-
Bobby


On Wed, Sep 6, 2017 at 3:05 PM Alexandre Vermeerbergen <
avermeerber...@gmail.com> wrote:

> Hello,
>
> I have noticed that when I run topologies in LocalCluster mode (for
> automated test purposes), there are residual files left in a directory
> corresponding to System.getProperty("user.dir")+"/logs".
>
> Inside this directory, a subdirectory called "workers-artifacts" is created
> when the topology is submitted, and it contains again a subdir then some
> files ; nothing is cleaned up when the topology shuts down.
>
> This causes our tests to fail in our automated test replay system, because
> tests are ran with a special user id which has no write permission in
> System.getProperty("user.dir")+"/logs".
>
> I have tried change this behavior by using these setters on the Config
> object passed to LocalCluster.submitTopology:
>
> Config conf = new Config();
> conf.setDebug(true);
>
> conf.put(Config.STORM_LOCAL_DIR, "/tmp/localdir");
> conf.put(Config.STORM_WORKERS_ARTIFACTS_DIR, "/tmp/localdir");
>
> but it changed nothing.
>
> I finally found a by-pass by putting this line at the beginning of our test
> code:
>
> System.setProperty("user.dir", "/tmp/localdir");
>
> But I feel that it's a dirty hack.
>
> What is the official way to control LocalCluster's behavior which regards
> to these directories & files ?
>
> Best regards,
> Alexandre Vermeerbergen
>

Re: [Propose] move website repository from svn to git

[Bobby Evans]

Thanks for clarifying.
We have not run into an issue were we actually have needed to fix the docs for 
an already released version, not saying that it will not happen in the future 
though as the docs are not perfect.  But updating the docs to describe a 
feature that someone is working on is a much more common situation in my 
opinion, so if we do need to update docs for a release that we already did, 
then it probably would involve a lot of applying the same changes to a number 
of different places.


- Bobby


On Monday, August 7, 2017, 9:09:23 AM CDT, Jungtaek Lim <kabh...@gmail.com> 
wrote:

Maybe I was not clear. I meant removing 'releases' directory from
storm-site repo, not storm repo, given that we have duplicated docs between
twos. If we can get rid of that we can reduce the size greatly, but as I
stated from another thread, it may need to decouple main docs and release
docs.

One thing I'm considering is fixing doc for version which is already
released. Assuming we had to fix doc for 1.1.1 (already released), which
doc we need to fix? Only 1.x-branch and 1.1.x-branch in storm repo? Or we
need to store docs in releases directory as it is and fix it as well?

2017년 8월 7일 (월) 오후 10:52, Bobby Evans <ev...@yahoo-inc.com.invalid>님이 작성:

> Please don't remove the .md docs from the storm repo.  Those docs document
> the current build.  Things that are common and almost never change we can
> move to a separate repo.  But things that change from one release to
> another should stay, because they show what the current build is like, and
> having the docs in a separate repo from the code means that the code is
> going to change all the time and the docs are going to get out of date.
>
>
> - Bobby
>
>
> On Monday, August 7, 2017, 6:14:40 AM CDT, Jungtaek Lim <kabh...@gmail.com>
> wrote:
>
> FYI: I've exported SVN repo of website and pushed to 'asf-site' branch of
> GIT repo.
>
> https://github.com/apache/storm-site
>
> Please note that 'content' directory will be used for serving website.
> (INFRA guided me to use this directory)
> Need to update README.md for the new instruction.
>
> Btw, this is still too way heavy, lots of files occupying huge space (2.3g)
> making git too way slow as same as svn. Better to reduce some more
> contents, maybe removing source docs (md files) which are also available in
> storm repo.
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2017년 8월 7일 (월) 오전 11:55, Jungtaek Lim <kabh...@gmail.com>님이 작성:
>
> > FYI: storm-site git repository is created, and according to notification
> > mail, github mirror will be created in a day.
> >
> > I filed another INFRA issues to associate website to the new repository.
> > https://issues.apache.org/jira/servicedesk/agent/INFRA/issue/INFRA-14810
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 8월 2일 (수) 오전 8:09, Jungtaek Lim <kabh...@gmail.com>님이 작성:
> >
> >> FYI: I just take a step to this, but blocked at creating git repository
> >> in reporeq.apache.org.
> >>
> >> Just filed https://issues.apache.org/jira/browse/INFRA-14765. In that
> >> issue I also asked how to serve website with non-main project
> repository.
> >>
> >> 2017년 7월 31일 (월) 오후 10:56, Bobby Evans <ev...@yahoo-inc.com.invalid>님이
> >> 작성:
> >>
> >>> +1
> >>> I am fine with moving to git, but I would like it to be a different
> repo.
> >>> Our current repo is at least 160MB already (which is a lot to download)
> >>> but nothing compared the the web site that has lots and lots of things
> >>> checked in (I estimate it at about 1.5GB on an older version I have
> locally)
> >>>
> >>>
> >>> - Bobby
> >>>
> >>>
> >>> On Monday, July 31, 2017, 1:58:03 AM CDT, Xin Wang <
> >>> data.xinw...@gmail.com> wrote:
> >>>
> >>> +1 for moving to git.  - Xin
> >>>
> >>>
> >>>
> >>> 2017-07-31 14:54 GMT+08:00 Jungtaek Lim <kabh...@gmail.com>:
> >>>
> >>> > Bump. I think this is worth to address soon, since some contributors
> >>> > occasionally submit patches regarding documentations.
> >>> > Personally SVN is no longer feel convenient to use. If we all feel
> the
> >>> > same, let's change then.
> >>> >
> >>> > -Jungtaek Lim (HeartSaVioR)
> >>> >
> >>> > 2017년 7월 13일 (목) 오전 9:16, Jungtaek Lim <kabh...@gmail.com>님이 작성:
> >>> >
> >>> > > Maybe we could try out Gitbox, though every committers should join
> >>> thei

Re: [Propose] move website repository from svn to git

[Bobby Evans]

Please don't remove the .md docs from the storm repo.  Those docs document the 
current build.  Things that are common and almost never change we can move to a 
separate repo.  But things that change from one release to another should stay, 
because they show what the current build is like, and having the docs in a 
separate repo from the code means that the code is going to change all the time 
and the docs are going to get out of date. 


- Bobby


On Monday, August 7, 2017, 6:14:40 AM CDT, Jungtaek Lim <kabh...@gmail.com> 
wrote:

FYI: I've exported SVN repo of website and pushed to 'asf-site' branch of
GIT repo.

https://github.com/apache/storm-site

Please note that 'content' directory will be used for serving website.
(INFRA guided me to use this directory)
Need to update README.md for the new instruction.

Btw, this is still too way heavy, lots of files occupying huge space (2.3g)
making git too way slow as same as svn. Better to reduce some more
contents, maybe removing source docs (md files) which are also available in
storm repo.

- Jungtaek Lim (HeartSaVioR)

2017년 8월 7일 (월) 오전 11:55, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> FYI: storm-site git repository is created, and according to notification
> mail, github mirror will be created in a day.
>
> I filed another INFRA issues to associate website to the new repository.
> https://issues.apache.org/jira/servicedesk/agent/INFRA/issue/INFRA-14810
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2017년 8월 2일 (수) 오전 8:09, Jungtaek Lim <kabh...@gmail.com>님이 작성:
>
>> FYI: I just take a step to this, but blocked at creating git repository
>> in reporeq.apache.org.
>>
>> Just filed https://issues.apache.org/jira/browse/INFRA-14765. In that
>> issue I also asked how to serve website with non-main project repository.
>>
>> 2017년 7월 31일 (월) 오후 10:56, Bobby Evans <ev...@yahoo-inc.com.invalid>님이
>> 작성:
>>
>>> +1
>>> I am fine with moving to git, but I would like it to be a different repo.
>>> Our current repo is at least 160MB already (which is a lot to download)
>>> but nothing compared the the web site that has lots and lots of things
>>> checked in (I estimate it at about 1.5GB on an older version I have locally)
>>>
>>>
>>> - Bobby
>>>
>>>
>>> On Monday, July 31, 2017, 1:58:03 AM CDT, Xin Wang <
>>> data.xinw...@gmail.com> wrote:
>>>
>>> +1 for moving to git.  - Xin
>>>
>>>
>>>
>>> 2017-07-31 14:54 GMT+08:00 Jungtaek Lim <kabh...@gmail.com>:
>>>
>>> > Bump. I think this is worth to address soon, since some contributors
>>> > occasionally submit patches regarding documentations.
>>> > Personally SVN is no longer feel convenient to use. If we all feel the
>>> > same, let's change then.
>>> >
>>> > -Jungtaek Lim (HeartSaVioR)
>>> >
>>> > 2017년 7월 13일 (목) 오전 9:16, Jungtaek Lim <kabh...@gmail.com>님이 작성:
>>> >
>>> > > Maybe we could try out Gitbox, though every committers should join
>>> their
>>> > > Github accounts to 'apache' group and enable 2FA.
>>> > >
>>> > > 2017년 7월 13일 (목) 오전 8:38, Jungtaek Lim <kabh...@gmail.com>님이 작성:
>>> > >
>>> > >> Did we render webpage with asf-site branch? I didn't recognize it.
>>> > >>
>>> > >> Yes I meant separate git repository, like 'storm-site'. I'm happy
>>> I'm
>>> > not
>>> > >> the only one who feels inconvenient with SVN repo.
>>> > >> Would it better to initiate VOTE for this?
>>> > >>
>>> > >> Thanks,
>>> > >> Jungtaek Lim (HeartSaVioR)
>>> > >>
>>> > >> 2017년 7월 13일 (목) 오전 4:30, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>>> > >>
>>> > >>> We were using git before, then a year ago moved back to subversion
>>> to
>>> > >>> implement versioned documentation [1].
>>> > >>>
>>> > >>> If we do decide to move back to git for this, I would recommend
>>> using a
>>> > >>> separate git repository so it doesn’t bloat our main code
>>> repository.
>>> > When
>>> > >>> generating javadoc for a new version, the svn commit to publish the
>>> > site
>>> > >>> can take around 20 minutes.
>>> > >>>
>>> > >>> -Taylor
>>> > >>>
>>> > >>> > On Jul 12, 2017, at 

Re: possible to have supervisors without _eventlogger and _acker tasks

[Bobby Evans]

It should, especially for the ackers.
The ackers receive lots and lots of small messages, and those messages come 
from all over your topology.  What is more if you have max.spout.pending set 
how quickly the messages can get to them and back to the spouts determines the 
throughput of your topology to some degree.  But it all depends on what 
actually is the bottleneck in your topologies.  If it is the network/network 
ping time, then scheduling all of the components of your topology close to each 
other is important.  If it is the CPU or Memory then you need to spread them 
out more to get more free resources on other nodes.  This is kind of what RAS 
tries to do but it does it just from guesses supplied by the topology owner.   
In future releases we expect to add in elasticity to RAS so that it can look at 
the actual resources being used and take that into account when scheduling, 
because each topology is different.

- Bobby


On Tuesday, August 1, 2017, 3:40:11 PM CDT, AMir Firouzi <firouz...@gmail.com> 
wrote:

Thanks Bobby for your instant & informative reply,
i actually respect these rules. i schedule all of these loggers and ackers,
but right now my scheduler put all the system tasks(loggers and acker
tasks) into one worker in one machine and i'm not getting the best
performance! I think it's because all of the tasks should transfer data to
these tasks in another machines and network latency slows down the storm.
but i'm wondering if i put some of these system tasks near other
(bolt/spout) tasks, would it effect the performance?
thanks again for your answer.

On Tue, Aug 1, 2017 at 6:20 PM Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:

> By default there are no `_eventlogger` tasks.  To have this feature
> enabled you need to turn it on by setting topology.eventlogger.executors to
> a positive number.  Ackers are on by default, but can be disabled by
> setting the number of topology.acker.executors to 0.  You should respect
> these when scheduling a topology because if they are supposed to be there
> and they are not scheduled messages will be sent to them, but they will be
> lost.  In the case of acking all of the tuples will time out.  In the case
> of the event logger the UI will show it working, but nothing will ever come
> out.
> Now that is on a per topology basis, not on a per worker basis.  These
> bolts are like any other bolt.  They can be in any worker your scheduler
> wants to put them in.  When inserting an acker bolt it is using a keyed
> grouping connected to just about everything in your topology, so where you
> place it is not that critical as it is going to be talking to everything.
> The event logger bolts are similar, but using a fields grouping based off
> of component id.
>
> https://github.com/apache/storm/blob/4c8a986f519cdf3e63bed47e9c4f723e4867267a/storm-client/src/jvm/org/apache/storm/daemon/StormCommon.java#L346-L357
> You could try to be smart to try and collocate the component with the
> logger for it, but honestly this feature slows your topology down so much
> already it is probably not worth trying to optimize it as it really will
> only be used when you need to do some serious debugging.
>
>
> - Bobby
>
>
> On Tuesday, August 1, 2017, 4:44:55 AM CDT, AMir Firouzi <
> firouz...@gmail.com> wrote:
>
> hi guys
> i'm working on my own scheduler for storm. i wonder what happens if i
> create a worker process and put some tasks in it(bolt/spout tasks) but no
> _eventlogger and _acker tasks. what happens? is it a problem? tuples
> transferred/emitted from within tasks in this worker will be skipped or
> they just use another _acker or _loggers in other workers?
>
> thanks in advance
>

Re: add new tasks to already assigned slots after scheduling for the first time

[Bobby Evans]

I am not completely sure why that was a requirement.  I know internally we have 
removed it, not sure if we pushed that change back or not yet.  On newer 
versions of storm 1.0.3+ that have the rewritten supervisor that check is 
invalid and you should be able to remove the check and just update the slot.  
You might be able to get away with it on older versions too, I just don't know. 
 I think it had something to do with race conditions in both the supervisor 
where if the scheduling changed without clearing it out first the supervisor 
would not know that it changed and not do the right thing.
- Bobby


On Tuesday, August 1, 2017, 4:16:10 PM CDT, AMir Firouzi  
wrote:

I need some migration features for my scheduler in storm. i schedule the
tasks first time based on some logic and then after a while i need to
migrate some tasks to another workers. but after doing so if i assign the
task to an already used slot storm(nimbus) nags about not being able to use
an already used slot. is it impossible to do so? otherwise i have to use
another slots while some old used slots have the capacity to contains more
tasks and also compacting related tasks to a slot reduces intra-worker
traffic.

thanks

Re: [Discussion]: Storm Improvemement Proposal (SIP) to discuss changes

[Bobby Evans]

You don't want it in the bylaws yet, Not a big deal, but I still think having 
the rules formally written up are a good start, even if they are not formally 
voted on and binding in the bylaws.
We can try it out see what works and what doesn't.


- Bobby


On Wednesday, August 2, 2017, 12:29:46 PM CDT, Hugo Da Cruz Louro 
<hlo...@hortonworks.com> wrote:

I am +1 with having a bit more structure in the process (SIP), but being 
cautious about limiting the overhead. I like the SIP idea, but I agree with 
Bobby’s point of -  "what if you missed SIP and don’t agree with something 
during PR phase". For that I suggest that SIP is an iterative process - just 
like engineering is. The proposer would come with a proposal (SIP) with 
different levels of detail (it could be a few bullet points in some cases), and 
then he would add to it what was discussed in the PR to make it more 
compelling. In a way SIP is what we already have in the PRs anyways, so it’s 
just a little effort to organize the info. I would really encourage an effort 
to consolidate the info in the SIP after the PR discussion is complete. 

A huge advantage of SIP is that it will be a source of technical/design 
documentation with historical context. This is very important for new 
contributors coming in and will help future refactoring, discussion across 
backwards compatibility issues, etc.

I understand the concerns about SIP potentially slowing down the contributions. 
However, I believe that true velocity comes from how quickly we can have 
multiple contributors pick new areas of the code and quickly expand on it. This 
is easier to achieve with good documentation, clean, modular code, and good 
unit testing that allows for quickly trying things out.

Hugo


> On Aug 2, 2017, at 7:46 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
> 
> My only concerns are 
> 1. It is going to add more overhead and slow down getting a feature out.  But 
> that may not be the case if there is less overhead during the review process 
> to offset writing up a large design, and it can be balanced with small 
> features not needing it. 2. that I am going to miss a SIP review because I am 
> busy and then when I complain about something wrong with a feature during a 
> code review I will get a lot of push back because it passed SIP. 
> but I can also see both sides of the coin on this.  To me SIP needs to be a 
> change to the bylaws, so if you want this change Harsha I would suggest that 
> you write up your proposed changes to the bylaws, and we can discuss them in 
> more detail, because it feels rather abstract to me right now.  And deciding 
> what is a large feature vs a small one also feels a bit abstract.
> 
> 
> - Bobby
> 
> 
> On Tuesday, August 1, 2017, 6:30:56 PM CDT, Jungtaek Lim <kabh...@gmail.com> 
> wrote:
> 
> Still +1 to introduce this only for non connectors. Maybe would want to
> skip this also for non connectors and non storm-core
> (storm-client/storm-server) like Flux, SQL, storm-webapp as well, but maybe
> still have small chance to need it.
> 
> Despite that I voted to +1, I still worry about efforts on reviewing SIP:
> this will only work if we (in dev@ list) are open to participate and review
> SIP in desired duration. Two sides of the coin: it might incurs more active
> community, but it will just fail if community is not enough active. Each
> SIP discussion is easy to be staled if we don't care about much, and if we
> also want to introduce vote for SIP, easier to be staled.
> 
> So we all should have willing to go with this. I'm OK to take additional
> load, but would like to hear others opinions as well.
> 
> - Jungtaek Lim (HeartSaVioR)
> 
> 2017년 8월 2일 (수) 오전 2:27, Harsha <st...@harsha.io>님이 작성:
> 
>> Trying to bring attention this again.
>> We currently have few big feature PRs going on and there is considerable
>> discussion about the design and Implementation etcc. My intention of
>> starting SIP is to add these details before someone goes and writes up a
>> PR and everyone has to go through reading of design and sometimes those
>> docs are not clear and we end up having long discussion on the PRs which
>> should mainly about the code review itself.
>> We should at least start making this process mandatory for any new big
>> feature especially to the storm-core. I am less concerned about
>> connectors and other parts which should have least resistive path and
>> they are usually easy to review.
>> If the devs put their thoughts and design and goes through discussion
>> and get everyone on the same line when the PR shows up it will be less
>> surprising and everyone involved know how the PR/Code supposed to work.
>> 
>> -Harsha
>> 
>> On Fri, Jun 9, 2017, at 

Re: [Discussion]: Storm Improvemement Proposal (SIP) to discuss changes

[Bobby Evans]

 we ever veer too far in the
> > wrong direction to bring up and improve or remove this process.
> >
> > We should also as a community strive to have better quality and I am
> > hoping this will give us a chance to not only let users know what are
> > changes coming in but also keep the dev list to have a chance and join
> > the discussion.
> >
> > -Harsha
> >
> > On Jun 9, 2017, 7:18 PM -0700, Arun Iyer <ai...@hortonworks.com>, wrote:
> > I am for documenting and upfront design reviews, but maybe we should
> > keep it less formal and make it part of the JIRA to start with.
> >
> > Do we have any upcoming features for which we would like to see a
> > proposal? May be start with a couple of proposals
> > and see it works out before making it formal.
> >
> >
> > Thanks,
> > Arun
> >
> >
> >
> > 6/9/17, 6:49 PM, "P. Taylor Goetz" <ptgo...@gmail.com> wrote:
> >
> > -0
> >
> > The KIP process feels kind of heavy. I'd rather start with a lighter
> > effort like improving JIRA submissions and pull requests (some pull
> > requests/JIRAs, even from committers and PMC members, are woefully
> > inadequate in terms of detail), and see how that works out.
> >
> > I share Bobby's concern that doing so might raise the bar for
> > contributions and potentially have a chilling effect. We don't want to
> > scare away contributors. Kafka has somewhat of a reputation for setting
> > potentially too high a bar. I'd rather not see that happen with this
> > community.
> >
> > I will say that I like the idea of proposals for big features, ideally
> > before any coding even begins -- so that others have a chance to
> > collaborate. But I'm hesitant to impose too much process, voting, etc.
> > That could scare people off.
> >
> > I think we should think carefully before going down this trail.
> >
> > -Taylor
> >
> > On Jun 9, 2017, at 8:57 PM, Priyank Shah <ps...@hortonworks.com> wrote:
> >
> > +1 for SIPs including a new connector. The person writing SIP can
> > provide details about the external system for which connector is being
> > written to let others know why a certain design decision was made. This
> > will make it easy for reviewers.
> >
> > On 6/9/17, 5:24 PM, "Satish Duggana" <satish.dugg...@gmail.com> wrote:
> >
> > +1 for SIPs. It is so useful as mentioned by others in earlier mails. It
> > would be very useful for new contributors and others who are looking out
> > for a feature design and decisions taken etc.
> >
> > Whenever a minor feature is added to a connector it may not need a
> > separate
> > SIP but the existing README can be updated with details for users. It
> > can
> > be discussed and decided apropos whether a SIP is really required for
> > any
> > enhancement which is not really big.
> >
> >
> > On Sat, Jun 10, 2017 at 5:13 AM, Roshan Naik <ros...@hortonworks.com>
> > wrote:
> >
> > If I am looking at the Kafka site correctly, I see that Kafka has a
> > total
> > of 167 KIPs so far.
> > So I assume that minor new features would not be parrt of the SIP ?
> >
> > Unlike Kafka, since Storm has a number of connectors (that keep
> > growing),
> > I am speculating the SIP process might get somewhat unwieldy if it were
> > to
> > track little changes in each of the connectors.
> >
> > Also thinking that a SIP may not be needed to justify a new connector,
> > but
> > useful if we are replacing an old connector with a new one.
> >
> > -roshan
> >
> >
> >
> > On 6/9/17, 3:19 PM, "Harsha" <st...@harsha.io> wrote:
> >
> > Hi Bobby,
> > In general, a KIP is required for adding New features,
> > config
> > changes or backward-incompatible changes. Don't require
> > adding a KIP for bug-fixes. Devs who wants to add any
> > features will write up a wiki which has JIRA link, mailing
> > list discussion link and outline the Motivation, Public
> > interface changes and protocol changes etc ..a good example
> > here is
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 48+Delegation+token+support+for+Kafka.
> > They can start the discussion thread once its ready and once everyone
> > agrees its in a good shape, a Vote thread starts . Once there are
> > required votes are in one can start the PR process and get it merged
> > in.
> > Each release we can collect what features/fixes especially
> > to
>

Re: [DISCUSS] Ideas for resolving storm-drpc-server compilation issue on IDE

[Bobby Evans]

Yes some of the names need to be improved and if you have suggestions before we 
release a 2.x release that would be great.
storm-client, is actually the client.  We have included the worker classpath in 
here too because if the classpath is different for the client that serializes 
the topology from the classpath for the worker that deserializes it,  hard to 
debug problems start to show up.  This is the most critical part as this is 
what the user sees.  Everything else is off  of the user's class path so we can 
"fix" the packages in minor releases of storm.  There are some things here too 
that we may want to move to storm-server, but I have not done a complete walk 
though.  we should before we do a 2.0.0 release.  Some beta or even alpha 
releases might be good.

storm-client-misc is badly named.  It should probably be moved to external and 
be called something more like http-forwarding-metrics.  In fact I filed 
STORM-2670 for this.  If others have suggestions for a name please let me know.

Now for the stuff that may not be ideal, but we can change around after a 
release.

storm-rename-hack - STORM-2671 it should just go away.

storm-core - this is mostly things that were left over as we moved other stuff 
around.  It has command line commands (can probably move to storm-server, or we 
could have a separate package just for them); Some pieces of the storm-ui (as 
you know); a few things in daemon that will probably go away with STORM-2671; 
and some testing code that still depends on clojure (as the clojure client 
pieces have moved off to other packages).  Once the tests have all migrated out 
of clojure that part can go away too. 

storm-server is supposed to be everything else.  Initially we split the http 
part off from the java part, and had a separate package for DRPC, because I was 
hopeful that I could find time to upgrade the http server in drpc to be newer 
and let us do async requests, but I never found the time to do it. 


- Bobby


On Tuesday, August 1, 2017, 8:49:40 PM CDT, Hugo Da Cruz Louro 
<hlo...@hortonworks.com> wrote:

I have been following this discussion thread as part of the storm-core-ui 
migration. I would like to bring up a couple of points:

* The names of the packages "storm-client" and "storm-server" are a bit 
misleading to me. Isn’t what we really mean here "storm-workers" and 
"storm-daemons” ? Even if not these names, we should pick names that as close 
as possible to the “physical system”.

* storm-client-misc
  * I noticed that this module only has two classes [1]. They are currently 
used in the module storm-starter and nowhere else. If that is the case, we 
should just put the classes in the module storm-starter. The concern is if some 
users may be using them in their deployments. Do you know of any users using 
these classes? Perhaps we could poll 
us...@storm.apache.org<mailto:us...@storm.apache.org> and find out.

  * the -misc extension is also very confusing to me. My first thought was that 
it was some sort of library dependency placeholder, or something like that. If 
at all possible, my suggestion would be for us to eliminate this module 
altogether.

  * Since we Storm 2.0 is a major release, if we find out that not many users 
(maybe none) are using the classes [1] we could probably just put the classes 
HttpForwardingMetricsConsumer, HttpForwardingMetricsServer in storm-starter. As 
for the concern of breaking backwards compatibility, document a workaround 
using storm-starter.

Thanks,
Hugo

[1] - HttpForwardingMetricsConsumer, HttpForwardingMetricsServer


On Jul 31, 2017, at 6:51 AM, Bobby Evans 
<ev...@yahoo-inc.com.INVALID<mailto:ev...@yahoo-inc.com.INVALID>> wrote:

Those look reasonable to me.


- Bobby


On Monday, July 31, 2017, 2:22:47 AM CDT, Jungtaek Lim 
<kabh...@gmail.com<mailto:kabh...@gmail.com>> wrote:

I agreed to minimize the target of shade & relocation artifacts minimal as
possible, but as we shaded almost everything (meaning non-relocation will
affect user experience) so may need to find exhaustive set of troublesome
artifacts and relocate at least them. (Maybe union of everyone's lists?)

For me Guava, HttpClient, Netty (maybe no need to shade for now if we don't
plan to upgrade to 4.x: package name differs) is in my list.

Would be better to initiate poll or discussion with separate thread?

- Jungtaek Lim (HeartSaVioR)

2017년 7월 20일 (목) 오전 2:27, Bobby Evans 
<ev...@yahoo-inc.com.invalid<mailto:ev...@yahoo-inc.com.invalid>>님이 작성:

I am fine with a separate project for relocated dependencies (or even just
separate packages, you do a maven install of them and not include them in
the IDE at all).  Shading still has some drawbacks, but I think in a few
cases it makes since.  I would prefer it if we picked a very small number
of dependencies that cause people issues and just shade those.  Guava is
the big one that I worry abo

Re: Considerably slow building website

[Bobby Evans]

Be careful when removing the javadocs.  There are links to the javadocs from 
within the docs themselves.


- Bobby


On Tuesday, August 1, 2017, 12:57:56 PM CDT, P. Taylor Goetz 
<ptgo...@gmail.com> wrote:

I cleaned up the download page to remove some of the older releases and added a 
link to archive.a.o for older releases. I will also clean up dist as requested 
by infra.

While I’m at it, I’ll clean up the javadoc so we only include javadoc for 
releases on the download page.

That should help a little bit, but I agree that the publishing process is 
painful and would welcome any improvements.

One option (I haven’t tested yet) might be to simply move the javadoc to the 
“publish” directory so it doesn’t get regenerated every time the site gets 
published. That would mean the javadoc links won’t work when running Jekyll 
locally, but I think it’s a fair trade off.

-Taylor
 
> On Aug 1, 2017, at 9:39 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
> 
> Rebuilding everything each time is sadly necessary as currently the 
> header/footer for all of the content is inline in each page.  So if we add a 
> new release every page changes.  To fix this we would have to change the 
> header to dynamically include the HTML from another file that gets updated on 
> it's own.
> We might also want to think about rearranging things a bit, and reduce the 
> number of releases that we have on the site.  Do we really need both 0.9.6 
> and 0.9.7, or 0.10.0 through 0.10.2.  Maybe there is a way to archive some of 
> these so they are a part of the final site, but are not generated each time? 
> (probably would need the header change at a minimum to work)
> 
> 
> - Bobby
> 
> 
> On Tuesday, August 1, 2017, 6:01:03 AM CDT, Jungtaek Lim <kabh...@gmail.com> 
> wrote:
> 
> I found I forgot to build website with "-d publish/" parameter. Now it
> reduced to 1347.585 secs but that is still way too long
> 
> I've done some tests on building website ('jekyll build -d publish/
> --profile'):
> 
> 1. as it is : 1347.585 secs
> 2. excluding 'releases' directories : 2.38 secs
> 3. excluding 'releases' directories, and including '2.0.0-SNAPSHOT'
> directory of releases : 45 secs
> 
> The build time is not stable but you can see how much the difference is. If
> we can separate building doc for each release, that should be best and it
> should reduce the build time greatly.
> 
> If we can't separate building doc, we may want to take alternative
> approach: reducing maintaining releases. You can imagine that if we keep
> adding docs for new releases in website repo it should increase overall
> build time. I guess we may be better to provide only the last version of
> version lines: 0.9.7, 0.10.2, 1.0.4, 1.1.0 (will be 1.1.1 soon),
> 2.0.0-SNAPSHOT, total 5 releases. If we respect semantic versioning, major
> changes shouldn't be introduced in bug-fix releases so don't need to
> maintain docs separately.
> 
> I would like to gather opinions around this along with moving website to
> git. Looking forward to hear others opinions.
> 
> Thanks,
> Jungtaek Lim (HeartSaVioR)
> 
> 2017년 8월 1일 (화) 오전 7:44, Jungtaek Lim <kabh...@gmail.com>님이 작성:
> 
>> Also found that we don't expose 1.0.4 in documentation dropdown and 1.0.4
>> directory is not created in 'publish/releases' directory. Maybe also missed
>> that.
>> 
>> 2017년 8월 1일 (화) 오전 7:36, Jungtaek Lim <kabh...@gmail.com>님이 작성:
>> 
>>> Hi devs,
>>> 
>>> I'm trying to modify release note on 1.0.4 one of user reported about
>>> wrong CHANGELOG. And surprisingly, it took about 50 mins to serve the
>>> website locally. Any hints to reduce the time? 50 mins for only building
>>> the website is really annoying and anyone don't want to wait for that if we
>>> modify "a" file.
>>> 
>>> And I found Storm 1.1.0 release note markdown file is missing. Taylor,
>>> could you add it back to the SVN repo?
>>> 
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>> 

Re: possible to have supervisors without _eventlogger and _acker tasks

[Bobby Evans]

By default there are no `_eventlogger` tasks.  To have this feature enabled you 
need to turn it on by setting topology.eventlogger.executors to a positive 
number.  Ackers are on by default, but can be disabled by setting the number of 
topology.acker.executors to 0.  You should respect these when scheduling a 
topology because if they are supposed to be there and they are not scheduled 
messages will be sent to them, but they will be lost.  In the case of acking 
all of the tuples will time out.  In the case of the event logger the UI will 
show it working, but nothing will ever come out.
Now that is on a per topology basis, not on a per worker basis.  These bolts 
are like any other bolt.  They can be in any worker your scheduler wants to put 
them in.  When inserting an acker bolt it is using a keyed grouping connected 
to just about everything in your topology, so where you place it is not that 
critical as it is going to be talking to everything.  The event logger bolts 
are similar, but using a fields grouping based off of component id.  
https://github.com/apache/storm/blob/4c8a986f519cdf3e63bed47e9c4f723e4867267a/storm-client/src/jvm/org/apache/storm/daemon/StormCommon.java#L346-L357
You could try to be smart to try and collocate the component with the logger 
for it, but honestly this feature slows your topology down so much already it 
is probably not worth trying to optimize it as it really will only be used when 
you need to do some serious debugging.


- Bobby


On Tuesday, August 1, 2017, 4:44:55 AM CDT, AMir Firouzi  
wrote:

hi guys
i'm working on my own scheduler for storm. i wonder what happens if i
create a worker process and put some tasks in it(bolt/spout tasks) but no
_eventlogger and _acker tasks. what happens? is it a problem? tuples
transferred/emitted from within tasks in this worker will be skipped or
they just use another _acker or _loggers in other workers?

thanks in advance

Re: Considerably slow building website

[Bobby Evans]

Rebuilding everything each time is sadly necessary as currently the 
header/footer for all of the content is inline in each page.  So if we add a 
new release every page changes.  To fix this we would have to change the header 
to dynamically include the HTML from another file that gets updated on it's own.
We might also want to think about rearranging things a bit, and reduce the 
number of releases that we have on the site.  Do we really need both 0.9.6 and 
0.9.7, or 0.10.0 through 0.10.2.  Maybe there is a way to archive some of these 
so they are a part of the final site, but are not generated each time? 
(probably would need the header change at a minimum to work)


- Bobby


On Tuesday, August 1, 2017, 6:01:03 AM CDT, Jungtaek Lim  
wrote:

I found I forgot to build website with "-d publish/" parameter. Now it
reduced to 1347.585 secs but that is still way too long

I've done some tests on building website ('jekyll build -d publish/
--profile'):

1. as it is : 1347.585 secs
2. excluding 'releases' directories : 2.38 secs
3. excluding 'releases' directories, and including '2.0.0-SNAPSHOT'
directory of releases : 45 secs

The build time is not stable but you can see how much the difference is. If
we can separate building doc for each release, that should be best and it
should reduce the build time greatly.

If we can't separate building doc, we may want to take alternative
approach: reducing maintaining releases. You can imagine that if we keep
adding docs for new releases in website repo it should increase overall
build time. I guess we may be better to provide only the last version of
version lines: 0.9.7, 0.10.2, 1.0.4, 1.1.0 (will be 1.1.1 soon),
2.0.0-SNAPSHOT, total 5 releases. If we respect semantic versioning, major
changes shouldn't be introduced in bug-fix releases so don't need to
maintain docs separately.

I would like to gather opinions around this along with moving website to
git. Looking forward to hear others opinions.

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 8월 1일 (화) 오전 7:44, Jungtaek Lim 님이 작성:

> Also found that we don't expose 1.0.4 in documentation dropdown and 1.0.4
> directory is not created in 'publish/releases' directory. Maybe also missed
> that.
>
> 2017년 8월 1일 (화) 오전 7:36, Jungtaek Lim 님이 작성:
>
>> Hi devs,
>>
>> I'm trying to modify release note on 1.0.4 one of user reported about
>> wrong CHANGELOG. And surprisingly, it took about 50 mins to serve the
>> website locally. Any hints to reduce the time? 50 mins for only building
>> the website is really annoying and anyone don't want to wait for that if we
>> modify "a" file.
>>
>> And I found Storm 1.1.0 release note markdown file is missing. Taylor,
>> could you add it back to the SVN repo?
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>

Re: [DISCUSS] Remove CHANGELOG file

[Bobby Evans]

So it looks like we all agree, now we just need someone to file a JIRA and a 
corresponding pull request.  The kafka script looks like a good place to start, 
but we can iterate on it in the pull request to try and address Taylor's 
concern about JIRA not being up to date.  I would love to do it, but I am 
really overloaded right now so if someone else wants to take lead on it that 
would be great.


- Bobby


On Monday, July 31, 2017, 1:45:14 PM CDT, P. Taylor Goetz <ptgo...@gmail.com> 
wrote:

I’m all for getting rid of the current process for CHANGELOG. My only concern 
with any JIRA-based solution is that we would need to be very good about 
setting the “Fix Version” field properly when merging a patch and updating the 
associated ticket. In the past I’ve seen a lot of patches merged without the 
associated JIRA updated. If we’re going to rely on JIRA as the source of truth 
for change logs, we need to be very conscientious about updating JIRA as 
necessary.

-Taylor

> On Jul 31, 2017, at 10:06 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
> 
> I am happy to switch as soon as someone has a working alternative.  The big 
> thing in my opinion is giving end users a clear list of all of the changes 
> that went into a release so they can review it for themselves.  However we do 
> it is fine with me as the current changelog file leaves a lot to be desired. 
> I personally would be fine with us updating the web page/release notes to 
> have a link to a JIRA query in it as a starting point.
> 
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20STORM%20AND%20fixVersion%20in%20(1.0.4)%20ORDER%20BY%20priority%20DESC
> or
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20STORM%20AND%20fixVersion%20in%20(1.1.1)%20ORDER%20BY%20priority%20DESC
> for example.
> Later on we can start looking at more complex alternatives that run the above 
> query and join it with the git revision history and possibly pull requests to 
> give a more complete view for what has happened.
> 
> - Bobby
> 
> 
> On Monday, July 31, 2017, 1:42:11 AM CDT, Jungtaek Lim <kabh...@gmail.com> 
> wrote:
> 
> Let me also put long ago discussion about this:
> 
> http://search-hadoop.com/m/Storm/8gnYyUdhVp1eajp31?subj=+DISCUSSION+More+convenient+way+to+maintain+committer+contributor+list+and+changelogs
> 
> 
> In my view, from long ago discussion, Haohui and Bobby agreed to not
> maintain CHANGELOG by hand. Haohui also suggested how to get them
> automatically, whereas I just would want to remove it, but that's also OK)
> We didn’t get agreement clearly about removing CHANGELOG but at least saw
> our needs to automate it.
> 
> 
> And in current discussion, again in my view, Roshan, Hugo, Stig agree to
> remove CHANGELOG. I’ve been continuously claiming to remove CHANGELOG, so 3
> PMC members and 1 contributor seem to agree on removing CHANGELOG, and at
> least 2 more PMC members to not maintain CHANGELOG manually.
> 
> 
> I will initiate a VOTE thread if we need to. Again, release managers would
> be affected by this change so I would want to hear Taylor’s opinion before
> going forward, but this is clear pain point for mergers so will initiate a
> VOTE thread in several days (at least in this week) if Taylor doesn’t put
> opinion on this or misses this discussion.
> 
> 
> Thanks,
> 
> Jungtaek Lim (HeartSaVioR)
> 
> 2017년 7월 28일 (금) 오전 10:53, Jungtaek Lim <kabh...@gmail.com>님이 작성:
> 
>> correction: other projects -> *some* other projects, though they're
>> popular projects (including in competition)
>> 
>> 2017년 7월 28일 (금) 오전 10:51, Jungtaek Lim <kabh...@gmail.com>님이 작성:
>> 
>>> I'm happy that there're other guys having same difficult and sharing same
>>> feeling.
>>> 
>>> This discussion has been initiating several times (from me) and getting
>>> some +1s for each thread but didn't reach to actual work.
>>> 
>>> We already utilize JIRA, and I'm subscribing issues@ and taking care of
>>> issues forgot to mark resolve and/or labeling fixed versions.
>>> It may sounds ideal for us to let reporters caring about their issues,
>>> but committers can also help that, and in fact merger is in responsible to
>>> take care of resolving the issue, so irrelevant to contributor for this
>>> side.
>>> 
>>> My other consideration is that which thing is convenient for release
>>> manager. Taylor took the release manager all the time (thanks for the great
>>> work!) and it is directly related to release announcement so would like to
>>> hear his opinion. If it is more convenient or he think he can tolerate
>>> that, we can just go on.
>>> 
>&g

Re: Writing orc files with storm via java API

[Bobby Evans]

It should be possible to make this work, but it is not going to be simple.  The 
real issue is the format of the orc file.  It is not one record at a time, like 
CSV or other supported formats are.  Sadly this is currently an assumption with 
the AbstractHdfsBolt.
https://github.com/apache/storm/blob/master/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/bolt/format/RecordFormat.java
So to support it we would need to make some modifications, not impossible, just 
not a drop in replacement.  If this is something you want to tackle and 
contribute back I think we would all love it.  You might also run into some 
issues with metadata for the format being written at the end of the file.
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC
I am not totally sure how easy it is to recover an ORC file if that footer is 
missing because a worker crashed.  You might end up with data loss in some 
cases if you are not extremely careful.  You might also need to modify the ORC 
APIs themselves to be able to support storing/recovering the metadata in an 
external location for recovery to truly fix it, and then store them in ZK on a 
flush until the file is rotated.

The Trident HDFState
https://github.com/apache/storm/blob/master/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/trident/HdfsState.java
might be a more appropriate place to start, as the updated state is written out 
in micro batches, but you still have to deal with the footer issues, as trident 
really cares about exactly once processing.

So overall it is not a simple problem, and relying on an external server like 
hive would make it a lot simpler.


- Bobby


On Tuesday, July 25, 2017, 8:38:42 AM CDT, Igor Kuzmenko  
wrote:

Is there any implementation of storm bolt which can write files to HDFS in
ORC format, without using Hive Streaming API?
I've found java API for writing ORC files 
and I'm guessing is there any existing Hive bolts that uses it or any plans
to create such?

Re: [DISCUSS] Remove CHANGELOG file

[Bobby Evans]

I am happy to switch as soon as someone has a working alternative.  The big 
thing in my opinion is giving end users a clear list of all of the changes that 
went into a release so they can review it for themselves.  However we do it is 
fine with me as the current changelog file leaves a lot to be desired. I 
personally would be fine with us updating the web page/release notes to have a 
link to a JIRA query in it as a starting point.

https://issues.apache.org/jira/issues/?jql=project%20%3D%20STORM%20AND%20fixVersion%20in%20(1.0.4)%20ORDER%20BY%20priority%20DESC
or
https://issues.apache.org/jira/issues/?jql=project%20%3D%20STORM%20AND%20fixVersion%20in%20(1.1.1)%20ORDER%20BY%20priority%20DESC
for example.
Later on we can start looking at more complex alternatives that run the above 
query and join it with the git revision history and possibly pull requests to 
give a more complete view for what has happened.

- Bobby


On Monday, July 31, 2017, 1:42:11 AM CDT, Jungtaek Lim  
wrote:

Let me also put long ago discussion about this:

http://search-hadoop.com/m/Storm/8gnYyUdhVp1eajp31?subj=+DISCUSSION+More+convenient+way+to+maintain+committer+contributor+list+and+changelogs


In my view, from long ago discussion, Haohui and Bobby agreed to not
maintain CHANGELOG by hand. Haohui also suggested how to get them
automatically, whereas I just would want to remove it, but that's also OK)
We didn’t get agreement clearly about removing CHANGELOG but at least saw
our needs to automate it.


And in current discussion, again in my view, Roshan, Hugo, Stig agree to
remove CHANGELOG. I’ve been continuously claiming to remove CHANGELOG, so 3
PMC members and 1 contributor seem to agree on removing CHANGELOG, and at
least 2 more PMC members to not maintain CHANGELOG manually.


I will initiate a VOTE thread if we need to. Again, release managers would
be affected by this change so I would want to hear Taylor’s opinion before
going forward, but this is clear pain point for mergers so will initiate a
VOTE thread in several days (at least in this week) if Taylor doesn’t put
opinion on this or misses this discussion.


Thanks,

Jungtaek Lim (HeartSaVioR)

2017년 7월 28일 (금) 오전 10:53, Jungtaek Lim 님이 작성:

> correction: other projects -> *some* other projects, though they're
> popular projects (including in competition)
>
> 2017년 7월 28일 (금) 오전 10:51, Jungtaek Lim 님이 작성:
>
>> I'm happy that there're other guys having same difficult and sharing same
>> feeling.
>>
>> This discussion has been initiating several times (from me) and getting
>> some +1s for each thread but didn't reach to actual work.
>>
>> We already utilize JIRA, and I'm subscribing issues@ and taking care of
>> issues forgot to mark resolve and/or labeling fixed versions.
>> It may sounds ideal for us to let reporters caring about their issues,
>> but committers can also help that, and in fact merger is in responsible to
>> take care of resolving the issue, so irrelevant to contributor for this
>> side.
>>
>> My other consideration is that which thing is convenient for release
>> manager. Taylor took the release manager all the time (thanks for the great
>> work!) and it is directly related to release announcement so would like to
>> hear his opinion. If it is more convenient or he think he can tolerate
>> that, we can just go on.
>>
>> Please note that other projects don't use merge commit. Most of the time
>> they squash commits in PR into one, labeling commit title as JIRA issue,
>> making commit list just as CHANGELOG. That's another thing we discussed
>> earlier and I think we need to discuss again, but that can be discussed
>> from another thread.
>>
>> Regarding maintaining contributors: easy to explain. Just take a look at
>> what Spark has been doing. Some other projects follow the approach as well.
>>
>> We can run the script to extract authors of git commits, and just " |
>> sort | uniq", and done. Pulling assigner from JIRA issue may be more
>> accurate, since it requires actual account whereas author information in
>> commit is not strictly required to identify them. We can apply hybrid
>> approach as well, but for starter just following git commits looks OK to me.
>>
>> IMHO they don't feel proud strongly only they're listed in contributors.
>> Looking at contribution graph works better in this case, given that it also
>> shows commit count and lines of change. (regardless of accuracy)
>> It may give more proud to mention them as release announce. It will lead
>> contributors to play consistently, trying to participate and be mentioned
>> for releases as many as possible. IMO Spark built a great strategy for this
>> side, and if we all think it is great, why not follow?
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>> 2017년 7월 28일 (금) 오전 6:58, Stig Rohde Døssing 님이
>> 작성:
>>
>>> We already have to keep JIRA updated, and keeping JIRA consistent is
>>> easier
>>> since there 

Re: [Propose] move website repository from svn to git

[Bobby Evans]

+1
I am fine with moving to git, but I would like it to be a different repo.
Our current repo is at least 160MB already (which is a lot to download) but 
nothing compared the the web site that has lots and lots of things checked in 
(I estimate it at about 1.5GB on an older version I have locally)


- Bobby


On Monday, July 31, 2017, 1:58:03 AM CDT, Xin Wang  
wrote:

+1 for moving to git.  - Xin



2017-07-31 14:54 GMT+08:00 Jungtaek Lim :

> Bump. I think this is worth to address soon, since some contributors
> occasionally submit patches regarding documentations.
> Personally SVN is no longer feel convenient to use. If we all feel the
> same, let's change then.
>
> -Jungtaek Lim (HeartSaVioR)
>
> 2017년 7월 13일 (목) 오전 9:16, Jungtaek Lim 님이 작성:
>
> > Maybe we could try out Gitbox, though every committers should join their
> > Github accounts to 'apache' group and enable 2FA.
> >
> > 2017년 7월 13일 (목) 오전 8:38, Jungtaek Lim 님이 작성:
> >
> >> Did we render webpage with asf-site branch? I didn't recognize it.
> >>
> >> Yes I meant separate git repository, like 'storm-site'. I'm happy I'm
> not
> >> the only one who feels inconvenient with SVN repo.
> >> Would it better to initiate VOTE for this?
> >>
> >> Thanks,
> >> Jungtaek Lim (HeartSaVioR)
> >>
> >> 2017년 7월 13일 (목) 오전 4:30, P. Taylor Goetz 님이 작성:
> >>
> >>> We were using git before, then a year ago moved back to subversion to
> >>> implement versioned documentation [1].
> >>>
> >>> If we do decide to move back to git for this, I would recommend using a
> >>> separate git repository so it doesn’t bloat our main code repository.
> When
> >>> generating javadoc for a new version, the svn commit to publish the
> site
> >>> can take around 20 minutes.
> >>>
> >>> -Taylor
> >>>
> >>> > On Jul 12, 2017, at 10:33 AM, Jungtaek Lim 
> wrote:
> >>> >
> >>> > Hi devs,
> >>> >
> >>> > I think we discussed moving website repository from SVN to GIT from a
> >>> long
> >>> > time ago, and we were OK on that, but action was not taken.
> >>> >
> >>> > Now I can see number of projects (Spark, Kafka, Beam, maybe more) are
> >>> using
> >>> > separate GIT repository for website.
> >>> > Although we may still need to have version specific document (doc
> >>> > directory) from code repository and copy Jekyll build result to
> website
> >>> > repo, anyone can look at the whole website code and craft pull
> >>> requests to
> >>> > help us. Git would be more convenient for ourselves than SVN (since
> >>> we're
> >>> > maintaining Storm from GIT).
> >>> >
> >>> > So I'd like to propose having a new repository 'storm-website' or
> >>> > 'storm-site' with 'asf-site' as default branch, and move SVN contents
> >>> to
> >>> > GIT.
> >>> > (Sure we need to ask INFRA for helping Storm website to be rendered
> >>> from a
> >>> > new GIT repo.)
> >>> >
> >>> > What do you think?
> >>> >
> >>> > Thanks,
> >>> > Jungtaek Lim (HeartSaVioR)
> >>>
> >>>
>



-- 
Thanks,
Xin

Re: [DISCUSS] Ideas for resolving storm-drpc-server compilation issue on IDE

[Bobby Evans]

Those look reasonable to me.


- Bobby


On Monday, July 31, 2017, 2:22:47 AM CDT, Jungtaek Lim <kabh...@gmail.com> 
wrote:

I agreed to minimize the target of shade & relocation artifacts minimal as
possible, but as we shaded almost everything (meaning non-relocation will
affect user experience) so may need to find exhaustive set of troublesome
artifacts and relocate at least them. (Maybe union of everyone's lists?)

For me Guava, HttpClient, Netty (maybe no need to shade for now if we don't
plan to upgrade to 4.x: package name differs) is in my list.

Would be better to initiate poll or discussion with separate thread?

- Jungtaek Lim (HeartSaVioR)

2017년 7월 20일 (목) 오전 2:27, Bobby Evans <ev...@yahoo-inc.com.invalid>님이 작성:

> I am fine with a separate project for relocated dependencies (or even just
> separate packages, you do a maven install of them and not include them in
> the IDE at all).  Shading still has some drawbacks, but I think in a few
> cases it makes since.  I would prefer it if we picked a very small number
> of dependencies that cause people issues and just shade those.  Guava is
> the big one that I worry about. Netty is a possibility and I think asm
> would be another, but it is a transitive dependency so it would require us
> with our own version of kryo exposing the kryo API but pulling in a shaded
> asm.
> The servlet-api concerns me, but it looks like it is tied to the
> IHttpCredentialsPlugin which should move to the server package anyways.
>
> The rest I am not concerned about, are things that are exposed to end
> users, or are for test and not actually shipped.
> $ mvn dependecy:tree...
> [INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ storm-client
> ---
> [INFO] org.apache.storm:storm-client:jar:2.0.0-SNAPSHOT
> [INFO] +- uk.org.lidalia:sysout-over-slf4j:jar:1.0.2:compile
> [INFO] +- org.slf4j:slf4j-api:jar:1.7.21:compile
> [INFO] +- org.apache.logging.log4j:log4j-api:jar:2.8.2:compile
> [INFO] +- org.apache.logging.log4j:log4j-core:jar:2.8.2:compile
> [INFO] +- org.apache.logging.log4j:log4j-slf4j-impl:jar:2.8.2:compile
> [INFO] +- org.slf4j:log4j-over-slf4j:jar:1.6.6:compile
> [INFO] +- com.google.guava:guava:jar:16.0.1:compile
> [INFO] +- org.apache.thrift:libthrift:jar:0.9.3:compile
> [INFO] |  \- org.apache.httpcomponents:httpcore:jar:4.4.1:compile
> [INFO] +- commons-io:commons-io:jar:2.5:compile
> [INFO] +- commons-lang:commons-lang:jar:2.5:compile
> [INFO] +- commons-collections:commons-collections:jar:3.2.2:compile
> [INFO] +- com.lmax:disruptor:jar:3.3.2:compile
> [INFO] +- com.googlecode.json-simple:json-simple:jar:1.1:compile
> [INFO] +- org.yaml:snakeyaml:jar:1.11:compile
> [INFO] +- io.netty:netty:jar:3.9.0.Final:compile
> [INFO] +- com.esotericsoftware:kryo:jar:3.0.3:compile
> [INFO] |  +- com.esotericsoftware:reflectasm:jar:1.10.1:compile
> [INFO] |  |  \- org.ow2.asm:asm:jar:5.0.3:compile
> [INFO] |  +- com.esotericsoftware:minlog:jar:1.3.0:compile
> [INFO] |  \- org.objenesis:objenesis:jar:2.1:compile
> [INFO] +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile
> [INFO] |  \- jline:jline:jar:0.9.94:compile
> [INFO] +- org.apache.curator:curator-framework:jar:2.12.0:compile
> [INFO] +- org.jgrapht:jgrapht-core:jar:0.9.0:compile
> [INFO] +- javax.servlet:servlet-api:jar:2.5:compile
> [INFO] +- org.apache.httpcomponents:httpclient:jar:4.3.3:compile
> [INFO] |  +- commons-logging:commons-logging:jar:1.1.3:compile
> [INFO] |  \- commons-codec:commons-codec:jar:1.6:compile
> [INFO] +- org.apache.curator:curator-client:jar:2.12.0:compile
> [INFO] +- junit:junit:jar:4.11:test
> [INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
> [INFO] +- org.mockito:mockito-core:jar:1.9.5:test
> [INFO] \- org.hamcrest:hamcrest-library:jar:1.3:test
> - Bobby
>
>
> On Wednesday, July 12, 2017, 9:45:43 AM CDT, Jungtaek Lim <
> kabh...@gmail.com> wrote:
>
> I'd like to bump on this again, since we have a few huge issues for Storm
> 2.0.0, and this issue is a kind of regression and effectively blocker.
> (Please note that current master branch removes shading for some libraries
> to make IDE happy.)
>
> At that time I didn't consider option 2 as possible solution, but now Flink
> is going with this option, and I can't find reason to not doing this.
>
> * Repository: https://github.com/apache/flink-shaded
> * Discussion thread:
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Changing-Flink-s-shading-model-td17419.html
>
> Thought?
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2017년 3월 31일 (금) 오후 3:12, Jungtaek Lim <kabh...@gmail.com>님이 작성:
>
> > Bobby,
> >
> > I've worked on separating worker and daemon classpath.
> >
> > - Issue: STORM-2441: Break down 'storm-c

Re: [VOTE] Release Apache Storm 1.1.1 (rc2)

[Bobby Evans]

+1 built from source (v1.1.1 41bfea87b1a002565333bd18a06d766af1ca3816)  Ran all 
of the unit tests and some manual tests.


- Bobby


On Thursday, July 27, 2017, 1:38:46 PM CDT, P. Taylor Goetz  
wrote:

This is a call to vote on releasing Apache Storm 1.1.1 (rc2)

Full list of changes in this release:

https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=41bfea87b1a002565333bd18a06d766af1ca3816

The tag/commit to be voted upon is v1.1.1:

https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=948ce7d63a31fae8c478785985d0ef79808e234e;hb=41bfea87b1a002565333bd18a06d766af1ca3816

The source archive being voted upon can be found here:

https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.1-rc2/apache-storm-1.1.1-src.tar.gz

Other release files, signatures and digests can be found here:

https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.1-rc2/

The release artifacts are signed with the following key:

https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd

The Nexus staging repository for this release is:

https://repository.apache.org/content/repositories/orgapachestorm-1050

Please vote on releasing this package as Apache Storm 1.1.1.

When voting, please list the actions taken to verify the release.

This vote will be open for at least 72 hours.

[ ] +1 Release this package as Apache Storm 1.1.1
[ ]  0 No opinion
[ ] -1 Do not release this package because...

Thanks to everyone who contributed to this release.

-Taylor

Re: Setting heap size parameters by workers.childopts and supervisor.childopts

[Bobby Evans]

worker.childops is the default value that is set by the system administrator in 
storm.yaml on each of the supervisor nodes.  topology.worker.childopts is what 
you set in your topology conf if you want to add something more to the command 
line.


- Bobby


On Tuesday, July 25, 2017, 11:50:04 PM CDT, sam mohel  
wrote:

i'm using 0.10.2 version . i tried to write in the code
conf.put(Config.WORKER_CHILDOPTS, "-Xmx4g");
conf.put(Config.SUPERVISOR_CHILDOPTS, "-Xmx4g");

but i didn't touch any affect . Did i write the right configurations ?
Does this value is the largest ?

Re: [VOTE] Release Apache Storm 1.1.1 (rc1)

[Bobby Evans]

We could do a 1.1.2 release sooner if needed.  Technically any committer can 
call for a release at any point in time.  If there is a reason to do a release 
(like an important fix for a critical component) then we can do it.


- Bobby


On Tuesday, July 25, 2017, 1:48:07 PM CDT, Alexandre Vermeerbergen 
 wrote:

Hi,

I guess 1.1.2 is going to be a few months away from now, so we'll have to
go with our own basic Kafka 0.10 Spout in the meantime...

You can discard my previous vote, we'd need to at least download 1.1.1 rc
and give it a try to make an objective vote.

Regards,
Alexandre Vermeerbergen

2017-07-25 19:38 GMT+02:00 P. Taylor Goetz :

> Hi Alexandre,
>
> STORM-2648 couldn’t be included because there is no patch available for it
> yet. Once there is a patch available, it can go into the next release, so
> it’s certainly possible for it to be available in 1.1.2.
>
> -Taylor
>
>
> > On Jul 25, 2017, at 1:12 PM, Alexandre Vermeerbergen <
> avermeerber...@gmail.com> wrote:
> >
> > Hello,
> >
> > -1 (non binding)
> >
> > Maybe it's a bit selfish, but I really count on
> > https://issues.apache.org/jira/browse/STORM-2648 being part of Storm
> 1.1.1,
> > because we have a requirement to use Kafka 0.10 consumer in topologies
> > requiring at most once behavior.
> >
> > We understood that we could use storm-kafka-client with autocommit, but
> > then we're missing ack/fails and complete latency.
> >
> > We know that we could by-pass this limitation by implementing our own
> Kafka
> > 0.10 spout, but if possible it would be great to have Storm 1.1.1's storm
> > kafka client cover the need of "at most once" requirements.
> >
> > Would it be possible to have this STORM-2648 to be part of 1.1.1 ?
> >
> > Best regards,
> > Alexandre Vermeerbergen
> >
> >
> > 2017-07-25 18:24 GMT+02:00 P. Taylor Goetz :
> >
> >> Yes. There were a number of important patches present in 1.x-branch that
> >> were not in 1.1.x-branch.
> >>
> >> When I tried to merge 1.x-branch to 1.1.x-branch, I ran into unexpected
> >> conflicts. I thought about deleting 1.1.x-branch and recreating it, but
> >> decided against it, not wanting lose changes there that we might want to
> >> keep in case we wanted to revisit the contents of that branch. In the
> end I
> >> decided to cut the release from 1.x-branch.
> >>
> >> Jungtaek, I believe you created 1.1.x-branch… Do you know why/how it
> >> diverged?
> >>
> >> -Taylor
> >>
> >>> On Jul 25, 2017, at 12:08 PM, Stig Rohde Døssing <
> stigdoess...@gmail.com>
> >> wrote:
> >>>
> >>> Is it on purpose that this is cut from 1.x-branch and not 1.1.x-branch?
> >>>
> >>> 2017-07-25 17:09 GMT+02:00 P. Taylor Goetz :
> >>>
>  This is a call to vote on releasing Apache Storm 1.1.1 (rc1)
> 
>  Full list of changes in this release:
> 
>  https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
>  plain;f=CHANGELOG.md;hb=88f0b8a45553ea960164fab18c736a5cdbae8e58
> 
>  The tag/commit to be voted upon is v1.1.1:
> 
>  https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=
>  89bf57855806d84dba8d9b7fe6e76f9074a6aa19;hb=
> >> 88f0b8a45553ea960164fab18c736a
>  5cdbae8e58
> 
>  The source archive being voted upon can be found here:
> 
>  https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
>  1.1-rc1/apache-storm-1.1.1-src.tar.gz
> 
>  Other release files, signatures and digests can be found here:
> 
>  https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.1-rc1/
> 
>  The release artifacts are signed with the following key:
> 
>  https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
>  plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> 
>  The Nexus staging repository for this release is:
> 
>  https://repository.apache.org/content/repositories/
> orgapachestorm-1049
> 
>  Please vote on releasing this package as Apache Storm 1.1.1.
> 
>  When voting, please list the actions taken to verify the release.
> 
>  This vote will be open for at least 72 hours.
> 
>  [ ] +1 Release this package as Apache Storm 1.1.1
>  [ ]  0 No opinion
>  [ ] -1 Do not release this package because...
> 
>  Thanks to everyone who contributed to this release.
> 
>  -Taylor
> >>
> >>
>
>

Re: [VOTE] Release Apache Storm 1.1.1 (rc1)

[Bobby Evans]

I built from source (v1.1.1 88f0b8a45553ea960164fab18c736a5cdbae8e58) ran all 
of the unit tests and ran some manual tests.
I am +1 on the release.


- Bobby


On Tuesday, July 25, 2017, 11:14:14 AM CDT, Kishorkumar Patil 
 wrote:

I built and ran some manual tests. 
+1 to release this package. 

-Kishor


On Tuesday, July 25, 2017, 12:08:32 PM EDT, Stig Rohde Døssing 
 wrote:

Is it on purpose that this is cut from 1.x-branch and not 1.1.x-branch?

2017-07-25 17:09 GMT+02:00 P. Taylor Goetz :

> This is a call to vote on releasing Apache Storm 1.1.1 (rc1)
>
> Full list of changes in this release:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> plain;f=CHANGELOG.md;hb=88f0b8a45553ea960164fab18c736a5cdbae8e58
>
> The tag/commit to be voted upon is v1.1.1:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=
> 89bf57855806d84dba8d9b7fe6e76f9074a6aa19;hb=88f0b8a45553ea960164fab18c736a
> 5cdbae8e58
>
> The source archive being voted upon can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> 1.1-rc1/apache-storm-1.1.1-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.1-rc1/
>
> The release artifacts are signed with the following key:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1049
>
> Please vote on releasing this package as Apache Storm 1.1.1.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 1.1.1
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor

Re: [VOTE] Release Apache Storm 1.0.4 (rc1)

[Bobby Evans]

I built from source (v1.0.4 a5e1c154b5b2ae74fd78bf10d4c130afb1ad4513), ran all 
of the unit tests, and ran a few manual tests on my laptop.
I am +1 on the release.


- Bobby


On Tuesday, July 25, 2017, 6:27:18 AM CDT, Stig Rohde Døssing 
 wrote:

The linked changelog (https://git-wip-us.apache.org/repos/asf?p=storm.git;a=
blob_plain;f=CHANGELOG.md;h=50878fab679973a1230466920006dc0746ffddd5;hb=
eac433b0beb3798c4723deb39b3c4fad446378f4) seems to contain only changes up
to 1.0.3. It's just a problem with the link, the changelog in the src tar
is correct.

Unpacked the binary tar to a new directory.
Started Nimbus, a supervisor and the UI daemon via the storm.cmd script.
The cluster was using the default configuration.
Unpacked the src tar, and built ExclamationTopology and the
storm-kafka-client KafkaSpoutTopologyMainNamedTopics against the Nexus
staging repository.
Submitted storm-starter ExclamationTopology
* Changed log level and verified that the log level changed
* Tried to use Logviewer, but this fails on Windows due to
"java.util.regex.PatternSyntaxException:
Unexpected internal error near index 1
 \
 ...
 at org.apache.storm.daemon.logviewer$get_topo_port_
workerlog.invoke(logviewer.clj:123)". This bug isn't new, so it's not a
blocker IMO. I think it's probably fixed on master with
https://github.com/apache/storm/pull/2204
* Deactivated/activated and killed the topology, this worked as expected
Submitted storm-kafka-client KafkaSpoutTopologyMainNamedTopics using the
directions at https://github.com/apache/storm/tree/v1.0.4/external/
storm-kafka-client#build-and-run-bundled-examples, running against a local
Kafka instance.

The topologies seemed to run fine once I got past a few Windows specific
bugs:
* https://issues.apache.org/jira/browse/STORM-2451 which has a workaround
listed on the issue. I don't know if it makes sense to pull the bugfix back
into 1.0.x?
* There's an error log when log files are rolled, seemingly due to Windows'
file locking behavior. Seems like more of an annoyance than a real issue,
since the files appear to roll correctly. The log is "STDERR Thread-2
[INFO] 2017-07-25 13:00:45,639 Thread-5-exclaim2-executor[7 7] ERROR Unable
to delete file E:\apache-storm-1.0.4\logs\workers-artifacts\production-
topology-1-1500978841\6700\worker.log: java.nio.file.FileSystemException
E:\apache-storm-1.0.4\logs\workers-artifacts\production-
topology-1-1500978841\6700\worker.log: The process cannot access the file
because it is being used by another process."

Other minor issues:
* Noticed in the src tar that the integration-test pom is pointing to
1.0.3-SNAPSHOT instead of 1.0.4. This is also an issue on the 1.0.x-branch
branch on github.
* Non-HA Nimbus won't start when previously deployed topologies are not in
storm-local, giving the message "o.a.s.zookeeper main-EventThread [INFO]
code for all active topologies not available locally, giving up
leadership.". This is annoying when trying to do a rolling upgrade, since
it is easy to screw up by installing the new Storm version in a new
directory rather than overwriting the old installation. I think we should
add storm.local.dir to the comments in conf/storm.yaml and encourage people
to set it to a directory outside the storm dir.
* It looks like the blob store may be using the wrong base path when
storm.local.dir is relative. I get the directory
STORM_HOME/bin/storm-local/blobs
in addition to the expected STORM_HOME/storm-local. I'm guessing the path
is based on which working directory "storm nimbus/supervisor" is being run
from.

I think since most of these issues have workarounds and were likely present
in 1.0.3, they shouldn't block the release.

It might make sense to fix the integration-test pom and maybe pull
STORM-2451 back too. Since those issues are so minor I don't want to block
the release for that alone though. +1

2017-07-24 20:24 GMT+02:00 P. Taylor Goetz :

> This is a call to vote on releasing Apache Storm 1.0.4 (rc1)
>
> Full list of changes in this release:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> plain;f=CHANGELOG.md;h=50878fab679973a1230466920006dc0746ffddd5;hb=
> eac433b0beb3798c4723deb39b3c4fad446378f4
>
> The tag/commit to be voted upon is v1.0.4:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> plain;f=CHANGELOG.md;hb=a5e1c154b5b2ae74fd78bf10d4c130afb1ad4513
>
> The source archive being voted upon can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> 0.4-rc1/apache-storm-1.0.4-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.4-rc1/
>
> The release artifacts are signed with the following key:
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> 

Re: The #storm-user IRC channel

[Bobby Evans]

I agree if it is not being use then lets remove it.


- Bobby


On Thursday, July 20, 2017, 9:57:57 AM CDT, Jungtaek Lim <kabh...@gmail.com> 
wrote:

I don't see the benefit of keeping IRC channel for users, so OK to remove
it from anywhere in Storm site or code repo.

Btw, Apache Beam maintains Slack channel but it's likely for contributors
and committers. Personally I like the approach, but in order to make this
successful, many of us need to tolerate having one more interactive
(ping-able) channel. If they're dedicated to Storm project it doesn't
matter, but if not it might really matter.

- Jungtaek Lim (HeartSaVioR)

2017년 7월 20일 (목) 오후 11:30, Bobby Evans <ev...@yahoo-inc.com>님이 작성:

> Moving to the dev mailing list as it is more than just the PMC that
> provides support for storm.
>
>
> - Bobby
>
>
>
> On Wednesday, July 19, 2017, 6:27:12 PM CDT, Hugo Louro <
> hmclo...@gmail.com> wrote:
>
>
> The first question is, do we want to provide chat support? It has the pro
> of being more interactive, but the con of making info more disperse.
>
> If the decision is to go with having an official chat, one can evaluate if
> there are better chat and/or more integrated alternatives, e.g Gitter
> <https://en.wikipedia.org/wiki/Gitter>.
>
> Thanks,
> Hugo
> On Wed, Jul 19, 2017 at 2:15 PM, Kyle Nusbaum <knusb...@yahoo-inc.com>
> wrote:
>
> We've had a #storm-user channel on Freenode forever, but we seem to have
> almost totally abandoned it. We still direct users there on our website: 
> http://storm.apache.org/
> getting-help.html <http://storm.apache.org/getting-help.html> but the
> information found in the channel is terribly out of date and incorrect.
>
>
> I peek in there every once in a while, and users frequently show up with
> questions. Probably many are going unanswered. I wonder if it makes our
> project seem a little dead if the help channel we list on our web page is
> so out of date and inactive.
>
> Currently Nathan is the only user with permission to change anything. I
> think we should either update the info on the channel and maybe try and do
> something with it, or remove it from the website.
>
> Thanks,
> -- Kyle
>
>

Re: The #storm-user IRC channel

[Bobby Evans]

Moving to the dev mailing list as it is more than just the PMC that provides 
support for storm.


- Bobby


On Wednesday, July 19, 2017, 6:27:12 PM CDT, Hugo Louro  
wrote:

The first question is, do we want to provide chat support? It has the pro of 
being more interactive, but the con of making info more disperse.
If the decision is to go with having an official chat, one can evaluate if 
there are better chat and/or more integrated alternatives, e.g Gitter.
Thanks,Hugo
On Wed, Jul 19, 2017 at 2:15 PM, Kyle Nusbaum  wrote:

We've had a #storm-user channel on Freenode forever, but we seem to have almost 
totally abandoned it. We still direct users there on our website: 
http://storm.apache.org/ getting-help.html but the information found in the 
channel is terribly out of date and incorrect.
I peek in there every once in a while, and users frequently show up with 
questions. Probably many are going unanswered. I wonder if it makes our project 
seem a little dead if the help channel we list on our web page is so out of 
date and inactive. 
Currently Nathan is the only user with permission to change anything. I think 
we should either update the info on the channel and maybe try and do something 
with it, or remove it from the website.

Thanks,
-- Kyle

Re: [DISCUSS] Ideas for resolving storm-drpc-server compilation issue on IDE

[Bobby Evans]

I am fine with a separate project for relocated dependencies (or even just 
separate packages, you do a maven install of them and not include them in the 
IDE at all).  Shading still has some drawbacks, but I think in a few cases it 
makes since.  I would prefer it if we picked a very small number of 
dependencies that cause people issues and just shade those.  Guava is the big 
one that I worry about. Netty is a possibility and I think asm would be 
another, but it is a transitive dependency so it would require us with our own 
version of kryo exposing the kryo API but pulling in a shaded asm.  
The servlet-api concerns me, but it looks like it is tied to the 
IHttpCredentialsPlugin which should move to the server package anyways.

The rest I am not concerned about, are things that are exposed to end users, or 
are for test and not actually shipped.
$ mvn dependecy:tree...
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ storm-client ---
[INFO] org.apache.storm:storm-client:jar:2.0.0-SNAPSHOT
[INFO] +- uk.org.lidalia:sysout-over-slf4j:jar:1.0.2:compile
[INFO] +- org.slf4j:slf4j-api:jar:1.7.21:compile 
[INFO] +- org.apache.logging.log4j:log4j-api:jar:2.8.2:compile
[INFO] +- org.apache.logging.log4j:log4j-core:jar:2.8.2:compile
[INFO] +- org.apache.logging.log4j:log4j-slf4j-impl:jar:2.8.2:compile
[INFO] +- org.slf4j:log4j-over-slf4j:jar:1.6.6:compile
[INFO] +- com.google.guava:guava:jar:16.0.1:compile
[INFO] +- org.apache.thrift:libthrift:jar:0.9.3:compile
[INFO] |  \- org.apache.httpcomponents:httpcore:jar:4.4.1:compile
[INFO] +- commons-io:commons-io:jar:2.5:compile
[INFO] +- commons-lang:commons-lang:jar:2.5:compile
[INFO] +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] +- com.lmax:disruptor:jar:3.3.2:compile
[INFO] +- com.googlecode.json-simple:json-simple:jar:1.1:compile
[INFO] +- org.yaml:snakeyaml:jar:1.11:compile
[INFO] +- io.netty:netty:jar:3.9.0.Final:compile
[INFO] +- com.esotericsoftware:kryo:jar:3.0.3:compile
[INFO] |  +- com.esotericsoftware:reflectasm:jar:1.10.1:compile
[INFO] |  |  \- org.ow2.asm:asm:jar:5.0.3:compile
[INFO] |  +- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO] |  \- org.objenesis:objenesis:jar:2.1:compile
[INFO] +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile
[INFO] |  \- jline:jline:jar:0.9.94:compile
[INFO] +- org.apache.curator:curator-framework:jar:2.12.0:compile
[INFO] +- org.jgrapht:jgrapht-core:jar:0.9.0:compile
[INFO] +- javax.servlet:servlet-api:jar:2.5:compile
[INFO] +- org.apache.httpcomponents:httpclient:jar:4.3.3:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.1.3:compile
[INFO] |  \- commons-codec:commons-codec:jar:1.6:compile
[INFO] +- org.apache.curator:curator-client:jar:2.12.0:compile
[INFO] +- junit:junit:jar:4.11:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- org.mockito:mockito-core:jar:1.9.5:test
[INFO] \- org.hamcrest:hamcrest-library:jar:1.3:test
- Bobby


On Wednesday, July 12, 2017, 9:45:43 AM CDT, Jungtaek Lim <kabh...@gmail.com> 
wrote:

I'd like to bump on this again, since we have a few huge issues for Storm
2.0.0, and this issue is a kind of regression and effectively blocker.
(Please note that current master branch removes shading for some libraries
to make IDE happy.)

At that time I didn't consider option 2 as possible solution, but now Flink
is going with this option, and I can't find reason to not doing this.

* Repository: https://github.com/apache/flink-shaded
* Discussion thread:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Changing-Flink-s-shading-model-td17419.html

Thought?

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 3월 31일 (금) 오후 3:12, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> Bobby,
>
> I've worked on separating worker and daemon classpath.
>
> - Issue: STORM-2441: Break down 'storm-core' to extract client (worker)
> artifacts <http://issues.apache.org/jira/browse/STORM-2441>
> - PR: https://github.com/apache/storm/pull/2034
>
> I don't address your suggestion about "classpath selection" and "hiding
> local mode". Please file issues if you would like to address.
>
> Btw, I exclude artifacts from shade & relocation list so still need to
> address dependency issue.
>
> Folks,
>
> any other ideas or opinions around dependency issue?
>
> IMHO Option 2 is clearer but not sure where we can create a new git repo
> (ASF git or even outside), and also it's not against LICENSEs to repackage
> shade & relocated artifacts to Maven.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2017년 3월 29일 (수) 오후 10:42, Bobby Evans <ev...@yahoo-inc.com.invalid>님이 작성:
>
> I am fine with those changes so long as we finish the separation of worker
> and daemon classpaths.  Otherwise we have made some very big changes for
> our end users that are going to have a hard time upgrading.
> If all we support is the optio

Re: New Committer/PMC Member: Stig Rohde Døssing

[Bobby Evans]

Welcome

- Bobby


On Saturday, July 15, 2017, 8:01:47 PM CDT, Matthias J. Sax  
wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Congrats!

On 7/14/17 6:17 PM, Xin Wang wrote:
> Congrats Stig!
> 
> - Xin
> 
> 2017-07-15 9:13 GMT+08:00 Satish Duggana
> :
> 
>> Congrats Stig!
>> 
>> ~Satish.
>> 
>> On Sat, Jul 15, 2017 at 4:18 AM, Jungtaek Lim 
>> wrote:
>> 
>>> Welcome Stig! Well deserved.
>>> 
>>> - Jungtaek Lim (HeartSaVioR) On Sat, 15 Jul 2017 at 4:25 AM
>>> Hugo Da Cruz Louro <
>> hlo...@hortonworks.com>
>>> wrote:
>>> 
 Welcome Stig. Looking forward to collaborating with you.
 
 Hugo
 
> On Jul 14, 2017, at 11:44 AM, P. Taylor Goetz
> 
>>> wrote:
> 
> Please join me in congratulating the latest Committer and
> PMC Member,
 Stig Rohde Døssing.
> 
> Stig has been very active contributing patches to Storm’s
> Kafka
 integration, participating in code reviews, and answering
 questions on
>>> the
 mailing lists.
> 
> Welcome Stig!
> 
> -Taylor
> 
 
 
>>> 
>> 
> 
-BEGIN PGP SIGNATURE-
Comment: GPGTools - https://gpgtools.org

iQIYBAEBCgAGBQJZarriAAoJELz8Z8hxAGOi5s4P3j/XBh45/nCrpuydt5CB9pEQ
/VtGB/DZ1HfJiBE0E5TgzEkuktt5A1UBf2XbNtmUZ4v7Tpjb+Cq/o9KbPWLKacp8
5bDZxyKIlIqZKK11Tj0bNqx1F3aQuMMUtFowU6oVum2OZPBsCHJD2a7H9nG1qEaP
YorJCyq3k3EpakZjO70DPX3igM4Aj4d17FyGlHqU+R8NtoDrktTLeKPoakcX2ZHO
7nXecowDPaX2y9hGrWGhI4nYfYFhNHhjQE1KXqVOmLN9vnQjPYl8YdqSANiVMZwy
XWK1R0KWj1fifXoUZCFlrYk14XBpWotDkPQfMbxCaZoy4vfCsakLQDyA7JUa+ksz
sitjvB/E8ZEWX9jvfP8ruA0qaQFBwzIzdH4szXL555xJQjYlr1Tes2q6j6yaa7nu
8C/flDifHEnnqEqASiTfAyn9i4Oss02/TWq1m87GckGB9WY927hA4CaSiKPg8nLz
Q/SPERXH81GwNLQrQ2Q67DcRZU5QuFZ1/+yoL93Rz/qIQDBXm/KYg9OC1E0ineOe
r4tD0LmVbA6HAMSATJJ0gX8gJSjUpWNKfAA9jTlz8Upk1zZQSi62BAKYgpCJZhYn
7uFivFkz0B78Wq33IUBrwcIgS20RwNTwQPOatm9O3Ut8n8Y2HlakftLDDg8mWSkH
8wdWg3UcVODgzQA=
=qJd+
-END PGP SIGNATURE-

Re: Few observations related to KafkaSpout implementation (1.1.0)

[Bobby Evans]

I believe that "better" is relative most of the time.  Optimizing a failure 
case costs something, unless of course the code is doing something horrible and 
there truly is a better way to do it that uses less memory, cpu, etc.  If so 
please put up a pull request and lets fix the thing.  However most of the time 
getting better performance costs something.  If we want to avoid going back to 
kafka for a failed tuple then we need to store it somewhere.  The simplest way 
to do it would be to cache the tuple in memory, but that will cost as much 
memory as all of the outstanding tuples for the spout.  It is not a clear win 
if failures are not common.
How frequently should failures happen?  I don't think we have anything like 
that documented, because it depends on a number of things, but most of the time 
it indicates that there is something that needs to be fixed about the topology. 
 For me if I see any failures in a topology and we are not in the middle of a 
rolling upgrade I am concerned.  A lot of teams ignore errors because they are 
running a lambda architecture and will have batch stomp on the top of it later, 
but I still get very nervous when I see it.

- Bobby


On Monday, July 10, 2017, 2:33:52 PM CDT, chandan singh <cks07...@gmail.com> 
wrote:

Sorry about being cryptic there. What I meant is that it will be much
better if we don't make assumptions about frequency of failure rates in
topologies. I know it is more of a commonsense but out of curiosity, can
you point me to any Storm documentation which makes a comment on preferable
failure rates. I was suggesting if we can offer the user an optimization
through clean API, the user will be free to decide on the rationale of
using it.

On Tue, Jul 11, 2017 at 12:06 AM, Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:

> I'm not sure what assumptions you want to make that this is preventing, or
> why they would be helpful.
>
> - Bobby
>
>
> On Monday, July 10, 2017, 12:14:53 PM CDT, chandan singh <
> cks07...@gmail.com> wrote:
>
> Hi Stig & Bobby
>
> Thanks for confirming my understanding.
>
> 1) Ensuring that calls to nexTuple(), ack()  and fail() are non-blocking
> has been a guideline on http://storm.apache.org/
> releases/1.1.0/Concepts.html
> for long. Copying verbatim here : "The main method on spouts is nextTuple.
> nextTuple either emits a new tuple into the topology or simply returns if
> there are no new tuples to emit. It is imperative that nextTuple does not
> block for any spout implementation, because Storm calls all the spout
> methods on the same thread." I admit that there is some chance my
> interpretation is partially incorrect but I have been following it in a
> custom spout till now. Even though the objective is different, there is a
> similar hint on Kafka official documentation. Please see under heading "2.
> Decouple Consumption and Processing" on
> https://kafka.apache.org/0110/javadoc/index.html?org/apache/
> kafka/clients/consumer/KafkaConsumer.html.
> Essentially, a thread polls Kafka and spout thread gets the messages
> through a shared queue. If pre-fetching is present in Kafka (I will read
> about it further), I assume we do not have to fetch in another thread but I
> am not sure how does the pre-fetching behave with re-seeking before every
> poll.
>
> 2) @Bobby, you are correct in pointing what needs to be optimized but the
> facts, sometimes, prevent us from making assumptions. We do optimize our
> retry loop such that we don't poll the messages again. I especially see
> problems when combined with exponential back off.  I am not sure how
> difficult or clean will it be to expose some sort of configuration to allow
> such optimization?  Do you think it will be worth trying out something?
>
> Thanks
> Chandan
>

Re: Few observations related to KafkaSpout implementation (1.1.0)

[Bobby Evans]

I'm not sure what assumptions you want to make that this is preventing, or why 
they would be helpful.

- Bobby


On Monday, July 10, 2017, 12:14:53 PM CDT, chandan singh  
wrote:

Hi Stig & Bobby

Thanks for confirming my understanding.

1) Ensuring that calls to nexTuple(), ack()  and fail() are non-blocking
has been a guideline on http://storm.apache.org/releases/1.1.0/Concepts.html
for long. Copying verbatim here : "The main method on spouts is nextTuple.
nextTuple either emits a new tuple into the topology or simply returns if
there are no new tuples to emit. It is imperative that nextTuple does not
block for any spout implementation, because Storm calls all the spout
methods on the same thread." I admit that there is some chance my
interpretation is partially incorrect but I have been following it in a
custom spout till now. Even though the objective is different, there is a
similar hint on Kafka official documentation. Please see under heading "2.
Decouple Consumption and Processing" on
https://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html.
Essentially, a thread polls Kafka and spout thread gets the messages
through a shared queue. If pre-fetching is present in Kafka (I will read
about it further), I assume we do not have to fetch in another thread but I
am not sure how does the pre-fetching behave with re-seeking before every
poll.

2) @Bobby, you are correct in pointing what needs to be optimized but the
facts, sometimes, prevent us from making assumptions. We do optimize our
retry loop such that we don't poll the messages again. I especially see
problems when combined with exponential back off.  I am not sure how
difficult or clean will it be to expose some sort of configuration to allow
such optimization?  Do you think it will be worth trying out something?

Thanks
Chandan

Re: Few observations related to KafkaSpout implementation (1.1.0)

[Bobby Evans]

For #2 it is a question of what do you optimize for.  Storm typically assumes 
that failures should be rare so we optimize for that.  We keep the minimal 
information around to be able to replay the message, but not a lot more.  If 
you are getting lots of failures, you really should be more concerned about why 
you are getting failures and not so much with how the failures are impacting 
the performance of the spout.  Or to put it another way the best performance 
optimization is to stop doing the thing that is slow.  If failures are the 
slowest part the best thing for performance is to stop failing.


- Bobby


On Monday, July 10, 2017, 9:10:38 AM CDT, Stig Rohde Døssing 
 wrote:

Hi Chandan,

I'm going to assume we're talking about the storm-kafka-client spout, and
not the storm-kafka spout.

1) Yes, we do polling and committing in the spout thread. I'm not aware of
why that would be against spout best practices? Simplicity is definitely a
reason, but I don't think anyone has actually checked what the throughput
of a threaded implementation would be compared to the current
implementation. Keep in mind that the KafkaConsumer is not thread safe, so
a threaded implementation would still need to do all consumer interaction
via a single thread. Also as far as I know the KafkaConsumer does
prefetching of messages (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records#KIP-41:KafkaConsumerMaxRecords-Prefetching),
so most calls to poll should not take very long.

What kind of threading did you have in mind?

2) No, that seems correct. When a message fails the spout remembers the
offset of the failed message, but does not keep the message in memory. When
the message becomes ready for retry, the spout seeks the consumer back to
the failed message's offset and fetches it again. The consumer will fetch
that message plus any later messages before it catches back up to where it
was before the message failed.

We choose to poll for the failed message again because keeping it around in
memory could be a problem if there are many failed messages, since they'd
potentially need to hang around for a while depending on the configured
backoff.

There's maybe a potential optimization here where we could try to seek
directly to the latest emitted offset instead of fetching everything from
the failed message forward, but it's not something we've looked at
implementing. I haven't looked at whether this would work, or what kind of
edge cases there may be.

Do you have suggestions for improving/replacing this loop?

2017-07-10 15:39 GMT+02:00 chandan singh :

> Hi
>
> I hope I am using the right mailing list. Please advice if I am wrong.
>
> I have few observations about the KafkaSpout and feel that some of these
> lead to inefficiencies. It will be of great help if someone can throw some
> light on the rationale behind the implementation.
>
> 1) Kafka polling and committing offsets is done in the spout thread which
> is somewhat against the spout best practices. Is simplicity the reason
> behind this design? Am I missing something?
>
> 2)  Poll-iterate-commit-seek loop seems inefficient in recurrent failure
> scenarios. Let say the first massage fails. We will keep polling the same
> set of messages at least as many times as that message is retried and
> probably more if we are using exponential back-off. Did I misunderstand the
> implementation?
>
> Regards
> Chandan
>

Re: DRPC problem

[Bobby Evans]

You are going to need to look at the logs for your topology and the logs for 
the drpc server to see if there is anything in there that indicates what is 
happening.


- Bobby


On Wednesday, June 28, 2017, 11:21:14 PM CDT, sam mohel  
wrote:

I submitted topology in local without any problem , but in production mode i 
couldn't as you can see in ui zeros values in columns except execute columns .i 
got after sometimes in terminal drpcexecutionexception(msg:request timed out)my 
configurations are Machine A and Machine B storm.yaml in Machine A is 
storm.zookeeper.servers:     - "192.168.x.x"     nimbus.host : "192.168.x.x"
 supervisor.childopts: "-Xmx4g" 
 worker.childopts: "-Xmx4g" 
storm.yaml in Machine B is storm.zookeeper.servers:     - "192.168.x.x"     
nimbus.host : "192.168.x.x"
 supervisor.childopts: "-Xmx4g" 
 worker.childopts: "-Xmx4g" 
i set drpc in the code 
Config conf = new Config(); List dprcServers = new ArrayList(); 
     dprcServers.add("192.168.x.x") ; conf.put(Config.DRPC_SERVERS, 
dprcServers); conf.put(Config.DRPC_PORT, 3772);// distributed mode Config conf 
= createTopologyConfiguration( prop, true); LocalDRPC drpc = null; 
StormSubmitter.submitTopology( args[0], conf, buildTopology(drpc));             
client=new DRPCClient("192.168.x.x", 3772);
i used same ip address for storm.zookeeper.servers ,  nimbus.host ,dprcServers 
and DRPCClient . Is that wrong ? and i ran nimbus , drpc,ui in Machine A ,   I 
ran supervisor in Machine Bi appreciate any help , Thanks 
​

Re: [DISCUSS] Storm 2.0 Roadmap

[Bobby Evans]

+1.
If the 1.1 and 1.2 lines start to become difficult to maintain we can look at 
putting them in maintenance mode too once we have a 2.x release.
I am a little nervous about merging a new feature into 1.x branch without first 
going to master, but I hope that it will not be too much work to port it to 
master, and I trust the devs on that branch to do the right thing.
On a related note we have not done much with feature branches before so I am 
not sure what we want to do about merging in the new metrics API branch to 1.x. 
 I know for me I have not had time to keep up with the development work going 
on there.  I would at least like to have a pull request put up for review 
before we merge it in.  This would fit with our current bylaws that do not 
mention feature branches.  If all of the changes have already followed the 
review process then technically I think it is OK to just merge it in, but I 
still would like to take some time to look at the changes, and especially the 
new APIs.

- Bobby


On Wednesday, June 28, 2017, 1:53:34 AM CDT, Jungtaek Lim <kabh...@gmail.com> 
wrote:

That's great news that metrics work is ready!

I'm +1 to Taylor's proposal, but in order to respect semantic versioning, I
propose some modifications from Taylor's proposal:

- create 1.1.x-branch with target version 1.1.1-SNAPSHOT and port back only
bug fixes to the 1.1.x-branch
- change the target version of 1.x-branch to 1.2.0-SNAPSHOT

If we also agree above, I would like to volunteer the back-port work.

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 6월 28일 (수) 오전 10:09, Harsha <st...@harsha.io>님이 작성:

> +1 for above stated approach on releasing 1.2.0 with metrics
> -Harsha
>
> On Tue, Jun 27, 2017, at 12:17 PM, P. Taylor Goetz wrote:
> > The work on metrics is ready for a pull request to 1.x-branch from the
> > feature branch. I’ve held off because we haven’t reached consensus on a
> > path forward with the 1.x release lines .
> >
> > I’d like to propose the following for the 1.x line:
> >
> > 1. Create a branch for 1.2 so we have a branch to review the metrics
> > stuff.
> > 2. Release 1.1.1
> > 3. Review/merge metrics work. Port metrics to master.
> > 4. Release 1.2.0
> > 5. Put the entire 1.x line into maintenance mode. Drop support for 1.0.x.
> > (we would only support 1.2.x and 1.1.x which are very closely aligned).
> >
> > Dropping support for 1.0.x line would eliminate the need to maintain one
> > of the fairly heavily diverged branches. The 1.2.x and 1.1.x would be
> > very closely aligned. I just up merged metrics_v2 against 1.x-branch
> > after a while, and there were no conflicts.
> >
> > That would give us a little more bandwidth to focus on 2.0 and needed bug
> > fixes to the 1.x line like some of the issues raised with
> > storm-kafka-client. We could even start releasing alpha/beta versions of
> > 2.0 in parallel to the steps above.
> >
> > Any thoughts on that approach?
> >
> > -Taylor
> >
> >
> > > On Jun 24, 2017, at 1:21 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
> > >
> > > Yes I prefer option 1, but it might depend on the progress of metrics
> V2.
> > > If it can be done within predictable near future I'm OK to pick option
> 2,
> > > but if not, we may be better to focus releasing 2.0.0 and make it
> really
> > > happen.
> > >
> > > Whichever we go, I feel it's time to track remaining work on Storm
> 2.0.0. I
> > > found some bugs on master branch so filed issues, and we've remaining
> port
> > > work (UI and logviewer). We've some other improvements target for
> 2.0.0:
> > > worker redesign, beam integration, and so on, and we don't track its
> > > progress at all. I don't think we should wait for features which
> progress
> > > is not transparent (in other words we don't know when it will be
> finished).
> > >
> > > - Jungtaek Lim (HeartSaVioR)
> > >
> > > 2017년 6월 24일 (토) 오전 5:19, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
> > >
> > >> Bobby/Jungtaek,
> > >>
> > >> Are you saying you want to forego the 1.2 “metrics_v2” release and
> include
> > >> it only in 2.0? (I ask because that work is already based on
> 1.x-branch,
> > >> and forward-porting it to master is relatively simple.) I’d kind of
> like
> > >> that work go out soon.
> > >>
> > >> If we go with option 1, I would want to see a 2.0 release (even if
> it’s a
> > >> “beta” or “preview) before putting the 1.x line into maintenance mode.
> > >>
> > >> -Taylor
> > >>
> > >>>

Re: [DISCUSS] Storm 2.0 Roadmap

[Bobby Evans]

Yes I would prefer to see us get a 2.x alpha release out.  Are the new metrics 
APIs going to be a breaking change?  I assume not, because we wouldn't want to 
put them in 1.x otherwise.  If that is the case then we don't need to wait for 
them before doing a 2.x release.

- Bobby


On Friday, June 23, 2017, 3:19:28 PM CDT, P. Taylor Goetz <ptgo...@gmail.com> 
wrote:

Bobby/Jungtaek,

Are you saying you want to forego the 1.2 “metrics_v2” release and include it 
only in 2.0? (I ask because that work is already based on 1.x-branch, and 
forward-porting it to master is relatively simple.) I’d kind of like that work 
go out soon. 

If we go with option 1, I would want to see a 2.0 release (even if it’s a 
“beta” or “preview) before putting the 1.x line into maintenance mode. 

-Taylor

> On Jun 23, 2017, at 9:51 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID> wrote:
> 
> I see 2 ways to address this.
> 1) We put the 1.x line into maintenance mode like with 0.10.  We don't 
> backport anything except bug fixes.2) We backport a lot of the backwards 
> compatible changes from 2.x to 1.x.
> My personal preference is 1.  It makes it clear the direction we want to go 
> in.  The biggest issue is that we probably would want to do a 2.x release 
> sooner rather then later.  Even if we don't get all of the features that 
> people want, if we just get a release out we can add in new features if they 
> are backwards compatible, or we can create a 3.x line that would have the 
> breaking changes in it.
> 
> - Bobby
> 
> 
> On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <kabh...@gmail.com> 
> wrote:
> 
> I'd like to bump this again instead of initiating new discussion thread.
> 
> I had having hard time to create and apply pull requests for both master
> and 1.x-branch and that's really painful and sometimes blocker for me to do
> merge step.
> Two branches are heavily diverged more than between 0.10 and 1.0.0, even
> IDE can't switch between the branch smoothly. We didn't even address
> checkstyle issue yet, but after addressing, it could be "completely"
> diverged. JDK version is another major issue, since the pull requests
> targeted for master branch are not checked against JDK 7, and some of them
> make some issues regarding JDK version while porting back.
> 
> So personally I really would like to see the plan for 1.x version line
> changed - skipping any minor releases including 1.2.0 - and have epic issue
> for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
> proposed plan was having 2.0.0 directly after 1.0.0)
> 
> Would like to hear everyone's opinions. If we have consensus to not having
> any minor releases for 1.x version line, I will not port back non-bugfix
> pull requests to 1.x-branch, and guide contributors to create pull requests
> against master branch, not 1.x version line.
> 
> Thanks,
> Jungtaek Lim (HeartSaVioR)
> 
> 2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <avermeerber...@gmail.com>님이
> 작성:
> 
>> +1 for Roshan's suggestion : in our Storm 1.x based supervision system,
>> we're very interested anything that can provide better throughput.
>> 
>> 2017-06-03 18:12 GMT+02:00 Roshan Naik <ros...@hortonworks.com>:
>> 
>>> For 2.0 beta … it would be good to incorporate some of the Worker
>>> improvements (STORM-2284)  IMO. Changes to messaging subsystem can be
>>> delivered sooner and my in-progress implementation suggests that it will
>>> yield substantial latency improvements. The 2.0 beta phase would really
>>> help kick the tires on the revised messaging system and the performance
>>> improvements will also be a good incentive for trying out the 2.0 line.
>>> 
>>> I notice multiple other bottlenecks that are holding back throughput a
>>> lot, which can be addressed in a subsequent 2.x minor release.
>>> -roshan
>>> 
>>> 
>>> On 6/3/17, 7:20 AM, "Jungtaek Lim" <kabh...@gmail.com> wrote:
>>> 
>>>    I also would love to see metrics V2 code sooner or later too. If we
>>> can get
>>>    it before releasing 2.0.0 that will be great, and then maybe we could
>>> just
>>>    move toward to 2.0.0, not adding any improvements to 1.x version
>> line.
>>>    (And that's what I would want to.)
>>> 
>>>    If we would really want to have 1.2.0, I suggest that we make the
>> 1.1.1
>>>    version correct right now rather than after releasing 1.1.1. We also
>>> merged
>>>    non-bugfix things to 1.x-branch but that's not what users expect. I
>>> agree
>>>    that work may be painful, but anyway need to do it.

Re: [DISCUSS] Storm 2.0 Roadmap

[Bobby Evans]

I see 2 ways to address this.
1) We put the 1.x line into maintenance mode like with 0.10.  We don't backport 
anything except bug fixes.2) We backport a lot of the backwards compatible 
changes from 2.x to 1.x.
My personal preference is 1.  It makes it clear the direction we want to go in. 
 The biggest issue is that we probably would want to do a 2.x release sooner 
rather then later.  Even if we don't get all of the features that people want, 
if we just get a release out we can add in new features if they are backwards 
compatible, or we can create a 3.x line that would have the breaking changes in 
it.

- Bobby


On Thursday, June 22, 2017, 7:39:55 PM CDT, Jungtaek Lim <kabh...@gmail.com> 
wrote:

I'd like to bump this again instead of initiating new discussion thread.

I had having hard time to create and apply pull requests for both master
and 1.x-branch and that's really painful and sometimes blocker for me to do
merge step.
Two branches are heavily diverged more than between 0.10 and 1.0.0, even
IDE can't switch between the branch smoothly. We didn't even address
checkstyle issue yet, but after addressing, it could be "completely"
diverged. JDK version is another major issue, since the pull requests
targeted for master branch are not checked against JDK 7, and some of them
make some issues regarding JDK version while porting back.

So personally I really would like to see the plan for 1.x version line
changed - skipping any minor releases including 1.2.0 - and have epic issue
for 2.0.0 and just go ahead. That was our proposed plan indeed. (even
proposed plan was having 2.0.0 directly after 1.0.0)

Would like to hear everyone's opinions. If we have consensus to not having
any minor releases for 1.x version line, I will not port back non-bugfix
pull requests to 1.x-branch, and guide contributors to create pull requests
against master branch, not 1.x version line.

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 6월 4일 (일) 오전 1:17, Alexandre Vermeerbergen <avermeerber...@gmail.com>님이
작성:

> +1 for Roshan's suggestion : in our Storm 1.x based supervision system,
> we're very interested anything that can provide better throughput.
>
> 2017-06-03 18:12 GMT+02:00 Roshan Naik <ros...@hortonworks.com>:
>
> > For 2.0 beta … it would be good to incorporate some of the Worker
> > improvements (STORM-2284)  IMO. Changes to messaging subsystem can be
> > delivered sooner and my in-progress implementation suggests that it will
> > yield substantial latency improvements. The 2.0 beta phase would really
> > help kick the tires on the revised messaging system and the performance
> > improvements will also be a good incentive for trying out the 2.0 line.
> >
> > I notice multiple other bottlenecks that are holding back throughput a
> > lot, which can be addressed in a subsequent 2.x minor release.
> > -roshan
> >
> >
> > On 6/3/17, 7:20 AM, "Jungtaek Lim" <kabh...@gmail.com> wrote:
> >
> >    I also would love to see metrics V2 code sooner or later too. If we
> > can get
> >    it before releasing 2.0.0 that will be great, and then maybe we could
> > just
> >    move toward to 2.0.0, not adding any improvements to 1.x version
> line.
> >    (And that's what I would want to.)
> >
> >    If we would really want to have 1.2.0, I suggest that we make the
> 1.1.1
> >    version correct right now rather than after releasing 1.1.1. We also
> > merged
> >    non-bugfix things to 1.x-branch but that's not what users expect. I
> > agree
> >    that work may be painful, but anyway need to do it.
> >
> >    - Jungtaek Lim (HeartSaVioR)
> >
> >    2017년 6월 3일 (토) 오전 3:49, Bobby Evans <ev...@yahoo-inc.com.invalid>님이
> > 작성:
> >
> >    > I would love to see the metrics V2 code come out sooner rather than
> >    > later.  +1.
> >    > My biggest blocker for a 1.x release is
> >    > https://github.com/apache/storm/pull/2142  Even though the pull
> > request
> >    > says it is minor it showed that we messed up pushing back some
> > changes for
> >    > pacemaker to open source (the code does not run at all which for me
> > is a
> >    > blocker) and I really want to get that fully fixed/tested before
> > another
> >    > release.
> >    > As for 2.x I think we are very close to being able to so a 2.x
> alpha
> >    > release.  I would like to see metrics V2 merged in simply because
> it
> > is a
> >    > big change for user facing APIs.  But after that I would love to
> see
> > us
> >    > starting to push forward on getting that out.
> >    >
> >    >
> >    > - Bobby
> >    >

Re: Which thread(s) do the tuple fanout?

[Bobby Evans]

Each bolt/spout has 2 threads of their own.  In one thread the bolt/spout 
executes on.  A second thread then batches/routes the tuples where they need to 
go.  This may be putting them in the input queue for another bolt/spout in the 
current worker or putting them on a queue to be sent to a different worker.


- Bobby


On Wednesday, June 14, 2017, 2:13:29 PM CDT, Madhav Ancha (BLOOMBERG/ 919 3RD 
A)  wrote:

Which thread(s) do the tuple fanout from a source blot to the subscriber bolts 
please.

Thanks
Madhav.

Re: [Proposal] Breaking down AutoCreds implementations into two classes

[Bobby Evans]

In practice I am OK with the change, as it does not impact any of the end user 
facing configs.  The incompatibility is for administrators, which I think is 
fine.  I am not convinced of some of the design choices, but those can be 
discussed on the JIRA.


- Bobby


On Thursday, June 15, 2017, 1:34:55 AM CDT, Jungtaek Lim  
wrote:

Hi devs,

While debugging storm-autocreds issue, I had hard time to follow the code
path since the class implements two different kinds of interfaces together.

Auto* classes implement three interfaces for two kinds of purposes:
- INimbusCredentialPlugin, ICredentialsRenewer (Nimbus plugin)
- IAutoCredentials (worker, topology submitter)

and it shares implementation of prepare().

I've already prepared PR for proposal:
https://github.com/apache/storm/pull/2161

Please note that I also removed cluster-wide configurations, since it was
also confusing me, and may open potential security vulnerability. (Even
small code change could open it)

This breaks backward compatibility, so would like to see everyone is OK to.

Would like to hear your thought.

Thanks,
Jungtaek Lim (HeartSaVioR)

Re: About client session timed out, have not

[Bobby Evans]

Turn on garbage collection logging on your workers.  We have found that GC is 
typically the culprit in cases like this.  A large stop the world GC happens 
which makes all threads wait for it so the heartbeats time out and things get 
rescheduled.
Once you know it is GC then you need to take a look at increasing your heap 
(probably a good first step) and then monitoring what is on the heap and do you 
have a memory leak or what?  This can some times be difficult as storm has lots 
of buffers so you may also want to turn on the logging metrics consumer so you 
can start to see if some of the buffers in your topology are full and if so by 
how much.

- Bobby


On Wednesday, June 14, 2017, 3:59:05 AM CDT, Li Wang  
wrote:

I merged to track the source of error. I found the first error message is
on nimbus.log:  "o.a.s.d.nimbus [INFO] Executor T1-1-1497424747:[8 8] not
alive". The nimbus detects some executors are not alive and thus make
reassignment which cause the worker restart.

I did not find any error in the supervisor and worker logs that would
causes an executor to be timeout. Since my topology is data-intensive, is
it possible that the heartbeat messages was not delivered in time and thus
caused the timeout of executors in nimbus? How does the an executor send
heartbeat to the nimbus? Is it through zookeeper, or just transmit via
netty like a metric tuple?

I will first try to solve the problem by increasing timeout value and will
let you guys know if it works.

Regards,
Li Wang

On 14 June 2017 at 11:27, Li Wang  wrote:

> Hi all,
>
> We deployed a data-intesive topology which involves in a lot of HDFS
> access via HDFS client. We found that after the topology has been executed
> for about half an hour, the topology throughput occasionally drops to zero
> for tens of seconds and sometimes the worker is shutdown without any error
> messages.
>
> I checked the log thoroughly, found nothing wrong but a info message  that
> reads “ClientCnxn [INFO] Client session timed out, have not head from
> server in 1ms for sessioned …”. I am not sure how this message is
> related to the wired behavior of my topology. But every time my topology
> behaves abnormally, this message happens to show up in the log.
>
> Any help or suggestion is highly appreciated.
>
> Thanks,
> Li Wang.
>
>
>

Re: [Discussion]: Storm Improvemement Proposal (SIP) to discuss changes

[Bobby Evans]

Can you please explain how KIP currently works and how you would like to see 
something similar in storm?
If we make the process more formal we will probably have less people 
contributing, but we will probably have better overall patches.  It is a 
balancing act and having never used KIP I would like to understand it better 
before going all in on it.
- Bobby


On Friday, June 9, 2017, 4:09:38 PM CDT, Stig Døssing 
 wrote:

This sounds like a good idea. KIPs seem to work well for Kafka. It's easy
for discussions to get lost or just not seen on the mailing list.

2017-06-09 21:36 GMT+02:00 Harsha :

> Hi All,
>          We’ve seen good adoption of KIP approach in Kafka community
>          and would like to see we adopt the similar approach for storm
>          as well.
> Its hard to keep track of proposed changes and mailing list threads to
> know what all changes that are coming into  and what design/backward
> incompatible changes being approved.  It will be good to have this
> documented and go through discussion then Vote phase to get them
> approved before we merge the PRs. This will keep everyone informed of
> what changes happened even if they are not following the mailing list
> they can go to wiki to see the list of changes went into a release.
> Community overall will be well informed of the changes as well. Would
> like to hear your thoughts.
>
> Thanks,
> Harsha
>

Re: [DISCUSS] Storm 2.0 Roadmap

[Bobby Evans]

I would love to see the metrics V2 code come out sooner rather than later.  +1.
My biggest blocker for a 1.x release is 
https://github.com/apache/storm/pull/2142  Even though the pull request says it 
is minor it showed that we messed up pushing back some changes for pacemaker to 
open source (the code does not run at all which for me is a blocker) and I 
really want to get that fully fixed/tested before another release.
As for 2.x I think we are very close to being able to so a 2.x alpha release.  
I would like to see metrics V2 merged in simply because it is a big change for 
user facing APIs.  But after that I would love to see us starting to push 
forward on getting that out.


- Bobby


On Friday, June 2, 2017, 1:39:46 PM CDT, P. Taylor Goetz  
wrote:

I’d like to bump this thread and start a discussion around our next release. 
Here are my thoughts.

There are a number of important fixes in 1.x-branch so I’d like to consider 
releasing 1.1.1 soon. I’d appreciate input on any open issues that should be 
resolved for that release.

I’d like us to consider releasing the metrics improvements in STORM-2153 [1] as 
version 1.2.0. That work is in the metrics_v2 feature branch right now and 
would need to be reviewed and merged. That work is against the 1.x-branch right 
now. I would recommend porting it to master *after* the review/merge since 
there will likely be changes as a result of the review.

> Maybe related to or not, but would we want to create a new branch
> "1.1.x-branch", and make "1.x-branch" target for 1.2?



If wee agree to the above, I would say yes. After the 1.1.1 release, we could 
create a 1.1.x-branch that would be the maintenance/release branch for that 
version line. 1.x-branch would then become the target for the 1.2.0 release.

There are a few fixes in the 0.10.x branch that probably warrant a release. 
After that we may want to back away from that version line a bit so we can 
focus more on newer versions.

In the past, we’ve shied away form doing “beta” releases, but I’m wondering if 
we might want to revisit that for the 2.0 release — the idea being that it 
would give early adopter users a chance to kick the tires on what’s coming in 
2.0 and provide feedback, find bugs, etc. to help make the final release more 
solid. I’m on the fence here and could go either way.

I’d appreciate any input others may have.


Thanks,

-Taylor


[1] https://issues.apache.org/jira/browse/STORM-2153



> On Mar 30, 2017, at 9:09 PM, Jungtaek Lim  wrote:
> 
> Maybe related to or not, but would we want to create a new branch
> "1.1.x-branch", and make "1.x-branch" target for 1.2?
> 
> I'm not clear we don't release 1.2 for moving toward to 2.0.0, so hence the
> question.
> 
> - Jungtaek Lim (HeartSaVioR)
> 
> 2017년 3월 29일 (수) 오전 1:56, Hugo Da Cruz Louro 님이 작성:
> 
>> +1 for finishing the porting to Java ahead of anything else - it will be a
>> significant milestone. I have a JIRA assigned concerning to the porting. I
>> will work on it for the 2.0 release.
>> 
>> it’s a priority to guarantee no performance regressions. As part of this
>> endeavor, explore an automated (or easy) way to run and assert major
>> performance benchmarks. Ideally any contributor should be able to fairly
>> easily test the impact of changes under certain performance test scenarios.
>> 
>> Beam Runner work should take into account the impact of incorporating new
>> JStorm features and Storm Worker Redesign<
>> https://issues.apache.org/jira/browse/STORM-2284>. Not very efficient to
>> start doing it, to  find out that it will have to chance in face of Storm
>> and worker redesign. That is, it should be done after it’s building blocks
>> are stable.
>> 
>> Thanks,
>> Hugo
>> 
>> On Mar 24, 2017, at 12:07 AM, Arun Mahadevan  ar...@apache.org>> wrote:
>> 
>> +1 to release with the porting completed. I think its mainly the UI server
>> and log viewer that’s pending.
>> 
>> We can start doing the regression and performance tests for whatever is
>> already ported.
>> 
>> If anyone is running the master branch in their pre-prod / prod
>> environments, it will be good to know and give us more confidence.
>> 
>> The other features can be added in follow up releases.
>> 
>> Regards,
>> Arun
>> 
>> 
>> On 3/24/17, 11:47 AM, "Satish Duggana"  satish.dugg...@gmail.com>> wrote:
>> 
>> +1 to have 2.0 with porting and performance(it should be at least as good
>> as 1.x release) issues addressed
>> 
>> We can target other tasks(mentioned by Taylor and Jungtaek) for 2.x-branch.
>> 
>> 
>> Exactly-once support:
>> While thinking through the exactlyonce support design, it is realized
>> better to avoid acking tuples and implement exactly once by snapshotting
>> barriers. It seems JStorm folks followed similar design, they claim it
>> gives better performance. This feature is essential for beam runner and we
>> can decide on respective 

Re: Updating APIs to fix checkstyle violations

[Bobby Evans]

As this is for a change going into 2.x I totally agree lets just change the 
API.  If we really want to do a good job we can also have a pull request in the 
1.x version that adds in the new API and deprecates the old one, so users are 
not caught off guard as much.


- Bobby

On Sunday, May 21, 2017, 6:11:19 AM CDT, Jungtaek Lim  
wrote: I prefer to not have too much exceptions from chosen style guide. So I'm
OK to change the method to setSslKeystore, but just would like to hear
others voices because we're breaking backward compatibility once again (we
did it at 1.1.0) just because of respecting code style.

- Jungtaek Lim (HeartSaVioR)

2017년 5월 21일 (일) 오후 8:07, Stig Rohde Døssing 님이 작성:

> Hi,
>
> As part of fixing the checkstyle violations on storm-kafka-client, we'd
> need to rename some methods on KafkaSpoutConfig. See
> https://github.com/apache/storm/pull/2117. Does anyone have an opinion on
> when/if it is acceptable to break the API for this reason?
>
> I'd also like opinions on whether method names like setSslKeystore are
> better than setSSLKeystore? The current checkstyle rules prohibit adjacent
> capital letters in non-final names due to this bit of the checkstyle
> config.
>
> 
>            
>            
> 
>

Re: Confusing documentation for Trident State Management

[Bobby Evans]

First of all I am not an expert on trident.  I understand it at a high level, 
but I have not done too much with the internals of it.  And yes we are all in 
agreement that the documentation for trident is quite bad.  It was something 
that Nathan finished just before he left Twitter so a lot of the deep knowledge 
about it is still with him.
In all distributed systems there really is no way to get truly exactly once 
processing.  All processing needs to be idempotent with retry.  Spark 
streaming, beam, trident, flink, all offer some variant of this.  they just 
differ in how far you would have to roll back in your processing.  The recently 
announced transactions in spark streaming can be thought of as the same thing 
but with an interesting twist because the input and the output are both going 
to the same system so they can atomically commit the the output and update the 
pointer for the input at the same time. 

The way trident does it is by dividing the input into a micro batch.  The 
guarantee is that the batches will be committed to some external system in 
order.  They may be processed out of order, but will be committed in order.  If 
one batch fails that batch will be replayed and all other outstanding batches 
will not be committed until the previous batch has completed.
This is why writing a result to an external system in storm requires you to 
write to a State
https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/trident/state/State.java
All state defines is how to commit data to it.  There is nothing about how that 
data is stored or what that data is.  Typically the data will be stored in a 
key value store but it could be anything.
https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/trident/state/map/MapState.java
How you implement those guarantees in the state is up to you.  If could be 
through transactions to a database, it could be through adding the transaction 
id to the key.

https://github.com/apache/storm/blob/master/external/storm-hbase/src/main/java/org/apache/storm/hbase/trident/state/HBaseState.java
HBase ignores the commits and assumes all of the operations are already 
idempotent.

https://github.com/apache/storm/blob/master/external/storm-hdfs/src/main/java/org/apache/storm/hdfs/trident/HdfsState.java
Hdfs does use the commits to rotate files and can then recover from replayed 
data if need be.

You might also want to look at OpaqueMap
https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/trident/state/map/OpaqueMap.java
to see how it tries to add the transaction id to different backing maps so that 
is can make the operations idempotent.


- Bobby

On Sunday, May 14, 2017, 3:49:11 PM CDT, Tech Id  
wrote:Hi,

We have been using Storm for more than an year now and generally very happy
with the product.
However I want to bring up a concern for the documentation of Trident.
Me and a few colleagues went over the documentation at
http://storm.apache.org/releases/current/Trident-state.html
But it is still unclear how the micro-batching achieves its goal.

Consider this statement for instance:
"*Suppose your topology computes word count and you want to store the word
counts in a key/value database ... what you can do is store the transaction
id with the count in the database as an atomic value. *"


*Question:*
The key/value database is the target of Trident topology or something
internal to Trident?
- If it is the target of the topology, how can we change the system from
being a key-value store to a key-value-txid store? It seems that we are
making the target itself as idempotent and not really providing
exactly-once semantics to Trident itself in a generic way because target's
idempotency is specific to the target and should not be considered an
exactly-once property of streaming.
- If the key-value store is something internal to Trident, then how are we
achieving exactly-once for the actual target? It seems that the actual
target and the key-value store are two different systems with no
distributed transaction. Clearly, one system can fail and the other can
succeed, thus loosing the exactly-once promise.


*Another Question:*
It seems that a micro-batch will be continuously replayed even if just one
word in that batch failed to update its count in the key-value store. If
that is true, the next batch will not even be sent to the target system
because that next batch might have the same word and since its previous
batch was not successful due to the same failing word, this next batch
would also fail. (Rule 3: "*State updates are ordered among batches. That
is, the state updates for batch 3 won't be applied until the state updates
for batch 2 have succeeded.*").
This means that the whole topology will be stuck at the failing batch.
This is much worse than single-tuple at-least-once delivery because a batch
may be having thousands to millions of messages and all of 

Re: NullPointerException seen in 1.1.0

[Bobby Evans]

Yes retry would be the most logical way to work around this.  The code is kind 
of odd in that there are two separate locations in ZK for this information.  
Leader election simply stores who the leader is at `/leader-lock`, but the 
information about all of the nimbus instances that are alive is stored under 
`/nimbuses`.  What you have run into is where they are not in sync with each 
other.  The leader lock said nimbus-A is the leader and nimbuses had no 
knowledge of nimbus-A at all.  If nimbus-A was crashing during this period of 
time then it is a race and we need to fix it with retry (I'll file a JIRA for 
this anyways as we should have this in no matter what).  If nimbus-A was not 
crashing then ZK some how messed up or we some how messed up.  The only way 
that could happen on our end is if for some reason we have two different 
connections to ZK, one for leader election and another for writing to nimbuses. 
 If that is not the case, and this is reproducible, then yes the first thing to 
do is to turn on debug logging, and try to grab the snapshot/edit logs for your 
ZK cluster right after this happened.  I am really hopeful that it was nimbus 
crashing.

- Bobby

On Sunday, May 14, 2017, 4:03:22 PM CDT, S G <sg.online.em...@gmail.com> 
wrote:Thanks Bobby,

This looks like a serious issue to me. Any ideas how I can provide more
information (like enable some logs etc) to gain more insight into this
problem?

It might be a good idea to add some retry logic or some waiting logic on
the node that comes up empty handed so that it handles the error more
gracefully rather than crashing with a NullPointerException?

Also, the leader election is supposed to happen through zookeeper, right?
Isn't the new leader becoming a leader after saving its state in zookeeper?
Because then the other nodes should not come empty handed.
If no, then it seems like a bug and the leader should persist the state in
zookeeper first before becoming a leader.


> looks like it is caused by trying to read a NimbusSummary for the leader
but not being able to find it
Instead of crashing, this should trigger a new leader election IMO with
some good warning messages in the logs.


Disclaimer: I have not seen the actual code that does the nimbus leader
election. Above are just some suggestions based on my limited knowledge. So
please forgive any outrageous/obvious ideas :)



On Tue, May 9, 2017 at 1:58 PM, Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:

> This looks like something odd is happening with leader election.  The
> exception looks like it is caused by trying to read a NimbusSummary for the
> leader but not being able to find it.  So it could mean that a leader is
> elected and is then crashing quickly enough that the other node when it
> tries to read this loses the race and comes up empty handed.  But if you
> only have a single nimbus configured then this is not the case and
> something else worse is happening.
>
>
> - Bobby
>
> On Monday, May 8, 2017, 4:41:13 PM CDT, S G <sg.online.em...@gmail.com>
> wrote:Hi,
>
> I am trying to upgrade from 1.0.2 to 1.1.0 version of Storm.
> And I see the below exception happening randomly on the Nimbus node.
> When it happens, Nimbus is unable to accept any new topologies.
>
>
> java.lang.NullPointerException: null
>        at
> clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:301)
> ~[clojure-1.7.0.jar:?]
>        at
> org.apache.storm.daemon.nimbus$mk_reified_nimbus$
> reify__10782.getLeader(nimbus.clj:2383)
> ~[storm-core-1.1.0.jar:1.1.0]
>        at
> org.apache.storm.generated.Nimbus$Processor$getLeader.
> getResult(Nimbus.java:3944)
> ~[storm-core-1.1.0.jar:1.1.0]
>        at
> org.apache.storm.generated.Nimbus$Processor$getLeader.
> getResult(Nimbus.java:3928)
> ~[storm-core-1.1.0.jar:1.1.0]
>        at
> org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39)
> ~[storm-core-1.1.0.jar:1.1.0]
>        at
> org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> ~[storm-core-1.1.0.jar:1.1.0]
>        at
> org.apache.storm.security.auth.SimpleTransportPlugin$
> SimpleWrapProcessor.process(SimpleTransportPlugin.java:162)
> ~[storm-core-1.1.0.jar:1.1.0]
>        at
> org.apache.storm.thrift.server.AbstractNonblockingServer$
> FrameBuffer.invoke(AbstractNonblockingServer.java:518)
> ~[storm-core-1.1.0.jar:1.1.0]
>        at
> org.apache.storm.thrift.server.Invocation.run(Invocation.java:18)
> ~[storm-core-1.1.0.jar:1.1.0]
>        at
> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> [?:1.8.0_51]
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> [?:1.8.0_51]
>        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_51]
>
>
> I have not been able to isolate what causes this exception.
> Any help would be appreciated.
>
> Thanks
> SG
>

Re: NullPointerException seen in 1.1.0

[Bobby Evans]

This looks like something odd is happening with leader election.  The exception 
looks like it is caused by trying to read a NimbusSummary for the leader but 
not being able to find it.  So it could mean that a leader is elected and is 
then crashing quickly enough that the other node when it tries to read this 
loses the race and comes up empty handed.  But if you only have a single nimbus 
configured then this is not the case and something else worse is happening.


- Bobby

On Monday, May 8, 2017, 4:41:13 PM CDT, S G  
wrote:Hi,

I am trying to upgrade from 1.0.2 to 1.1.0 version of Storm.
And I see the below exception happening randomly on the Nimbus node.
When it happens, Nimbus is unable to accept any new topologies.


java.lang.NullPointerException: null
        at
clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:301)
~[clojure-1.7.0.jar:?]
        at
org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__10782.getLeader(nimbus.clj:2383)
~[storm-core-1.1.0.jar:1.1.0]
        at
org.apache.storm.generated.Nimbus$Processor$getLeader.getResult(Nimbus.java:3944)
~[storm-core-1.1.0.jar:1.1.0]
        at
org.apache.storm.generated.Nimbus$Processor$getLeader.getResult(Nimbus.java:3928)
~[storm-core-1.1.0.jar:1.1.0]
        at
org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:39)
~[storm-core-1.1.0.jar:1.1.0]
        at
org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
~[storm-core-1.1.0.jar:1.1.0]
        at
org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:162)
~[storm-core-1.1.0.jar:1.1.0]
        at
org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
~[storm-core-1.1.0.jar:1.1.0]
        at
org.apache.storm.thrift.server.Invocation.run(Invocation.java:18)
~[storm-core-1.1.0.jar:1.1.0]
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_51]
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_51]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_51]


I have not been able to isolate what causes this exception.
Any help would be appreciated.

Thanks
SG

Re: ClassCastException when trying to run topology on a remote Nimbus server

[Bobby Evans]

a 0.9.x or 0.10.x topology will not be compatible with a 1.x cluster.  We 
changed the package names from backtype to org.apache in between the two.  So 
even though the client can talk to the nimbus through thrift the topology will 
not run, with exceptions like you saw.  The best solution is to upgrade your 
topology to compile against 1.x and this will require a code change.


- Bobby

On Tuesday, May 9, 2017, 2:57:06 PM CDT, Stig Døssing 
 wrote:What happens if you upgrade your storm-core?

Also as far as I know there shouldn't be anything on the server using the
backtype.storm packages anymore. Please make sure you're not deploying the
storm-core jar to the server (set it to provided scope in Maven if you use
that). I'm not too familiar with running old storm-core jars with new
topologies, but are you setting the client.jartransformer.class as
mentioned in https://storm.apache.org/releases/1.1.0/index.html?

2017-05-09 11:15 GMT+02:00 Yovav Waichman :

> Hi,
>
> I’m a developer using Storm in our production environment.
> When running my topology locally, everything works fine.
> However, when trying to run storm with my topology on a remote Nimbus
> server, I get the following exception:
>
> Exception in thread "main" java.lang.ExceptionInInitializerError
>        at backtype.storm.topology.TopologyBuilder.createTopology(
> TopologyBuilder.java:106)
>        at com.jivesoftware.playbox.engine.Main.main(Main.java:77)
> Caused by: java.lang.ClassCastException: org.apache.storm.serialization.
> GzipThriftSerializationDelegate cannot be cast to
> backtype.storm.serialization.SerializationDelegate
>        at backtype.storm.utils.Utils.getSerializationDelegate(
> Utils.java:432)
>        at backtype.storm.utils.Utils.(Utils.java:72)
>        ... 2 more
>
>
> My server has Strom version 1.1.0 installed and my topology is using
> storm-core 0.9.4 version.
> It seems that there’s a conflict between the versions,
> GzipThriftSerializationDelegate from package
> org.apache.storm.serialization is being cast to
> backtype.storm.serialization.SerializationDelegate instead of being cast
> to
> org.apache.storm.serialization.SerializationDelegate.
>
> I would appreciate your help on that,
>
> Thanks in advance,
> Yovav

Re: [VOTE] Java Code Style Standard for Apache Storm.

[Bobby Evans]

Thanks to everyone that voted.
Of the 10 people that voted 8 voted for google as their first choice.  1 voted 
for Hadoop and 1 voted for a modified form of Google.
With that I think google wins.  I will start by merging in the checkstyle pull 
request for google 
https://github.com/apache/storm/pull/2093
After that I think we should start looking at how to auto format the existing 
code base and drop the exception count.


- Bobby

On Tuesday, May 2, 2017, 1:12:39 AM CDT, Arun Mahadevan <ar...@apache.org> 
wrote:+1 for Google Java Style Guide (with 4 spaces indentation instead of 2 
and line wrap set to 120 instead of 100)

Thanks,
Arun

On 5/2/17, 12:32 AM, "Bobby Evans" <ev...@yahoo-inc.com.INVALID> wrote:

>Just a reminder to everyone that voting ends on Wednesday.
>
>
>- Bobby
>
>On Friday, April 28, 2017, 1:33:37 PM CDT, Kishorkumar Patil 
><kpa...@yahoo-inc.com.INVALID> wrote:
>[1] Google Java Style Guide<https://google.github.io/styleguide/javaguide.html>
>
>[2] Sun Java Code 
>Conventions<http://www.oracle.com/technetwork/java/codeconventions-150003.pdf>
>
>[3] HBase 
>style<https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml>
> (sorry all I could find was an eclipse XML config)
>
>[4] Hadoop style<https://wiki.apache.org/hadoop/CodeReviewChecklist> which is 
>described as java but with 2 spaces instead of 4. 
>
>-Kishor
>
>
>    On Thursday, April 27, 2017 10:01 PM, Satish Duggana <sati...@apache.org> 
>wrote:
> 
>
> [1] Google Java Style Guide<https://google.github.
>io/styleguide/javaguide.html>
>
>[2] Hadoop style<https://wiki.apache.org/hadoop/CodeReviewChecklist> which
>is described as java but with 2 spaces instead of 4.
>
>[3] Sun Java Code Conventions<http://www.oracle.com/technetwork/java/
>codeconventions-150003.pdf>
>
>[4] HBase style<https://github.com/apache/hbase/blob/master/dev-
>support/hbase_eclipse_formatter.xml> (sorry all I could find was an eclipse
>XML config)
>
>Thanks,
>~Satish.
>
>On Thu, Apr 27, 2017 at 7:50 PM, Kyle Nusbaum <
>knusb...@yahoo-inc.com.invalid> wrote:
>
>> [1] Google Java Style Guide<https://google.github.
>> io/styleguide/javaguide.html>
>>
>> [2] Sun Java Code Conventions<http://www.oracle.com/technetwork/java/
>> codeconventions-150003.pdf>
>>
>> [3] HBase style<https://github.com/apache/hbase/blob/master/dev-
>> support/hbase_eclipse_formatter.xml> (sorry all I could find was an
>> eclipse XML config)
>>
>> [4] Hadoop style<https://wiki.apache.org/hadoop/CodeReviewChecklist>
>> which is described as java but with 2 spaces instead of 4.
>> Thanks,
>> -- Kyle
>>
>> On Thursday, April 27, 2017, 3:51:25 AM CDT, Julien Nioche <
>> lists.digitalpeb...@gmail.com> wrote:Non-binding : [1 ] Google Java Style
>> Guide
>>
>> On 26 April 2017 at 19:50, Bobby Evans <ev...@yahoo-inc.com.invalid>
>> wrote:
>>
>> > We would like to adopt a code style standard for Apache Storm.  Please
>> > rank the following with 1 being the most desired and 4 being the least
>> > desired (5 if you have a write in choice).  This is not an official vote
>> as
>> > per the ByLaws, but we will probably go with whichever wins the most 1
>> > votes (unless it is really close and then we may go with some STeVe like
>> > ranking, but I want to avoid it because it is hard and I might get the
>> math
>> > wrong) This is open to everyone so please vote.
>> >
>> >
>> > [ ] Google Java Style Guide<https://google.github.
>> > io/styleguide/javaguide.html>
>> >
>> > [ ] Sun Java Code Conventions<http://www.oracle.com/technetwork/java/
>> > codeconventions-150003.pdf>
>> >
>> > [ ] HBase style<https://github.com/apache/hbase/blob/master/dev-
>> > support/hbase_eclipse_formatter.xml> (sorry all I could find was an
>> > eclipse XML config)
>> >
>> > [ ] Hadoop style<https://wiki.apache.org/hadoop/CodeReviewChecklist>
>> > which is described as java but with 2 spaces instead of 4.
>> >
>> > [ ] Other (Please specify)
>> >
>> >
>> > I apologize if the formatting is bad I am forced to use a corporate
>> > sponsored mail client that somehow messed up plain text formatting.  I
>> > don't know how.
>> >
>> >
>> > - Bobby
>>
>>
>>
>>
>> --
>>
>> *Open Source Solutions for Text Engineering*
>>
>> http://www.digitalpebble.com
>> http://digitalpebble.blogspot.com/
>> #digitalpebble <http://twitter.com/digitalpebble>
>>
>
>
>  

Re: Need Help

[Bobby Evans]

Taylor, are you still working on the BEAM impl?  If so could you use Kamal's 
help at all?

- Bobby

On Thursday, April 27, 2017, 1:32:44 PM CDT, Kamal Bhatt  
wrote:Hello,

I am a developer interested to work on Storm tasks based on  Java Streaming
API and/or  supporting storm as a runtime for apache beam.

I have some experience working in Storm project like translating  clojure
tests to java and Admin Command.

Could someone from the group working on these tasks help me get started. I
appreciate any help/direction on this.

Thanks,
Kamal

Re: [VOTE] Java Code Style Standard for Apache Storm.

[Bobby Evans]

Just a reminder to everyone that voting ends on Wednesday.


- Bobby

On Friday, April 28, 2017, 1:33:37 PM CDT, Kishorkumar Patil 
<kpa...@yahoo-inc.com.INVALID> wrote:
[1] Google Java Style Guide<https://google.github.io/styleguide/javaguide.html>

[2] Sun Java Code 
Conventions<http://www.oracle.com/technetwork/java/codeconventions-150003.pdf>

[3] HBase 
style<https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml>
 (sorry all I could find was an eclipse XML config)

[4] Hadoop style<https://wiki.apache.org/hadoop/CodeReviewChecklist> which is 
described as java but with 2 spaces instead of 4. 

-Kishor


    On Thursday, April 27, 2017 10:01 PM, Satish Duggana <sati...@apache.org> 
wrote:
 

 [1] Google Java Style Guide<https://google.github.
io/styleguide/javaguide.html>

[2] Hadoop style<https://wiki.apache.org/hadoop/CodeReviewChecklist> which
is described as java but with 2 spaces instead of 4.

[3] Sun Java Code Conventions<http://www.oracle.com/technetwork/java/
codeconventions-150003.pdf>

[4] HBase style<https://github.com/apache/hbase/blob/master/dev-
support/hbase_eclipse_formatter.xml> (sorry all I could find was an eclipse
XML config)

Thanks,
~Satish.

On Thu, Apr 27, 2017 at 7:50 PM, Kyle Nusbaum <
knusb...@yahoo-inc.com.invalid> wrote:

> [1] Google Java Style Guide<https://google.github.
> io/styleguide/javaguide.html>
>
> [2] Sun Java Code Conventions<http://www.oracle.com/technetwork/java/
> codeconventions-150003.pdf>
>
> [3] HBase style<https://github.com/apache/hbase/blob/master/dev-
> support/hbase_eclipse_formatter.xml> (sorry all I could find was an
> eclipse XML config)
>
> [4] Hadoop style<https://wiki.apache.org/hadoop/CodeReviewChecklist>
> which is described as java but with 2 spaces instead of 4.
> Thanks,
> -- Kyle
>
> On Thursday, April 27, 2017, 3:51:25 AM CDT, Julien Nioche <
> lists.digitalpeb...@gmail.com> wrote:Non-binding : [1 ] Google Java Style
> Guide
>
> On 26 April 2017 at 19:50, Bobby Evans <ev...@yahoo-inc.com.invalid>
> wrote:
>
> > We would like to adopt a code style standard for Apache Storm.  Please
> > rank the following with 1 being the most desired and 4 being the least
> > desired (5 if you have a write in choice).  This is not an official vote
> as
> > per the ByLaws, but we will probably go with whichever wins the most 1
> > votes (unless it is really close and then we may go with some STeVe like
> > ranking, but I want to avoid it because it is hard and I might get the
> math
> > wrong) This is open to everyone so please vote.
> >
> >
> > [ ] Google Java Style Guide<https://google.github.
> > io/styleguide/javaguide.html>
> >
> > [ ] Sun Java Code Conventions<http://www.oracle.com/technetwork/java/
> > codeconventions-150003.pdf>
> >
> > [ ] HBase style<https://github.com/apache/hbase/blob/master/dev-
> > support/hbase_eclipse_formatter.xml> (sorry all I could find was an
> > eclipse XML config)
> >
> > [ ] Hadoop style<https://wiki.apache.org/hadoop/CodeReviewChecklist>
> > which is described as java but with 2 spaces instead of 4.
> >
> > [ ] Other (Please specify)
> >
> >
> > I apologize if the formatting is bad I am forced to use a corporate
> > sponsored mail client that somehow messed up plain text formatting.  I
> > don't know how.
> >
> >
> > - Bobby
>
>
>
>
> --
>
> *Open Source Solutions for Text Engineering*
>
> http://www.digitalpebble.com
> http://digitalpebble.blogspot.com/
> #digitalpebble <http://twitter.com/digitalpebble>
>


  

Re: [VOTE] Java Code Style Standard for Apache Storm.

[Bobby Evans]

I forgot to mention that we will have the vote run for 7 days just to give 
everyone a chance that wants to vote.  So it will close Wednesday May 5th.


- Bobby

On Wednesday, April 26, 2017, 1:50:56 PM CDT, Bobby Evans 
<ev...@yahoo-inc.com.INVALID> wrote:We would like to adopt a code style 
standard for Apache Storm.  Please rank the following with 1 being the most 
desired and 4 being the least desired (5 if you have a write in choice).  This 
is not an official vote as per the ByLaws, but we will probably go with 
whichever wins the most 1 votes (unless it is really close and then we may go 
with some STeVe like ranking, but I want to avoid it because it is hard and I 
might get the math wrong) This is open to everyone so please vote.


[ ] Google Java Style Guide<https://google.github.io/styleguide/javaguide.html>

[ ] Sun Java Code 
Conventions<http://www.oracle.com/technetwork/java/codeconventions-150003.pdf>

[ ] HBase 
style<https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml>
 (sorry all I could find was an eclipse XML config)

[ ] Hadoop style<https://wiki.apache.org/hadoop/CodeReviewChecklist> which is 
described as java but with 2 spaces instead of 4.

[ ] Other (Please specify)


I apologize if the formatting is bad I am forced to use a corporate sponsored 
mail client that somehow messed up plain text formatting.  I don't know how.


- Bobby

Re: [VOTE] Java Code Style Standard for Apache Storm.

[Bobby Evans]

[ 1 ] Google Java Style 
Guide<https://google.github.io/styleguide/javaguide.html>

[ 2 ] Sun Java Code 
Conventions<http://www.oracle.com/technetwork/java/codeconventions-150003.pdf>

[ 4 ] HBase 
style<https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml>
 (sorry all I could find was an eclipse XML config)

[ 3 ] Hadoop style<https://wiki.apache.org/hadoop/CodeReviewChecklist> which is 
described as java but with 2 spaces instead of 4.

[ ] Other (Please specify)
- Bobby

On Wednesday, April 26, 2017, 1:50:56 PM CDT, Bobby Evans 
<ev...@yahoo-inc.com.INVALID> wrote:We would like to adopt a code style 
standard for Apache Storm.  Please rank the following with 1 being the most 
desired and 4 being the least desired (5 if you have a write in choice).  This 
is not an official vote as per the ByLaws, but we will probably go with 
whichever wins the most 1 votes (unless it is really close and then we may go 
with some STeVe like ranking, but I want to avoid it because it is hard and I 
might get the math wrong) This is open to everyone so please vote.


[ ] Google Java Style Guide<https://google.github.io/styleguide/javaguide.html>

[ ] Sun Java Code 
Conventions<http://www.oracle.com/technetwork/java/codeconventions-150003.pdf>

[ ] HBase 
style<https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml>
 (sorry all I could find was an eclipse XML config)

[ ] Hadoop style<https://wiki.apache.org/hadoop/CodeReviewChecklist> which is 
described as java but with 2 spaces instead of 4.

[ ] Other (Please specify)


I apologize if the formatting is bad I am forced to use a corporate sponsored 
mail client that somehow messed up plain text formatting.  I don't know how.


- Bobby

[VOTE] Java Code Style Standard for Apache Storm.

[Bobby Evans]

We would like to adopt a code style standard for Apache Storm.  Please rank the 
following with 1 being the most desired and 4 being the least desired (5 if you 
have a write in choice).  This is not an official vote as per the ByLaws, but 
we will probably go with whichever wins the most 1 votes (unless it is really 
close and then we may go with some STeVe like ranking, but I want to avoid it 
because it is hard and I might get the math wrong) This is open to everyone so 
please vote.


[ ] Google Java Style Guide

[ ] Sun Java Code 
Conventions

[ ] HBase 
style
 (sorry all I could find was an eclipse XML config)

[ ] Hadoop style which is 
described as java but with 2 spaces instead of 4.

[ ] Other (Please specify)


I apologize if the formatting is bad I am forced to use a corporate sponsored 
mail client that somehow messed up plain text formatting.  I don't know how.


- Bobby

Re: [DISCUSS] Code Style

[Bobby Evans]

I really don't care what style we use so long as we have one and it is 
enforced.  Checkstyle is highly configurable so just about any style we pick we 
can enforce with it.  Most Java IDEs that I know of either use the checkstyle 
format itself or at least support a checkstyle plugin to let you know if you 
are off.
I have my own opinions about this but to avoid too much further discussion (aka 
religious war) I would propose that we have a simple vote on what standard 
people what.  So far I have heard
Google Java Style GuideSun 
Java Code 
Conventions 
HBase 
style
 (sorry all I could find was an eclipse XML config)Hadoop 
style which is described as 
java but with 2 spaces instead of 4.

If no one has any objections I will send out a new thread to vote on this.  I 
will probably also include a write in option too.


- Bobby

On Wednesday, April 26, 2017, 2:23:46 AM CDT, Satish Duggana 
 wrote:+1 to what Hugo and Taylor said. Lets go with 
google java style guide for
now and modify any of those rules if it is really required.

On Wed, Apr 26, 2017 at 5:23 AM, P. Taylor Goetz  wrote:

> Code convention discussions can get pretty heated/religious in nature,
> which we should seek to avoid. I'm largely in alignment with Hugo on this:
> Start with a good base like the Google or Sun/Oracle style guides, and
> establish consensus on where we feel a desire to differ.
>
> Rather than haggle over certain individual conventions, I propose we
> choose an existing style guide to base ours on and tweak it as we see fit.
> I have definite opinions about java code style, but I'll hold off on
> sharing them for now.
>
> Let's pick a starting point first. I believe the google and sun/oracle
> guides are both good starting points. Let's also try not to drag this out
> too much. I've seen similar conversations absorb more time and effort than
> it would take to update the associated codebase.
>
> -Taylor
>
> > On Apr 25, 2017, at 7:13 PM, Hugo Da Cruz Louro 
> wrote:
> >
> > Hi,
> >
> > We should just follow the Google Java Style Guide io/styleguide/javaguide.html>. There are many opinions on this subject,
> but I conjecture that this guide, which is very similar to the original Sun
> Java Code Conventions codeconventions-150003.pdf>  has a lot of research and reasoning behind
> it. It’s widely adopted, and I think that we should just follow it. The
> only exception I would really make is the column limit. I would change 100
> to 120 or 130, but not longer than that. Very long lines makes the code
> really hard to read.
> >
> > There are already existing check style, IntelliJ, and even Git hooks
> enforcing/validating this style, so adopting it would make it easier to
> incorporate it in the dev tools.
> >
> > Regardless of the variants to the aforementioned style, I do not think
> that we should use names starting with underscore _ in any circumstance.
> Not only it is not a Java convention, as it does not add anything to the
> code. The reason behind using _ was before the widespread uses of IDE’s, to
> mark class fields. IDEs do that with smart coloring.
> >
> > Thanks,
> > Hugo
> >
> > On Apr 25, 2017, at 3:50 PM, Jungtaek Lim  w...@gmail.com>> wrote:
> >
> > I would like to review the style guide of other projects, like HBase, and
> > so.
> >
> > Btw, IMHO, I don't like using underscore as prefix for fields. We're
> using
> > Java and the expression "class member variable" is from C++, and also
> > underscore style came from C++. We need to avoid mixing other languages'
> > style as well.
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 4월 26일 (수) 오전 7:10, Kyle Nusbaum  invalid>님이
> > 작성:
> >
> > Now that most of our code is in Java, I think it might be time to revisit
> > the issue of having some official and enforced code style.
> > I don't have very strong feelings about most of it, but here is what I
> was
> > thinking as a start, since I've seen this style quite a bit, and I've
> found
> > it makes code pretty easy to read:
> > 1. Indentation is 4 spaces per level (no tabs)
> > 2. All class member variables begin with underscore3. No wildcard
> > imports4. if / else / for / etc. always get curly braces.
> > There are obviously tons of other things, and we can get as picky as we
> > want, but I think enforcing at least these rules would go a long way to
> > making the code more consistent.
> >
> > Thoughts?
> >
> > Thanks,
> > -- Kyle
> >
>

Re: Fwd: Distribution of workers across cluster with Resource Aware Scheduler

[Bobby Evans]

No there is currently no way to do this with the default strategy.  You can 
write your own resource aware strategy to place the workers differently though. 
  You just need to write a class that implements IStrategy.  It can also be 
selected on a per topology basis.

The main reason for doing this is because it reduces the network load 
associated with a topology and hopefully speeds it up as a local transfer is 
much faster than a remote one.
If you are having problems with it then it might mean that we are not 
scheduling for the correct resources.  One big hole that we are aware of is 
network.  If you need/want network as a resource then please let us know.  We 
should be able to at least implement something simple similar to CPU and 
Memory, but a fully functional one that takes the complete network topography 
into account is going to be difficult to do.

- Bobby

On Tuesday, April 11, 2017, 10:32:49 PM CDT, Pracheer Agarwal 
 wrote:-- Forwarded message --
From: Pracheer Agarwal 
Date: Tue, Apr 11, 2017 at 9:07 AM
Subject: Distribution of workers across cluster with Resource Aware
Scheduler
To: u...@storm.apache.org


Hi All,

We are in the process of migrating to storm version 1.0.3. The pattern that
we observe with resource aware scheduler is that it launches workers on one
single machine till the time machine is resource wise exhausted.  So few of
the machines are fully utilized while few are sitting idle. Can I tune
storm(RAS) to spread the workers across the clusters ? Are there parameters
to control this feature ?

Regards
Pracheer

Re: zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss

[Bobby Evans]

That is a little hard to tell.  It looks like your ZK server crashed and was 
not relaunched.


- Bobby

On Friday, April 7, 2017, 1:35:43 AM CDT, xx  wrote:hi, 
everyone!  Lylian for your help!I'm running a storm cluster.I have a 
nimbus,zookeeper,kafka server, and a supervisor in one node.Furthermore, I 
deploy all of them in this node.so the zookeeper status is  standalone.
version information list:Storm 0.10.0Zookeeper-3.4.6Kafka_2.9.2-0.8.2.0JDK 1.7

when I deploy three topology in this node, it seems like work well .but two 
days later, it suddenly stop with a crash. nimbus, supervisor, worker ,storm 
ui, are killed.
2017-04-05 11:13:11.627 b.s.d.nimbus [INFO] [req 63] Access from:  principal: 
op:fileDownload2017-04-05 11:13:11.629 b.s.d.nimbus [INFO] [req 64] Access 
from:  principal: op:fileDownload2017-04-05 11:13:11.631 b.s.d.nimbus [INFO] 
[req 1] Access from:  principal: op:fileDownload2017-04-05 11:13:11.632 
b.s.d.nimbus [INFO] [req 2] Access from:  principal: op:fileDownload2017-04-05 
11:13:11.633 b.s.d.nimbus [INFO] [req 3] Access from:  principal: 
op:fileDownload2017-04-05 11:13:11.633 b.s.d.nimbus [INFO] [req 4] Access from: 
 principal: op:fileDownload2017-04-05 11:13:11.635 b.s.d.nimbus [INFO] [req 5] 
Access from:  principal: op:fileDownload2017-04-05 11:13:40.660 b.s.d.nimbus 
[INFO] [req 6] Access from:  principal: op:getClusterInfo2017-04-05 
11:14:03.901 b.s.d.nimbus [INFO] [req 7] Access from:  principal: 
op:getClusterInfo2017-04-05 11:14:35.821 b.s.d.nimbus [INFO] [req 8] Access 
from:  principal: op:getClusterInfo2017-04-05 19:30:32.541 
o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed out, have not heard from 
server in 13337ms for sessionid 0x15b23c37c6400c6, closing socket connection 
and attempting reconnect2017-04-05 19:30:32.644 
o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: 
SUSPENDED2017-04-05 19:30:32.651 b.s.cluster [WARN] Received event 
:disconnected::none: with disconnected Zookeeper.2017-04-05 19:30:33.138 
o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server 
xiang/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown 
error)2017-04-05 19:30:33.139 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection 
established to xiang/127.0.0.1:2181, initiating session2017-04-05 19:30:33.141 
o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server 
xiang/127.0.0.1:2181, sessionid = 0x15b23c37c6400c6, negotiated timeout = 
22017-04-05 19:30:33.141 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] 
State change: RECONNECTED2017-04-05 19:30:59.303 o.a.s.s.o.a.z.ClientCnxn 
[INFO] Client session timed out, have not heard from server in 13337ms for 
sessionid 0x15b23c37c6400c6, closing socket connection and attempting 
reconnect2017-04-05 19:30:59.404 o.a.s.s.o.a.c.f.s.ConnectionStateManager 
[INFO] State change: SUSPENDED2017-04-05 19:30:59.405 b.s.cluster [WARN] 
Received event :disconnected::none: with disconnected Zookeeper.2017-04-05 
19:31:00.145 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to 
server xiang/10.240.47.205:2181. Will not attempt to authenticate using SASL 
(unknown error)2017-04-05 19:31:00.146 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket 
connection established to xiang/10.240.47.205:2181, initiating 
session2017-04-05 19:31:00.147 o.a.s.s.o.a.z.ClientCnxn [INFO] Session 
establishment complete on server xiang/10.240.47.205:2181, sessionid = 
0x15b23c37c6400c6, negotiated timeout = 22017-04-05 19:31:00.148 
o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: 
RECONNECTED2017-04-05 19:31:13.487 o.a.s.s.o.a.z.ClientCnxn [INFO] Client 
session timed out, have not heard from server in 13340ms for sessionid 
0x15b23c37c6400c6, closing socket connection and attempting reconnect2017-04-05 
19:31:13.588 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change: 
SUSPENDED2017-04-05 19:31:13.588 b.s.cluster [WARN] Received event 
:disconnected::none: with disconnected Zookeeper.2017-04-05 19:31:14.436 
o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server 
xiang/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown 
error)2017-04-05 19:31:14.437 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection 
established to xiang/127.0.0.1:2181, initiating session2017-04-05 19:31:14.438 
o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server 
xiang/127.0.0.1:2181, sessionid = 0x15b23c37c6400c6, negotiated timeout = 
22017-04-05 19:31:14.439 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] 
State change: RECONNECTED2017-04-05 19:31:27.772 o.a.s.s.o.a.z.ClientCnxn 
[INFO] Client session timed out, have not heard from server in 13334ms for 
sessionid 0x15b23c37c6400c6, closing socket connection and attempting 
reconnect2017-04-05 19:31:27.873 o.a.s.s.o.a.c.f.s.ConnectionStateManager 
[INFO] State change: SUSPENDED2017-04-05 19:31:27.874 b.s.cluster [WARN] 
Received event :disconnected::none: with disconnected 

Re: [DISCUSS] - Reconcile Multiple SECURITY.md Files in Master and 1.x-Branch

[Bobby Evans]

My guess is that it was a merge error.  The one under docs is the one on the 
web site.  When we moved the web site source under docs I thought I did a git 
mv of the SECURITY.md in the base directory, but I guess I forgot to.
+1 for just keeping the one under the docs directory.


- Bobby

On Monday, April 3, 2017, 7:55:36 PM CDT, Jungtaek Lim  
wrote:Yeah they're duplicated docs and we have lots of duplicated docs indeed.

The issue is that someone would find the document from source directory,
and others would find the document from website. IMHO they may need to be
in both directories, but not ideal indeed, since I also saw that they're
not in sync often.

It would be nice if we can find a solution. RIght now I don't have an idea
for that.

- Jungtaek Lim (HeartSaVioR)

2017년 4월 4일 (화) 오전 7:01, Hugo Da Cruz Louro 님이 작성:

> Hi,
>
> I noticed that we have two SECURITY.md files with different contents in
> both Master and 1.x-branch.
>
> Master
> https://github.com/apache/storm/blob/master/SECURITY.md
> https://github.com/apache/storm/blob/master/docs/SECURITY.md
>
> 1.x-branch
> https://github.com/apache/storm/blob/1.x-branch/SECURITY.md
> https://github.com/apache/storm/blob/1.x-branch/docs/SECURITY.md
>
> Is there a particular reason to have two files? If not, I would suggest
> that we reconcile them in one file only at a location that we agree. Which
> location would you suggest?
>
> Thanks,
> Hugo
>