Re: [Discuss] ARM CI for Storm

[Jungtaek Lim]

t; > > Of course, welcome another developers join to maintain the aarch64
> > CI,
> > > > and
> > > > > take effort on aarch64 work together.
> > > > >
> > > > > Thanks for your attention.
> > > > >
> > > > > Regards,
> > > > > Yikun
> > > > > 
> > > > > Jiang Yikun（Kero）
> > > > > Mail: yikunk...@gmail.com
> > > > >
> > > > >
> > > > > Stig Rohde Døssing  于2019年6月21日周五
> 上午2:23写道：
> > > > >
> > > > > > I guess there is no interest in maintaining ARM compatibility, or
> > at
> > > > > least
> > > > > > no one currently wants to take on the effort. Let's not add the
> > > Openlab
> > > > > CI
> > > > > > then, we can always do it later if someone expresses interest
> (and
> > > > > > willingness to maintain).
> > > > > >
> > > > > > Den ons. 12. jun. 2019 kl. 17.11 skrev Stig Rohde Døssing <
> > > > > > stigdoess...@gmail.com>:
> > > > > >
> > > > > > > Good point, let's see if there's anyone with an ARM
> environment.
> > > > > > >
> > > > > > > Den tir. 11. jun. 2019 kl. 23.07 skrev Jungtaek Lim <
> > > > kabh...@gmail.com
> > > > > >:
> > > > > > >
> > > > > > >> I guess the point is not related to open CI build. The point
> is
> > > > > whether
> > > > > > we
> > > > > > >> really want to support ARM. I'm seeing OpenLab request for
> other
> > > > > Apache
> > > > > > >> projects as well, so I'd rather not treat their request as
> > > > commitment
> > > > > of
> > > > > > >> putting efforts to make builds on ARM green.
> > > > > > >>
> > > > > > >> I'm a bit hesitant to add some environment on the support list
> > > > unless
> > > > > > >> there're enough engineers willing to work on maintaining. If
> the
> > > > work
> > > > > is
> > > > > > >> left to existing maintainers, we will be stuck on failing
> tests
> > on
> > > > ARM
> > > > > > CI
> > > > > > >> but no environment to play locally.
> > > > > > >>
> > > > > > >> So I'd be +1 if there're at least two folks experienced with
> ARM
> > > > env.
> > > > > > >> volunteer to maintain the compatibility part. Otherwise -1
> here.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Jungtaek Lim (HeartSaVioR)
> > > > > > >>
> > > > > > >> On Mon, Jun 10, 2019 at 7:27 PM Stig Rohde Døssing <
> > > > > > >> stigdoess...@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi,
> > > > > > >> >
> > > > > > >> > An issue was opened suggesting adding ARM CI to the Storm
> > build,
> > > > > using
> > > > > > >> > OpenLab https://issues.apache.org/jira/browse/STORM-3401,
> > > > > associated
> > > > > > PR
> > > > > > >> > here https://github.com/apache/storm/pull/3023
> > > > > > >> >
> > > > > > >> > As far as I can tell, we need to allow OpenLab access to the
> > > > Github
> > > > > > PRs,
> > > > > > >> > similar to how Travis has permission to access our PRs. The
> > app
> > > > that
> > > > > > >> needs
> > > > > > >> > access is https://github.com/apps/theopenlab-ci. Note that
> > the
> > > > app
> > > > > > will
> > > > > > >> > have access to the PRs only, not the repo code.
> > > > > > >> >
> > > > > > >> > Is anyone opposed to me asking infra to allow OpenLab access
> > to
> > > > our
> > > > > > PRs?
> > > > > > >> > I'll let this thread sit for a week or so, if there is no
> > > > opposition
> > > > > > >> I'll
> > > > > > >> > ask infra to give the app access.
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> Name : Jungtaek Lim
> > > > > > >> Blog : http://medium.com/@heartsavior
> > > > > > >> Twitter : http://twitter.com/heartsavior
> > > > > > >> LinkedIn : http://www.linkedin.com/in/heartsavior
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> >
>


-- 
Name : Jungtaek Lim
Blog : http://medium.com/@heartsavior
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior

Re: [Discuss] ARM CI for Storm

[Jungtaek Lim]

Sorry for the late reply, I'm on business trip to US this week and missed
this mail at that time.

Maybe we could just start with ARM CI build but just treat it as
"optional", and once we are happy with the ARM CI build result, we could
count it in. If it goes unstable, ignore it again. Does that make sense?

Another possible issue is that contributor may not want to deal with ARM64
as well. Personally I don't deal with Windows machine, because I don't have
one, but that's not an only reason. I also hope not to deal with Windows
machine cause if something goes wrong specific to Windows, I might have no
idea to fix that.

So (maybe) dedicated help would be really needed to keep up supporting ARM,
but in worst case we can ignore the build, so no big deal.

On Sat, Jul 6, 2019 at 4:29 AM Stig Rohde Døssing 
wrote:

> Jungtaek, does the workflow outlined by Yikun work for you?
>
> Den tor. 27. jun. 2019 kl. 18.00 skrev Roshan Naik
> :
>
> > Thanks for volunteering.
> >
> >
> > Sent from Yahoo Mail for iPhone
> >
> >
> > On Wednesday, June 26, 2019, 7:08 PM, Yikun Jiang 
> > wrote:
> >
> > Yes, we will definitely help to fix ARM test failures in PRs, and the
> > OpenLab CI will tell us which PR perhaps has ARM compatible problem, if
> > it's an easy fix problem, I think authors can fix it by themselves. If
> not,
> > we will help them to address it.
> >
> > For the lack of ARM environment, if the developer are interested in
> testing
> > and debugging their storm patch in ARM  environment but don't have ARM
> > environment , we can also provide tmp ARM environment to them for
> testing.
> >
> > Regards,
> > Yikun
> > 
> > Jiang Yikun（Kero）
> > Mail: yikunk...@gmail.com
> >
> >
> > Stig Rohde Døssing  于2019年6月26日周三 上午1:48写道：
> >
> > > It sounds pretty low risk for us, if you're volunteering to help fix
> any
> > > ARM-specific CI failures that may crop up. Will this include helping to
> > fix
> > > ARM test failures in PRs? Most people are unlikely to have an ARM
> > > environment they can use to test, and I'd prefer not to ask
> contributors
> > to
> > > fix ARM test failures themselves.
> > >
> > > Den tir. 25. jun. 2019 kl. 11.08 skrev Yikun Jiang <
> yikunk...@gmail.com
> > >:
> > >
> > > > Sorry for late reply, actually, we have a developer team that willing
> > to
> > > > work on this, and I'm the owner of storm ARM CI in the OpenLab team.
> We
> > > not
> > > > only want to enable the OpenLab CI, but also want to maintain the arm
> > CI
> > > > job and fix the CI issue in Storm project. That means if the storm
> > > project
> > > > has some ARM compatible problem, I will very happy to fix it.
> > > >
> > > > As I mentioned before, one of the OpenLab goal is to make more open
> > > source
> > > > software to be more compatible for aarch64 platform. And the Storm
> > > project
> > > > is one of the most important one in BigData area, so we would like to
> > > > propose to work on aarch64 related works in Storm. we plan to start
> the
> > > > aarch64 related work from to add aarch64 build job for Storm, and our
> > > > initial plan as below:
> > > >
> > > > 1. We first propose enable a periodical job in OpenLab project to
> make
> > > sure
> > > > the storm can compile and build successfully in arm64 env.
> > > > 2. We hope can run the OpenLab CI in every pr to ensure that any pr
> > after
> > > > Step 1 will not break the arm64 build.
> > > > 3. We can add more complex test cases on aarch64, like unit tests and
> > > > functional tests, step by step. It's a log term works.
> > > >
> > > > Of course, welcome another developers join to maintain the aarch64
> CI,
> > > and
> > > > take effort on aarch64 work together.
> > > >
> > > > Thanks for your attention.
> > > >
> > > > Regards,
> > > > Yikun
> > > > --------
> > > > Jiang Yikun（Kero）
> > > > Mail: yikunk...@gmail.com
> > > >
> > > >
> > > > Stig Rohde Døssing  于2019年6月21日周五 上午2:23写道：
> > > >
> > > > > I guess there is no interest in maintaining ARM compatibility, or
> at
> > > > least
> > > > > no one currently wants to take on the effort

Re: [Discuss] ARM CI for Storm

[Jungtaek Lim]

I guess the point is not related to open CI build. The point is whether we
really want to support ARM. I'm seeing OpenLab request for other Apache
projects as well, so I'd rather not treat their request as commitment of
putting efforts to make builds on ARM green.

I'm a bit hesitant to add some environment on the support list unless
there're enough engineers willing to work on maintaining. If the work is
left to existing maintainers, we will be stuck on failing tests on ARM CI
but no environment to play locally.

So I'd be +1 if there're at least two folks experienced with ARM env.
volunteer to maintain the compatibility part. Otherwise -1 here.

Thanks,
Jungtaek Lim (HeartSaVioR)

On Mon, Jun 10, 2019 at 7:27 PM Stig Rohde Døssing 
wrote:

> Hi,
>
> An issue was opened suggesting adding ARM CI to the Storm build, using
> OpenLab https://issues.apache.org/jira/browse/STORM-3401, associated PR
> here https://github.com/apache/storm/pull/3023
>
> As far as I can tell, we need to allow OpenLab access to the Github PRs,
> similar to how Travis has permission to access our PRs. The app that needs
> access is https://github.com/apps/theopenlab-ci. Note that the app will
> have access to the PRs only, not the repo code.
>
> Is anyone opposed to me asking infra to allow OpenLab access to our PRs?
> I'll let this thread sit for a week or so, if there is no opposition I'll
> ask infra to give the app access.
>


-- 
Name : Jungtaek Lim
Blog : http://medium.com/@heartsavior
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior

Re: [VOTE] Release Apache Storm 1.2.3 (rc2)

[Jungtaek Lim]

We have one binding vote needed to pass the RC. Could any PMC participate
and finalize the vote?

On Mon, May 13, 2019 at 7:06 PM Roshan Naik 
wrote:

>  +1- verified binary distribution- setup local cluster - ran some perf
> topos with two workers and acking enabled- basic checking of the UI
> -roshan
>
>
> On Saturday, May 11, 2019, 1:21:26 PM EDT, Alexandre Vermeerbergen <
> avermeerber...@gmail.com> wrote:
>
>  Oh and I forgot to mention that all my tests are made using
> AdoptOpenJDK11.0.3 with OpenJ9.
> It all runs very fine with this Java version & VM type.
> We just had to patch our Hadoop dependencies (for HDFS & HBase Bolts)
> to fix badly written Java version detection code in these
> dependencies.
>
>
> Le sam. 11 mai 2019 à 19:18, Alexandre Vermeerbergen
>  a écrit :
> >
> > +1 (non binding)
> >
> > - Downloaded binary distribution & used it to update my Storm cloud
> service
> > - Downloaded source distribution, compiled it and used the generated
> > .jar files to update my Storm builder dependencies (storm-core.jar,
> > storm-kafa-client.jar, flux*.jar)
> > - Run my clusters with ~15 topologies
> >
> > Disclaimer : I have not yet ran test at big scale with this Storm
> > 1.2.3 update, but I noticed from release notes that I was already
> > running for several months with a Storm 1.2.3 pre-version updated my
> > all commits that sounded useful to us (in particular the ones allowing
> > to use recent Kafka brokers), so basically I'm confident to run 1.2.3
> > final at scale within very short time after it'll be released.
> >
> > Kind regards,
> > Alexandre Vermeerbergen
> >
> > Le ven. 10 mai 2019 à 18:28, P. Taylor Goetz  a
> écrit :
> > >
> > > This is a call to vote on releasing Apache Storm 1.2.3 (rc2)
> > >
> > > Full list of changes in this release:
> > >
> > >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.3-rc2/RELEASE_NOTES.html
> > >
> > > The tag/commit to be voted upon is v1.2.3:
> > >
> > >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=ac0e39d7cad48b3ec863c8a3b711d36511d4daf6;hb=4e162a47c8219546ab9639401363a8f1b5e51119
> > >
> > > The source archive being voted upon can be found here:
> > >
> > >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.3-rc2/apache-storm-1.2.3-src.tar.gz
> > >
> > > Other release files, signatures and digests can be found here:
> > >
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.3-rc2/
> > >
> > > The release artifacts are signed with the following key:
> > >
> > >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > >
> > > The Nexus staging repository for this release is:
> > >
> > > https://repository.apache.org/content/repositories/orgapachestorm-1081
> > >
> > > Please vote on releasing this package as Apache Storm 1.2.3.
> > >
> > > When voting, please list the actions taken to verify the release.
> > >
> > > This vote will be open for at least 72 hours.
> > >
> > > [ ] +1 Release this package as Apache Storm 1.2.3
> > > [ ]  0 No opinion
> > > [ ] -1 Do not release this package because...
> > >
> > > Thanks to everyone who contributed to this release.
> > >
> > > -Taylor



-- 
Name : Jungtaek Lim
Blog : http://medium.com/@heartsavior
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior

Re: [VOTE] Release Apache Storm 1.2.3 (rc2)

[Jungtaek Lim]

+1 (binding)

> source

- verify file (MD5, SHA)
-- source, tar.gz : OK
-- source, zip : OK

- extract file
-- source, tar.gz : OK
-- source, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- build source with JDK 8
-- source, tar.gz : OK

- build source dist
-- source, tar.gz : OK

- build binary dist
-- source, tar.gz : OK

> binary

- verify file (MD5, SHA)
-- binary, tar.gz : OK
-- binary, zip : OK

- extract file
-- binary, tar.gz : OK
-- binary, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- launch daemons : OK

- run ExclamationTopology (local) : OK

- run RollingTopWords (remote) : OK
  - activate / deactivate / rebalance / kill : OK
  - logviewer (worker dir, daemon dir) : OK
  - change log level : OK
  - thread dump, heap dump, restart worker : OK
  - log search : OK

Thanks,
Jungtaek Lim (HeartSaVioR)

On Sat, May 11, 2019 at 1:28 AM P. Taylor Goetz  wrote:

> This is a call to vote on releasing Apache Storm 1.2.3 (rc2)
>
> Full list of changes in this release:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.3-rc2/RELEASE_NOTES.html
>
> The tag/commit to be voted upon is v1.2.3:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=ac0e39d7cad48b3ec863c8a3b711d36511d4daf6;hb=4e162a47c8219546ab9639401363a8f1b5e51119
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.3-rc2/apache-storm-1.2.3-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.3-rc2/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1081
>
> Please vote on releasing this package as Apache Storm 1.2.3.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 1.2.3
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor



-- 
Name : Jungtaek Lim
Blog : http://medium.com/@heartsavior
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior

Re: [VOTE] Release Apache Storm 2.0.0 RC7

[Jungtaek Lim]

+1 (binding)

> source

- verify file (MD5, SHA)
-- source, tar.gz : OK
-- source, zip : OK

- extract file
-- source, tar.gz : OK
-- source, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- build source with JDK 8
-- source, tar.gz : OK

- build source dist
-- source, tar.gz : OK

- build binary dist
-- source, tar.gz : OK

> binary

- verify file (MD5, SHA)
-- binary, tar.gz : OK
-- binary, zip : OK

- extract file
-- binary, tar.gz : OK
-- binary, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- launch daemons : OK

- run ExclamationTopology (local) : OK

- run RollingTopWords (remote) : OK
  - activate / deactivate / rebalance / kill : OK
  - logviewer (worker dir, daemon dir) : OK
  - change log level : OK
  - thread dump, heap dump, restart worker : OK
  - log search : OK

Thanks,
Jungtaek Lim (HeartSaVioR)

On Wed, May 1, 2019 at 4:38 AM Kishorkumar Patil
 wrote:

> Built from the source code. All Unit tests passed.
> Ran some basic tests with WordCountTopology, BasicDRPCTopology.
> Run ThroughputVsLatency for comparison and it looks good- with 99.9% of
> events processed within 5.4ms.
> +1
>
>
> On Tue, Apr 30, 2019 at 2:16 PM Derek Dagit 
> wrote:
>
> > OK ran the distribution from the tarball, and built and ran from the src
> > tarball on macOS with JDK8. Tested WordCountTopology on both and it
> seemed
> > fine.
> >
> > +1
> >
> > On Tue, Apr 30, 2019 at 11:55 AM Derek Dagit  wrote:
> >
> > > Sent too soon:
> > >
> > > - I verified sigs and sums
> > > - Built the source from the tarball on macOS
> > >   - I saw a couple of test errors because the local stormconf.ser was
> > gone
> > > when the supervisor tried to open it, and this resulted in a crash. I
> > > re-ran the tests and they passed.
> > >   - I do not think this is a blocker, but it is probable worth looking
> > > into.
> > >
> > > I'll continue to evaluate the RC, but wanted to send this out in case
> > > anyone else has seen something similar.
> > >
> > > On Tue, Apr 30, 2019 at 11:52 AM Derek Dagit  wrote:
> > >
> > >> - I verified sigs and sums
> > >>
> > >> On Mon, Apr 29, 2019 at 5:49 PM P. Taylor Goetz 
> > >> wrote:
> > >>
> > >>> This is a call to vote on releasing Apache Storm 2.0.0 (rc7)
> > >>>
> > >>> Full list of changes in this release:
> > >>>
> > >>>
> > >>>
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc7/RELEASE_NOTES.html
> > >>>
> > >>> The tag/commit to be voted upon is v2.0.0:
> > >>>
> > >>>
> > >>>
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=007863edd95e838b3df414928c6fa3f28244ab49;hb=2ba95bbd1c911d4fc6363b1c4b9c4c6d86ac9aae
> > >>>
> > >>> The source archive being voted upon can be found here:
> > >>>
> > >>>
> > >>>
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc7/apache-storm-2.0.0-src.tar.gz
> > >>>
> > >>> Other release files, signatures and digests can be found here:
> > >>>
> > >>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc7/
> > >>>
> > >>> The release artifacts are signed with the following key:
> > >>>
> > >>>
> > >>>
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > >>>
> > >>> The Nexus staging repository for this release is:
> > >>>
> > >>>
> https://repository.apache.org/content/repositories/orgapachestorm-1079
> > >>>
> > >>> Please vote on releasing this package as Apache Storm 2.0.0.
> > >>>
> > >>> When voting, please list the actions taken to verify the release.
> > >>>
> > >>> This vote will be open for at least 72 hours.
> > >>>
> > >>> [ ] +1 Release this package as Apache Storm 2.0.0
> > >>> [ ]  0 No opinion
> > >>> [ ] -1 Do not release this package because...
> > >>>
> > >>> Thanks to everyone who contributed to this release.
> > >>>
> > >>> -Taylor
> > >>
> > >>
> >
>


-- 
Name : Jungtaek Lim
Blog : http://medium.com/@heartsavior
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior

Re: [VOTE] Release Apache Storm 2.0.0 RC5

[Jungtaek Lim]

According to how other projects are doing right now, looks like we are not
doing.

https://github.com/apache/flink/blob/master/NOTICE-binary
https://github.com/apache/spark/blob/master/LICENSE-binary
https://github.com/apache/kafka/blob/trunk/LICENSE

If I understand correct, aether in storm-submit-tool is also EPL v1.0.

Btw, I just ran the command "mvn generate-resources
-Dlicense.skipAggregateAddThirdParty=false" and see the output: the format
of output looks good though the output is a bit verbose. (Attached the
output file.)

Would it be OK to include the file without cleaning up?

2019년 3월 25일 (월) 오후 8:19, Stig Rohde Døssing 님이 작성:

> 0
>
> Built and ran tests from source zip.
> Ran ExclamationTopology on local install set up from binary zip.
> Verified no unexpected error logs.
> Ran integration test locally.
> Clicked around in UI for a bit, checked that logviewer works.
> Ran the license check plugin, and verified that all dependency licenses are
> either category A or B.
>
> We have some category B dependencies, e.g. JAXB and Jersey under CDDL.
> https://apache.org/legal/resolved.html#appropriately-labelled-condition
> mentions that we should list category B dependencies somewhere visible to
> users. Do we do this currently?
>
> Den fre. 22. mar. 2019 kl. 21.23 skrev P. Taylor Goetz  >:
>
> > This is a call to vote on releasing Apache Storm 2.0.0 (rc5)
> >
> > Full list of changes in this release:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc5/RELEASE_NOTES.html
> >
> > The tag/commit to be voted upon is v2.0.0:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=7e0a711e4ed5315f04f9f726caff61e0f169e320;hb=b5823809bd4b438e789a36f163f318d4b161ad13
> >
> > The source archive being voted upon can be found here:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc5/apache-storm-2.0.0-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc5/
> >
> > The release artifacts are signed with the following key:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1076
> >
> > Please vote on releasing this package as Apache Storm 2.0.0.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 2.0.0
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
>

List of third-party dependencies grouped by their license type.


Apache License

* HttpClient (commons-httpclient:commons-httpclient:3.0.1 - 
http://jakarta.apache.org/commons/httpclient/)
* HttpClient (commons-httpclient:commons-httpclient:3.1 - 
http://jakarta.apache.org/httpcomponents/httpclient-3.x/)
* HttpClient (org.apache.httpcomponents:httpclient:4.2.5 - 
http://hc.apache.org/httpcomponents-client)
* HttpCore (org.apache.httpcomponents:httpcore:4.2.4 - 
http://hc.apache.org/httpcomponents-core-ga)

Apache License, Version 2.0

* ActiveMQ :: Broker (org.apache.activemq:activemq-broker:5.15.8 - 
http://activemq.apache.org/activemq-broker)
* ActiveMQ :: Client (org.apache.activemq:activemq-client:5.15.8 - 
http://activemq.apache.org/activemq-client)
* ActiveMQ :: KahaDB Store 
(org.apache.activemq:activemq-kahadb-store:5.15.8 - 
http://activemq.apache.org/activemq-kahadb-store)
* ActiveMQ :: MQTT Protocol (org.apache.activemq:activemq-mqtt:5.15.8 - 
http://activemq.apache.org/activemq-mqtt)
* ActiveMQ :: Openwire Legacy Support 
(org.apache.activemq:activemq-openwire-legacy:5.15.8 - 
http://activemq.apache.org/activemq-openwire-legacy)
* ActiveMQ Protocol Buffers Implementation and Compiler 
(org.apache.activemq.protobuf:activemq-protobuf:1.1 - 
http://activemq.apache.org/activemq-protobuf)
* Aether :: API (org.sonatype.aether:aether-api:1.7 - 
http://aether.sonatype.org/aether-api/)
* Aether :: Implementation (org.sonatype.aether:aether-impl:1.7 - 
http://aether.sonatype.org/aether-impl/)
* Aether :: SPI (org.sonatype.aether:aether-spi:1.7 - 
http://aether.sonatype.org/aether-spi/)
* Aether :: Utilities (org.sonatype.aether:aether-util:1.7 - 
http://aether.sonatype.org/aether-util/)
* Aggregate Designer Algorithm 
(net.hydromatic:aggdesigner-algorithm:6.0 - 
http://github.com/julianhyde/aggdesigner/aggdesigner-algorithm)
* aircompressor (io.airlift:aircompressor:0.3 - 
http://github.com/airlift/aircompressor)
* Annotation 1.0 
(org.apache.geronimo.specs:geronimo-annotation_1.0_spec:1.1.1 - 

Re: Storm 2.0 blogs ?

[Jungtaek Lim]

Thanks all for bringing nice ideas and items to announce and publicize!

Though not a topic for blog, I would like to let us remind the change of
modules (`storm-core` -> `storm-client` and `storm-server`) in release
announcement.

>From Storm 2.0.0, most of the cases end users would need to add only
`storm-client` as a dependency in their topology jar, which greatly reduces
unnecessary dependencies. If my memory is right, there's a case end users
still need to add `storm-server` as a dependency as well - run local
cluster directly instead of storm command - but not sure it is still valid
after we introduced `storm local`.
(Bobby, could you confirm this?)

The way of launching topology in local mode is also changed as well: `storm
local`instead of `storm jar`. The command is documented in storm.py, but
looks like missing in docs/Command-line-client.md so need to add it there.

https://github.com/apache/storm/blob/8a475696e908c53f1c06bf1a8f373d8ac0483427/bin/storm.py#L327-L353

Our strategy on dependency resolution has also changed as well: instead of
shading most of dependencies, we put our best effort to reduce dependencies
for worker (storm-client), and shade only some of dependencies which are
known to be troublesome for version conflict. I guess it's worth to mention
in release announcement too.

https://github.com/apache/storm/blob/8a475696e908c53f1c06bf1a8f373d8ac0483427/shaded-deps/pom.xml#L36-L158

- Jungtaek Lim (HeartSaVioR)

2019년 1월 23일 (수) 오전 4:39, Arun Mahadevan 님이 작성:

> Nice suggestions Roshan and Taylor.
>
> I think we can start with a release announcement (maybe with pointers to
> the docs) and follow it up with blog posts that deep dives into the
> different features and improvements.
>
> A few from my side for the release announcement would be,
>
> - Streams API - A typed API for users to express their streaming
> computations more easily and supports functional style operations. [1]
> - Windowing enhancements and support for window state checkpointing [2]
>
> Would also try to do some write up for blogs around this once we figured
> out how to go about the blog series.
>
> - Arun
>
> [1] https://github.com/apache/storm/blob/master/docs/Stream-API.md
> [2]
>
> https://github.com/apache/storm/blob/master/docs/Windowing.md#stateful-windowing
>
>
>
> On Tue, 22 Jan 2019 at 09:57, P. Taylor Goetz  wrote:
>
> > Awesome. I’ve been thinking about this as well.
> >
> > The release notes don’t really tell the story of everything in this
> > release, and some of the new features and improvements elude my memory.
> >
> > If others could point out new features that they’d like to be pointed out
> > in the release announcement, please list them in this thread. Better yet
> > would be a blurb describing the feature/improvement and what benefits it
> > brings users. Then I can stitch them together into a release
> announcement.
> > Basically crowd/dev-sourcing the release announcement.
> >
> > Or, instead of one big announcement, do we want to do a series of blog
> > posts? If we go the latter route, the first post should probably cover
> the
> > most significant features. What would those be? Roshan has a pretty good
> > start.
> >
> > What would others add?
> >
> > If anyone wants to help out, let me know.
> >
> > -Taylor
> >
> > > On Jan 22, 2019, at 10:37 AM, Bobby Evans  wrote:
> > >
> > > I totally agree, especially with the performance improvements in it.
> > >
> > > Thanks,
> > >
> > > Bobby
> > >
> > > On Tue, Jan 22, 2019 at 12:40 AM Roshan Naik
> > 
> > > wrote:
> > >
> > >> Now that  2.0 has all the votes it needs to move forward, maybe a good
> > >> time to think of some blogs to go with this long awaited release.
> > >> Some potential topics that come to mind are:
> > >> 1- Overview of new features and major changes since 1.x2-
> > Re-architecture
> > >> (messaging, threading, back pressure)3- Micro benchmarks4- Revisit the
> > >> famous Yahoo benchmark5- Window state persistence6- SQL enhancements7-
> > new
> > >> Metrics stuff8- Kafka related changes9- Security10- An area you have
> > worked
> > >> on ?11- Other ideas ?
> > >> Anyone interested in contributing blogs ?
> > >> FYI: I am working on content for topics 2 & 3.
> > >>
> > >> -roshan
> >
> >
>

Re: [NOTICE] Migration of storm repos to gitbox is done

[Jungtaek Lim]

Thanks for the effort, Ethan!

Just a reminder for all committers: If my understanding is right, now all
committers should set up your account into ASF Gitbox to be registered as
storm team and be able to get write access. I'm seeing only subset of
members in storm committer now.

2019년 1월 21일 (월) 오후 5:27, Roshan Naik 님이 작성:

> Thanks Ethan.
>
>
> Sent from Yahoo Mail for iPhone
>
>
> On Sunday, January 20, 2019, 7:33 PM, Ethan Li 
> wrote:
>
> Hi Team,
>
> The migration to https://gitbox.apache.org/ is done (
> https://issues.apache.org/jira/browse/INFRA-17705 <
> https://issues.apache.org/jira/browse/INFRA-17705>).
>
> Thanks,
>
> - Ethan
>
>
>

Re: [VOTE] Release Apache Storm 1.1.4 (rc1)

[Jungtaek Lim]

+1 (binding)

> source

- verify file (MD5, SHA)
-- source, tar.gz : OK
-- source, zip : OK

- extract file
-- source, tar.gz : OK
-- source, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- build source with JDK 8
-- source, tar.gz : OK

- build source dist
-- source, tar.gz : OK

- build binary dist
-- source, tar.gz : OK

> binary

- verify file (MD5, SHA)
-- binary, tar.gz : OK
-- binary, zip : OK

- extract file
-- binary, tar.gz : OK
-- binary, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- launch daemons : OK

- run ExclamationTopology (local) : OK

- run RollingTopWords (remote) : OK
  - activate / deactivate / rebalance / kill : OK
  - logviewer (worker dir, daemon dir) : OK
  - change log level : OK
  - thread dump, heap dump, restart worker : OK
  - log search : OK

Thanks,
Jungtaek Lim (HeartSaVioR)

2019년 1월 10일 (목) 오전 8:06, Ethan Li 님이 작성:

> +1
>
> - Built from the src, ran all the unit tests and integration tests.
> - Set up a single-node cluster and submit a word count topology.
> - Checked the UI.
> They look good.
>
> Thanks
> Ethan
>
>
> > On Jan 9, 2019, at 12:37 PM, Bobby Evans  wrote:
> >
> > +1 I built the code from the git tag and all the unit tests passed.  I
> also
> > ran some manual tests.
> >
> > Thanks,
> >
> > Bobby
> >
> > On Wed, Jan 9, 2019 at 11:01 AM P. Taylor Goetz 
> wrote:
> >
> >> This is a call to vote on releasing Apache Storm 1.1.4 (rc1)
> >>
> >> Full list of changes in this release:
> >>
> >>
> >>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.4-rc1/RELEASE_NOTES.html
> >>
> >> The tag/commit to be voted upon is v1.1.4:
> >>
> >>
> >>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=dfb1f971c35ca985a35692aed6d28be3d7e707de;hb=babab9648a6deb3d444c45bdd6d0ec30804e91f1
> >>
> >> The source archive being voted upon can be found here:
> >>
> >>
> >>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.4-rc1/apache-storm-1.1.4-src.tar.gz
> >>
> >> Other release files, signatures and digests can be found here:
> >>
> >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.4-rc1/
> >>
> >> The release artifacts are signed with the following key:
> >>
> >>
> >>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >>
> >> The Nexus staging repository for this release is:
> >>
> >> https://repository.apache.org/content/repositories/orgapachestorm-1075
> >>
> >> Please vote on releasing this package as Apache Storm 1.1.4.
> >>
> >> When voting, please list the actions taken to verify the release.
> >>
> >> This vote will be open for at least 72 hours.
> >>
> >> [ ] +1 Release this package as Apache Storm 1.1.4
> >> [ ]  0 No opinion
> >> [ ] -1 Do not release this package because...
> >>
> >> Thanks to everyone who contributed to this release.
> >>
> >> -Taylor
>
>

Re: [VOTE] Release Apache Storm 1.2.3 (rc1)

[Jungtaek Lim]

+1 (binding)

> source

- verify file (MD5, SHA)
-- source, tar.gz : OK
-- source, zip : OK

- extract file
-- source, tar.gz : OK
-- source, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- build source with JDK 8
-- source, tar.gz : OK

- build source dist
-- source, tar.gz : OK

- build binary dist
-- source, tar.gz : OK

> binary

- verify file (MD5, SHA)
-- binary, tar.gz : OK
-- binary, zip : OK

- extract file
-- binary, tar.gz : OK
-- binary, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- launch daemons : OK

- run ExclamationTopology (local) : OK

- run RollingTopWords (remote) :
  - activate / deactivate / rebalance / kill : OK
  - logviewer (worker dir, daemon dir) : OK
  - change log level : OK
  - thread dump, heap dump, restart worker : OK
  - log search : OK

Thanks,
Jungtaek Lim (HeartSaVioR)

2019년 1월 15일 (화) 오후 4:55, Alexandre Vermeerbergen 님이
작성:

> Hello Taylor,
>
> +1
>
> DownloadedStorm1.2.3 RC1 + build 1.2.3 RC1 from sources to get up to
> date storm-kafka-client
>
> Upgraded by integration cluster with 1.2.3 RC.
> Configuration:
> - CentOS7, Java 11 from AdoptOpenJDK11.0.1 with Hotspot  on all VMs
> - 1 Nimbus VM, 7 Supervisor,VMs, 3 Zookeeper VMs, 19 topologies with
> high dependency on Storm
> Kafka Client, using Kafka Client 2.0.1 connected to a distinct Kafka
> Brokers Cluster (5 VMs) based on Kafka 1.1.1 (this later one still
> runs with Java 8).
>
> Kind regards,
> Alexandre
>
> Le mar. 8 janv. 2019 à 21:20, P. Taylor Goetz  a écrit
> :
> >
> > This is a call to vote on releasing Apache Storm 1.2.3 (rc1)
> >
> > Full list of changes in this release:
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.3-rc1/RELEASE_NOTES.html
> >
> > The tag/commit to be voted upon is v1.2.3:
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=5eaaa51591f2e4dc3e31f22bcc581af4a1f39c03;hb=6ba98b215857656e0186887b5d1a6a5aceee10c4
> >
> > The source archive being voted upon can be found here:
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.3-rc1/apache-storm-1.2.3-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.3-rc1/
> >
> > The release artifacts are signed with the following key:
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1074
> >
> > Please vote on releasing this package as Apache Storm 1.2.3.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 1.2.3
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
>

Re: [VOTE] Migration of git repos to gitbox.apache.org

[Jungtaek Lim]

Thanks Ethan for leading the effort on migration! +1 from me.

Thanks,
Jungtaek Lim (HeartSaVioR)

2019년 1월 19일 (토) 오전 4:48, Govind Menon 님이 작성:

> +1
>
> On Fri, 18 Jan 2019 at 13:33, Stig Rohde Døssing 
> wrote:
>
> > +1
> >
> > Den fre. 18. jan. 2019 kl. 16.59 skrev Kishorkumar Patil
> > :
> >
> > > +1
> > >
> > > On Fri, Jan 18, 2019 at 10:47 AM Ethan Li 
> > > wrote:
> > >
> > > > This is a call to vote on migration of git repos to
> gitbox.apache.org
> > <
> > > > http://gitbox.apache.org/>.
> > > >
> > > > The following repositories on git-wip-us belong to our project:
> > > > - storm-site.git
> > > > - storm.git
> > > >
> > > > All repositories not migrated on February 7th will be mass migrated
> > > > without warning. It’s better to work with the infrastructure team to
> > > avoid
> > > > the mass that day.
> > > >
> > > > The notice email is at
> > > >
> > >
> >
> https://lists.apache.org/thread.html/2bb953946ee6652a054329df2814f512f9186e863ec841555f9cc26a@%3Cdev.storm.apache.org%3E
> > > > <
> > > >
> > >
> >
> https://lists.apache.org/thread.html/2bb953946ee6652a054329df2814f512f9186e863ec841555f9cc26a@%3Cdev.storm.apache.org%3E
> > > > >
> > > >
> > > > This vote will be open for at least 24 hours.
> > > >
> > > > Thanks
> > > >
> > > > - Ethan
> > >
> > > --
> > > -Kishor
> > >
> >
>

Re: [VOTE] Release Apache Storm 2.0.0 (rc4)

[Jungtaek Lim]

Hello all,

+1 (binding)

> source

- verify file (MD5, SHA)
-- source, tar.gz : OK
-- source, zip : OK

- extract file
-- source, tar.gz : OK
-- source, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- build source with JDK 8 (-Pall-tests && -Pexternals)
-- source, tar.gz : OK

- build source dist
-- source, tar.gz : OK

- build binary dist
-- source, tar.gz : OK

> binary

- verify file (MD5, SHA)
-- binary, tar.gz : OK
-- binary, zip : OK

- extract file
-- binary, tar.gz : OK
-- binary, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- launch daemons : OK

- run ExclamationTopology (local) : OK

- run RollingTopWords (remote) : OK
  - activate / deactivate / rebalance / kill : OK
  - logviewer (worker dir, daemon dir) : OK
  - change log level : OK
  - thread dump, heap dump, restart worker : OK
  - log search : OK

I don't think perf. degrade is a blocker for this release, since we are
still providing far better performance then 1.x. As Roshan is planning to
address this, that could be fixed in 2.0.x hopefully.

Thanks,
Jungtaek Lim (heartSaVioR)

2019년 1월 19일 (토) 오전 5:23, Roshan Naik 님이 작성:

>  Was verifying the performance on 2.0 and noticed that the inter worker
> messaging has fallen off a cliff  With ConstSpoutNullBoltTopo (producer
> batch size=1k, ackers=0, workers=2) it was ~3.2mill/sec down  and now its
> down to  ~1.6mill/sec. Something has impacted this recently. Will try to
> narrow down the issue by tomorrow hopefully. Single worker numbers look
> good.
> -roshan
> On Friday, January 11, 2019, 2:46:14 AM PST, Stig Rohde Døssing <
> stigdoess...@gmail.com> wrote:
>
>  Thanks Arun,
>
> I don't have an opinion on it, just wanted to make sure that it wasn't an
> oversight.
>
> +1
>
> Set up Storm from the binary zip file
> Built storm-kafka-client-examples from the zip while pointing Maven at the
> Nexus repo
> Ran one of the example Kafka topologies with a single-node cluster against
> a local Kafka. Verified there were no errors in the logs, clicked around in
> UI.
>
> Den tor. 10. jan. 2019 kl. 22.25 skrev Arun Mahadevan :
>
> > This is for users to use the "auto credentials" mechanism (delegation
> > tokens) with HDFS/Hive/Hbase.
> >
> > We have been shipping it since 1.x (I think since 1.2.0 release) so that
> > users can just add that directory to class path rather than building it
> > separately to get the right dependencies. We could consider removing it
> > from the main binary and ship it separately but it will need changes to
> the
> > build, release and documentation and users will need to download and
> > install it separately.
> >
> >
> > Thanks,
> > Arun
> >
> > On Thu, 10 Jan 2019 at 10:28, Stig Rohde Døssing  >
> > wrote:
> >
> > > I think this was remarked on by Roshan in the last RC, but the binary
> > > distribution has become significantly larger since 1.x. It looks like
> > this
> > > is down to storm-autocreds not being added to the exclusion list in
> > > storm-dist/binary/final-package/src/main/assembly/binary.xml.
> > >
> > > Since the module isn't excluded, external/storm-autocreds contains the
> > > module jar, plus all dependency jars. Is this an accident, or do we
> want
> > to
> > > include these jars in the distribution?
> > >
> > > Den ons. 9. jan. 2019 kl. 19.48 skrev Ethan Li <
> > ethanopensou...@gmail.com
> > > >:
> > >
> > > > +1
> > > >
> > > > - Built from the src, ran all the unit tests and integration tests.
> > > > - Set up a single-node cluster and submit ThroughputVsLatency
> topology.
> > > > - Checked the UI.
> > > > They look good.
> > > >
> > > > Thanks
> > > > Ethan
> > > >
> > > > > On Jan 9, 2019, at 8:48 AM, Bobby Evans  wrote:
> > > > >
> > > > > +1 built from the git tag.  Ran all of the unit tests and ran some
> > > manual
> > > > > tests they all passed.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Bobby
> > > > >
> > > > > On Tue, Jan 8, 2019 at 6:30 PM Xin Wang 
> > > wrote:
> > > > >
> > > > >> +1
> > > > >>
> > > > >> Built it and ran all of the tests.  Everything passed.
> > > > >>
> > > > >> -Xin
> > > > >>
> > > > >> Kishorkumar Patil  于2019年1月9日周三
> 上午5:08写道：
> > > > >>

Re: New Storm Committer and PMC Member

[Jungtaek Lim]

Congrats Govind! Keep up the good work.

-Jungtaek Lim (HeartSaVioR)

2019년 1월 10일 (목) 오전 11:47, Xin Wang 님이 작성:

> Congrats Govind!
>
> -Xin
>
> Arun Mahadevan  于2019年1月10日周四 上午3:23写道：
>
> > Congratulations Govind, keep up the good work.
> >
> > - Arun
> >
> > On Wed, 9 Jan 2019 at 11:13, Kishorkumar Patil 
> > wrote:
> >
> > > Congratulations Govind!
> > >
> > > -Kishor
> > >
> > >
> > > On Wed, Jan 9, 2019 at 1:00 PM Stig Rohde Døssing <
> > stigdoess...@gmail.com>
> > > wrote:
> > >
> > > > Congratulations.
> > > >
> > > > Den ons. 9. jan. 2019 kl. 19.55 skrev Roshan Naik
> > > > :
> > > >
> > > > > Congratulations Govind. Roshan
> > > > >
> > > > >
> > > > > Sent from Yahoo Mail for iPhone
> > > > >
> > > > >
> > > > > On Wednesday, January 9, 2019, 10:47 AM, Ethan Li <
> > > > > ethanopensou...@gmail.com> wrote:
> > > > >
> > > > > Congratulations! Govind. Well deserved!
> > > > >
> > > > > Ethan
> > > > >
> > > > > > On Jan 9, 2019, at 12:40 PM, Hugo Louro 
> > wrote:
> > > > > >
> > > > > > Congratulations Govind. Very well deserved. Thank you for all
> your
> > > > > > contributions and dedication to the Storm project.
> > > > > >
> > > > > > Best,
> > > > > > Hugo
> > > > > >
> > > > > > On Wed, Jan 9, 2019 at 10:36 AM Bobby Evans 
> > > wrote:
> > > > > >
> > > > > >> I am happy to announce that Govind Menon has just been added as
> > the
> > > > > latest
> > > > > >> Committer and PMC member to the Apache Storm Project.  Please
> join
> > > me
> > > > in
> > > > > >> congratulating him on this and thanking him for his
> contributions
> > so
> > > > > far.
> > > > > >>
> > > > > >> Thanks,
> > > > > >>
> > > > > >> Bobby Evans
> > > > > >>
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> Thanks,
> Xin
>

Re: [VOTE] Release Apache Storm 2.0.0 (rc2)

[Jungtaek Lim]

Thanks all for the quick turnaround! Here's my +1 (binding).

> source

- verify file (signature, MD5, SHA)
-- source, tar.gz : OK
-- source, zip : OK

- extract file
-- source, tar.gz : OK
-- source, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- build source with JDK 8 (-Pall-tests && -Pexternals)
-- source, tar.gz : OK

- build source dist
-- source, tar.gz : OK

- build binary dist
-- source, tar.gz : OK

> binary

- verify file (signature, MD5, SHA)
-- binary, tar.gz : OK
-- binary, zip : OK

- extract file
-- binary, tar.gz : OK
-- binary, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- launch daemons : OK

- run RollingTopWords (local) : OK

- run RollingTopWords (remote) : OK
  - activate / deactivate / rebalance / kill : OK
  - logviewer (worker dir, daemon dir) :OK
  - change log level : OK
  - thread dump, heap dump, restart worker : OK
  - log search :OK

Note that "profiling worker" and "topology log search" works now which were
failing in RC1.

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 10월 11일 (목) 오전 3:02, Stig Rohde Døssing 님이 작성:

> +1
>
> Built and ran unit tests from the tag.
> Ran ExclamationTopology locally using the Storm tar, verified that UI looks
> as expected, that logviewer works, and that there were no errors in the
> logs.
> Verified the signature and SHA512 for the source and binary tars.
>
> We should consider deleting the md5 files, Apache's release policy
> recommends against including them in a release
> https://www.apache.org/dev/release-distribution#sigs-and-sums.
>
>
> Den ons. 10. okt. 2018 kl. 17.02 skrev Bobby Evans :
>
> > +1
> >
> > built and ran all of the unit tests from the tag.
> > Ran some small perf tests on a single node cluster.  Things look really
> > good there.
> >
> >
> > On a side note our CI pipeline has been running and passing builds very
> > close to this release too.  (we are following master currently) and it is
> > looking good.
> >
> > Thanks,
> >
> > Bobby
> >
> > On Tue, Oct 9, 2018 at 4:02 PM Kishorkumar Patil  >
> > wrote:
> >
> > > +1  to release this package.
> > >
> > > I ran basic tests, and fucntionality tested manually some of the UI
> > > features and profiling issues reported as part of the blockers. I did
> not
> > > notice any silent failures either - or any failures/exception in the
> > logs.
> > >
> > > Regards
> > > -Kishor
> > >
> > >
> > > On Tue, Oct 9, 2018 at 4:05 PM P. Taylor Goetz 
> > wrote:
> > >
> > > > This is a call to vote on releasing Apache Storm 2.0.0 (rc2)
> > > >
> > > > Full list of changes in this release:
> > > >
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc2/RELEASE_NOTES.html
> > > >
> > > > The tag/commit to be voted upon is v2.0.0:
> > > >
> > > >
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=f8d04910dc3fd14534c186232ecf7882d8916f67;hb=f8d04910dc3fd14534c186232ecf7882d8916f67
> > > >
> > > > The source archive being voted upon can be found here:
> > > >
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc2/apache-storm-2.0.0-src.tar.gz
> > > >
> > > > Other release files, signatures and digests can be found here:
> > > >
> > > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc2/
> > > >
> > > > The release artifacts are signed with the following key:
> > > >
> > > >
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > > >
> > > > The Nexus staging repository for this release is:
> > > >
> > > >
> https://repository.apache.org/content/repositories/orgapachestorm-1071
> > > >
> > > > Please vote on releasing this package as Apache Storm 2.0.0.
> > > >
> > > > When voting, please list the actions taken to verify the release.
> > > >
> > > > This vote will be open for at least 72 hours.
> > > >
> > > > [ ] +1 Release this package as Apache Storm 2.0.0
> > > > [ ]  0 No opinion
> > > > [ ] -1 Do not release this package because...
> > > >
> > > > Thanks to everyone who contributed to this release.
> > > >
> > > > -Taylor
> > > >
> > >
> >
>

Re: Cleaning Up Old Pull Requests STORM-3250

[Jungtaek Lim]

+1 It doesn't look like there're any critical PRs in the list, and it is
pretty less chance we could connect with PR authors.

-Jungtaek Lim (HeartSaVioR)

2018년 10월 9일 (화) 오전 5:08, Kishorkumar Patil 님이 작성:

> +1.
> It would be nice to clean up old clutter while we are getting ready for
> days past 2.x
>
> -Kishor
>
>
> On Mon, Oct 8, 2018 at 3:52 PM Bobby Evans  wrote:
>
> > +1
> >
> > Sounds good to me
> >
> > On Mon, Oct 8, 2018 at 2:50 PM Derek Dagit 
> > wrote:
> >
> > > Currently there are over 150 open pull requests on the Apache Storm
> > GitHub
> > > project. Over 100 of these have not been modified in 2018.
> > >
> > > It seems we are unlikely to handle each one of these without
> significant
> > > effort and time. Looking at many of them, they seem to be abandoned by
> > the
> > > requester.
> > >
> > > I propose in STORM-3250 that we close all pull requests that have not
> > been
> > > updated in 2018 and leave any corresponding Jira issues as they are. If
> > > there are any pull requests among these should remain open, please let
> me
> > > know. I plan to wait at least a week before requesting any changes.
> > >
> > >
> > >
> > >
> >
> https://github.com/apache/storm/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+updated%3A%3C2018-01-01
> > >
> > > --
> > > Derek
> > >
> >
>

Re: [VOTE] Release Apache Storm 2.0.0 (rc1)

[Jungtaek Lim]

Just filed STORM-3238 [1] for topology log search error, and STORM-3239 [2]
for worker profile.

- Jungtaek Lim (HeartSaVioR)

1. https://issues.apache.org/jira/browse/STORM-3238
2. https://issues.apache.org/jira/browse/STORM-3239

2018년 9월 28일 (금) 오전 12:13, Govind Menon 님이 작성:

> I have a fix for the worker profiling that I'm testing now - I should have
> a PR up today.
>
> On Thu, Sep 27, 2018 at 3:50 AM Jungtaek Lim  wrote:
>
> > Sorry I was in a long holidays in South Korea and am just back today.
> I'll
> > reproduce them and file JIRAs but it is pretty simple to reproduce, basic
> > search in topology didn't work as well as operations in worker profiling
> > didn't work. So if someone would like to tackle them it should be pretty
> > easy to reproduce for someone as well.
> >
> > -Jungtaek Lim (HeartSaVioR)
> >
> > 2018년 9월 26일 (수) 오후 9:02, Kishorkumar Patil 님이
> > 작성:
> >
> > > I have patch up for  WorkerToken Auto renewal fix. I think this should
> be
> > > added to 2.x RC.
> > >
> > > https://issues.apache.org/jira/browse/STORM-3235
> > > https://github.com/apache/storm/pull/2850
> > >
> > > -Kishor
> > >
> > > On Tue, Sep 25, 2018 at 8:04 PM P. Taylor Goetz 
> > wrote:
> > >
> > > > Jungtaek, can you elaborate on the issues you ran into when testing
> the
> > > rc
> > > > and/or file JIRAs?
> > > >
> > > > -Taylor
> > > >
> > > > > On Sep 25, 2018, at 4:02 PM, Bobby Evans  wrote:
> > > > >
> > > > > Nope.  I plan to start looking into them, unless someone else is
> > > already
> > > > > doing so.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Bobby
> > > > >
> > > > > On Tue, Sep 25, 2018 at 2:48 PM P. Taylor Goetz  >
> > > > wrote:
> > > > >
> > > > >> Thanks for the update. I will cancel the vote.
> > > > >>
> > > > >> Do the fixes you merged cover the bugs that Jungtaek pointed out?
> > > > >>
> > > > >> -Taylor
> > > > >>
> > > > >>> On Sep 25, 2018, at 1:56 PM, Bobby Evans 
> wrote:
> > > > >>>
> > > > >>> Just FYI Taylor I just merged in 3 JIRA to the master branch and
> > will
> > > > >>> probably merge in the fix for the UI that Govind mentioned, but
> all
> > > of
> > > > >> them
> > > > >>> would be okay to go into 2.0.0 as well.  Since this RC looks like
> > it
> > > is
> > > > >> not
> > > > >>> going to pass if you want to move the tag up on the master
> branch,
> > > that
> > > > >>> would be fine, once we have the fixes in.
> > > > >>>
> > > > >>> Thanks,
> > > > >>>
> > > > >>> Bobby
> > > > >>>
> > > > >>> On Tue, Sep 25, 2018 at 9:31 AM Govind Menon
> >  > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>>> Hi all,
> > > > >>>>
> > > > >>>> I introduced a bug with the visualization as part of the UI
> > > migration
> > > > >> with
> > > > >>>> the following fix - https://github.com/apache/storm/pull/2849.
> > It's
> > > > >> not a
> > > > >>>> show stopper by any means and there's a workaround (specifically
> > > > hiding
> > > > >>>> system stats on the UI). I think we should include this in the
> RC.
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>> Govind.
> > > > >>>>
> > > > >>>>
> > > > >>>> On Mon, Sep 24, 2018 at 3:09 PM Alexandre Vermeerbergen <
> > > > >>>> avermeerber...@gmail.com> wrote:
> > > > >>>>
> > > > >>>>> Hello All,
> > > > >>>>>
> > > > >>>>> I will need a little bit time to test this new release with our
> > 15+
> > > > >>>>> topologies, but it's definitely worth and exiting !
> > > > >>>>> "stay tuned" :)
> > > > >>>>>
> > > 

Re: [GitHub] storm issue #1136: [STORM-1565] Multi-Lang Performance Improvements

[Jungtaek Lim]

Andrew,

could you please initiate discussion thread instead? I feel the patch is no
longer simple one to just get +1 and merge in, because everyone lose
context on the patch.

Btw, as a contributor of multilang, the concept of supporting multiple
languages are good, but not sure it can bring heavily optimized for major
languages, mostly Python. One of example is supporting "subprocess
timeout": it is actually incomplete because Interpreter languages work like
the way, GIL.

Note that the opinion is totally my own: as we move on 2.0 which has
Streams API, I'd be happy if some Python engineers put their efforts to
migrate it to python. The progress of Streams API is stuck for now, but it
is still valid for at-least-once use cases and give much convenience while
constructing topology. It should remedy the need to learn how to use for
both things, Storm and other third parties. Someone might be concerned, but
I think that's the way we need to go.

-Jungtaek Lim (HeartSaVioR)

2018년 9월 27일 (목) 오전 8:53, P. Taylor Goetz 님이 작성:

> Andrew,
>
> Thanks for bringing the private conversation back to the dev@ list so
> everyone has more context.
>
> That being said, I’d encourage the use of the public mailing lists as much
> as possible. Private conversations exclude the larger community from
> participating in potentially fruitful discussions.
>
> Don’t get me wrong, I applaud you doing the right thing in bringing this
> back to the community. I just wish it could have happened sooner.
>
> All good. We just do better next time. :)
>
> -Taylor
>
> > On Sep 26, 2018, at 5:58 PM, amontalenti  wrote:
> >
> > Github user amontalenti commented on the issue:
> >
> >https://github.com/apache/storm/pull/1136
> >
> >Was chatting with @roshannaik and @dan-blanchard today, and this PR
> came up.
> >
> >Someone on Storm team may benefit from taking a look at this as part
> of the performance revisions being done for Storm 2.0. As mentioned in
> [#428 in streamparse](
> https://github.com/Parsely/streamparse/issues/428#issuecomment-380135343)
> -- the community project for running Python multi-lang topologies with
> Storm -- getting this merged somewhere in the Storm codebase would open up
> the possibility to switch serializer from Kryo & JSON to msgpack
> throughout, which would speed up multi-lang use cases considerably. This PR
> includes a pure Java implementation of a msgpack serializer, as well as
> pointers to the right msgpack library in the Java community; it just needs
> to be reviewed, tested, and merged.
> >
> >
> > ---
>

Re: [VOTE] Release Apache Storm 2.0.0 (rc1)

[Jungtaek Lim]

Sorry I was in a long holidays in South Korea and am just back today. I'll
reproduce them and file JIRAs but it is pretty simple to reproduce, basic
search in topology didn't work as well as operations in worker profiling
didn't work. So if someone would like to tackle them it should be pretty
easy to reproduce for someone as well.

-Jungtaek Lim (HeartSaVioR)

2018년 9월 26일 (수) 오후 9:02, Kishorkumar Patil 님이 작성:

> I have patch up for  WorkerToken Auto renewal fix. I think this should be
> added to 2.x RC.
>
> https://issues.apache.org/jira/browse/STORM-3235
> https://github.com/apache/storm/pull/2850
>
> -Kishor
>
> On Tue, Sep 25, 2018 at 8:04 PM P. Taylor Goetz  wrote:
>
> > Jungtaek, can you elaborate on the issues you ran into when testing the
> rc
> > and/or file JIRAs?
> >
> > -Taylor
> >
> > > On Sep 25, 2018, at 4:02 PM, Bobby Evans  wrote:
> > >
> > > Nope.  I plan to start looking into them, unless someone else is
> already
> > > doing so.
> > >
> > > Thanks,
> > >
> > > Bobby
> > >
> > > On Tue, Sep 25, 2018 at 2:48 PM P. Taylor Goetz 
> > wrote:
> > >
> > >> Thanks for the update. I will cancel the vote.
> > >>
> > >> Do the fixes you merged cover the bugs that Jungtaek pointed out?
> > >>
> > >> -Taylor
> > >>
> > >>> On Sep 25, 2018, at 1:56 PM, Bobby Evans  wrote:
> > >>>
> > >>> Just FYI Taylor I just merged in 3 JIRA to the master branch and will
> > >>> probably merge in the fix for the UI that Govind mentioned, but all
> of
> > >> them
> > >>> would be okay to go into 2.0.0 as well.  Since this RC looks like it
> is
> > >> not
> > >>> going to pass if you want to move the tag up on the master branch,
> that
> > >>> would be fine, once we have the fixes in.
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Bobby
> > >>>
> > >>> On Tue, Sep 25, 2018 at 9:31 AM Govind Menon  >
> > >>> wrote:
> > >>>
> > >>>> Hi all,
> > >>>>
> > >>>> I introduced a bug with the visualization as part of the UI
> migration
> > >> with
> > >>>> the following fix - https://github.com/apache/storm/pull/2849. It's
> > >> not a
> > >>>> show stopper by any means and there's a workaround (specifically
> > hiding
> > >>>> system stats on the UI). I think we should include this in the RC.
> > >>>>
> > >>>> Thanks,
> > >>>> Govind.
> > >>>>
> > >>>>
> > >>>> On Mon, Sep 24, 2018 at 3:09 PM Alexandre Vermeerbergen <
> > >>>> avermeerber...@gmail.com> wrote:
> > >>>>
> > >>>>> Hello All,
> > >>>>>
> > >>>>> I will need a little bit time to test this new release with our 15+
> > >>>>> topologies, but it's definitely worth and exiting !
> > >>>>> "stay tuned" :)
> > >>>>>
> > >>>>> Best regards,
> > >>>>> Alexandre Vermeerbergen
> > >>>>>
> > >>>>> Le ven. 21 sept. 2018 à 20:23, P. Taylor Goetz  >
> > a
> > >>>>> écrit :
> > >>>>>>
> > >>>>>> This is a call to vote on releasing Apache Storm 2.0.0 (rc1)
> > >>>>>>
> > >>>>>> Full list of changes in this release:
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc1/RELEASE_NOTES.html
> > >>>>>>
> > >>>>>> The tag/commit to be voted upon is v2.0.0:
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=062bb2f158a66b802a3ad7bdda14fb8f70cf9b88;hb=062bb2f158a66b802a3ad7bdda14fb8f70cf9b88
> > >>>>>>
> > >>>>>> The source archive being voted upon can be found here:
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc1/apache-storm-2.0.0-src.tar.gz
> > >>>>>>
> > >>>>>> Other release files, signatures and digests can be found here:
> > >>>>>>
> > >>>>>>
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc1/
> > >>>>>>
> > >>>>>> The release artifacts are signed with the following key:
> > >>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > >>>>>>
> > >>>>>> The Nexus staging repository for this release is:
> > >>>>>>
> > >>>>>>
> > >>
> https://repository.apache.org/content/repositories/orgapachestorm-1070
> > >>>>>>
> > >>>>>> Please vote on releasing this package as Apache Storm 2.0.0.
> > >>>>>>
> > >>>>>> When voting, please list the actions taken to verify the release.
> > >>>>>>
> > >>>>>> This vote will be open for at least 72 hours.
> > >>>>>>
> > >>>>>> [ ] +1 Release this package as Apache Storm 2.0.0
> > >>>>>> [ ]  0 No opinion
> > >>>>>> [ ] -1 Do not release this package because...
> > >>>>>>
> > >>>>>> Thanks to everyone who contributed to this release.
> > >>>>>>
> > >>>>>> -Taylor
> > >>>>>
> > >>>>
> > >>
> > >>
> >
> >
>

Re: [VOTE] Release Apache Storm 2.0.0 (rc1)

[Jungtaek Lim]

First of all, thanks Bobby for driving all the Storm 2.0.0 remaining works
to make Storm 2.0.0 reach in finish line. Also thanks Taylor to drive
release process.

Since it is a major release, I would like to see most of features work so
that we don't have regression. Unfortunately some of basic features don't
look like working. Not a critical one but it would be better to address it.

Here's the details. I'm doing the test as same as other minor releases. I
don't have any integration test, and Bobby stated he saw passing
integration test (I guess they're internal tests in Oath) so I'd like to
rely on this and skip going through testing complete feature set.

> source

- verify file (signature, MD5, SHA)
-- source, tar.gz : OK
-- source, zip : OK

- extract file
-- source, tar.gz : OK
-- source, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- build source with JDK 8
-- source, tar.gz : OK

- build source dist
-- source, tar.gz : OK

- build binary dist
-- source, tar.gz : OK

> binary

- verify file (signature, MD5, SHA)
-- binary, tar.gz : OK
-- binary, zip : OK

- extract file
-- binary, tar.gz : OK
-- binary, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- launch daemons : OK

- run RollingTopWords (local) : OK

- run RollingTopWords (remote) : FAIL
  - activate / deactivate / rebalance / kill : OK
  - logviewer (worker dir, daemon dir) :OK
  - change log level : OK
  - thread dump, heap dump, restart worker : FAIL (Don't work)
  - log search : topology search doesn't work, deep search work

No +1 here but once we fix the worker profile as well as topology log
search I would be +1 for next RC.

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 9월 22일 (토) 오전 5:34, Bobby Evans 님이 작성:

> +1 I rebuilt the code from the git tag and all of the unit/integration
> tests passed.  I also ran a single node cluster with both some simple
> functionality and performance tests and everything looked good.
>
> On a side note we just deployed a version based off of the same code as
> 2.0.0 to staging (think just some packaging differences and a few plugins),
> but with the compatibility layer to run older topologies and everything is
> looking good so far.
>
> Thanks,
>
> Bobby
>
> On Fri, Sep 21, 2018 at 1:25 PM P. Taylor Goetz 
> wrote:
>
> > This is a call to vote on releasing Apache Storm 2.0.0 (rc1)
> >
> > Full list of changes in this release:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc1/RELEASE_NOTES.html
> >
> > The tag/commit to be voted upon is v2.0.0:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=062bb2f158a66b802a3ad7bdda14fb8f70cf9b88;hb=062bb2f158a66b802a3ad7bdda14fb8f70cf9b88
> >
> > The source archive being voted upon can be found here:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc1/apache-storm-2.0.0-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc1/
> >
> > The release artifacts are signed with the following key:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1070
> >
> > Please vote on releasing this package as Apache Storm 2.0.0.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 2.0.0
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
> >
>

Re: Regarding releasing Apache Storm 2.0.0

[Jungtaek Lim]

 > > > > > p.s. I don't want to create branch-2.x or branch-2.0.x until
> >> > > absolutely
> >> > > > > > necessary, I don't see any major features with pull requests
> up
> >> but
> >> > > if
> >> > > > > you
> >> > > > > > do run across one please send something out before merging it
> >> in,
> >> > so
> >> > > we
> >> > > > > can
> >> > > > > > set up the branches properly at that time.
> >> > > > >
> >> > > > >
> >> > > > > Agree. We can always branch off the release tag/commit.
> >> > > > >
> >> > > > > -Taylor
> >> > > > >
> >> > > > >
> >> > > > > > On Sep 10, 2018, at 12:25 PM, Bobby Evans 
> >> > wrote:
> >> > > > > >
> >> > > > > > It has been nearly a month since this was originally sent out,
> >> and
> >> > > this
> >> > > > > is
> >> > > > > > not the first of these kinds of emails to go out about a 2.0.0
> >> > > release.
> >> > > > > I
> >> > > > > > think we have made a lot of really good progress on getting
> >> ready
> >> > > for a
> >> > > > > 2.0
> >> > > > > > release, and I really would like to see it happen before
> another
> >> > > month
> >> > > > > > passes.
> >> > > > > >
> >> > > > > > We have a 2.0 based deploy in some of our staging clusters,
> >> > currently
> >> > > > > > following the master branch with a little that is Yahoo
> >> specific on
> >> > > top.
> >> > > > > We
> >> > > > > > would like to start pushing towards production with it soon.
> >> > > > > >
> >> > > > > > There are a few issues that we are aware of.
> >> > > > > >
> >> > > > > >
> >> > >
> >> https://issues.apache.org/jira/issues/?jql=project%20%3D%20STORM%20AND%
> >> > > > > 20affectedVersion%20in%20(2.0.0)%20AND%20resolution%20%3D%
> >> > > > > 20Unresolved%20ORDER%20BY%20priority%20DESC
> >> > > > > >
> >> > > > > > There are no blockers still open, and only 4 issues listed as
> >> > > critical.
> >> > > > > >
> >> > > > > > If others have any open issues that feel need to be addressed
> >> prior
> >> > > to a
> >> > > > > > 2.0.0 release please respond to this with the JIRA number.  I
> >> would
> >> > > like
> >> > > > > to
> >> > > > > > set a goal/tentative date of Sep 17th (one week from today) to
> >> put
> >> > > > > together
> >> > > > > > a release candidate for a 2.0.0 release, and unless there are
> >> major
> >> > > > > > blockers that show up I think we can do it.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > >
> >> > > > > > Bobby Evans
> >> > > > > >
> >> > > > > > p.s. I don't want to create branch-2.x or branch-2.0.x until
> >> > > absolutely
> >> > > > > > necessary, I don't see any major features with pull requests
> up
> >> but
> >> > > if
> >> > > > > you
> >> > > > > > do run across one please send something out before merging it
> >> in,
> >> > so
> >> > > we
> >> > > > > can
> >> > > > > > set up the branches properly at that time.
> >> > > > > >
> >> > > > > >
> >> > > > > > On Wed, Jul 18, 2018 at 10:47 PM Jungtaek Lim <
> >> kabh...@gmail.com>
> >> > > wrote:
> >> > > > > >
> >> > > > > >> I'd like to say first, thanks Stig to take up remaining
> issues.
> >> > > Thanks
> >> > > > > to
> >> > > > > >> his efforts, according to the epic, we have onl

Re: [DISCUSS] Remove Ganglia metrics reporter

[Jungtaek Lim]

For Storm 2.0 I'm OK to drop supporting Ganglia reporter as same as why we
removed storm-druid in Storm 2.0.

When we drop supporting Ganglia reporter in 1.x it would be backward
incompatible. We just introduced Metrics V2 in Storm 1.2 and I guess few
people use Ganglia reporter, so I feel even OK to drop supporting for 1.x,
but just want to be sure it has no outstanding impact.

2018년 8월 18일 (토) 오후 9:37, Stig Rohde Døssing 님이 작성:

> Hi,
>
> The metrics v2 system contains a Ganglia reporter based on Dropwizard
> Metrics' metrics-ganglia module. One of the dependencies of this module is
> licensed under LGPL. In order to avoid including LGPL software in our
> releases, we would have to exclude this module and ask users to manually
> install the relevant jars.
>
> Dropwizard Metrics has dropped support for metrics-ganglia due to the
> license on the LGPL dependency
> https://github.com/dropwizard/metrics/issues/1319. We won't be able to
> upgrade to Metrics version 4+ unless we remove the metrics-ganglia
> dependency.
>
> I'd like to know whether anyone is not okay with removing the Ganglia
> reporter from master and deprecating it in 1.x? If we want to keep Ganglia
> reporting, we should try to find an alternative to metrics-ganglia so we
> don't get stuck on a specific Metrics version.
>

Re: Regarding releasing Apache Storm 2.0.0

[Jungtaek Lim]

I'd like to say first, thanks Stig to take up remaining issues. Thanks to
his efforts, according to the epic, we have only one major issue left:
porting UI to Java [1], and pull request [2] is available for that.
There're another issues [3] [4] targeting 2.0.0 (since it is backward
incompatible) but they are all about removing deprecated things, so easier
to be reviewed and make decisions.

Once we have a patch for that now, IMHO it would be good to review and ship
in 2.0.0 if it wouldn't take a month or so. We could do some sanity tests
in parallel, so waiting for UI port would not block much time on releasing
Storm 2.0.0.

- Jungtaek Lim (HeartSaVioR)

1. https://issues.apache.org/jira/browse/STORM-1311
2. https://github.com/apache/storm/pull/2752
3. https://issues.apache.org/jira/browse/STORM-2947
4. https://issues.apache.org/jira/browse/STORM-3156


2018년 7월 11일 (수) 오전 5:12, Alexandre Vermeerbergen 님이
작성:

> +1 would love to try it when an RC is avail!
>
> Alexandre Vermeerbergen
>
> 2018-07-10 21:15 GMT+02:00 Arun Mahadevan :
> > +1 to get it out soon.
> >
> >
> >
> >
> > On 7/10/18, 11:52 AM, "P. Taylor Goetz"  wrote:
> >
> >>+1 Sounds good to me.
> >>
> >>-Taylor
> >>
> >>> On Jul 10, 2018, at 2:18 AM, Jungtaek Lim  wrote:
> >>>
> >>> Hi devs,
> >>>
> >>> I hopefully have a time to sort out issues regarding Storm 2.0.0 and
> link
> >>> to epic issue.
> >>>
> >>> https://issues.apache.org/jira/browse/STORM-2714
> >>> (require login to Apache JIRA to see issues in epic)
> >>>
> >>> I guess we are close to the release, mostly left reviewing some pending
> >>> pull requests, and some manual sanity tests.
> >>>
> >>> Given that master branch is relatively stabilized for Travis CI build,
> as
> >>> well as style check and Java port make codebase better (at least for
> me), I
> >>> would really want to make Storm 2.0.0 released sooner than later, and
> rely
> >>> majorly on 2.x version line.
> >>>
> >>> So I would propose dev folks to concentrate on remaining tasks for
> Storm
> >>> 2.0.0 till we announce release. WDYT?
> >>>
> >>> Thanks,
> >>> Jungtaek Lim (HeartSaVioR)
> >>
> >
>

Regarding releasing Apache Storm 2.0.0

[Jungtaek Lim]

Hi devs,

I hopefully have a time to sort out issues regarding Storm 2.0.0 and link
to epic issue.

https://issues.apache.org/jira/browse/STORM-2714
(require login to Apache JIRA to see issues in epic)

I guess we are close to the release, mostly left reviewing some pending
pull requests, and some manual sanity tests.

Given that master branch is relatively stabilized for Travis CI build, as
well as style check and Java port make codebase better (at least for me), I
would really want to make Storm 2.0.0 released sooner than later, and rely
majorly on 2.x version line.

So I would propose dev folks to concentrate on remaining tasks for Storm
2.0.0 till we announce release. WDYT?

Thanks,
Jungtaek Lim (HeartSaVioR)

Re: Requesting review for a couple of PRs

[Jungtaek Lim]

Stig,

I started reviewing your pull requests. There would be conflicts between
1.x-branch as well as conflicts between pull requests, so please follow up
some following-up requests if any.

Btw I'll rebase STORM-2406 to make it getting review again.

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 7월 8일 (일) 오전 1:54, Stig Rohde Døssing 님이 작성:

> Hugo,
>
> Sounds good, thanks.
>
> Jungtaek,
>
> I don't think there are any missing tasks in STORM-2953. Once the listed
> tasks are done, we should be good to delete storm-kafka. I've been holding
> off on looking at https://issues.apache.org/jira/browse/STORM-2972 because
> I was hoping that https://github.com/apache/storm/pull/2443 could go in
> first.
>
> 2018-07-07 0:21 GMT+02:00 Hugo Louro :
>
> > I’ll take a pass at reviewing them as well.
> > Thanks,
> > Hugo
> >
> > > On Jul 6, 2018, at 1:41 PM, Jungtaek Lim  wrote:
> > >
> > > Yeah I have been hoping that some other folks who are familiar with
> > > storm-kafka-client jump in and review in time, but unfortunately it
> > didn't
> > > happen. Will try to review those PRs, hopefully within couple of days.
> > >
> > > Btw, if there're missing PRs to resolve STORM-2953
> > > <https://issues.apache.org/jira/browse/STORM-2953> I'd like to ask you
> > to
> > > also add them here, so we can resolve the one which is effectively
> > blocker
> > > for Storm 2.0.
> > >
> > > Thanks,
> > > Jungtaek Lim (HeartSaVioR)
> > >
> > > 2018년 7월 6일 (금) 오후 9:08, Stig Rohde Døssing 님이
> > 작성:
> > >
> > >> Hi devs,
> > >>
> > >> There are a couple of PRs open for changes in storm-kafka-client that
> > would
> > >> be nice to get reviewed. The PRs are
> > >> https://github.com/apache/storm/pull/2652
> > >> https://github.com/apache/storm/pull/2648 and
> > >> https://github.com/apache/storm/pull/2590. If someone could find a
> > couple
> > >> of minutes to give them a look, I'd appreciate it.
> > >>
> > >> Thanks.
> > >>
> >
>

Re: Requesting review for a couple of PRs

[Jungtaek Lim]

Yeah I have been hoping that some other folks who are familiar with
storm-kafka-client jump in and review in time, but unfortunately it didn't
happen. Will try to review those PRs, hopefully within couple of days.

Btw, if there're missing PRs to resolve STORM-2953
<https://issues.apache.org/jira/browse/STORM-2953> I'd like to ask you to
also add them here, so we can resolve the one which is effectively blocker
for Storm 2.0.

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 7월 6일 (금) 오후 9:08, Stig Rohde Døssing 님이 작성:

> Hi devs,
>
> There are a couple of PRs open for changes in storm-kafka-client that would
> be nice to get reviewed. The PRs are
> https://github.com/apache/storm/pull/2652
> https://github.com/apache/storm/pull/2648 and
> https://github.com/apache/storm/pull/2590. If someone could find a couple
> of minutes to give them a look, I'd appreciate it.
>
> Thanks.
>

Re: Powered-By Storm: Southcom - Mezzo fluid mediation

[Jungtaek Lim]

Hi Paolo,

sorry for the late.

Stig helps to add your company to the powered-by page via
https://github.com/apache/storm-site/pull/6

The page is now updated: http://storm.apache.org/Powered-By.html

Thanks,
Jungtaek Lim (HeartSaVioR)


2018년 6월 16일 (토) 오전 1:40, Paolo Galeotti 님이 작성:

> Nathan,
>
> Thanks to Storm we have redefined a telecom critical process, mediation.
> Classical mediation was file based, but we envision file as a stream of
> records that we can redirect into Mezzo.
>
> "Mezzo is a distributed real time mediation platform, build upon Storm. The
> mediation process is described in an acyclic graph (Storm topology) of
> nodes that we called a flow. Each node (Storm bolts) of the flow has one or
> more mediation rules and receives one record from a previous node and
> returns the mediated record to the next one. The rules are programmable in
> a high language and editable with the flow editor."
>
> we would like to be named on your page
> http://storm.apache.org/Powered-By.html
>
> thanks for your time
>
> Paolo Galeotti
>

Re: [VOTE] Release Apache Storm 1.2.2 (rc3)

[Jungtaek Lim]

REMINDER: The vote has been opened from 6 days ago, and still has only two
binding +1s. Please verify the release candidate and vote.

2018년 5월 21일 (월) 오후 11:16, Julien Nioche 님이
작성:

> +1 non binding
>
> compiled + ran StormCrawler topology without problems
>
> Thanks
>
> Julien
>
>
>
> On 17 May 2018 at 20:59, P. Taylor Goetz  wrote:
>
> > This is a call to vote on releasing Apache Storm 1.2.2 (rc3)
> >
> > Full list of changes in this release:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > 2.2-rc3/RELEASE_NOTES.html
> >
> > The tag/commit to be voted upon is v1.2.2:
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=
> >
> d29eaa70280dd0ac4b7d393387fab4a91eee9e9d;hb=d2d6f40344e6cc92ab07f3a462d577
> > ef6b61f8b1
> >
> > The source archive being voted upon can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > 2.2-rc3/apache-storm-1.2.2-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.2-rc3/
> >
> > The release artifacts are signed with the following key:
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> > plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1068
> >
> > Please vote on releasing this package as Apache Storm 1.2.2.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for 72 hours or until at least 3 PMC members vote
> > +1.
> >
> > [ ] +1 Release this package as Apache Storm 1.2.2
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
> >
>
>
>
> --
>
> *Open Source Solutions for Text Engineering*
>
> http://www.digitalpebble.com
> http://digitalpebble.blogspot.com/
> #digitalpebble 
>

Re: [VOTE] Release Apache Storm 1.1.3 (rc2)

[Jungtaek Lim]

REMINDER: The vote has been opened from 5 days ago, and still has only one
binding +1s. Please verify the release candidate and vote.

2018년 5월 19일 (토) 오전 10:49, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> This release candidate contains md5/sha for src archives, unlike 1.2.2
> RCs. May worth to check that.
>
> +1 (binding)
>
> On top to my verification on 1.2.3 rc1 (was +1),
>
> - verification on source/binary dist succeeded
> - build succeeded on source
> - spinning single cluster from binary dist succeeded
> - ran RollingTopWords
>
> The difference between rc1 and rc2 is regarding storm-kafka-client patch,
> so skipped checking storm-core features.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2018년 5월 19일 (토) 오전 4:39, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>
>> This is a call to vote on releasing Apache Storm 1.1.3 (rc2)
>>
>> Full list of changes in this release:
>>
>>
>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.3-rc2/RELEASE_NOTES.html
>>
>> The tag/commit to be voted upon is v1.1.3:
>>
>>
>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=e983fef3d4dcd35a821cecb462e198505458d1a4;hb=c51c8fc251856291597a5007a07e433f26951464
>>
>> The source archive being voted upon can be found here:
>>
>>
>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.3-rc2/apache-storm-1.1.3-src.tar.gz
>>
>> Other release files, signatures and digests can be found here:
>>
>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.3-rc2/
>>
>> The release artifacts are signed with the following key:
>>
>>
>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>>
>> The Nexus staging repository for this release is:
>>
>> https://repository.apache.org/content/repositories/orgapachestorm-1069
>>
>> Please vote on releasing this package as Apache Storm 1.1.3.
>>
>> When voting, please list the actions taken to verify the release.
>>
>> This vote will be open for 72 hours or until at least 3 PMC members vote
>> +1.
>>
>> [ ] +1 Release this package as Apache Storm 1.1.3
>> [ ]  0 No opinion
>> [ ] -1 Do not release this package because...
>>
>> Thanks to everyone who contributed to this release.
>>
>> -Taylor
>>
>

Re: [VOTE] Release Apache Storm 1.1.3 (rc2)

[Jungtaek Lim]

This release candidate contains md5/sha for src archives, unlike 1.2.2 RCs.
May worth to check that.

+1 (binding)

On top to my verification on 1.2.3 rc1 (was +1),

- verification on source/binary dist succeeded
- build succeeded on source
- spinning single cluster from binary dist succeeded
- ran RollingTopWords

The difference between rc1 and rc2 is regarding storm-kafka-client patch,
so skipped checking storm-core features.

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 5월 19일 (토) 오전 4:39, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:

> This is a call to vote on releasing Apache Storm 1.1.3 (rc2)
>
> Full list of changes in this release:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.3-rc2/RELEASE_NOTES.html
>
> The tag/commit to be voted upon is v1.1.3:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=e983fef3d4dcd35a821cecb462e198505458d1a4;hb=c51c8fc251856291597a5007a07e433f26951464
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.3-rc2/apache-storm-1.1.3-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.3-rc2/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1069
>
> Please vote on releasing this package as Apache Storm 1.1.3.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for 72 hours or until at least 3 PMC members vote
> +1.
>
> [ ] +1 Release this package as Apache Storm 1.1.3
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
>

Re: [VOTE] Release Apache Storm 1.2.2 (rc3)

[Jungtaek Lim]

There's no md5/sha file for src tar.gz and zip (again), and gpg
verification warns that Taylor's key is expired.

If above issues are not blockers on release, I'm +1 (binding).

On top to my verification on 1.2.2 rc2 (was +1),

- verification on binary dist succeeded
- build succeeded on source
- spinning single cluster from binary dist succeeded
- ran RollingTopWords and verified STORM-2988[1] and STORM-3069[2] runs
well which are added between rc2 and rc3

Thanks,
Jungtaek Lim (HeartSaVioR)

1. https://issues.apache.org/jira/browse/STORM-2988
2. https://issues.apache.org/jira/browse/STORM-3069

2018년 5월 19일 (토) 오전 7:20, Alexandre Vermeerbergen <avermeerber...@gmail.com>님이
작성:

> Hello,
>
> I won't have time to test this RC3 before mid next-week and yet with
> modest load.
>
> Anyway, I already heavily tested with high load (~15 topologies
> running on a 5 Supervisor VMs cluster with Kafka 0.10.2.0 libs against
> a Kafka 1.0.1 cluster) with  Storm 1.1.2 RC2 (previous Release
> candidate) with Stig's latest fixes on Storm Kafka Client.
> Results are very good in term of stability (better than with 1.2.0),
> and performance is same as with Storm 1.2.0 (our current production
> setup).
> We have issues with Kafka Spout (we need time to give detailed
> feedbacks) which are not regressions, and thus nothing blocker for
> 1.2.2 RC3.
>
> So, in case it would be too late for me to give a feedback on Storm
> 1.2.2 RC3, let me give an anticipated  [+1] (non binding)
>
> Best regards,
> Alexandre Vermeerbergen
>
> 2018-05-17 21:59 GMT+02:00 P. Taylor Goetz <ptgo...@gmail.com>:
> > This is a call to vote on releasing Apache Storm 1.2.2 (rc3)
> >
> > Full list of changes in this release:
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.2-rc3/RELEASE_NOTES.html
> >
> > The tag/commit to be voted upon is v1.2.2:
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=d29eaa70280dd0ac4b7d393387fab4a91eee9e9d;hb=d2d6f40344e6cc92ab07f3a462d577ef6b61f8b1
> >
> > The source archive being voted upon can be found here:
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.2-rc3/apache-storm-1.2.2-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.2-rc3/
> >
> > The release artifacts are signed with the following key:
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1068
> >
> > Please vote on releasing this package as Apache Storm 1.2.2.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for 72 hours or until at least 3 PMC members vote
> +1.
> >
> > [ ] +1 Release this package as Apache Storm 1.2.2
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
>

Re: [VOTE] Release Apache Storm 1.1.3 (rc1)

[Jungtaek Lim]

+1 (binding)

> source

- verify file (signature, MD5, SHA)
-- source, tar.gz : OK
-- source, zip : OK

- extract file
-- source, tar.gz : OK
-- source, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- build source with JDK 7
-- source, tar.gz : OK

- build source dist
-- source, tar.gz : OK

- build binary dist
-- source, tar.gz : OK

> binary

- verify file (signature, MD5, SHA)
-- binary, tar.gz : OK
-- binary, zip : OK

- extract file
-- binary, tar.gz : OK
-- binary, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- launch daemons : OK

- run RollingTopWords (local) : OK

- run RollingTopWords (remote) : OK
  - activate / deactivate / rebalance / kill : OK
  - logviewer (worker dir, daemon dir) : OK
  - change log level : OK
  - thread dump, heap dump, restart worker : OK
  - log search : OK

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 5월 5일 (토) 오전 2:30, Bobby Evans <ev...@oath.com.invalid>님이 작성:

> +1
>
> I built from the git tag, ran all of the unit tests launched a single node
> cluster and ran a few simple tests on it.
>
> On Thu, May 3, 2018 at 12:21 PM P. Taylor Goetz <ptgo...@gmail.com> wrote:
>
> > This is a call to vote on releasing Apache Storm 1.1.3 (rc1)
> >
> > Full list of changes in this release:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.3-rc1/RELEASE_NOTES.html
> >
> > The tag/commit to be voted upon is v1.1.3:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=caffcf810cf28b8b8c7ea6e15a555a7160de1384;hb=e590a3289a9f7b9806a4c5cb78a2c06b4fbf019f
> >
> > The source archive being voted upon can be found here:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.3-rc1/apache-storm-1.1.3-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.3-rc1/
> >
> > The release artifacts are signed with the following key:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1065
> >
> > Please vote on releasing this package as Apache Storm 1.1.3.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for 72 hours or until at least 3 PMC members vote
> > +1.
> >
> > [ ] +1 Release this package as Apache Storm 1.1.3
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
> >
>

Re: [VOTE] Release Apache Storm 1.2.2 (rc2)

[Jungtaek Lim]

I just realized that there's no md5/sha file for src tar.gz and zip, so
please disregard the verification on source  tar.gz and source zip. Please
note that signature check still succeeds. It might be better if we can get
missed md5/sha files and verify files exhaustively.

2018년 5월 6일 (일) 오후 9:31, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> +1 (binding)
>
> > source
>
> - verify file (signature, MD5, SHA)
> -- source, tar.gz : OK
> -- source, zip : OK
>
> - extract file
> -- source, tar.gz : OK
> -- source, zip : OK
>
> - diff-ing extracted files between tar.gz and zip : OK
>
> - build source with JDK 7
> -- source, tar.gz : OK
>
> - build source dist
> -- source, tar.gz : OK
>
> - build binary dist
> -- source, tar.gz : OK
>
> > binary
>
> - verify file (signature, MD5, SHA)
> -- binary, tar.gz : OK
> -- binary, zip : OK
>
> - extract file
> -- binary, tar.gz : OK
> -- binary, zip : OK
>
> - diff-ing extracted files between tar.gz and zip : OK
>
> - launch daemons : OK
>
> - run RollingTopWords (local) : OK
>
> - run RollingTopWords (remote) : OK
>   - activate / deactivate / rebalance / kill : OK
>   - logviewer (worker dir, daemon dir) : OK
>   - change log level : OK
>   - thread dump, heap dump, restart worker : OK
>   - log search : OK
>
> I don't feel STORM-3059 is a blocker so OK to go ahead on release, but I'm
> also OK to resolve STORM-3059 quickly and roll out RC3.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2018년 5월 6일 (일) 오후 8:35, Stig Rohde Døssing <stigdoess...@gmail.com>님이 작성:
>
>> Alexandre,
>>
>> Could you try running just
>>
>> mvn clean install -DskipTests
>>
>> I think for some reason one of the dependencies isn't being built as it
>> should by Maven.
>>
>> 2018-05-06 12:59 GMT+02:00 Alexandre Vermeerbergen <
>> avermeerber...@gmail.com
>> >:
>>
>> > Hello Stig,
>> >
>> > Thanks, I followed your instructions, but got a build failure:
>> >
>> > [INFO] --- maven-dependency-plugin:2.8:unpack (unpack) @ storm-core ---
>> > [INFO] Configured Artifact: org.apache.storm:multilang-
>> > ruby:1.2.3-SNAPSHOT:jar
>> > Downloading: http://repository.apache.org/snapshots/org/apache/storm/
>> > multilang-ruby/1.2.3-SNAPSHOT/maven-metadata.xml
>> > Downloading: https://clojars.org/repo/org/apache/storm/multilang-ruby/1
>> .
>> > 2.3-SNAPSHOT/maven-metadata.xml
>> > Downloading: https://clojars.org/repo/org/apache/storm/multilang-ruby/1
>> .
>> > 2.3-SNAPSHOT/multilang-ruby-1.2.3-SNAPSHOT.jar
>> > Downloading: http://repository.apache.org/snapshots/org/apache/storm/
>> > multilang-ruby/1.2.3-SNAPSHOT/multilang-ruby-1.2.3-SNAPSHOT.jar
>> > [INFO] 
>> > 
>> > [INFO] Reactor Summary:
>> > [INFO]
>> > [INFO] Storm .. SUCCESS [
>> > 4.279 s]
>> > [INFO] maven-shade-clojure-transformer  SUCCESS [
>> > 4.568 s]
>> > [INFO] storm-maven-plugins  SUCCESS [
>> > 3.992 s]
>> > [INFO] Storm Core . FAILURE
>> > [03:12 min]
>> > [INFO] storm-kafka-client . SKIPPED
>> > [INFO] 
>> > 
>> > [INFO] BUILD FAILURE
>> > [INFO] 
>> > 
>> > [INFO] Total time: 03:28 min
>> > [INFO] Finished at: 2018-05-06T12:27:58+02:00
>> > [INFO] Final Memory: 86M/1460M
>> > [INFO] 
>> > 
>> > [ERROR] Failed to execute goal
>> > org.apache.maven.plugins:maven-dependency-plugin:2.8:unpack (unpack)
>> > on project storm-core: Unable to find artifact. Could not find
>> > artifact org.apache.storm:multilang-ruby:jar:1.2.3-SNAPSHOT in clojars
>> > (https://clojars.org/repo/)
>> > [ERROR]
>> > [ERROR] Try downloading the file manually from the project website.
>> > [ERROR]
>> > [ERROR] Then, install it using the command:
>> > [ERROR] mvn install:install-file -DgroupId=org.apache.storm
>> > -DartifactId=multilang-ruby -Dversion=1.2.3-SNAPSHOT -Dpackaging=jar
>> > -Dfile=/path/to/file
>> > [ERROR]
>> > [ERROR] Alternatively, if you host your own repository you can de

Re: [VOTE] Release Apache Storm 1.2.2 (rc2)

[Jungtaek Lim]

+1 (binding)

> source

- verify file (signature, MD5, SHA)
-- source, tar.gz : OK
-- source, zip : OK

- extract file
-- source, tar.gz : OK
-- source, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- build source with JDK 7
-- source, tar.gz : OK

- build source dist
-- source, tar.gz : OK

- build binary dist
-- source, tar.gz : OK

> binary

- verify file (signature, MD5, SHA)
-- binary, tar.gz : OK
-- binary, zip : OK

- extract file
-- binary, tar.gz : OK
-- binary, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- launch daemons : OK

- run RollingTopWords (local) : OK

- run RollingTopWords (remote) : OK
  - activate / deactivate / rebalance / kill : OK
  - logviewer (worker dir, daemon dir) : OK
  - change log level : OK
  - thread dump, heap dump, restart worker : OK
  - log search : OK

I don't feel STORM-3059 is a blocker so OK to go ahead on release, but I'm
also OK to resolve STORM-3059 quickly and roll out RC3.

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 5월 6일 (일) 오후 8:35, Stig Rohde Døssing <stigdoess...@gmail.com>님이 작성:

> Alexandre,
>
> Could you try running just
>
> mvn clean install -DskipTests
>
> I think for some reason one of the dependencies isn't being built as it
> should by Maven.
>
> 2018-05-06 12:59 GMT+02:00 Alexandre Vermeerbergen <
> avermeerber...@gmail.com
> >:
>
> > Hello Stig,
> >
> > Thanks, I followed your instructions, but got a build failure:
> >
> > [INFO] --- maven-dependency-plugin:2.8:unpack (unpack) @ storm-core ---
> > [INFO] Configured Artifact: org.apache.storm:multilang-
> > ruby:1.2.3-SNAPSHOT:jar
> > Downloading: http://repository.apache.org/snapshots/org/apache/storm/
> > multilang-ruby/1.2.3-SNAPSHOT/maven-metadata.xml
> > Downloading: https://clojars.org/repo/org/apache/storm/multilang-ruby/1.
> > 2.3-SNAPSHOT/maven-metadata.xml
> > Downloading: https://clojars.org/repo/org/apache/storm/multilang-ruby/1.
> > 2.3-SNAPSHOT/multilang-ruby-1.2.3-SNAPSHOT.jar
> > Downloading: http://repository.apache.org/snapshots/org/apache/storm/
> > multilang-ruby/1.2.3-SNAPSHOT/multilang-ruby-1.2.3-SNAPSHOT.jar
> > [INFO] 
> > 
> > [INFO] Reactor Summary:
> > [INFO]
> > [INFO] Storm .. SUCCESS [
> > 4.279 s]
> > [INFO] maven-shade-clojure-transformer  SUCCESS [
> > 4.568 s]
> > [INFO] storm-maven-plugins  SUCCESS [
> > 3.992 s]
> > [INFO] Storm Core . FAILURE
> > [03:12 min]
> > [INFO] storm-kafka-client . SKIPPED
> > [INFO] 
> > 
> > [INFO] BUILD FAILURE
> > [INFO] 
> > 
> > [INFO] Total time: 03:28 min
> > [INFO] Finished at: 2018-05-06T12:27:58+02:00
> > [INFO] Final Memory: 86M/1460M
> > [INFO] 
> > 
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-dependency-plugin:2.8:unpack (unpack)
> > on project storm-core: Unable to find artifact. Could not find
> > artifact org.apache.storm:multilang-ruby:jar:1.2.3-SNAPSHOT in clojars
> > (https://clojars.org/repo/)
> > [ERROR]
> > [ERROR] Try downloading the file manually from the project website.
> > [ERROR]
> > [ERROR] Then, install it using the command:
> > [ERROR] mvn install:install-file -DgroupId=org.apache.storm
> > -DartifactId=multilang-ruby -Dversion=1.2.3-SNAPSHOT -Dpackaging=jar
> > -Dfile=/path/to/file
> > [ERROR]
> > [ERROR] Alternatively, if you host your own repository you can deploy
> > the file there:
> > [ERROR] mvn deploy:deploy-file -DgroupId=org.apache.storm
> > -DartifactId=multilang-ruby -Dversion=1.2.3-SNAPSHOT -Dpackaging=jar
> > -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
> > [ERROR]
> > [ERROR]
> > [ERROR] org.apache.storm:multilang-ruby:jar:1.2.3-SNAPSHOT
> > [ERROR]
> > [ERROR] from the specified remote repositories:
> > [ERROR] central (http://repo1.maven.org/maven2/, releases=true,
> > snapshots=false),
> > [ERROR] clojars (https://clojars.org/repo/, releases=true,
> > snapshots=true),
> > [ERROR] apache.snapshots (http://repository.apache.org/snapshots,
> > releases=false, snapshots=true)
> > [ERROR] -> [Help 1]
> >
> > I guess something's missing in the instruction or in the dependency
> > declarat

Re: New Committer/PMC Member: Ethan Li

[Jungtaek Lim]

Congratulations! Welcome Ethan!
On Tue, 17 Apr 2018 at 7:30 AM Stig Rohde Døssing 
wrote:

> Congratulations
>
> 2018-04-16 21:16 GMT+02:00 Erik Weathers :
>
> > congrats Ethan!
> >
> > On Mon, Apr 16, 2018 at 10:51 AM, P. Taylor Goetz 
> > wrote:
> >
> > > Congratulations! Welcome Ethan!
> > >
> > > -Taylor
> > >
> > > > On Apr 16, 2018, at 12:23 PM, Bobby Evans  wrote:
> > > >
> > > > Please Join with me in welcoming Ethan Li as the newest Apache Storm
> > > > committer and PMC member.
> > > >
> > > > Great work!
> > >
> > >
> >
>

Re: New Committer/PMC Member: Roshan Naik

[Jungtaek Lim]

Congrats Roshan!

2018년 4월 6일 (금) 오전 11:39, P. Taylor Goetz 님이 작성:

> Please join me in congratulating and welcoming Roshan Naik as the latest
> Apache Storm committer and PMC member.
>
> Welcome Roshan!
>
> -Taylor
>
>

Re: New Metrics Reporting API - for 2.0.0

[Jungtaek Lim]

Aaron,

please feel free to take up STORM-2908 and STORM-2909 as well. I'd be much
appreciated. I asked Taylor to take them up but didn't work.
They're coupled with Storm 2.0.0 release so sooner is better, but no hard
due date for now.

-Jungtaek Lim (HeartSaVioR)

2018년 3월 14일 (수) 오전 12:35, Aaron Gresch <agre...@gmail.com>님이 작성:

> I was wondering what the plan/timeline is for STORM-2909 for the Storm 2
> release.
>
> If I can help, please let me know.
>
> Thanks,
> Aaron
>

Re: Regarding "Powered By"

[Jungtaek Lim]

Hi Harsh,

Great to hear and thanks for using Apache Storm for your critical
production environment!
All kinds of contributions welcome: documentation, sharing use cases,
sharing ideas to improve, code contributions, being early adopter to try
out master branch or verify release candidate, and so.

If you would want to add your company to the list of "Powered By", you
could edit the page in below repository and submit a pull request.

https://github.com/apache/storm-site

Thanks again!
Jungtaek Lim (HeartSaVioR)


2018년 3월 1일 (목) 오전 12:06, Harsh Choudhary <ha...@crispanalytics.com>님이 작성:

> Hi
>
> We, at Crisp Analytics, have been using Apache Storm in production for
> around 4 years now. We have built a real-time system which is fed by
> industrial sensor data from around the globe.
>
> Our use case and usage of Apache Storm both evolved a lot in this period
> from a system of ingestion to real-time predictions and alarm system.
>
> We read a continuous stream of industrial meter readings and ingest it into
> our system and also send it for further real-time transformation. Storm
> helps us in transforming the streaming data, calculate predictions on the
> fly, store the various stages of transformed data and feed the data to
> further systems.
>
> It has been a very great journey and we are continuously innovating to use
> Storm to its full potential. We thank the whole community behind it and
> honored to be the part of it.
>
> *Thanks!*
>
> *Harsh Choudhary*
>
> *Big Data Architect *| *Crisp Analytics*
>
> Mobile: +91 89 01 333 191 | Email: ha...@crispanalytics.com
>
> Crisp Analytics Blog - http://crispanalytics.com/blog
>

Re: Building history of storm worker assignments

[Jungtaek Lim]

Hi Nikhil,

While Nimbus would be a best place to hook the information of assignment
and record, but Nimbus doesn't provide the hook (if I'm not missing here).
As you already stated, you could leverage IWorkerHook to record the start
of worker which looks enough on your case. It only records when worker is
successfully launched. If there's some cases which worker continuously
fails to launch, the assignment will not be recorded. Please note that
shutdown is not guaranteed to be called.

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 2월 28일 (수) 오전 3:37, Nikhil Bafna <nikhil.bafn...@gmail.com>님이 작성:

> Would implementiong IWorkerHook be more appropriate?
>
> On Tue, Feb 27, 2018 at 11:19 PM, Nikhil Bafna <nikhil.bafn...@gmail.com>
> wrote:
>
> >
> > Hello,
> >
> > I want to build a centralised history that looks like
> >
> > timestamp : topology_name : component_name : topology_id : component_id
> : VM hostname : VM IP : Worker port
> >
> > What would be the best to go about it in Storm? I can think of
> >
> >1. Reporting this from prepare() method of a spout/bolt
> >2. Write a custom scheduler that reports the assignments
> >
> > I had explore event hook, but that doesn't have an event for
> > worker/executor task assignment. The UI only reports the current
> > assignments. I can devise a poll based mechanism from the REST API, but
> > it's not guaranteed to not miss events.
> >
> > --
> > Nikhil Bafna
> >
>
>
>
> --
> Nikhil Bafna
>

Re: New Committer/PMC Member: Erik Weathers

[Jungtaek Lim]

Welcome Erik! Congrats!

-Jungtaek Lim (HeartSaVioR)

2018년 2월 23일 (금) 오전 7:05, Stig Rohde Døssing <stigdoess...@gmail.com>님이 작성:

> Congratulations Erik. Happy to see you join.
>
> 2018-02-22 20:53 GMT+01:00 Alexandre Vermeerbergen <
> avermeerber...@gmail.com
> >:
>
> > Hi,
> >
> > Welcome to Erik...
> > ... a Stormy Weather(s) sounds like a fantastic match indeed!
> >
> > Alexandre Vermeerbergen (Storm addict)
> >
> > 2018-02-22 20:49 GMT+01:00 P. Taylor Goetz <ptgo...@apache.org>:
> > > The Apache Storm PMC has voted to add Erik Weathers as a Committer and
> > PMC Member.
> > >
> > > Please join me in congratulating Erik on his new role!
> > >
> > > -Taylor
> >
>

[DISCUSS] Handling dependencies for Storm 2.0.0 (STORM-2882)

[Jungtaek Lim]

Hi devs,

I'm initiating this one for part of Storm 2.0.0. Actually we had some
discussion from earlier thread [1] but it was originated for IDE
complaining and we completely remove shading/relocating in Storm 2.0.0 at
that time, ends up with exposing all the transitive dependencies in
'storm-client' to user topology code. We've reduced lots of dependencies
while breaking down storm-core, but troublesome dependencies (for example,
Guava, Jackson, Netty, and so on) are still left.

There're two questions we should address:

1. Do we want to (and need to) shade/relocate the dependencies?
(We may also want to see it as regression issue, since end users should
tackle with dependencies if we don't shade/relocate.)

2. If we want to relocate the dependencies, how?

If we would want to relocate them, there's a good reference on Flink: Flink
creates separate repository [2] which contains pom for relocation for each
target artifact. Links for related discussion [3] and issue [4] are
available, so if you are interested on Flink's approach, please refer the
links. I guess it may require considerable efforts, so if someone finds
simpler and easier way that should be great.

Please share your voices regarding two questions, and idea/knowledge of
how. Thanks in advance.

Jungtaek Lim (HeartSaVioR)

1.
https://mail-archives.apache.org/mod_mbox/incubator-storm-dev/201703.mbox/%3ccaf5108gjjqzsyywcp99bgpgec7tufe-tbt9fi0m78ah8rkm...@mail.gmail.com%3E
2. https://github.com/apache/flink-shaded
3.
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Changing-Flink-s-shading-model-td17419.html
4. https://issues.apache.org/jira/browse/FLINK-6529

Re: [DISCUSS] Plan on Storm 2.0.0

[Jungtaek Lim]

Anyway, I'll initiate some discussion threads for current issues targeted
to Storm 2.0.0. Sorry in prior to bulk of discussions.

2018년 2월 20일 (화) 오전 11:25, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> I think that's good idea. "blockers" should have priority as blocker (so
> they're self-classifiable), but may be better to classify "very desirable"
> and "good to have".
> Would we want to break down epic issue, or determine with other field
> (priority, or label)?
>
> 2018년 2월 20일 (화) 오전 9:49, Roshan Naik <ros...@hortonworks.com>님이 작성:
>
>> Would be good to callout which of issues listed in Storm 2.0 epic are
>> considered “blockers” vs “very desirable” vs “good to have”
>>
>> -roshan
>>
>>
>> On 2/19/18, 3:29 PM, "Jungtaek Lim" <kabh...@gmail.com> wrote:
>>
>> Hi devs,
>>
>> We've just released Storm 1.2.1 (not officially announced but vote
>> passed
>> anyway) and we're in agreement that no further minor version on 1.x
>> version
>> line, so it's time to focus on Storm 2.0.0 so that we can bring far
>> long
>> undetermined milestone to reality.
>>
>> Here is the epic issue for Storm 2.0.0:
>> https://issues.apache.org/jira/browse/STORM-2714
>>
>> We have no restriction to file issues and submit patches so there're
>> more
>> changes available outside of epic issue, but once we reach consensus
>> to
>> bring Storm 2.0.0 fairly sooner, I propose we (team, or individual)
>> add all
>> the issues to epic which we target to include to Storm 2.0.0, so that
>> we
>> can track all the remaining items from the epic issue, and decide to
>> postpone some non-blocker dragging items to out of Storm 2.0.0. Makes
>> sense?
>>
>> I have some backlog issues for myself which would need somewhat huge
>> efforts so not sure they can be included to Storm 2.0.0. I'll add
>> some to
>> epic if I think I can find time to do. If you have items which are
>> planned
>> to be included to Storm 2.0.0, please add them to epic issue.
>>
>> I hope that we can release Storm 2.0.0 in 1Q (1 month and several days
>> left): even if it doesn't happen, we could release beta version of
>> Storm
>> 2.0.0 in 1Q and we say "feature freeze" and concentrate on
>> stabilizing the
>> release and making Storm 2.0.0 happen sooner than later. If you have
>> items
>> for Storm 2.0.0 which requires more than 1 month to be finished, I
>> think it
>> would be worth to share the items and discuss.
>>
>> Please add your voice on anything about Storm 2.0.0 plan. It would be
>> much
>> appreciated if someone puts efforts on testing and stabilizing the
>> current
>> master branch (it would take much time and efforts and more hands are
>> definitely better).
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>>
>>

Re: Verification about my observation for current State implementation

[Jungtaek Lim]

> You can only change the parallelism (number of executors). AFAIK there is
no option to change the number of tasks and maybe I am missing if it
changed recently.

Yeah looks like I misunderstand the feature. It addresses number of
executors.

I'm wondering the benefit of having N tasks : 1 executor. It even makes me
confusing continuously, and I don't find beneficial use cases for that.
Most of things would be IO bound processing, and should be covered with
asynchronous processing. I hope we just simplify this one.

Another perspective of rebalance: suppose end users provide parallelism
bigger than current task count. If task count is not changed, rebalance is
not working as expected from users. If task count is changed, we are
breaking precondition of above statement.

We should take into consideration that end users modify topology code and
resubmit. In this case task count can be (and should be) changed, and we
should support state resharding in this case. Without resharding each task
fails to load all the KVs for keys routing to the task. It doesn't seem to
the issue about whether supporting dynamic task change or not.

- Jungtaek Lim (HeartSaVioR)

ps. btw, internal structure of Storm topology is not capable of scaling
task counts without restructuring since task id is increased sequentially
across components. I suspect it would block elasticity so that should be
also the thing to address.

2018년 2월 21일 (수) 오전 8:33, Arun Mahadevan <ar...@apache.org>님이 작성:

> >>
> >>1. Right now users need to vary the database/table name in the state
> >provider config per topology.
> >
> >Redis Cluster doesn't support multi database and it is uncommon usage for
> >Redis to use even tens of databases, so end users can't do the needed
> thing
> >with Redis state backend. For HBase state backend they can do, but
> creating
> >table requires administrator permission of HBase, which someone may not
> >want to open to multiple users. So I feel including topology name in
> >namespace looks like "mandatory" rather than "good to have".
>
>
> I think this should be a fairly straightforward change and we should do it.
>
> >>
> >>2. As of now storm only supports rebalancing the tasks (task count
> >remains the same).
> >
> >That is not correct. We are still missed to provide the feature in UI (I'm
> >not familiar with front-end so I had in mind but I didn't work on that)
> but
> >you can rebalance with different parallelism of component via cli.
> >
> >Below is the rebalance command provided via cli. Quoting command line
> >client document[1]:
> >
> >rebalance
> >Syntax: storm rebalance topology-name [-w wait-time-secs] [-n
> >new-num-workers] [-e component=parallelism]*
>
>
> You can only change the parallelism (number of executors). AFAIK there is
> no option to change the number of tasks and maybe I am missing if it
> changed recently. As long as the number of tasks are fixed, we don’t have
> to worry about re-sharding the keys since the namespaces are at task level.
> Autoscaling may or may not change this design (we may only allow increasing
> or reducing the parallelism and not the tasks) and it requires more thought
> on how it would affect the user managed state at core spout/bolt level
> (outside of the higher level abstractions) and so IMO, we can tackle it
> later.
>
> Thanks,
> Arun
>
>
>
> On 2/20/18, 2:57 PM, "Jungtaek Lim" <kabh...@gmail.com> wrote:
>
> >> 1. Right now users need to vary the database/table name in the state
> >provider config per topology.
> >
> >Redis Cluster doesn't support multi database and it is uncommon usage for
> >Redis to use even tens of databases, so end users can't do the needed
> thing
> >with Redis state backend. For HBase state backend they can do, but
> creating
> >table requires administrator permission of HBase, which someone may not
> >want to open to multiple users. So I feel including topology name in
> >namespace looks like "mandatory" rather than "good to have".
> >
> >> 2. As of now storm only supports rebalancing the tasks (task count
> >remains the same).
> >
> >That is not correct. We are still missed to provide the feature in UI (I'm
> >not familiar with front-end so I had in mind but I didn't work on that)
> but
> >you can rebalance with different parallelism of component via cli.
> >
> >Below is the rebalance command provided via cli. Quoting command line
> >client document[1]:
> >
> >rebalance
> >Syntax: storm rebalance topology-name [-w wait-time-secs] [-n
> >new-num-workers] [-e component=parallelism]*
> >
> 

Re: Verification about my observation for current State implementation

[Jungtaek Lim]

> 1. Right now users need to vary the database/table name in the state
provider config per topology.

Redis Cluster doesn't support multi database and it is uncommon usage for
Redis to use even tens of databases, so end users can't do the needed thing
with Redis state backend. For HBase state backend they can do, but creating
table requires administrator permission of HBase, which someone may not
want to open to multiple users. So I feel including topology name in
namespace looks like "mandatory" rather than "good to have".

> 2. As of now storm only supports rebalancing the tasks (task count
remains the same).

That is not correct. We are still missed to provide the feature in UI (I'm
not familiar with front-end so I had in mind but I didn't work on that) but
you can rebalance with different parallelism of component via cli.

Below is the rebalance command provided via cli. Quoting command line
client document[1]:

rebalance
Syntax: storm rebalance topology-name [-w wait-time-secs] [-n
new-num-workers] [-e component=parallelism]*

And we're sure that elasticity is "the thing" to provide, and for
elasticity we may even want to adjust parallelism on the fly (ideally,
would not that easy).

> IMO, users should have the flexibility to store multiple key-values in
the state in the core API. ... I think this flexibility also helps us to
provide useful abstractions on top.

I agree most beneficial aspect from core API is flexibility and less
restriction. I'm just worrying about state flexibility which should be
handled with caution. For example, without proper restriction (especially
grouping) duplicated keys in different tasks can happen, and even some end
users are brave to create their own state reshard tool, they need to deal
with merging the values for same key. Most end users may not recognize the
situation and hard to deal with, which doesn't seem friendly.

I feel we need to at least document recommended usage (selecting key, using
field grouping with selected key, selecting same key as key of state KV)
with notice/caution effects for other usages, and also need to provide
state reshard tool for such usage. The tool can be used for states handled
with streams API.

> 3. Re-sharding would be based on the keys (re-hashing). The component id,
task-id would be used to map to the namespace. So I am not clear about the
concern you have raised.

Yes resharding would be based on the keys, which means keys should be moved
to different task id. So to move the key in state backend, we need to parse
and modify the namespace part, and we use delimiter which is allowed to
topology name as well as component name. (I don't think namespace is badly
composed. The problem is not restricting topology name and component name.
We are allowing too many flexibility and being stuck from that.)

Windowed state is another thing we should deal with. I still didn't have
time to deep dive with, but to support resharding, window (and effectively
any states) should be grouped by key so that it can be placed to right
(might be new) task after relaunching topology. I'm asking that
current window/partition/windowsystem
is capable of.

> IMO before we do this, we can introduce stateful exactly once processing
via the streams API as the first step.

While I definitely agree that introducing stateful exactly-once processing
is major feature and the thing we should support sooner than later, I'm not
100% sure about which thing to do first. We're supporting at-least-once
stateful processing in 1.x version line and haven't provided "state
reshard" feature for a long time which makes actual users directly suffer,
so that's another major feature to me.

- Jungtaek Lim (HeartSaVioR)

1. http://storm.apache.org/releases/1.2.1/Command-line-client.html


2018년 2월 21일 (수) 오전 4:22, Arun Iyer <ai...@hortonworks.com>님이 작성:

> correction: task count remains the same (executor count can vary).
>
>
>
>
> On 2/20/18, 11:20 AM, "Arun Iyer on behalf of Arun Mahadevan" <
> ai...@hortonworks.com on behalf of ar...@apache.org> wrote:
>
> >Hi Jungtaek,
> >
> >
> >1. Right now users need to vary the database/table name in the state
> provider config per topology. Agree, its better to include topology in the
> namespace.
> >
> >2. IMO, users should have the flexibility to store multiple key-values in
> the state in the core API. As of now storm only supports rebalancing the
> tasks (executor count remains the same). So users need not worry about
> re-sharding their state as long as they use the right grouping. I think
> this flexibility also helps us to provide useful abstractions on top.
> >
> >3. Re-sharding would be based on the keys (re-hashing). The component id,
> task-id would be used to map to the namespace. So I am not clear about the
> concern you have raised.
> >
> >We c

Re: [DISCUSS] Plan on Storm 2.0.0

[Jungtaek Lim]

I think that's good idea. "blockers" should have priority as blocker (so
they're self-classifiable), but may be better to classify "very desirable"
and "good to have".
Would we want to break down epic issue, or determine with other field
(priority, or label)?

2018년 2월 20일 (화) 오전 9:49, Roshan Naik <ros...@hortonworks.com>님이 작성:

> Would be good to callout which of issues listed in Storm 2.0 epic are
> considered “blockers” vs “very desirable” vs “good to have”
>
> -roshan
>
>
> On 2/19/18, 3:29 PM, "Jungtaek Lim" <kabh...@gmail.com> wrote:
>
> Hi devs,
>
> We've just released Storm 1.2.1 (not officially announced but vote
> passed
> anyway) and we're in agreement that no further minor version on 1.x
> version
> line, so it's time to focus on Storm 2.0.0 so that we can bring far
> long
> undetermined milestone to reality.
>
> Here is the epic issue for Storm 2.0.0:
> https://issues.apache.org/jira/browse/STORM-2714
>
> We have no restriction to file issues and submit patches so there're
> more
> changes available outside of epic issue, but once we reach consensus to
> bring Storm 2.0.0 fairly sooner, I propose we (team, or individual)
> add all
> the issues to epic which we target to include to Storm 2.0.0, so that
> we
> can track all the remaining items from the epic issue, and decide to
> postpone some non-blocker dragging items to out of Storm 2.0.0. Makes
> sense?
>
> I have some backlog issues for myself which would need somewhat huge
> efforts so not sure they can be included to Storm 2.0.0. I'll add some
> to
> epic if I think I can find time to do. If you have items which are
> planned
> to be included to Storm 2.0.0, please add them to epic issue.
>
> I hope that we can release Storm 2.0.0 in 1Q (1 month and several days
> left): even if it doesn't happen, we could release beta version of
> Storm
> 2.0.0 in 1Q and we say "feature freeze" and concentrate on stabilizing
> the
> release and making Storm 2.0.0 happen sooner than later. If you have
> items
> for Storm 2.0.0 which requires more than 1 month to be finished, I
> think it
> would be worth to share the items and discuss.
>
> Please add your voice on anything about Storm 2.0.0 plan. It would be
> much
> appreciated if someone puts efforts on testing and stabilizing the
> current
> master branch (it would take much time and efforts and more hands are
> definitely better).
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
>
>

Verification about my observation for current State implementation

[Jungtaek Lim]

Hi,

I'd like to verify my observation on current State implementation is
correct, so that we could fix them if necessary and make plan for
improvement.

1. State is stored with namespace prefix which typically composes to
(component id, task id) pair and it doesn't look like having classification
for topology. Is this correct observation? If then I think that's worth to
call it as 'critical' and it must be fixed.

2. We're allowing end-users to put key of state, and also no restriction
for grouping on stateful component. I feel such flexibility breaks the
possibility to reshard state and end-users are required to implement their
own reshard tool according to their topology state key distribution logic.
I expect it will not happen on streams API (since it should be done with
keyed stream) but wouldn't it better to also restrict such flexibility also
for core API?

3. Suppose we are going to support state resharding (for allowing change of
parallelism) and we restrict to apply field grouping with key while
connecting stateful component.
Then key-value can be moved based on key (though finding and replacing task
id may not be trivial if component name has '-'... we have same issue on
metric name, so maybe time to restrict characters on topology name as well
as component name?).
Is it also true for window/partition/windowsystem state? I didn't take a
deep look on window state (I would find a time) but it would be great if
someone knowing the detail makes it clear.

Thanks in advance,
Jungtaek Lim (HeartSaVioR)

[DISCUSS] Plan on Storm 2.0.0

[Jungtaek Lim]

Hi devs,

We've just released Storm 1.2.1 (not officially announced but vote passed
anyway) and we're in agreement that no further minor version on 1.x version
line, so it's time to focus on Storm 2.0.0 so that we can bring far long
undetermined milestone to reality.

Here is the epic issue for Storm 2.0.0:
https://issues.apache.org/jira/browse/STORM-2714

We have no restriction to file issues and submit patches so there're more
changes available outside of epic issue, but once we reach consensus to
bring Storm 2.0.0 fairly sooner, I propose we (team, or individual) add all
the issues to epic which we target to include to Storm 2.0.0, so that we
can track all the remaining items from the epic issue, and decide to
postpone some non-blocker dragging items to out of Storm 2.0.0. Makes sense?

I have some backlog issues for myself which would need somewhat huge
efforts so not sure they can be included to Storm 2.0.0. I'll add some to
epic if I think I can find time to do. If you have items which are planned
to be included to Storm 2.0.0, please add them to epic issue.

I hope that we can release Storm 2.0.0 in 1Q (1 month and several days
left): even if it doesn't happen, we could release beta version of Storm
2.0.0 in 1Q and we say "feature freeze" and concentrate on stabilizing the
release and making Storm 2.0.0 happen sooner than later. If you have items
for Storm 2.0.0 which requires more than 1 month to be finished, I think it
would be worth to share the items and discuss.

Please add your voice on anything about Storm 2.0.0 plan. It would be much
appreciated if someone puts efforts on testing and stabilizing the current
master branch (it would take much time and efforts and more hands are
definitely better).

Thanks,
Jungtaek Lim (HeartSaVioR)

Re: [DISCUSS] consider EOL for version lines

[Jungtaek Lim]

Did download page being reverted only from published site? I can see
release notes are published (so I guess gitpubsub works well) but download
page doesn't represent current storm-site. Is it due to LGPL issue on Storm
1.2.0?

2018년 2월 17일 (토) 오전 2:18, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:

> As part of the release process I updated the download page [1] to move
> older releases to archive.a.o as requested by INFRA. I also removed the
> “Current Release” sections for 0.9.x and 0.10.x to make it more clear that
> those lines are no longer supported.
>
> If we want to add a more explicit statement to the download page, we can
> certainly do that as well.
>
> -Taylor
>
> [1] http://storm.apache.org/downloads.html
>
> > On Feb 13, 2018, at 4:45 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
> >
> > We're in discussion to split out storm-kafka-client to have separate
> > release version and cycle. Since we have applied necessary changes to
> > storm-core of 1.1.x/1.0.x, unless storm-kafka-client needs more changes
> on
> > storm-core, storm-core 1.1.x/1.0.x should be compatible with
> > storm-kafka-client be compatible with storm-core 1.x.
> >
> > I'm OK to keep the 1.0.x version lines specifically for storm-mesos (and
> > then we would end up keeping 1.1.x version line too) if we can't avoid
> the
> > case, but I couldn't guarantee all the bug fixes should be ported back to
> > 1.0.x. We even often don't port back the bug fixes to 1.0.x version lines
> > when 1.1 is the active minor version of Storm unless the fixes are
> > critical. That means we may not port back the bug fixes to 1.1.x after
> > announcing 1.2.0 and 1.2.x-branch is cut. As we all know, it greatly
> > reduces the maintenance cost. It does make sense for storm-kafka-client
> to
> > do the backport for 1.1.x/1.0.x branches because huge changes are just
> > applied (though I guess the experiment described above may resolve such
> > issue), but not for others including storm-core except critical/blocker
> > issues.
> >
> > So in storm-mesos view, it would be better to catch up with highest
> > possible version (that's why I hope to adopt storm-mesos in storm repo,
> > maybe not likely to happen for now), and I understand storm-mesos can't
> for
> > now because of Storm's issue. I wish the investigation of "Storm on X"
> > would occur actively sooner than later, so that it can be included as
> > earlier version of Storm 2.x (ideally 2.0.0 but it is just an ideal
> view).
> >
> > Anyway, looks like there is no objection to announce 0.10.x/0.9.x
> > explicitly EOL.
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> > 2018년 2월 14일 (수) 오전 6:02, Erik Weathers <eweath...@groupon.com.invalid
> >님이
> > 작성:
> >
> >> Thanks for keeping storm-mesos in mind Stig. :)  I'd be most worried
> about
> >> any issues we might see with the backported storm-kafka-client and how
> we
> >> *might* need to fix bugs in 1.0.x.  At least it should be easy to
> >> cherry-pick fixes back into 1.0.x after the backport-stomping of
> >> STORM-2937.
> >>
> >> Look forward to working with Bobby to get a long term plan for storm to
> run
> >> on mesos in 2.x+.
> >>
> >> - Erik
> >>
> >> On Tue, Feb 13, 2018 at 11:26 AM, Stig Rohde Døssing <
> >> stigdoess...@gmail.com
> >>> wrote:
> >>
> >>> +1 to maintain 3 version lines, though we may want to look at what we
> can
> >>> do for storm mesos, which I think it currently stuck on 1.0.x.
> >>>
> >>> 2018-02-13 20:17 GMT+01:00 Hugo Da Cruz Louro <hlo...@hortonworks.com
> >:
> >>>
> >>>> +1 to maintain 3 version lines. Let’s properly announce that in our
> >>> portal
> >>>> and users list such that users know what’s coming.
> >>>>
> >>>> Agree with focusing on 2.0 which has a lot of improvements, rather
> than
> >>>> 1.x, x >= 3.
> >>>>
> >>>>> On Feb 13, 2018, at 10:43 AM, Alexandre Vermeerbergen <
> >>>> avermeerber...@gmail.com> wrote:
> >>>>>
> >>>>> +1 (non binding) to maintaining less version lines, provided that
> >>>>> 1.2.x branch is maintained long enough to allow progressive adoption
> >>>>> of 2.x
> >>>>>
> >>>>> Alexandre Vermeerbergen
> >>>>>
> >>>>> 2018-02-13 19:38 GMT+01:00 Priyank Shah <ps...@hortonwork

Re: [VOTE] Release Apache Storm 1.2.1 (rc1)

[Jungtaek Lim]

+1 (binding)

* verified src/bin with targz/zip archives
* checked the diff between extracted targz/zip (no difference)
* oncrpc.jar is not included in binary distribution

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 2월 17일 (토) 오전 10:07, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:

> Thank you to all who have voted thus far. Although we have enough PMC +1’s
> to release, I’d like to hold the vote open until at least 24 hours have
> passed so those in other timezones have a chance to weigh in an understand
> what’s going on.
>
> -Taylor
>
> > On Feb 16, 2018, at 7:04 PM, Arun Mahadevan <ar...@apache.org> wrote:
> >
> > +1 (binding)
> >
> > - Downloaded .zip/.tar.gz and verified md5.
> >
> > - Verified the oncrpc jar is not included in the distribution
> >
> > - Ran a few sample topologies, checked log viewer.
> >
> > - Built the source code from .zip file
> >
> >
> > Thanks,
> > Arun
> >
> >
> >
> > On 2/16/18, 3:27 PM, "Satish Duggana" <satish.dugg...@gmail.com> wrote:
> >
> >> +1 (binding)
> >>
> >> On Sat, Feb 17, 2018 at 3:03 AM, P. Taylor Goetz <ptgo...@gmail.com>
> wrote:
> >>
> >>> +1 (binding)
> >>>
> >>> Verified that the LGPL dependency is no longer included in the binary
> >>> distribution.
> >>>
> >>> -Taylor
> >>>
> >>>> On Feb 16, 2018, at 4:01 PM, P. Taylor Goetz <ptgo...@gmail.com>
> wrote:
> >>>>
> >>>> This is a call to vote on releasing Apache Storm 1.2.1 (rc1)
> >>>>
> >>>> NOTE: The primary purpose of this release is to remove an
> LGPL-licensed
> >>> binary that was inadvertently included in the 1.2.0 binary release.
> >>>>
> >>>> Full list of changes in this release:
> >>>>
> >>>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> >>> 2.1-rc1/RELEASE_NOTES.html
> >>>>
> >>>> The tag/commit to be voted upon is v1.2.1:
> >>>>
> >>>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=
> >>>
> 89646c86c667e0a35aed45c8063d777ae1f32b30;hb=d156d25d991311eaa1f5131d3dc347
> >>> 87f87ce684
> >>>>
> >>>> The source archive being voted upon can be found here:
> >>>>
> >>>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> >>> 2.1-rc1/apache-storm-1.2.1-src.tar.gz
> >>>>
> >>>> Other release files, signatures and digests can be found here:
> >>>>
> >>>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.1-rc1/
> >>>>
> >>>> The release artifacts are signed with the following key:
> >>>>
> >>>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> >>> plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >>>>
> >>>> The Nexus staging repository for this release is:
> >>>>
> >>>>
> https://repository.apache.org/content/repositories/orgapachestorm-1061
> >>>>
> >>>> Please vote on releasing this package as Apache Storm 1.2.1.
> >>>>
> >>>> When voting, please list the actions taken to verify the release.
> >>>>
> >>>> This vote will be open for 72 hours or until at least 3 PMC members
> vote
> >>> +1.
> >>>>
> >>>> [ ] +1 Release this package as Apache Storm 1.2.1
> >>>> [ ]  0 No opinion
> >>>> [ ] -1 Do not release this package because...
> >>>>
> >>>> Thanks to everyone who contributed to this release.
> >>>>
> >>>> -Taylor
> >>>
> >>>
> >
>
>

Re: [DISCUSS] consider EOL for version lines

[Jungtaek Lim]

We're in discussion to split out storm-kafka-client to have separate
release version and cycle. Since we have applied necessary changes to
storm-core of 1.1.x/1.0.x, unless storm-kafka-client needs more changes on
storm-core, storm-core 1.1.x/1.0.x should be compatible with
storm-kafka-client be compatible with storm-core 1.x.

I'm OK to keep the 1.0.x version lines specifically for storm-mesos (and
then we would end up keeping 1.1.x version line too) if we can't avoid the
case, but I couldn't guarantee all the bug fixes should be ported back to
1.0.x. We even often don't port back the bug fixes to 1.0.x version lines
when 1.1 is the active minor version of Storm unless the fixes are
critical. That means we may not port back the bug fixes to 1.1.x after
announcing 1.2.0 and 1.2.x-branch is cut. As we all know, it greatly
reduces the maintenance cost. It does make sense for storm-kafka-client to
do the backport for 1.1.x/1.0.x branches because huge changes are just
applied (though I guess the experiment described above may resolve such
issue), but not for others including storm-core except critical/blocker
issues.

So in storm-mesos view, it would be better to catch up with highest
possible version (that's why I hope to adopt storm-mesos in storm repo,
maybe not likely to happen for now), and I understand storm-mesos can't for
now because of Storm's issue. I wish the investigation of "Storm on X"
would occur actively sooner than later, so that it can be included as
earlier version of Storm 2.x (ideally 2.0.0 but it is just an ideal view).

Anyway, looks like there is no objection to announce 0.10.x/0.9.x
explicitly EOL.

- Jungtaek Lim (HeartSaVioR)

2018년 2월 14일 (수) 오전 6:02, Erik Weathers <eweath...@groupon.com.invalid>님이
작성:

> Thanks for keeping storm-mesos in mind Stig. :)  I'd be most worried about
> any issues we might see with the backported storm-kafka-client and how we
> *might* need to fix bugs in 1.0.x.  At least it should be easy to
> cherry-pick fixes back into 1.0.x after the backport-stomping of
> STORM-2937.
>
> Look forward to working with Bobby to get a long term plan for storm to run
> on mesos in 2.x+.
>
> - Erik
>
> On Tue, Feb 13, 2018 at 11:26 AM, Stig Rohde Døssing <
> stigdoess...@gmail.com
> > wrote:
>
> > +1 to maintain 3 version lines, though we may want to look at what we can
> > do for storm mesos, which I think it currently stuck on 1.0.x.
> >
> > 2018-02-13 20:17 GMT+01:00 Hugo Da Cruz Louro <hlo...@hortonworks.com>:
> >
> > > +1 to maintain 3 version lines. Let’s properly announce that in our
> > portal
> > > and users list such that users know what’s coming.
> > >
> > > Agree with focusing on 2.0 which has a lot of improvements, rather than
> > > 1.x, x >= 3.
> > >
> > > > On Feb 13, 2018, at 10:43 AM, Alexandre Vermeerbergen <
> > > avermeerber...@gmail.com> wrote:
> > > >
> > > > +1 (non binding) to maintaining less version lines, provided that
> > > > 1.2.x branch is maintained long enough to allow progressive adoption
> > > > of 2.x
> > > >
> > > > Alexandre Vermeerbergen
> > > >
> > > > 2018-02-13 19:38 GMT+01:00 Priyank Shah <ps...@hortonworks.com>:
> > > >> +1 to maintaining 3 version lines as suggested by Jungtaek.
> > > >>
> > > >> On 2/13/18, 9:51 AM, "Arun Iyer on behalf of Arun Mahadevan" <
> > > ai...@hortonworks.com on behalf of ar...@apache.org> wrote:
> > > >>
> > > >>+1 to maintain 3 version lines.
> > > >>
> > > >>I think the next focus should be 2.0.0 than 1.3.0.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>On 2/12/18, 11:40 PM, "Jungtaek Lim" <kabh...@gmail.com> wrote:
> > > >>
> > > >>> Hi devs,
> > > >>>
> > > >>> I've noticed that we are providing 4 different version lines
> (1.1.x,
> > > 1.0.x,
> > > >>> 0.10.x, 0.9.x) in download page, and I expect we will add one more
> > for
> > > >>> 1.2.0. Moreover, we have one more develop version line (2.0.0 -
> > master)
> > > >>> which most of development happens there.
> > > >>>
> > > >>> Recently we're releasing 3 version lines (1.0.6 / 1.1.2 / 1.2.0)
> > > >>> simultaneously and it took heavy effort to track all the RCs and
> > > verify all
> > > >>> of them. I guess release manager would take more overhead of
> > > releasing, and
> > > >>> it doesn't make sense for me if we continue maintaining all of
> them.
> > > >>>
> > > >>> Ideally I'd like to propose maintaining three version lines: 2.0.0
> > > (next
> > > >>> major) / 1.3.0 (next minor - may not happen) / 1.2.1 (next bugfix)
> > and
> > > >>> making others EOL (that respects semantic versioning and even other
> > > >>> projects tend to maintain only two version lines), but if someone
> > > feels too
> > > >>> aggressive, I propose at least we explicitly announce EOL to 0.x
> > > version
> > > >>> lines and get rid of any supports (downloads) for them.
> > > >>>
> > > >>> Would like to hear your opinion.
> > > >>>
> > > >>> Thanks,
> > > >>> Jungtaek Lim (HeartSaVioR)
> > > >>
> > > >>
> > > >>
> > > >
> > >
> > >
> >
>

Re: [DISCUSS] [Storm-Kafka] - Maintenance, Branch Support, and Deprecation Plans for storm-kafka and storm-kaka-client

[Jungtaek Lim]

I forgot the one, but this is great time to revisit this, since we got
resolved many storm-kafka-client issues hence feeling it as fairly stable.

I'm +1 on deprecate storm-kafka at 1.x version lines and remove it at
2.0.0. Do we want to have explicit VOTE to receive more opinions here?

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 7월 20일 (목) 오전 5:10, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:

> +1 I’m fine with taking this approach.
>
> -Taylor
>
> > On Jul 19, 2017, at 2:04 PM, Stig Rohde Døssing <stigdoess...@gmail.com>
> wrote:
> >
> > +1 for removing storm-kafka from master, since we shouldn't encourage
> > people to use a component that won't work on new Kafka versions. As you
> > both mentioned, the 1.x version of storm-kafka should still be usable on
> a
> > 2.0 cluster, so it will still be available in case people need it. A wiki
> > page for tracking current missing pieces for storm-kafka-client sounds
> good.
> >
> > 2017-07-19 19:09 GMT+02:00 Harsha <st...@harsha.io>:
> >
> >> +1 on moving away from storm-kafka for Storm 2.0. For existing users we
> >> can provide any critical bug fixes and provide it as part of 1.x
> >> releases. They can still use the existing 1.x storm-kafka against 2.0.
> >> Since kafka itself is moving away from older APIs continuing two
> >> versions of kafka connector doesnt’ make sense and honestly splits the
> >> usage which doesn’t give us any feedback on new storm-kafka-client.
> >> Thanks,
> >> Harsha
> >>
> >> On Wed, Jul 19, 2017, at 09:20 AM, Hugo Da Cruz Louro wrote:
> >>> Hi,
> >>>
> >>> The goal of this email is to summarize and unify the discussion started
> >>> across several email threads (Storm 2.0
> >>> Roadmap<http://search-hadoop.com/?project=Storm=%22%
> >> 5BDISCUSS%5D+Storm+2.0+Roadmap%22>,
> >>> 1.1.1 Release
> >>> Planning<http://search-hadoop.com/m/Storm/8gnYyGagLDWv1qG?
> >> subj=Release+Planning+for+1+1+1+and+others+>,
> >>> Lag
> >>> Issues<http://search-hadoop.com/m/Storm/8gnYyLmjIjYr692?
> >> subj=Lag+issues+using+Storm+1+1+1+latest+build+with+
> >> StormKafkaClient+1+1+1+vs+old+StormKafka+spouts>)
> >>> concerning the maintenance, branch support, and eventual deprecation of
> >>> storm-kafka and storm-kafka-client.
> >>>
> >>> It was proposed in an earlier
> >>> discussion<http://search-hadoop.com/?project=Storm=%
> >> 22%5BDISCUSS%5D+Storm+2.0+Roadmap%22>
> >>> the plan to deprecate storm-kafka in prol of storm-kafka-client. To
> >>> clarify, the idea is not to completely eliminate storm-kafka, but
> rather
> >>> keep supporting it in the 1.x-branch, while removing it from master
> (i.e.
> >>> Storm 2.0 onwards). That is, storm-kafka-client will then become the
> only
> >>> Storm Kafka option available for Storm 2.0 onwards, given that we have
> >>> enough confidence in its stability by the time of the Storm 2.0
> release.
> >>>
> >>> The main reason for this proposal is the fact that the Kafka community
> >>> agreed<https://cwiki.apache.org/confluence/display/KAFKA/
> >> KIP-109:+Old+Consumer+Deprecation>
> >>> to deprecate the old consumer APIs starting in version 0.10.2, and will
> >>> remove them in the next major version (0.12). This implies that
> >>> storm-kafka will not work for Kafka 0.12 onwards. Important features
> >>> missing in the old Kafka consumer are: security, new message format,
> and
> >>> fetching offsets based on time stamp (KIP-79).
> >>>
> >>> In earlier discussions the Storm community has shown concerns about the
> >>> performance and stability of the storm-kafka-client. Those concerns are
> >>> valid and were mirrored by the Kafka community in their early
> deprecation
> >>> discussions. I align with what was said in the Kafka
> >>> discussion<http://search-hadoop.com/m/Kafka/uyzND1e4bUP1Rjq721>: the
> >>> storm-kafka-client has bugs, but so does storm-kafka, and all the
> >>> development is currently going into storm-kafka-client, which will be
> >>> even more prevalent in face of Kafka discontinuing the old consumer
> >>> API’s. The only way to stabilize a complex component such as
> >>> storm-kafka-client is to test it extensively in all its variants, which
> >>> inevitably comes from users using it. Furthermore, removing storm-kafka
> >

[DISCUSS] consider EOL for version lines

[Jungtaek Lim]

Hi devs,

I've noticed that we are providing 4 different version lines (1.1.x, 1.0.x,
0.10.x, 0.9.x) in download page, and I expect we will add one more for
1.2.0. Moreover, we have one more develop version line (2.0.0 - master)
which most of development happens there.

Recently we're releasing 3 version lines (1.0.6 / 1.1.2 / 1.2.0)
simultaneously and it took heavy effort to track all the RCs and verify all
of them. I guess release manager would take more overhead of releasing, and
it doesn't make sense for me if we continue maintaining all of them.

Ideally I'd like to propose maintaining three version lines: 2.0.0 (next
major) / 1.3.0 (next minor - may not happen) / 1.2.1 (next bugfix) and
making others EOL (that respects semantic versioning and even other
projects tend to maintain only two version lines), but if someone feels too
aggressive, I propose at least we explicitly announce EOL to 0.x version
lines and get rid of any supports (downloads) for them.

Would like to hear your opinion.

Thanks,
Jungtaek Lim (HeartSaVioR)

Re: [VOTE] Release Apache Storm 1.0.6 (rc3)

[Jungtaek Lim]

I just went ahead verifying current RC except serialization UID issue in
Fields. I could also vote for RC4 immediately if necessary.

+1 (binding)

> source

- verify file (signature, MD5, SHA)
-- source, tar.gz : OK
-- source, zip : OK

- extract file
-- source, tar.gz : OK
-- source, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- build source with JDK 7
-- source, tar.gz : OK

- build source dist
-- source, tar.gz : OK

- build binary dist
-- source, tar.gz : OK

> binary

- verify file (signature, MD5, SHA)
-- binary, tar.gz : OK
-- binary, zip : OK

- extract file
-- binary, tar.gz : OK
-- binary, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- launch daemons : OK

- run RollingTopWords (local) : OK

- run RollingTopWords (remote) : OK
  - activate / deactivate / rebalance / kill : OK
  - logviewer (worker dir, daemon dir) : OK
  - change log level : OK
  - thread dump, heap dump, restart worker : OK
  - log search : OK

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 2월 9일 (금) 오후 6:18, Erik Weathers <eweath...@groupon.com.invalid>님이 작성:

> I'm fine submitting a PR to back that line out (or any of you committer
> folks could just rip it out).
>
> But I'd like to understand Storm a bit better as part of making this
> decision. :-)  Am I correct in assuming it would only be a problem if the
> serialized Fields were stored somewhere (e.g., ZooKeeper, local filesystem)
> and then read back in after the Nimbus/Workers are brought back up after
> the upgrade?  Seems Fields is used in a *lot* of places, and I don't know
> precisely what is serialized for reused upon Storm Nimbus/Worker daemon
> restarts.  I believe there are examples of Fields being used to create
> Spout or Bolt objects that are used to create the StormTopology object,
> which I believe is serialized into ZooKeeper.  But I'm not clear if it's
> directly the Fields object itself or some kind of translation from that
> into the thrift objects that make up StormTopology.
>
> I also don't know exactly when kryo is applicable in Storm.  I've never
> done anything with kryo directly.
>
> - Erik
>
> On Thu, Feb 8, 2018 at 10:00 PM, P. Taylor Goetz <ptgo...@gmail.com>
> wrote:
>
> > *serialized* ;)
> >
> > > On Feb 9, 2018, at 12:48 AM, P. Taylor Goetz <ptgo...@gmail.com>
> wrote:
> > >
> > > I’d have to check (can’t right now), but I think that class gets
> > sterilized via kryo. If that’s not the case, yes, it could cause
> problems.
> > >
> > > I think the safest option would be to remove the serialversionuid.
> > >
> > > -Taylor
> > >
> > >> On Feb 8, 2018, at 5:36 PM, Erik Weathers
> <eweath...@groupon.com.INVALID>
> > wrote:
> > >>
> > >> Something I just realized -- in the storm-kafka-client stomping into
> > >> 1.0.x-branch PR, I backported a change to Fields.java which added a
> > >> serialVersionUID.
> > >> Could that potentially break topologies when you upgrade storm-core on
> > the
> > >> servers (nimbus, workers) from 1.0.{1..5} to 1.0.6?   I'm not super
> > >> familiar with the serialization that occurs in Storm and whether that
> > could
> > >> break people.
> > >>
> > >> https://github.com/apache/storm/pull/2550/files#diff-71a428d
> > 508c4f5af0bfe3cc186e8edcf
> > >>
> > >> - Erik
> > >>
> > >>> On Thu, Feb 8, 2018 at 1:25 PM, Bobby Evans <ev...@oath.com.invalid>
> > wrote:
> > >>>
> > >>> +1 I built the code from the git tag, ran all the unit tests (which
> > passed
> > >>> the first time), and ran some tests on a single node cluster.
> > >>>
> > >>> It all looked good.
> > >>>
> > >>> - Bobby
> > >>>
> > >>>> On Thu, Feb 8, 2018 at 1:22 PM P. Taylor Goetz <ptgo...@gmail.com>
> > wrote:
> > >>>>
> > >>>> This is a call to vote on releasing Apache Storm 1.0.6 (rc3)
> > >>>>
> > >>>> Full list of changes in this release:
> > >>>>
> > >>>>
> > >>>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > >>> 0.6-rc3/RELEASE_NOTES.html
> > >>>>
> > >>>> The tag/commit to be voted upon is v1.0.6:
> > >>>>
> > >>>>
> > >>>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=
> > >>> e68365f9f947ddd1794b2edef2149fdfaa1590a2;hb=7993db01580ce62d
> > 44866dc00e0a72
&g

Re: [VOTE] Release Apache Storm 1.1.2 (rc3)

[Jungtaek Lim]

+1 (binding)

> source

- verify file (signature, MD5, SHA)
-- source, tar.gz : OK
-- source, zip : OK

- extract file
-- source, tar.gz : OK
-- source, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- build source with JDK 7 (without all-tests flag: integration tests are
failing with high chance, hence relying on Bobby's verification)
-- source, tar.gz : OK

- build source dist
-- source, tar.gz : OK

- build binary dist
-- source, tar.gz : OK

> binary

- verify file (signature, MD5, SHA)
-- binary, tar.gz : OK
-- binary, zip : OK

- extract file
-- binary, tar.gz : OK
-- binary, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- launch daemons : OK

- run RollingTopWords (local) : OK

- run RollingTopWords (remote) : OK
  - activate / deactivate / rebalance / kill : OK
  - logviewer (worker dir, daemon dir) : OK
  - change log level : OK
  - thread dump, heap dump, restart worker : OK
  - log search : OK

Thanks,
Jungtaek Lim (HeartSaVioR)


2018년 2월 9일 (금) 오전 5:59, Bobby Evans <ev...@oath.com.invalid>님이 작성:

> +1 I built from the tag and did some basic tests on a single node cluster.
>
> I am disappointed that it took me 7 times to get all of the unit tests to
> pass for the build, but I don't think it should block a release.
>
> - Bobby
>
> On Thu, Feb 8, 2018 at 11:48 AM P. Taylor Goetz <ptgo...@gmail.com> wrote:
>
> > This is a call to vote on releasing Apache Storm 1.1.2 (rc3)
> >
> > Full list of changes in this release:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.2-rc3/RELEASE_NOTES.html
> >
> > The tag/commit to be voted upon is v1.1.2:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=3aab00d053d3576d3f8a37bcc3c7e51072ead22e;hb=0bb7a66a9b26ad96afc81b27dd45f93ae8969c44
> >
> > The source archive being voted upon can be found here:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.2-rc3/apache-storm-1.1.2-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.2-rc3/
> >
> > The release artifacts are signed with the following key:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1059
> >
> > Please vote on releasing this package as Apache Storm 1.1.2.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 1.1.2
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
> >
>

Re: [VOTE] Release Apache Storm 1.0.6 (rc3)

[Jungtaek Lim]

I'm not 100% sure either, but I assume Storm serializes topology
information into thrift structure (at least Grouping.fields() takes
List instead of Fields), so unless nodes are using different
versions of Storm (which we never support), that would be safe. If it's not
safe, we already broke rolling upgrade from under 1.1.0 to 1.1.0 since same
modification was done from 1.1.0.

2018년 2월 9일 (금) 오전 7:37, Erik Weathers 님이 작성:

> Something I just realized -- in the storm-kafka-client stomping into
> 1.0.x-branch PR, I backported a change to Fields.java which added a
> serialVersionUID.
> Could that potentially break topologies when you upgrade storm-core on the
> servers (nimbus, workers) from 1.0.{1..5} to 1.0.6?   I'm not super
> familiar with the serialization that occurs in Storm and whether that could
> break people.
>
>
> https://github.com/apache/storm/pull/2550/files#diff-71a428d508c4f5af0bfe3cc186e8edcf
>
> - Erik
>
> On Thu, Feb 8, 2018 at 1:25 PM, Bobby Evans 
> wrote:
>
> > +1 I built the code from the git tag, ran all the unit tests (which
> passed
> > the first time), and ran some tests on a single node cluster.
> >
> > It all looked good.
> >
> > - Bobby
> >
> > On Thu, Feb 8, 2018 at 1:22 PM P. Taylor Goetz 
> wrote:
> >
> > > This is a call to vote on releasing Apache Storm 1.0.6 (rc3)
> > >
> > > Full list of changes in this release:
> > >
> > >
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > 0.6-rc3/RELEASE_NOTES.html
> > >
> > > The tag/commit to be voted upon is v1.0.6:
> > >
> > >
> > > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=
> >
> e68365f9f947ddd1794b2edef2149fdfaa1590a2;hb=7993db01580ce62d44866dc00e0a72
> > 66984638d0
> > >
> > > The source archive being voted upon can be found here:
> > >
> > >
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > 0.6-rc3/apache-storm-1.0.6-src.tar.gz
> > >
> > > Other release files, signatures and digests can be found here:
> > >
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.6-rc3/
> > >
> > > The release artifacts are signed with the following key:
> > >
> > >
> > > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> > plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > >
> > > The Nexus staging repository for this release is:
> > >
> > > https://repository.apache.org/content/repositories/orgapachestorm-1060
> > >
> > > Please vote on releasing this package as Apache Storm 1.0.6.
> > >
> > > When voting, please list the actions taken to verify the release.
> > >
> > > This vote will be open for at least 72 hours.
> > >
> > > [ ] +1 Release this package as Apache Storm 1.0.6
> > > [ ]  0 No opinion
> > > [ ] -1 Do not release this package because...
> > >
> > > Thanks to everyone who contributed to this release.
> > >
> > > -Taylor
> > >
> >
>

Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch / 1.0.x-branch with 1.x-branch

[Jungtaek Lim]

Update: I also merged the patch for 1.0.x-branch. Thanks all for taking
valuable efforts and time to participate this topic and making it possible
so quickly. Now release phase of both 1.1.2 and 1.0.6 can be restart.

2018년 2월 8일 (목) 오전 6:30, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> Update: I just merged the patch for 1.1.x-branch so release phase of Storm
> 1.1.2 can be restarted. Patch for 1.0.x-branch from Erik is available and
> got some +1s but waiting for 24hrs rule.
>
> 2018년 2월 7일 (수) 오전 5:03, Stig Rohde Døssing <stigdoess...@gmail.com>님이 작성:
>
>> Took a look at backporting to 1.0.x. We'll have to update the time
>> simulation code (Time.java in storm-core) to support nanoseconds, as Erik
>> noted, but this isn't a breaking change and only affects tests.
>>
>> This PR https://github.com/apache/storm/pull/1995/files#diff-
>> 72647db30ffd6005dc01c4d1f75d2c68 made a breaking change to
>> IOpaquePartitionedTridentSpoutExecutor, so we'll have to do the same on
>> 1.0.x.
>>
>> 2018-02-06 19:13 GMT+01:00 P. Taylor Goetz <ptgo...@gmail.com>:
>>
>> > Just a heads up: While this gets sorted out I’m going to proceed with a
>> > 1.2.0 RC.
>> >
>> > -Taylor
>> >
>> > > On Feb 5, 2018, at 10:46 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
>> > >
>> > > UPDATE: Submitted a pull request https://github.com/apache/
>> > storm/pull/2549 for
>> > > STORM-2936 (against 1.1.x-branch)
>> > >
>> > > Erik, please change the status to "IN PROGRESS" if someone is working
>> > on. I
>> > > would find the free time and just do it if there's no one working in
>> > > progress.
>> > >
>> > > 2018년 2월 6일 (화) 오전 10:39, Jungtaek Lim <kabh...@gmail.com>님이 작성:
>> > >
>> > >> Thanks for quick response Erik!
>> > >>
>> > >> Just filed two issues :
>> > >> https://issues.apache.org/jira/browse/STORM-2936 (for 1.1.x-branch)
>> > >> https://issues.apache.org/jira/browse/STORM-2937 (for 1.0.x-branch)
>> > >>
>> > >> We have another discussion around making storm-kafka-client be
>> > experiment
>> > >> of managing separately (independent of Storm release). So the three
>> > >> versions which are in release phase might be the last releases of
>> > >> "battery-included" of storm-kafka-client if our experiment works
>> well.
>> > >>
>> > >> If we would want to make the change for storm-kafka-client, it might
>> be
>> > >> better to put the change and release before start experimenting, but
>> > that's
>> > >> just a thought on my own. In opposite way, we could even start
>> > experiment
>> > >> and make change of storm-core of 1.0.x-branch to be compatible with
>> that
>> > >> version of storm-kafka-client. We could even do it for 1.1.x-branch,
>> but
>> > >> the change is almost done so it doesn't look like needed to postpone
>> it.
>> > >>
>> > >> Would like to here everyone's voice on this.
>> > >>
>> > >> -Jungtaek Lim (HeartSaVioR)
>> > >>
>> > >> 2018년 2월 6일 (화) 오전 10:23, Erik Weathers
>> <eweath...@groupon.com.invalid
>> > >님이
>> > >> 작성:
>> > >>
>> > >>> Thanks for the quick response Jungtaek!
>> > >>>
>> > >>> Yes, my teammates and myself would like to help on this.  Is there
>> an
>> > >>> existing JIRA for the work you've been doing on the other branches?
>> > >>>
>> > >>> I propose we don't make this block 1.0.6 -- we can just release
>> 1.0.7
>> > >>> quickly when the backport is done, if that is amenable.
>> > >>> That strategy also might be cleaner since it would avoid other
>> changes
>> > in
>> > >>> 1.0.6 being lumped together with this.
>> > >>>
>> > >>> - Erik
>> > >>>
>> > >>> On Mon, Feb 5, 2018 at 5:16 PM, Jungtaek Lim <kabh...@gmail.com>
>> > wrote:
>> > >>>
>> > >>>> UPDATE: I've finished working on overwriting storm-kafka-client
>> > >>> 1.x-branch
>> > >>>> to 1.1.x-branch. Not yet pushed to ASF git, but pushed to my fork
>> > first
>> > >>> to
>> > >>&g

Re: [VOTE] Release Apache Storm 1.2.0 (rc4)

[Jungtaek Lim]

+1 (binding)

> source

- verify file (signature, MD5, SHA)
-- source, tar.gz : OK
-- source, zip : OK

- extract file
-- source, tar.gz : OK
-- source, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- build source with JDK 7
-- source, tar.gz : OK

> binary

- verify file (signature, MD5, SHA)
-- binary, tar.gz : OK
-- binary, zip : OK

- extract file
-- binary, tar.gz : OK
-- binary, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

I already verified others from RC3, and also verified that
storm-kafka-monitor source/javadoc files are no longer exist in toollib
directory.

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 2월 8일 (목) 오전 7:24, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:

> This is a call to vote on releasing Apache Storm 1.2.0 (rc4)
>
> Note that the only difference between rc4 and rc3 is the fix for
> https://issues.apache.org/jira/browse/STORM-2942.
>
> Full list of changes in this release:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc4/RELEASE_NOTES.html
>
> The tag/commit to be voted upon is v1.2.0:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=cef4d49e222e53656f38c40d754d4f41799cd9a9;hb=2a0097f9a20b9df494caadb87c87d4e4db01a7ed
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc4/apache-storm-1.2.0-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc4/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1058
>
> Please vote on releasing this package as Apache Storm 1.2.0.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 1.2.0
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
>

Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch / 1.0.x-branch with 1.x-branch

[Jungtaek Lim]

Update: I just merged the patch for 1.1.x-branch so release phase of Storm
1.1.2 can be restarted. Patch for 1.0.x-branch from Erik is available and
got some +1s but waiting for 24hrs rule.

2018년 2월 7일 (수) 오전 5:03, Stig Rohde Døssing <stigdoess...@gmail.com>님이 작성:

> Took a look at backporting to 1.0.x. We'll have to update the time
> simulation code (Time.java in storm-core) to support nanoseconds, as Erik
> noted, but this isn't a breaking change and only affects tests.
>
> This PR https://github.com/apache/storm/pull/1995/files#diff-
> 72647db30ffd6005dc01c4d1f75d2c68 made a breaking change to
> IOpaquePartitionedTridentSpoutExecutor, so we'll have to do the same on
> 1.0.x.
>
> 2018-02-06 19:13 GMT+01:00 P. Taylor Goetz <ptgo...@gmail.com>:
>
> > Just a heads up: While this gets sorted out I’m going to proceed with a
> > 1.2.0 RC.
> >
> > -Taylor
> >
> > > On Feb 5, 2018, at 10:46 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
> > >
> > > UPDATE: Submitted a pull request https://github.com/apache/
> > storm/pull/2549 for
> > > STORM-2936 (against 1.1.x-branch)
> > >
> > > Erik, please change the status to "IN PROGRESS" if someone is working
> > on. I
> > > would find the free time and just do it if there's no one working in
> > > progress.
> > >
> > > 2018년 2월 6일 (화) 오전 10:39, Jungtaek Lim <kabh...@gmail.com>님이 작성:
> > >
> > >> Thanks for quick response Erik!
> > >>
> > >> Just filed two issues :
> > >> https://issues.apache.org/jira/browse/STORM-2936 (for 1.1.x-branch)
> > >> https://issues.apache.org/jira/browse/STORM-2937 (for 1.0.x-branch)
> > >>
> > >> We have another discussion around making storm-kafka-client be
> > experiment
> > >> of managing separately (independent of Storm release). So the three
> > >> versions which are in release phase might be the last releases of
> > >> "battery-included" of storm-kafka-client if our experiment works well.
> > >>
> > >> If we would want to make the change for storm-kafka-client, it might
> be
> > >> better to put the change and release before start experimenting, but
> > that's
> > >> just a thought on my own. In opposite way, we could even start
> > experiment
> > >> and make change of storm-core of 1.0.x-branch to be compatible with
> that
> > >> version of storm-kafka-client. We could even do it for 1.1.x-branch,
> but
> > >> the change is almost done so it doesn't look like needed to postpone
> it.
> > >>
> > >> Would like to here everyone's voice on this.
> > >>
> > >> -Jungtaek Lim (HeartSaVioR)
> > >>
> > >> 2018년 2월 6일 (화) 오전 10:23, Erik Weathers <eweath...@groupon.com.invalid
> > >님이
> > >> 작성:
> > >>
> > >>> Thanks for the quick response Jungtaek!
> > >>>
> > >>> Yes, my teammates and myself would like to help on this.  Is there an
> > >>> existing JIRA for the work you've been doing on the other branches?
> > >>>
> > >>> I propose we don't make this block 1.0.6 -- we can just release 1.0.7
> > >>> quickly when the backport is done, if that is amenable.
> > >>> That strategy also might be cleaner since it would avoid other
> changes
> > in
> > >>> 1.0.6 being lumped together with this.
> > >>>
> > >>> - Erik
> > >>>
> > >>> On Mon, Feb 5, 2018 at 5:16 PM, Jungtaek Lim <kabh...@gmail.com>
> > wrote:
> > >>>
> > >>>> UPDATE: I've finished working on overwriting storm-kafka-client
> > >>> 1.x-branch
> > >>>> to 1.1.x-branch. Not yet pushed to ASF git, but pushed to my fork
> > first
> > >>> to
> > >>>> trigger Travis CI to see how the build goes well.
> > >>>>
> > >>>> https://github.com/HeartSaVioR/storm/commit/76b8a7d3a6f91e66
> > >>>> 612e87da8589f5723f05218a
> > >>>> https://travis-ci.org/HeartSaVioR/storm/builds/337819430
> > >>>>
> > >>>> Thanks for the input regarding 1.0.x version, Erik. I guess then we
> > >>> have no
> > >>>> alternative here: someone has to fix storm-kafka-client as well as
> > >>>> storm-core, since including shaded storm-core doesn't make sense for
> > >>>> official 

Re: [VOTE] Release Apache Storm 1.2.0 (rc3)

[Jungtaek Lim]

Filed https://issues.apache.org/jira/browse/STORM-2943 for
storm-kafka-monitor src/javadoc inclusion issue and assigned to Taylor.


2018년 2월 8일 (목) 오전 5:32, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> Thanks for addressing md5/sha issue, Taylor. Verified src.tar.gz and
> src.zip.
>
> Alexandre, thanks again for thoughtful verification of release. It much
> helped for us and I really appreciate it. Please keep up the good work.
>
> I feel what Alexandre reported is a blocker since it leads strange error
> in point of user side, worth to cancel current RC. Changing my vote to -1
> (binding).
>
> -Jungtaek Lim (HeartSaVioR)
>
> 2018년 2월 8일 (목) 오전 5:20, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>
>> Yes. It will be fixed. Thanks again for pointing it out.
>>
>> -Taylor
>>
>> > On Feb 7, 2018, at 2:57 PM, Alexandre Vermeerbergen <
>> avermeerber...@gmail.com> wrote:
>> >
>> > Hello Taylor,
>> >
>> > Yes same issue in Storm 1.1.0 : we have a production storm cluster
>> > based on this release, but then we were using our own Kafka spout, no
>> > we weren't impacted by this packaging issue.
>> >
>> > Could the extraneous files be cleaned up from 1.2.0-final?
>> >
>> > Best regards,
>> > Alexandre Vermeerbergen
>> >
>> > 2018-02-07 19:18 GMT+01:00 P. Taylor Goetz <ptgo...@gmail.com>:
>> >> Hi Alexandre,
>> >>
>> >> Thanks for the review. You’re right, the javadoc/source jars should
>> not be there, though it looks like this has been the case for a long time
>> and would have affected previous releases.
>> >>
>> >> It looks like the problem was introduced in 1.1.0. Have you seen the
>> same issue in other 1.1.x releases, or just seeing this now?
>> >>
>> >> -Taylor
>> >>
>> >>> On Feb 7, 2018, at 12:40 PM, Alexandre Vermeerbergen <
>> avermeerber...@gmail.com> wrote:
>> >>>
>> >>> Hello All,
>> >>>
>> >>> I hate to be the one who always give bad news, but as a matter of
>> >>> facts, Storm 1.2.0 RC3 installation from binary artifacts (both
>> >>> apache-storm-1.2.0-src.tar.gz and apache-storm-1.2.0.zip) leads to "by
>> >>> default KO Kafka monitor" in Nimbus UI (which dirty exceptions in
>> >>> ui.log)
>> >>>
>> >>> Here's for example what I get from apache-storm-1.2.0-src.tar.gz
>> >>> downloaded from
>> >>>
>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc3/apache-storm-1.2.0-src.tar.gz
>> :
>> >>>
>> >>> $ tar ztvf apache-storm-1.2.0.tar.gz apache-storm-1.2.0/toollib
>> >>> -rwxrwxrwx ptgoetz/staff 16999 2018-02-06 21:22
>> >>> apache-storm-1.2.0/toollib/storm-kafka-monitor-1.2.0-sources.jar
>> >>> -rwxrwxrwx ptgoetz/staff 93461 2018-02-06 21:22
>> >>> apache-storm-1.2.0/toollib/storm-kafka-monitor-1.2.0-javadoc.jar
>> >>> -rwxrwxrwx ptgoetz/staff 21591320 2018-02-06 21:22
>> >>> apache-storm-1.2.0/toollib/storm-kafka-monitor-1.2.0.jar
>> >>>
>> >>> And here's what I see in ui.log:
>> >>>
>> >>> org.apache.storm.kafka.spout.KafkaSpout
>> >>> 2018-02-07 16:49:57.153 o.a.s.u.TopologySpoutLag qtp1997623038-18
>> >>> [WARN] Exception message:Error: Could not find or load main class
>> >>>
>> .usr.local.Storm.storm-stable.toollib.storm-kafka-monitor-1.2.0-javadoc.jar
>> >>>
>> >>> org.apache.storm.utils.ShellUtils$ExitCodeException: Error: Could not
>> >>> find or load main class
>> >>>
>> .usr.local.Storm.storm-stable.toollib.storm-kafka-monitor-1.2.0-javadoc.jar
>> >>>
>> >>>   at
>> org.apache.storm.utils.ShellUtils.runCommand(ShellUtils.java:231)
>> >>> ~[storm-core-1.2.0.jar:1.2.0]
>> >>>   at org.apache.storm.utils.ShellUtils.run(ShellUtils.java:161)
>> >>> ~[storm-core-1.2.0.jar:1.2.0]
>> >>>   at
>> org.apache.storm.utils.ShellUtils$ShellCommandExecutor.execute(ShellUtils.java:371)
>> >>> ~[storm-core-1.2.0.jar:1.2.0]
>> >>>   at
>> org.apache.storm.utils.ShellUtils.execCommand(ShellUtils.java:461)
>> >>> ~[storm-core-1.2.0.jar:1.2.0]
>> >>>   at
>> org.apache.storm.utils.ShellUtils.execCommand(ShellUtils.java:444)
>> >>> ~[storm-core-1

Re: [VOTE] Release Apache Storm 1.2.0 (rc3)

[Jungtaek Lim]

Thanks for addressing md5/sha issue, Taylor. Verified src.tar.gz and
src.zip.

Alexandre, thanks again for thoughtful verification of release. It much
helped for us and I really appreciate it. Please keep up the good work.

I feel what Alexandre reported is a blocker since it leads strange error in
point of user side, worth to cancel current RC. Changing my vote to -1
(binding).

-Jungtaek Lim (HeartSaVioR)

2018년 2월 8일 (목) 오전 5:20, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:

> Yes. It will be fixed. Thanks again for pointing it out.
>
> -Taylor
>
> > On Feb 7, 2018, at 2:57 PM, Alexandre Vermeerbergen <
> avermeerber...@gmail.com> wrote:
> >
> > Hello Taylor,
> >
> > Yes same issue in Storm 1.1.0 : we have a production storm cluster
> > based on this release, but then we were using our own Kafka spout, no
> > we weren't impacted by this packaging issue.
> >
> > Could the extraneous files be cleaned up from 1.2.0-final?
> >
> > Best regards,
> > Alexandre Vermeerbergen
> >
> > 2018-02-07 19:18 GMT+01:00 P. Taylor Goetz <ptgo...@gmail.com>:
> >> Hi Alexandre,
> >>
> >> Thanks for the review. You’re right, the javadoc/source jars should not
> be there, though it looks like this has been the case for a long time and
> would have affected previous releases.
> >>
> >> It looks like the problem was introduced in 1.1.0. Have you seen the
> same issue in other 1.1.x releases, or just seeing this now?
> >>
> >> -Taylor
> >>
> >>> On Feb 7, 2018, at 12:40 PM, Alexandre Vermeerbergen <
> avermeerber...@gmail.com> wrote:
> >>>
> >>> Hello All,
> >>>
> >>> I hate to be the one who always give bad news, but as a matter of
> >>> facts, Storm 1.2.0 RC3 installation from binary artifacts (both
> >>> apache-storm-1.2.0-src.tar.gz and apache-storm-1.2.0.zip) leads to "by
> >>> default KO Kafka monitor" in Nimbus UI (which dirty exceptions in
> >>> ui.log)
> >>>
> >>> Here's for example what I get from apache-storm-1.2.0-src.tar.gz
> >>> downloaded from
> >>>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc3/apache-storm-1.2.0-src.tar.gz
> :
> >>>
> >>> $ tar ztvf apache-storm-1.2.0.tar.gz apache-storm-1.2.0/toollib
> >>> -rwxrwxrwx ptgoetz/staff 16999 2018-02-06 21:22
> >>> apache-storm-1.2.0/toollib/storm-kafka-monitor-1.2.0-sources.jar
> >>> -rwxrwxrwx ptgoetz/staff 93461 2018-02-06 21:22
> >>> apache-storm-1.2.0/toollib/storm-kafka-monitor-1.2.0-javadoc.jar
> >>> -rwxrwxrwx ptgoetz/staff 21591320 2018-02-06 21:22
> >>> apache-storm-1.2.0/toollib/storm-kafka-monitor-1.2.0.jar
> >>>
> >>> And here's what I see in ui.log:
> >>>
> >>> org.apache.storm.kafka.spout.KafkaSpout
> >>> 2018-02-07 16:49:57.153 o.a.s.u.TopologySpoutLag qtp1997623038-18
> >>> [WARN] Exception message:Error: Could not find or load main class
> >>>
> .usr.local.Storm.storm-stable.toollib.storm-kafka-monitor-1.2.0-javadoc.jar
> >>>
> >>> org.apache.storm.utils.ShellUtils$ExitCodeException: Error: Could not
> >>> find or load main class
> >>>
> .usr.local.Storm.storm-stable.toollib.storm-kafka-monitor-1.2.0-javadoc.jar
> >>>
> >>>   at
> org.apache.storm.utils.ShellUtils.runCommand(ShellUtils.java:231)
> >>> ~[storm-core-1.2.0.jar:1.2.0]
> >>>   at org.apache.storm.utils.ShellUtils.run(ShellUtils.java:161)
> >>> ~[storm-core-1.2.0.jar:1.2.0]
> >>>   at
> org.apache.storm.utils.ShellUtils$ShellCommandExecutor.execute(ShellUtils.java:371)
> >>> ~[storm-core-1.2.0.jar:1.2.0]
> >>>   at
> org.apache.storm.utils.ShellUtils.execCommand(ShellUtils.java:461)
> >>> ~[storm-core-1.2.0.jar:1.2.0]
> >>>   at
> org.apache.storm.utils.ShellUtils.execCommand(ShellUtils.java:444)
> >>> ~[storm-core-1.2.0.jar:1.2.0]
> >>>   at
> org.apache.storm.utils.TopologySpoutLag.getLagResultForKafka(TopologySpoutLag.java:163)
> >>> ~[storm-core-1.2.0.jar:1.2.0]
> >>>   at
> org.apache.storm.utils.TopologySpoutLag.getLagResultForNewKafkaSpout(TopologySpoutLag.java:189)
> >>> ~[storm-core-1.2.0.jar:1.2.0]
> >>>   at
> org.apache.storm.utils.TopologySpoutLag.lag(TopologySpoutLag.java:57)
> >>> ~[storm-core-1.2.0.jar:1.2.0]
> >>>   at org.apache.storm.ui.core$topology_lag.invoke(core.clj:805)
> 

Re: [VOTE] Release Apache Storm 1.2.0 (rc3)

[Jungtaek Lim]

CORRECTION: didn't check md5/sha from source targz/zip since the files are
not presented. Will check again once they are available.

2018년 2월 7일 (수) 오후 10:49, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> +1  (binding)
>
> > source
>
> - verify file (signature, MD5, SHA)
> -- source, tar.gz : OK
> -- source, zip : OK
>
> - extract file
> -- source, tar.gz : OK
> -- source, zip : OK
>
> - diff-ing extracted files between tar.gz and zip : OK
>
> - build source with JDK 7
> -- source, tar.gz : OK
>
> - build source dist
> -- source, tar.gz : OK
>
> - build binary dist
> -- source, tar.gz : OK
>
> > binary
>
> - verify file (signature, MD5, SHA)
> -- binary, tar.gz : OK
> -- binary, zip : OK
>
> - extract file
> -- binary, tar.gz : OK
> -- binary, zip : OK
>
> - diff-ing extracted files between tar.gz and zip : OK
>
> - launch daemons : OK
>
> - run RollingTopWords (local) : OK
>
> - run RollingTopWords (remote) : OK
>   - activate / deactivate / rebalance / kill : OK
>   - logviewer (worker dir, daemon dir) : OK
>   - change log level : OK
>   - thread dump, heap dump, restart worker : OK
>   - log search : OK
>
> Both STORM-2853 [1] and STORM-2912 [2] are confirmed to be resolved in
> verification phase.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 1. https://issues.apache.org/jira/browse/STORM-2853
> 2. https://issues.apache.org/jira/browse/STORM-2912
>
>
> 2018년 2월 7일 (수) 오후 9:39, Jungtaek Lim <kabh...@gmail.com>님이 작성:
>
>> Looks like missing md5/sha for apache-storm-1.2.0-src.tar.gz as well as
>> apache-storm-1.2.0-src.zip.
>>
>> I'll continue verifying without that.
>>
>> 2018년 2월 7일 (수) 오전 6:01, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>>
>>> This is a call to vote on releasing Apache Storm 1.2.0 (rc3)
>>>
>>> Full list of changes in this release:
>>>
>>>
>>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc3/RELEASE_NOTES.html
>>>
>>> The tag/commit to be voted upon is v1.2.0:
>>>
>>>
>>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=7be1308bfea0d71efac141888cdcfc9e708cfff5;hb=b20228de004dda52e414a9ade6e60ff0756df9d1
>>>
>>> The source archive being voted upon can be found here:
>>>
>>>
>>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc3/apache-storm-1.2.0-src.tar.gz
>>>
>>> Other release files, signatures and digests can be found here:
>>>
>>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc3/
>>>
>>> The release artifacts are signed with the following key:
>>>
>>>
>>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>>>
>>> The Nexus staging repository for this release is:
>>>
>>> https://repository.apache.org/content/repositories/orgapachestorm-1057
>>>
>>> Please vote on releasing this package as Apache Storm 1.2.0.
>>>
>>> When voting, please list the actions taken to verify the release.
>>>
>>> This vote will be open for at least 72 hours.
>>>
>>> [ ] +1 Release this package as Apache Storm 1.2.0
>>> [ ]  0 No opinion
>>> [ ] -1 Do not release this package because...
>>>
>>> Thanks to everyone who contributed to this release.
>>>
>>> -Taylor
>>>
>>

Re: [VOTE] Release Apache Storm 1.2.0 (rc3)

[Jungtaek Lim]

+1  (binding)

> source

- verify file (signature, MD5, SHA)
-- source, tar.gz : OK
-- source, zip : OK

- extract file
-- source, tar.gz : OK
-- source, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- build source with JDK 7
-- source, tar.gz : OK

- build source dist
-- source, tar.gz : OK

- build binary dist
-- source, tar.gz : OK

> binary

- verify file (signature, MD5, SHA)
-- binary, tar.gz : OK
-- binary, zip : OK

- extract file
-- binary, tar.gz : OK
-- binary, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- launch daemons : OK

- run RollingTopWords (local) : OK

- run RollingTopWords (remote) : OK
  - activate / deactivate / rebalance / kill : OK
  - logviewer (worker dir, daemon dir) : OK
  - change log level : OK
  - thread dump, heap dump, restart worker : OK
  - log search : OK

Both STORM-2853 [1] and STORM-2912 [2] are confirmed to be resolved in
verification phase.

Thanks,
Jungtaek Lim (HeartSaVioR)

1. https://issues.apache.org/jira/browse/STORM-2853
2. https://issues.apache.org/jira/browse/STORM-2912


2018년 2월 7일 (수) 오후 9:39, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> Looks like missing md5/sha for apache-storm-1.2.0-src.tar.gz as well as
> apache-storm-1.2.0-src.zip.
>
> I'll continue verifying without that.
>
> 2018년 2월 7일 (수) 오전 6:01, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>
>> This is a call to vote on releasing Apache Storm 1.2.0 (rc3)
>>
>> Full list of changes in this release:
>>
>>
>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc3/RELEASE_NOTES.html
>>
>> The tag/commit to be voted upon is v1.2.0:
>>
>>
>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=7be1308bfea0d71efac141888cdcfc9e708cfff5;hb=b20228de004dda52e414a9ade6e60ff0756df9d1
>>
>> The source archive being voted upon can be found here:
>>
>>
>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc3/apache-storm-1.2.0-src.tar.gz
>>
>> Other release files, signatures and digests can be found here:
>>
>> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc3/
>>
>> The release artifacts are signed with the following key:
>>
>>
>> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>>
>> The Nexus staging repository for this release is:
>>
>> https://repository.apache.org/content/repositories/orgapachestorm-1057
>>
>> Please vote on releasing this package as Apache Storm 1.2.0.
>>
>> When voting, please list the actions taken to verify the release.
>>
>> This vote will be open for at least 72 hours.
>>
>> [ ] +1 Release this package as Apache Storm 1.2.0
>> [ ]  0 No opinion
>> [ ] -1 Do not release this package because...
>>
>> Thanks to everyone who contributed to this release.
>>
>> -Taylor
>>
>

Re: [VOTE] Release Apache Storm 1.2.0 (rc3)

[Jungtaek Lim]

Looks like missing md5/sha for apache-storm-1.2.0-src.tar.gz as well as
apache-storm-1.2.0-src.zip.

I'll continue verifying without that.

2018년 2월 7일 (수) 오전 6:01, P. Taylor Goetz 님이 작성:

> This is a call to vote on releasing Apache Storm 1.2.0 (rc3)
>
> Full list of changes in this release:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc3/RELEASE_NOTES.html
>
> The tag/commit to be voted upon is v1.2.0:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=7be1308bfea0d71efac141888cdcfc9e708cfff5;hb=b20228de004dda52e414a9ade6e60ff0756df9d1
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc3/apache-storm-1.2.0-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.2.0-rc3/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1057
>
> Please vote on releasing this package as Apache Storm 1.2.0.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 1.2.0
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
>

Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch / 1.0.x-branch with 1.x-branch

[Jungtaek Lim]

UPDATE: Submitted a pull request https://github.com/apache/storm/pull/2549 for
STORM-2936 (against 1.1.x-branch)

Erik, please change the status to "IN PROGRESS" if someone is working on. I
would find the free time and just do it if there's no one working in
progress.

2018년 2월 6일 (화) 오전 10:39, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> Thanks for quick response Erik!
>
> Just filed two issues :
> https://issues.apache.org/jira/browse/STORM-2936 (for 1.1.x-branch)
> https://issues.apache.org/jira/browse/STORM-2937 (for 1.0.x-branch)
>
> We have another discussion around making storm-kafka-client be experiment
> of managing separately (independent of Storm release). So the three
> versions which are in release phase might be the last releases of
> "battery-included" of storm-kafka-client if our experiment works well.
>
> If we would want to make the change for storm-kafka-client, it might be
> better to put the change and release before start experimenting, but that's
> just a thought on my own. In opposite way, we could even start experiment
> and make change of storm-core of 1.0.x-branch to be compatible with that
> version of storm-kafka-client. We could even do it for 1.1.x-branch, but
> the change is almost done so it doesn't look like needed to postpone it.
>
> Would like to here everyone's voice on this.
>
> -Jungtaek Lim (HeartSaVioR)
>
> 2018년 2월 6일 (화) 오전 10:23, Erik Weathers <eweath...@groupon.com.invalid>님이
> 작성:
>
>> Thanks for the quick response Jungtaek!
>>
>> Yes, my teammates and myself would like to help on this.  Is there an
>> existing JIRA for the work you've been doing on the other branches?
>>
>> I propose we don't make this block 1.0.6 -- we can just release 1.0.7
>> quickly when the backport is done, if that is amenable.
>> That strategy also might be cleaner since it would avoid other changes in
>> 1.0.6 being lumped together with this.
>>
>> - Erik
>>
>> On Mon, Feb 5, 2018 at 5:16 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
>>
>> > UPDATE: I've finished working on overwriting storm-kafka-client
>> 1.x-branch
>> > to 1.1.x-branch. Not yet pushed to ASF git, but pushed to my fork first
>> to
>> > trigger Travis CI to see how the build goes well.
>> >
>> > https://github.com/HeartSaVioR/storm/commit/76b8a7d3a6f91e66
>> > 612e87da8589f5723f05218a
>> > https://travis-ci.org/HeartSaVioR/storm/builds/337819430
>> >
>> > Thanks for the input regarding 1.0.x version, Erik. I guess then we
>> have no
>> > alternative here: someone has to fix storm-kafka-client as well as
>> > storm-core, since including shaded storm-core doesn't make sense for
>> > official Storm release.
>> >
>> > I guess it doesn't take many hour(s), hence may not worth to sync and
>> talk
>> > offline. I just wanted to judge whether we are OK to make change of
>> > storm-core in bugfix version lines, but maybe the judgement itself can
>> be
>> > possible after finishing the change, so I'll just go ahead making the
>> > change.
>> > Since this is blocking release candidate, we should get it ASAP. That's
>> why
>> > I'm eager to go ahead making the change. If you could spend time now
>> > helping with making the change ASAP, please leave short notice (maybe
>> with
>> > JIRA issue?) and go ahead.
>> >
>> > Thanks,
>> > Jungtaek Lim (HeartSaVioR)
>> >
>> > 2018년 2월 6일 (화) 오전 9:41, Erik Weathers <eweath...@groupon.com.invalid
>> >님이
>> > 작성:
>> >
>> > > hey Jungtaek,
>> > >
>> > > Thanks for continuing to pursue this!
>> > >
>> > > The issue for Storm not working on Mesos is due to a fundamental
>> change
>> > to
>> > > the core scheduling logic in Storm:
>> > >
>> > >-
>> > >
>> > > https://issues.apache.org/jira/browse/STORM-2126?focusedComm
>> > entId=16136150=com.atlassian.jira.plugin.system.
>> > issuetabpanels%3Acomment-tabpanel#comment-16136150
>> > >
>> > > The yet-to-be-ironed-out solution that Bobby was brainstorming about
>> > isn't
>> > > a short term fix as far as I understand it.  I believe it to be many
>> many
>> > > months (years?) out for it to actually be workable.  Per my naive
>> > > understanding of the proposal, we'd probably have to completely
>> rewrite
>> > the
>> > > Storm-on-Mesos framework.  So it's probably the right long-term
&g

Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch / 1.0.x-branch with 1.x-branch

[Jungtaek Lim]

Thanks for quick response Erik!

Just filed two issues :
https://issues.apache.org/jira/browse/STORM-2936 (for 1.1.x-branch)
https://issues.apache.org/jira/browse/STORM-2937 (for 1.0.x-branch)

We have another discussion around making storm-kafka-client be experiment
of managing separately (independent of Storm release). So the three
versions which are in release phase might be the last releases of
"battery-included" of storm-kafka-client if our experiment works well.

If we would want to make the change for storm-kafka-client, it might be
better to put the change and release before start experimenting, but that's
just a thought on my own. In opposite way, we could even start experiment
and make change of storm-core of 1.0.x-branch to be compatible with that
version of storm-kafka-client. We could even do it for 1.1.x-branch, but
the change is almost done so it doesn't look like needed to postpone it.

Would like to here everyone's voice on this.

-Jungtaek Lim (HeartSaVioR)

2018년 2월 6일 (화) 오전 10:23, Erik Weathers <eweath...@groupon.com.invalid>님이
작성:

> Thanks for the quick response Jungtaek!
>
> Yes, my teammates and myself would like to help on this.  Is there an
> existing JIRA for the work you've been doing on the other branches?
>
> I propose we don't make this block 1.0.6 -- we can just release 1.0.7
> quickly when the backport is done, if that is amenable.
> That strategy also might be cleaner since it would avoid other changes in
> 1.0.6 being lumped together with this.
>
> - Erik
>
> On Mon, Feb 5, 2018 at 5:16 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
>
> > UPDATE: I've finished working on overwriting storm-kafka-client
> 1.x-branch
> > to 1.1.x-branch. Not yet pushed to ASF git, but pushed to my fork first
> to
> > trigger Travis CI to see how the build goes well.
> >
> > https://github.com/HeartSaVioR/storm/commit/76b8a7d3a6f91e66
> > 612e87da8589f5723f05218a
> > https://travis-ci.org/HeartSaVioR/storm/builds/337819430
> >
> > Thanks for the input regarding 1.0.x version, Erik. I guess then we have
> no
> > alternative here: someone has to fix storm-kafka-client as well as
> > storm-core, since including shaded storm-core doesn't make sense for
> > official Storm release.
> >
> > I guess it doesn't take many hour(s), hence may not worth to sync and
> talk
> > offline. I just wanted to judge whether we are OK to make change of
> > storm-core in bugfix version lines, but maybe the judgement itself can be
> > possible after finishing the change, so I'll just go ahead making the
> > change.
> > Since this is blocking release candidate, we should get it ASAP. That's
> why
> > I'm eager to go ahead making the change. If you could spend time now
> > helping with making the change ASAP, please leave short notice (maybe
> with
> > JIRA issue?) and go ahead.
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> > 2018년 2월 6일 (화) 오전 9:41, Erik Weathers <eweath...@groupon.com.invalid>님이
> > 작성:
> >
> > > hey Jungtaek,
> > >
> > > Thanks for continuing to pursue this!
> > >
> > > The issue for Storm not working on Mesos is due to a fundamental change
> > to
> > > the core scheduling logic in Storm:
> > >
> > >-
> > >
> > > https://issues.apache.org/jira/browse/STORM-2126?focusedComm
> > entId=16136150=com.atlassian.jira.plugin.system.
> > issuetabpanels%3Acomment-tabpanel#comment-16136150
> > >
> > > The yet-to-be-ironed-out solution that Bobby was brainstorming about
> > isn't
> > > a short term fix as far as I understand it.  I believe it to be many
> many
> > > months (years?) out for it to actually be workable.  Per my naive
> > > understanding of the proposal, we'd probably have to completely rewrite
> > the
> > > Storm-on-Mesos framework.  So it's probably the right long-term
> solution,
> > > but it isn't anything that should impact this discussion.
> > >
> > > > The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and
> > include
> > > it into their topology jar, it will also not work with Storm 1.0.x. It
> > > even can't
> > > compile.
> > >
> > > FWIW, I'm pretty sure that I was able to successfully run
> > > storm-kafka-client-1.1.x on a 1.0.5 storm cluster, but only after
> shading
> > > in storm-core-1.1.x to the topology uber jar.   There was *at least* a
> > > change to some timer-related class in storm-core in 1.1.x (something
> > about
> > > milliseconds IIRC -- it's been 1.5 months since I did it, need to
> revis

Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch / 1.0.x-branch with 1.x-branch

[Jungtaek Lim]

UPDATE: I've finished working on overwriting storm-kafka-client 1.x-branch
to 1.1.x-branch. Not yet pushed to ASF git, but pushed to my fork first to
trigger Travis CI to see how the build goes well.

https://github.com/HeartSaVioR/storm/commit/76b8a7d3a6f91e66612e87da8589f5723f05218a
https://travis-ci.org/HeartSaVioR/storm/builds/337819430

Thanks for the input regarding 1.0.x version, Erik. I guess then we have no
alternative here: someone has to fix storm-kafka-client as well as
storm-core, since including shaded storm-core doesn't make sense for
official Storm release.

I guess it doesn't take many hour(s), hence may not worth to sync and talk
offline. I just wanted to judge whether we are OK to make change of
storm-core in bugfix version lines, but maybe the judgement itself can be
possible after finishing the change, so I'll just go ahead making the
change.
Since this is blocking release candidate, we should get it ASAP. That's why
I'm eager to go ahead making the change. If you could spend time now
helping with making the change ASAP, please leave short notice (maybe with
JIRA issue?) and go ahead.

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 2월 6일 (화) 오전 9:41, Erik Weathers <eweath...@groupon.com.invalid>님이 작성:

> hey Jungtaek,
>
> Thanks for continuing to pursue this!
>
> The issue for Storm not working on Mesos is due to a fundamental change to
> the core scheduling logic in Storm:
>
>-
>
> https://issues.apache.org/jira/browse/STORM-2126?focusedCommentId=16136150=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16136150
>
> The yet-to-be-ironed-out solution that Bobby was brainstorming about isn't
> a short term fix as far as I understand it.  I believe it to be many many
> months (years?) out for it to actually be workable.  Per my naive
> understanding of the proposal, we'd probably have to completely rewrite the
> Storm-on-Mesos framework.  So it's probably the right long-term solution,
> but it isn't anything that should impact this discussion.
>
> > The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and include
> it into their topology jar, it will also not work with Storm 1.0.x. It
> even can't
> compile.
>
> FWIW, I'm pretty sure that I was able to successfully run
> storm-kafka-client-1.1.x on a 1.0.5 storm cluster, but only after shading
> in storm-core-1.1.x to the topology uber jar.   There was *at least* a
> change to some timer-related class in storm-core in 1.1.x (something about
> milliseconds IIRC -- it's been 1.5 months since I did it, need to revisit
> the process I followed).
>
> I'm happy to help with backporting / stomping storm-kafka-client in 1.0.x.
> Maybe we can talk offline about it?
>
> - Erik
>
> On Mon, Feb 5, 2018 at 4:20 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
>
> > UPDATE: Looks like we changed some parts of storm-core while fixing
> > storm-kafka-client issues (especially went in 1.1.0), hence overwriting
> > also incurs changes of storm-core. It doesn't look like a big deal for
> > 1.1.x-branch, but there looks like needed many changes for 1.0.x-branch.
> >
> > The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and include
> it
> > into their topology jar, it will also not work with Storm 1.0.x. It even
> > can't compile.
> >
> > 1.0.x version line was long lived (22 months) even we released Storm
> 1.1.0
> > at 11 months ago. Instead of struggling 1.0.x-branch to up to date, I'd
> > like to suggest that we define 1.0.x-branch as deprecated with guiding to
> > update to latest 1.1.x version or 1.2.0 (after release), and try to
> resolve
> > storm-mesos issue with Storm 1.1.0 ASAP to resolve Erik's concern.
> >
> > Makes sense? I'll continue working on 1.1.x-branch and update anyway.
> >
> > -Jungtaek Lim (HeartSaVioR)
> >
> > 2018년 2월 6일 (화) 오전 7:53, Jungtaek Lim <kabh...@gmail.com>님이 작성:
> >
> > > OK. No more opinion/vote in 5 days. I'll treat consensus was made, and
> go
> > > ahead making change: overwrite storm-kafka-client 1.2.0 to two branches
> > > 1.1.x/1.0.x.
> > >
> > > -Jungtaek Lim (HeartSaVioR)
> > >
> > > 2018년 2월 1일 (목) 오전 10:48, Jungtaek Lim <kabh...@gmail.com>님이 작성:
> > >
> > >> This discussion got 4 +1 (binding) and no -1. Moreover two active
> > >> maintainers for storm-kafka-client (Hugo and Stig) voted +1.
> > >>
> > >> Do we want to hold on for hearing more voices, or treating above
> > opinions
> > >> as consensus and reflect the change?
> > >>
> > >> Btw, I think we need to sort out the sequences between two topics:
> > >> separ

Re: [CANCELED] [VOTE] Release Apache Storm 1.2.0 (rc2)

[Jungtaek Lim]

I'm working on copying storm-kafka-client, but as I noticed to other thread
[1], copying incurs change of storm-core. While 1.1.x-branch requires small
change, but 1.0.x-branch requires non-trivial change and I have to track
the changes related to storm-kafka-client.

So I've suggested the alternative: deprecation of 1.0.x-branch in other
thread [1]. Please voice your opinion regarding this.

Thanks,
Jungtaek Lim (HeartSaVIoR)

1.
https://lists.apache.org/thread.html/3a1ac1f1c396723d9ff9effaf8ee839e0ada5df3a88bee9df0d4602c@%3Cdev.storm.apache.org%3E

2018년 2월 6일 (화) 오전 7:02, Stig Rohde Døssing <stigdoess...@gmail.com>님이 작성:

> It's been merged, thanks for the quick review. I think storm-kafka-client
> should be ready to be copied to the other two branches.
>
> 2018-02-05 22:00 GMT+01:00 P. Taylor Goetz <ptgo...@gmail.com>:
>
> > Yes,  here’s the pull request:
> >
> > https://github.com/apache/storm/pull/2547
> >
> > Once that’s merged, I can sync storm-kafka-client from 1.x-branch to
> > 1.1.x-branch and 1.0.x-branch and cut new RCs.
> >
> > -Taylor
> >
> > On Feb 5, 2018, at 1:27 PM, Arun Mahadevan <ar...@apache.org> wrote:
> >
> > STORM-2918 has been merged to 1.x branch.
> >
> >
> > Now looks like we waiting for 1.x versions of
> >
> > https://github.com/apache/storm/pull/2538 and
> > https://github.com/apache/storm/pull/2537 ?
> >
> > Can we get the next RC as soon as the above two are merged to 1.x ?
> >
> > Thanks,
> > Arun
> >
> >
> >
> > On 2/1/18, 12:55 AM, "dbis...@gmail.com on behalf of Artem Ervits" <
> > dbis...@gmail.com on behalf of artemerv...@gmail.com> wrote:
> >
> > -1
> > Please include https://issues.apache.org/jira/browse/STORM-2918
> >
> > On Jan 31, 2018 1:59 PM, "Stig Rohde Døssing" <stigdoess...@gmail.com>
> > wrote:
> >
> > The log indicates a bug. We can remove the WARN messages pretty easily,
> but
> > we'd still be throwing and catching an exception for each processed
> record.
> >
> > We're storing some data in Kafka alongside committed offsets that lets us
> > only apply the EARLIEST and LATEST strategies for where the consumer
> should
> > start if the topology is redeployed, rather than applying them every time
> > the worker restarts. This was added as a fix to a bug I introduced in
> > https://issues.apache.org/jira/browse/STORM-2666, where we error if the
> > spout tries to emit offsets that have already been committed (that never
> > happens during normal operation, but if the spout is restarted and using
> > EARLIEST it can happen). The new behavior of EARLIEST/LATEST is also much
> > more useful than the old one IMO.
> >
> > The bug is that we're only storing the metadata if the spout is
> configured
> > for at-least-once. We also support at-most-once and an "anything goes"
> > setting, and when either of those are used, the spout tries to read the
> > metadata that isn't there and complains about it. If we just remove the
> > log, EARLIEST and LATEST will behave differently for at-least-once and
> > at-most-once/no-guarantee.
> >
> > The patch makes changes to ensure that we store metadata in all cases.
> >
> > 2018-01-31 19:47 GMT+01:00 Arun Mahadevan <ar...@apache.org>:
> >
> > Are we waiting for https://github.com/apache/storm/pull/2538 to start
> >
> > the
> >
> > next RC ?
> >
> > This patch seem to contain more changes than just fixing a logging issue.
> > Can we only address the concern raised about logs filling with WARN
> > messages and address the rest in the next release or does the logs
> >
> > indicate
> >
> > some bug which is a blocker for 1.2.0 ?
> >
> > - Arun
> >
> >
> >
> >
> > On 1/29/18, 12:30 PM, "P. Taylor Goetz" <ptgo...@gmail.com> wrote:
> >
> > Cancelling this RC in order to address Stig’s concerns.
> >
> > -Taylor
> >
> > On Jan 26, 2018, at 4:11 PM, P. Taylor Goetz <ptgo...@gmail.com>
> >
> > wrote:
> >
> >
> > This is a call to vote on releasing Apache Storm 1.2.0 (rc2)
> >
> > Full list of changes in this release:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> >
> > 2.0-rc2/RELEASE_NOTES.html
> >
> >
> > The tag/commit to be voted upon is v1.2.0:
> >
> > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=
> >
> > 17a4645d7d65f5a7a08a50b5185c0fc52e82692f;hb=
> >
> > 458aa

Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch / 1.0.x-branch with 1.x-branch

[Jungtaek Lim]

UPDATE: Looks like we changed some parts of storm-core while fixing
storm-kafka-client issues (especially went in 1.1.0), hence overwriting
also incurs changes of storm-core. It doesn't look like a big deal for
1.1.x-branch, but there looks like needed many changes for 1.0.x-branch.

The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and include it
into their topology jar, it will also not work with Storm 1.0.x. It even
can't compile.

1.0.x version line was long lived (22 months) even we released Storm 1.1.0
at 11 months ago. Instead of struggling 1.0.x-branch to up to date, I'd
like to suggest that we define 1.0.x-branch as deprecated with guiding to
update to latest 1.1.x version or 1.2.0 (after release), and try to resolve
storm-mesos issue with Storm 1.1.0 ASAP to resolve Erik's concern.

Makes sense? I'll continue working on 1.1.x-branch and update anyway.

-Jungtaek Lim (HeartSaVioR)

2018년 2월 6일 (화) 오전 7:53, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> OK. No more opinion/vote in 5 days. I'll treat consensus was made, and go
> ahead making change: overwrite storm-kafka-client 1.2.0 to two branches
> 1.1.x/1.0.x.
>
> -Jungtaek Lim (HeartSaVioR)
>
> 2018년 2월 1일 (목) 오전 10:48, Jungtaek Lim <kabh...@gmail.com>님이 작성:
>
>> This discussion got 4 +1 (binding) and no -1. Moreover two active
>> maintainers for storm-kafka-client (Hugo and Stig) voted +1.
>>
>> Do we want to hold on for hearing more voices, or treating above opinions
>> as consensus and reflect the change?
>>
>> Btw, I think we need to sort out the sequences between two topics:
>> separating storm-kafka-client as independent release cycle, and this. I
>> guess some of us agreed former topic doesn't related to current RC, but I
>> think this topic can be (should be) reflected to current RC ongoing.
>>
>> -Jungtaek Lim (HeartSaVioR)
>>
>> 2018년 2월 1일 (목) 오전 4:08, Hugo Da Cruz Louro <hlo...@hortonworks.com>님이
>> 작성:
>>
>>> +1 to replace storm-kafka-client in 1.0.x branch.
>>> Hugo
>>>
>>> > On Jan 31, 2018, at 11:03 AM, Stig Rohde Døssing <
>>> stigdoess...@gmail.com> wrote:
>>> >
>>> > +1 to replace storm-kafka-client in 1.0.x branch. Breaking semantic
>>> > versioning is really nasty, but I think it is the lesser evil in this
>>> case.
>>> >
>>> > 2018-01-31 5:14 GMT+01:00 Harsha <st...@harsha.io>:
>>> >
>>> >> +1 to replace storm-kafka-client in 1.0.x branch
>>> >> -Harsha
>>> >> On Tue, Jan 30, 2018, at 7:04 PM, Jungtaek Lim wrote:
>>> >>> Bump up this thread so that we could reach consensus earlier. Given
>>> that
>>> >> we
>>> >>> got concern related to this, I think it is ideal to release
>>> 1.1.x/1.0.x
>>> >>> with making decision and applying the change if we want.
>>> >>>
>>> >>> 2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <kabh...@gmail.com>님이 작성:
>>> >>>
>>> >>>> Erik's concern brought from 1.0.6 RC1, because they can't use Storm
>>> >> 1.1.0
>>> >>>> or higher (Storm 1.1.0 broke storm-mesos.). While he could take an
>>> >>>> workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we decide to
>>> >>>> replace) with Storm 1.0.6, it would be better if we don't allow
>>> leaving
>>> >>>> storm-kafka-client in 1.0.x in inconsistent state.
>>> >>>>
>>> >>>> IMHO, breaking backward compatibility is worse, but leaving broken
>>> >> thing
>>> >>>> is worst. Hence I'm +1 to replace all, with noticing that it may
>>> bring
>>> >>>> backward incompatibility in release announce.
>>> >>>>
>>> >>>> -Jungtaek Lim (HeartSaVioR)
>>> >>>>
>>> >>>> 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>>> >>>>
>>> >>>>> As I mentioned else thread I’m open to this but would defer to
>>> >> community
>>> >>>>> consensus.
>>> >>>>>
>>> >>>>> If there’s concern about doing this for 1.0.x, one option would be
>>> >> skip
>>> >>>>> that version line and only apply it to 1.2.0 and 1.1.x.
>>> >>>>>
>>> >>>>> -Taylor
>>> >>>>>
>>> >>>>>> On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <kabh...@g

Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch / 1.0.x-branch with 1.x-branch

[Jungtaek Lim]

OK. No more opinion/vote in 5 days. I'll treat consensus was made, and go
ahead making change: overwrite storm-kafka-client 1.2.0 to two branches
1.1.x/1.0.x.

-Jungtaek Lim (HeartSaVioR)

2018년 2월 1일 (목) 오전 10:48, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> This discussion got 4 +1 (binding) and no -1. Moreover two active
> maintainers for storm-kafka-client (Hugo and Stig) voted +1.
>
> Do we want to hold on for hearing more voices, or treating above opinions
> as consensus and reflect the change?
>
> Btw, I think we need to sort out the sequences between two topics:
> separating storm-kafka-client as independent release cycle, and this. I
> guess some of us agreed former topic doesn't related to current RC, but I
> think this topic can be (should be) reflected to current RC ongoing.
>
> -Jungtaek Lim (HeartSaVioR)
>
> 2018년 2월 1일 (목) 오전 4:08, Hugo Da Cruz Louro <hlo...@hortonworks.com>님이 작성:
>
>> +1 to replace storm-kafka-client in 1.0.x branch.
>> Hugo
>>
>> > On Jan 31, 2018, at 11:03 AM, Stig Rohde Døssing <
>> stigdoess...@gmail.com> wrote:
>> >
>> > +1 to replace storm-kafka-client in 1.0.x branch. Breaking semantic
>> > versioning is really nasty, but I think it is the lesser evil in this
>> case.
>> >
>> > 2018-01-31 5:14 GMT+01:00 Harsha <st...@harsha.io>:
>> >
>> >> +1 to replace storm-kafka-client in 1.0.x branch
>> >> -Harsha
>> >> On Tue, Jan 30, 2018, at 7:04 PM, Jungtaek Lim wrote:
>> >>> Bump up this thread so that we could reach consensus earlier. Given
>> that
>> >> we
>> >>> got concern related to this, I think it is ideal to release
>> 1.1.x/1.0.x
>> >>> with making decision and applying the change if we want.
>> >>>
>> >>> 2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <kabh...@gmail.com>님이 작성:
>> >>>
>> >>>> Erik's concern brought from 1.0.6 RC1, because they can't use Storm
>> >> 1.1.0
>> >>>> or higher (Storm 1.1.0 broke storm-mesos.). While he could take an
>> >>>> workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we decide to
>> >>>> replace) with Storm 1.0.6, it would be better if we don't allow
>> leaving
>> >>>> storm-kafka-client in 1.0.x in inconsistent state.
>> >>>>
>> >>>> IMHO, breaking backward compatibility is worse, but leaving broken
>> >> thing
>> >>>> is worst. Hence I'm +1 to replace all, with noticing that it may
>> bring
>> >>>> backward incompatibility in release announce.
>> >>>>
>> >>>> -Jungtaek Lim (HeartSaVioR)
>> >>>>
>> >>>> 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>> >>>>
>> >>>>> As I mentioned else thread I’m open to this but would defer to
>> >> community
>> >>>>> consensus.
>> >>>>>
>> >>>>> If there’s concern about doing this for 1.0.x, one option would be
>> >> skip
>> >>>>> that version line and only apply it to 1.2.0 and 1.1.x.
>> >>>>>
>> >>>>> -Taylor
>> >>>>>
>> >>>>>> On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <kabh...@gmail.com>
>> >> wrote:
>> >>>>>>
>> >>>>>> Hi devs,
>> >>>>>>
>> >>>>>> This is initial post to separate out discussion topic from vote
>> >> thread,
>> >>>>> and
>> >>>>>> continue discussing.
>> >>>>>>
>> >>>>>> Background of the topic:
>> >>>>>> 1. Only 1.x-branch of storm-kafka-client got stabilized.
>> >> (relatively)
>> >>>>>> 2. We would avoid to port back patches to 1.1.x and 1.0.x because
>> >>>>> they're
>> >>>>>> diverged too much.
>> >>>>>>
>> >>>>>> Downside:
>> >>>>>> Backward compatibility might be broken for 1.1.x and 1.0.x. Not
>> >> sure for
>> >>>>>> 1.1.x, but at least 1.0.x, since supported Kafka client version is
>> >>>>>> different, and if my memory is right, we already applied backward
>> >>>>>> incompatible change into storm-kafka-client 1.1.0.
>> >>>>>>
>> >>>>>> Please put your opinion regarding topic. You're encouraged to copy
>> >> your
>> >>>>>> previous post in vote thread which helps to centralize opinions in
>> >>>>> current
>> >>>>>> thread.
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Jungtaek Lim (HeartSaVioR)
>> >>>>>
>> >>>>>
>> >>
>>
>>

Re: Storm multilang - .net core

[Jungtaek Lim]

CORRECTION: I don't work on multilang for now. "had some works" on
multilang and introduced small changes on multilang protocol.

2018년 2월 2일 (금) 오전 9:12, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> Please remove or bcc. for company mail/mailing list which don't allow
> sending mail outside of MS. My previous mail bounced. I just removed CCC
> from recipient in current mail.
>
> 2018년 2월 2일 (금) 오전 9:08, Jungtaek Lim <kabh...@gmail.com>님이 작성:
>
>> Mauro,
>>
>> First of all, thanks for proposing contribution of valuable effort on
>> Apache Storm! Really appreciated.
>>
>> I don't know about C#/.net but I have been working on the change of
>> multilang, so adding my voice here.
>>
>> 1.
>> Major concern from my side for code donation is sustainability. Not sure
>> but according to my experience, most of Storm PMCs and contributors don't
>> look using Windows at their dev. environment. It doesn't strictly mean they
>> don't have experience with C#, but relatively higher chance. I'm sorry to
>> the Azure team, but I recently found a forgotten major pull request on
>> storm-eventhubs, which doesn't look like no one could review and maintain.
>> It might become real pain if we receive the code which we can't/don't
>> maintain, hence I'd rather not vote +1 unless at least two PMCs know C#
>> well, understand the code completely, and willing to volunteer to maintain.
>>
>> 2.
>> As you may know, all of multilang adapters in Storm repo are actually
>> close to "example" of implementations. They're just implementations of the
>> protocol, and don't provide any language specific optimization as well as
>> language-standard code style. Most of Python users in Storm community would
>> rather use StreamParse, and it is not uncommon to see StreamParse question
>> in user group (whereas they have their own Github repo and issue as well).
>> I would like to see adapter (and more) projects really looking attractive
>> for other languages as well.
>>
>> 3.
>> How it affect releasing Storm? We don't publish them as package in its
>> language package environment. If NuGet is one of them, we need to add the
>> sequence of release phase for NuGet while releasing Storm, which was not
>> happened yet, and will become another pain. Moreover, if it's the case, I
>> don't feel needs for having strict coupled between Storm and .net adapter
>> package. For user side it's not a "battery-included" and there's no
>> difference between maintaining inside/outside. You could freely use
>> user/dev list to announce new .net adapter release and such announcements
>> are happening from other projects as well.
>>
>> 4.
>> It's completely a thought on my own, but I feel more needs of having more
>> language-native way of supporting language instead of keep improving
>> multilang. (Not meant to discontinue.) We will introduce streams API in
>> Storm 2.0.0, a higher-level API like Trident but typed and record-to-record
>> processing. We haven't supported Trident in multilang, but I'd like to see
>> support streams API in non-JVM language, not only defining protocol, but
>> also have it as first-class support (users should be able to construct
>> their own topology with only using the language. there's thrift but not
>> that convenient.). IMHO, according to Spark's "Lesson learned" (Databricks
>> had a poll and published it), I think it's really clear that Python should
>> be first (and only, R might be another good to have).
>>
>> Thanks for reading a wall of text. Regardless of whether we could include
>> .net adapter into Storm core or not, thanks again for crafting .net adapter
>> and proposing for donation.
>>
>> Jungtaek Lim (HeartSaVioR)
>>
>> 2018년 2월 2일 (금) 오전 2:03, Mauro Giusti <mau...@microsoft.com.invalid>님이
>> 작성:
>>
>>> Hello Storm devs -
>>>
>>> We are working on a project that uses Storm and C# / .net core
>>> components.
>>>
>>> As part of that, we would love to contribute to the Storm project with a
>>> multilang protocol sample that uses our C# component/adapter.
>>> The adapter will be a NuGet package, and we intend to publish the source
>>> code of this component as well.
>>>
>>> With that in mind, I have two questions:
>>>
>>>   *   Would you prefer the code for the .net adapter (not the sample:
>>> that will be on the Storm repo, just the adapter) to be hosted inside the
>>> Storm repo or in a separate GitHub repo? We see pros and cons: we might
>>> need to introduce .net core compilation in the Storm repo if we host the
>>> adapter there, the code will be a bit more scattered and harder to test if
>>> we host the adapter outside.
>>>   *   Can you help us review / answer design questions about our adapter
>>> and the multilang protocol? Is there a specific set of people that is more
>>> knowledgeable about this and/or wants to help in this specific area?
>>>
>>> Thank you,
>>> Mauro Giusti
>>> Azure Team, Microsoft.
>>>
>>

Re: Storm multilang - .net core

[Jungtaek Lim]

Please remove or bcc. for company mail/mailing list which don't allow
sending mail outside of MS. My previous mail bounced. I just removed CCC
from recipient in current mail.

2018년 2월 2일 (금) 오전 9:08, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> Mauro,
>
> First of all, thanks for proposing contribution of valuable effort on
> Apache Storm! Really appreciated.
>
> I don't know about C#/.net but I have been working on the change of
> multilang, so adding my voice here.
>
> 1.
> Major concern from my side for code donation is sustainability. Not sure
> but according to my experience, most of Storm PMCs and contributors don't
> look using Windows at their dev. environment. It doesn't strictly mean they
> don't have experience with C#, but relatively higher chance. I'm sorry to
> the Azure team, but I recently found a forgotten major pull request on
> storm-eventhubs, which doesn't look like no one could review and maintain.
> It might become real pain if we receive the code which we can't/don't
> maintain, hence I'd rather not vote +1 unless at least two PMCs know C#
> well, understand the code completely, and willing to volunteer to maintain.
>
> 2.
> As you may know, all of multilang adapters in Storm repo are actually
> close to "example" of implementations. They're just implementations of the
> protocol, and don't provide any language specific optimization as well as
> language-standard code style. Most of Python users in Storm community would
> rather use StreamParse, and it is not uncommon to see StreamParse question
> in user group (whereas they have their own Github repo and issue as well).
> I would like to see adapter (and more) projects really looking attractive
> for other languages as well.
>
> 3.
> How it affect releasing Storm? We don't publish them as package in its
> language package environment. If NuGet is one of them, we need to add the
> sequence of release phase for NuGet while releasing Storm, which was not
> happened yet, and will become another pain. Moreover, if it's the case, I
> don't feel needs for having strict coupled between Storm and .net adapter
> package. For user side it's not a "battery-included" and there's no
> difference between maintaining inside/outside. You could freely use
> user/dev list to announce new .net adapter release and such announcements
> are happening from other projects as well.
>
> 4.
> It's completely a thought on my own, but I feel more needs of having more
> language-native way of supporting language instead of keep improving
> multilang. (Not meant to discontinue.) We will introduce streams API in
> Storm 2.0.0, a higher-level API like Trident but typed and record-to-record
> processing. We haven't supported Trident in multilang, but I'd like to see
> support streams API in non-JVM language, not only defining protocol, but
> also have it as first-class support (users should be able to construct
> their own topology with only using the language. there's thrift but not
> that convenient.). IMHO, according to Spark's "Lesson learned" (Databricks
> had a poll and published it), I think it's really clear that Python should
> be first (and only, R might be another good to have).
>
> Thanks for reading a wall of text. Regardless of whether we could include
> .net adapter into Storm core or not, thanks again for crafting .net adapter
> and proposing for donation.
>
> Jungtaek Lim (HeartSaVioR)
>
> 2018년 2월 2일 (금) 오전 2:03, Mauro Giusti <mau...@microsoft.com.invalid>님이 작성:
>
>> Hello Storm devs -
>>
>> We are working on a project that uses Storm and C# / .net core components.
>>
>> As part of that, we would love to contribute to the Storm project with a
>> multilang protocol sample that uses our C# component/adapter.
>> The adapter will be a NuGet package, and we intend to publish the source
>> code of this component as well.
>>
>> With that in mind, I have two questions:
>>
>>   *   Would you prefer the code for the .net adapter (not the sample:
>> that will be on the Storm repo, just the adapter) to be hosted inside the
>> Storm repo or in a separate GitHub repo? We see pros and cons: we might
>> need to introduce .net core compilation in the Storm repo if we host the
>> adapter there, the code will be a bit more scattered and harder to test if
>> we host the adapter outside.
>>   *   Can you help us review / answer design questions about our adapter
>> and the multilang protocol? Is there a specific set of people that is more
>> knowledgeable about this and/or wants to help in this specific area?
>>
>> Thank you,
>> Mauro Giusti
>> Azure Team, Microsoft.
>>
>

Re: Storm multilang - .net core

[Jungtaek Lim]

Mauro,

First of all, thanks for proposing contribution of valuable effort on
Apache Storm! Really appreciated.

I don't know about C#/.net but I have been working on the change of
multilang, so adding my voice here.

1.
Major concern from my side for code donation is sustainability. Not sure
but according to my experience, most of Storm PMCs and contributors don't
look using Windows at their dev. environment. It doesn't strictly mean they
don't have experience with C#, but relatively higher chance. I'm sorry to
the Azure team, but I recently found a forgotten major pull request on
storm-eventhubs, which doesn't look like no one could review and maintain.
It might become real pain if we receive the code which we can't/don't
maintain, hence I'd rather not vote +1 unless at least two PMCs know C#
well, understand the code completely, and willing to volunteer to maintain.

2.
As you may know, all of multilang adapters in Storm repo are actually close
to "example" of implementations. They're just implementations of the
protocol, and don't provide any language specific optimization as well as
language-standard code style. Most of Python users in Storm community would
rather use StreamParse, and it is not uncommon to see StreamParse question
in user group (whereas they have their own Github repo and issue as well).
I would like to see adapter (and more) projects really looking attractive
for other languages as well.

3.
How it affect releasing Storm? We don't publish them as package in its
language package environment. If NuGet is one of them, we need to add the
sequence of release phase for NuGet while releasing Storm, which was not
happened yet, and will become another pain. Moreover, if it's the case, I
don't feel needs for having strict coupled between Storm and .net adapter
package. For user side it's not a "battery-included" and there's no
difference between maintaining inside/outside. You could freely use
user/dev list to announce new .net adapter release and such announcements
are happening from other projects as well.

4.
It's completely a thought on my own, but I feel more needs of having more
language-native way of supporting language instead of keep improving
multilang. (Not meant to discontinue.) We will introduce streams API in
Storm 2.0.0, a higher-level API like Trident but typed and record-to-record
processing. We haven't supported Trident in multilang, but I'd like to see
support streams API in non-JVM language, not only defining protocol, but
also have it as first-class support (users should be able to construct
their own topology with only using the language. there's thrift but not
that convenient.). IMHO, according to Spark's "Lesson learned" (Databricks
had a poll and published it), I think it's really clear that Python should
be first (and only, R might be another good to have).

Thanks for reading a wall of text. Regardless of whether we could include
.net adapter into Storm core or not, thanks again for crafting .net adapter
and proposing for donation.

Jungtaek Lim (HeartSaVioR)

2018년 2월 2일 (금) 오전 2:03, Mauro Giusti <mau...@microsoft.com.invalid>님이 작성:

> Hello Storm devs -
>
> We are working on a project that uses Storm and C# / .net core components.
>
> As part of that, we would love to contribute to the Storm project with a
> multilang protocol sample that uses our C# component/adapter.
> The adapter will be a NuGet package, and we intend to publish the source
> code of this component as well.
>
> With that in mind, I have two questions:
>
>   *   Would you prefer the code for the .net adapter (not the sample: that
> will be on the Storm repo, just the adapter) to be hosted inside the Storm
> repo or in a separate GitHub repo? We see pros and cons: we might need to
> introduce .net core compilation in the Storm repo if we host the adapter
> there, the code will be a bit more scattered and harder to test if we host
> the adapter outside.
>   *   Can you help us review / answer design questions about our adapter
> and the multilang protocol? Is there a specific set of people that is more
> knowledgeable about this and/or wants to help in this specific area?
>
> Thank you,
> Mauro Giusti
> Azure Team, Microsoft.
>

Re: Build failures on all 1.x version lines

[Jungtaek Lim]

Stig,

it's possible but we don't run test master branch with JDK 7, so we all
don't know the tests in storm-hdfs will succeed with JDK 7. It is occurred
consistently in only JDK 7, and I'm just expecting no patch was targeted to
fix that issue. (If it was, it must be pushed to 1.x version lines.)

Taylor,

yes it's too flaky, not only in my experiment but also build results on
Github pull requests page. It would be really tough to see build succeeds
10 times (maybe 5 times?) in a row, even 1.x version line with JDK 7 would
break more often. IMHO, once we support JDK 7 in 1.x version line, tests
should follow the environment: it must succeed with JDK 7 most of the time.

Hopefully the flaky tests I filed for older versions and forgot are not
seen recently, so closed such issues and filed new issues. I hope they
would be easy to fix but it may not, so investigation from multiple
contributors are definitely needed.

2018년 2월 2일 (금) 오전 5:12, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:

> I can confirm everything you’ve experienced. The maven release process
> runs all tests 3 (current SNAPSHOT, release version, post-release SNAPSHOT)
> times and they must pass for the build to succeed. That tends to highlight
> flaky tests.
>
> In the last 6 release candidates, only one made it through on the first
> try. IIRC it was 1.0.x. Sometimes switching JDKs helps, sometimes rebooting
> helps, sometimes I turn to voodoo… ;)
>
> For the last round of RCs, I’ve been using the following JDK:
>
> java version "1.8.0_152"
> Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
> Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)
>
> -Taylor
>
> > On Feb 1, 2018, at 6:31 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
> >
> > Another update: I've tracked first page of pull requests and filed build
> > failure issues under STORM-915.
> > https://issues.apache.org/jira/browse/STORM-915
> >
> > 2018년 2월 1일 (목) 오후 6:29, Jungtaek Lim <kabh...@gmail.com>님이 작성:
> >
> >> UPDATE: I've run build 3 times per branch from Travis CI, which ensures
> >> the test result is no longer related to my (possible) messed up local
> dev.
> >>
> >> 1. 1.x-branch (https://github.com/HeartSaVioR/storm/tree/1.x-branch)
> >>
> >> * storm-core, JDK7 : OK, FAIL, OK (66.6%)
> >> * storm-core, JDK8 : OK, OK, FAIL (66.6%)
> >> * !storm-core, JDK7 : ERROR, ERROR, ERROR (0%)
> >> * !storm-core, JDK8 : OK, OK, OK (100%)
> >>
> >> 2. 1.1.x-branch (https://github.com/HeartSaVioR/storm/tree/1.1.x-branch
> )
> >>
> >> * storm-core, JDK7 : OK, OK, OK (100%)
> >> * storm-core, JDK8 : OK, FAIL, OK (66.6%)
> >> * !storm-core, JDK7 : ERROR, ERROR, ERROR (0%)
> >> * !storm-core, JDK8 : OK, OK, OK (100%)
> >>
> >> 3. 1.0.x-branch (https://github.com/HeartSaVioR/storm/tree/1.0.x-branch
> )
> >>
> >> * storm-core, JDK7 : FAIL, FAIL, OK (33.3%)
> >> * storm-core, JDK8 : OK, FAIL, OK (66.6%)
> >> * !storm-core, JDK7 : ERROR, ERROR, ERROR (0%)
> >> * !storm-core, JDK8 : OK, OK, OK (100%)
> >>
> >> I didn't see consistent test failure except storm-hdfs test stuck in
> JDK 7
> >> which you may be already aware of (It must be fixed unless this cannot
> be
> >> fixed), but the chance of build failure is still high even except
> >> integration-test.
> >>
> >> Test failures occur from below tests (all failures are occurring both
> JDK
> >> 7 and JDK 8):
> >>
> >> * org.apache.storm.metrics-test
> >> * org.apache.storm.nimbus-test
> >> * integration.org.apache.storm.integration-test
> >>
> >> I'm relieved that consistent failure in my local was my env. issue
> >> (actually I found iPhone USB connect assigned private IP to my macOS
> and it
> >> completely messed up other things like manual test as well) but the test
> >> failures still need to be investigated since we don't have any plan to
> drop
> >> supporting 1.x version lines.
> >>
> >> - Jungtaek Lim (HeartSaVioR)
> >>
> >> 2018년 2월 1일 (목) 오후 3:13, Jungtaek Lim <kabh...@gmail.com>님이 작성:
> >>
> >>> Hi devs,
> >>>
> >>> I have been verifying release candidates for 1.2.0/1.1.2/1.0.6 and
> >>> encounter high chance of build failures across all the versions.
> >>> I thought the issue is only with JDK 7, but sometimes also with JDK 8.
> >>>
> >>> Since it affects my verification of release candidates ongoing, I would
> >>> like to see whether it is the issue from my environment (OSX 10.12.6,
> >>> Oracle Java SE 1.7.0_79-b15), or there is something we should fix.
> >>>
> >>> I'm now trying out rebuilding branches with Travis CI.
> >>>
> >>> 1.x : https://travis-ci.org/HeartSaVioR/storm/builds/335962894
> >>> 1.1.x : https://travis-ci.org/HeartSaVioR/storm/builds/335962725
> >>> 1.0.x : https://travis-ci.org/HeartSaVioR/storm/builds/335962792
> >>>
> >>> Please try out building branches with JDK7/8 (multiple times would be
> >>> really appreciated) and share whether the build passes smoothly or not.
> >>>
> >>> Thanks,
> >>> Jungtaek Lim (HeartSaVioR)
> >>>
> >>
>
>

Re: Build failures on all 1.x version lines

[Jungtaek Lim]

Another update: I've tracked first page of pull requests and filed build
failure issues under STORM-915.
https://issues.apache.org/jira/browse/STORM-915

2018년 2월 1일 (목) 오후 6:29, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> UPDATE: I've run build 3 times per branch from Travis CI, which ensures
> the test result is no longer related to my (possible) messed up local dev.
>
> 1. 1.x-branch (https://github.com/HeartSaVioR/storm/tree/1.x-branch)
>
> * storm-core, JDK7 : OK, FAIL, OK (66.6%)
> * storm-core, JDK8 : OK, OK, FAIL (66.6%)
> * !storm-core, JDK7 : ERROR, ERROR, ERROR (0%)
> * !storm-core, JDK8 : OK, OK, OK (100%)
>
> 2. 1.1.x-branch (https://github.com/HeartSaVioR/storm/tree/1.1.x-branch)
>
> * storm-core, JDK7 : OK, OK, OK (100%)
> * storm-core, JDK8 : OK, FAIL, OK (66.6%)
> * !storm-core, JDK7 : ERROR, ERROR, ERROR (0%)
> * !storm-core, JDK8 : OK, OK, OK (100%)
>
> 3. 1.0.x-branch (https://github.com/HeartSaVioR/storm/tree/1.0.x-branch)
>
> * storm-core, JDK7 : FAIL, FAIL, OK (33.3%)
> * storm-core, JDK8 : OK, FAIL, OK (66.6%)
> * !storm-core, JDK7 : ERROR, ERROR, ERROR (0%)
> * !storm-core, JDK8 : OK, OK, OK (100%)
>
> I didn't see consistent test failure except storm-hdfs test stuck in JDK 7
> which you may be already aware of (It must be fixed unless this cannot be
> fixed), but the chance of build failure is still high even except
> integration-test.
>
> Test failures occur from below tests (all failures are occurring both JDK
> 7 and JDK 8):
>
> * org.apache.storm.metrics-test
> * org.apache.storm.nimbus-test
> * integration.org.apache.storm.integration-test
>
> I'm relieved that consistent failure in my local was my env. issue
> (actually I found iPhone USB connect assigned private IP to my macOS and it
> completely messed up other things like manual test as well) but the test
> failures still need to be investigated since we don't have any plan to drop
> supporting 1.x version lines.
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2018년 2월 1일 (목) 오후 3:13, Jungtaek Lim <kabh...@gmail.com>님이 작성:
>
>> Hi devs,
>>
>> I have been verifying release candidates for 1.2.0/1.1.2/1.0.6 and
>> encounter high chance of build failures across all the versions.
>> I thought the issue is only with JDK 7, but sometimes also with JDK 8.
>>
>> Since it affects my verification of release candidates ongoing, I would
>> like to see whether it is the issue from my environment (OSX 10.12.6,
>> Oracle Java SE 1.7.0_79-b15), or there is something we should fix.
>>
>> I'm now trying out rebuilding branches with Travis CI.
>>
>> 1.x : https://travis-ci.org/HeartSaVioR/storm/builds/335962894
>> 1.1.x : https://travis-ci.org/HeartSaVioR/storm/builds/335962725
>> 1.0.x : https://travis-ci.org/HeartSaVioR/storm/builds/335962792
>>
>> Please try out building branches with JDK7/8 (multiple times would be
>> really appreciated) and share whether the build passes smoothly or not.
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>

Re: Build failures on all 1.x version lines

[Jungtaek Lim]

UPDATE: I've run build 3 times per branch from Travis CI, which ensures the
test result is no longer related to my (possible) messed up local dev.

1. 1.x-branch (https://github.com/HeartSaVioR/storm/tree/1.x-branch)

* storm-core, JDK7 : OK, FAIL, OK (66.6%)
* storm-core, JDK8 : OK, OK, FAIL (66.6%)
* !storm-core, JDK7 : ERROR, ERROR, ERROR (0%)
* !storm-core, JDK8 : OK, OK, OK (100%)

2. 1.1.x-branch (https://github.com/HeartSaVioR/storm/tree/1.1.x-branch)

* storm-core, JDK7 : OK, OK, OK (100%)
* storm-core, JDK8 : OK, FAIL, OK (66.6%)
* !storm-core, JDK7 : ERROR, ERROR, ERROR (0%)
* !storm-core, JDK8 : OK, OK, OK (100%)

3. 1.0.x-branch (https://github.com/HeartSaVioR/storm/tree/1.0.x-branch)

* storm-core, JDK7 : FAIL, FAIL, OK (33.3%)
* storm-core, JDK8 : OK, FAIL, OK (66.6%)
* !storm-core, JDK7 : ERROR, ERROR, ERROR (0%)
* !storm-core, JDK8 : OK, OK, OK (100%)

I didn't see consistent test failure except storm-hdfs test stuck in JDK 7
which you may be already aware of (It must be fixed unless this cannot be
fixed), but the chance of build failure is still high even except
integration-test.

Test failures occur from below tests (all failures are occurring both JDK 7
and JDK 8):

* org.apache.storm.metrics-test
* org.apache.storm.nimbus-test
* integration.org.apache.storm.integration-test

I'm relieved that consistent failure in my local was my env. issue
(actually I found iPhone USB connect assigned private IP to my macOS and it
completely messed up other things like manual test as well) but the test
failures still need to be investigated since we don't have any plan to drop
supporting 1.x version lines.

- Jungtaek Lim (HeartSaVioR)

2018년 2월 1일 (목) 오후 3:13, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> Hi devs,
>
> I have been verifying release candidates for 1.2.0/1.1.2/1.0.6 and
> encounter high chance of build failures across all the versions.
> I thought the issue is only with JDK 7, but sometimes also with JDK 8.
>
> Since it affects my verification of release candidates ongoing, I would
> like to see whether it is the issue from my environment (OSX 10.12.6,
> Oracle Java SE 1.7.0_79-b15), or there is something we should fix.
>
> I'm now trying out rebuilding branches with Travis CI.
>
> 1.x : https://travis-ci.org/HeartSaVioR/storm/builds/335962894
> 1.1.x : https://travis-ci.org/HeartSaVioR/storm/builds/335962725
> 1.0.x : https://travis-ci.org/HeartSaVioR/storm/builds/335962792
>
> Please try out building branches with JDK7/8 (multiple times would be
> really appreciated) and share whether the build passes smoothly or not.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>

Build failures on all 1.x version lines

[Jungtaek Lim]

Hi devs,

I have been verifying release candidates for 1.2.0/1.1.2/1.0.6 and
encounter high chance of build failures across all the versions.
I thought the issue is only with JDK 7, but sometimes also with JDK 8.

Since it affects my verification of release candidates ongoing, I would
like to see whether it is the issue from my environment (OSX 10.12.6,
Oracle Java SE 1.7.0_79-b15), or there is something we should fix.

I'm now trying out rebuilding branches with Travis CI.

1.x : https://travis-ci.org/HeartSaVioR/storm/builds/335962894
1.1.x : https://travis-ci.org/HeartSaVioR/storm/builds/335962725
1.0.x : https://travis-ci.org/HeartSaVioR/storm/builds/335962792

Please try out building branches with JDK7/8 (multiple times would be
really appreciated) and share whether the build passes smoothly or not.

Thanks,
Jungtaek Lim (HeartSaVioR)

Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch / 1.0.x-branch with 1.x-branch

[Jungtaek Lim]

This discussion got 4 +1 (binding) and no -1. Moreover two active
maintainers for storm-kafka-client (Hugo and Stig) voted +1.

Do we want to hold on for hearing more voices, or treating above opinions
as consensus and reflect the change?

Btw, I think we need to sort out the sequences between two topics:
separating storm-kafka-client as independent release cycle, and this. I
guess some of us agreed former topic doesn't related to current RC, but I
think this topic can be (should be) reflected to current RC ongoing.

-Jungtaek Lim (HeartSaVioR)

2018년 2월 1일 (목) 오전 4:08, Hugo Da Cruz Louro <hlo...@hortonworks.com>님이 작성:

> +1 to replace storm-kafka-client in 1.0.x branch.
> Hugo
>
> > On Jan 31, 2018, at 11:03 AM, Stig Rohde Døssing <stigdoess...@gmail.com>
> wrote:
> >
> > +1 to replace storm-kafka-client in 1.0.x branch. Breaking semantic
> > versioning is really nasty, but I think it is the lesser evil in this
> case.
> >
> > 2018-01-31 5:14 GMT+01:00 Harsha <st...@harsha.io>:
> >
> >> +1 to replace storm-kafka-client in 1.0.x branch
> >> -Harsha
> >> On Tue, Jan 30, 2018, at 7:04 PM, Jungtaek Lim wrote:
> >>> Bump up this thread so that we could reach consensus earlier. Given
> that
> >> we
> >>> got concern related to this, I think it is ideal to release 1.1.x/1.0.x
> >>> with making decision and applying the change if we want.
> >>>
> >>> 2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <kabh...@gmail.com>님이 작성:
> >>>
> >>>> Erik's concern brought from 1.0.6 RC1, because they can't use Storm
> >> 1.1.0
> >>>> or higher (Storm 1.1.0 broke storm-mesos.). While he could take an
> >>>> workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we decide to
> >>>> replace) with Storm 1.0.6, it would be better if we don't allow
> leaving
> >>>> storm-kafka-client in 1.0.x in inconsistent state.
> >>>>
> >>>> IMHO, breaking backward compatibility is worse, but leaving broken
> >> thing
> >>>> is worst. Hence I'm +1 to replace all, with noticing that it may bring
> >>>> backward incompatibility in release announce.
> >>>>
> >>>> -Jungtaek Lim (HeartSaVioR)
> >>>>
> >>>> 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
> >>>>
> >>>>> As I mentioned else thread I’m open to this but would defer to
> >> community
> >>>>> consensus.
> >>>>>
> >>>>> If there’s concern about doing this for 1.0.x, one option would be
> >> skip
> >>>>> that version line and only apply it to 1.2.0 and 1.1.x.
> >>>>>
> >>>>> -Taylor
> >>>>>
> >>>>>> On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <kabh...@gmail.com>
> >> wrote:
> >>>>>>
> >>>>>> Hi devs,
> >>>>>>
> >>>>>> This is initial post to separate out discussion topic from vote
> >> thread,
> >>>>> and
> >>>>>> continue discussing.
> >>>>>>
> >>>>>> Background of the topic:
> >>>>>> 1. Only 1.x-branch of storm-kafka-client got stabilized.
> >> (relatively)
> >>>>>> 2. We would avoid to port back patches to 1.1.x and 1.0.x because
> >>>>> they're
> >>>>>> diverged too much.
> >>>>>>
> >>>>>> Downside:
> >>>>>> Backward compatibility might be broken for 1.1.x and 1.0.x. Not
> >> sure for
> >>>>>> 1.1.x, but at least 1.0.x, since supported Kafka client version is
> >>>>>> different, and if my memory is right, we already applied backward
> >>>>>> incompatible change into storm-kafka-client 1.1.0.
> >>>>>>
> >>>>>> Please put your opinion regarding topic. You're encouraged to copy
> >> your
> >>>>>> previous post in vote thread which helps to centralize opinions in
> >>>>> current
> >>>>>> thread.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Jungtaek Lim (HeartSaVioR)
> >>>>>
> >>>>>
> >>
>
>

Re: [VOTE] Release Apache Storm 1.1.2 (rc2)

[Jungtaek Lim]

Stig, ah yes, I didn't mix up but I didn't explain enough. Sorry about
that. You made great efforts to make tests stable and I ported them (not
sure I did everything but maybe most of things) back to only 1.x-branch,
not 1.1.x/1.0.x. I would like to see the possibility about fixed this
earlier. If not, looks like someone including me need to investigate the
issue. Switching JDK helps so really odd though.

2018년 2월 1일 (목) 오전 3:47, Stig Rohde Døssing <stigdoess...@gmail.com>님이 작성:

> I meant 1.1.2 RC.
>
> 2018-01-31 9:44 GMT+01:00 Jungtaek Lim <kabh...@gmail.com>:
>
> > Build with JDK 7 fails 3 times in same test: "metrics-test" in a row.
> Looks
> > like building with JDK 8 helps build to pass, but since we support JDK 7
> in
> > 1.x version line, I would like to see build pass with JDK 7. It is hard
> to
> > say intermittent failure, yes it may be intermittent, but really high
> > change.
> >
> > I'll try to build Storm 1.2.0 RC2 as well as Storm 1.0.6 RC2, and see the
> > difference. I'll also try to build with Travis CI to double-check my
> local
> > dev. for JDK 7 is messed or not. If there's difference in result, I'll
> > track down why the difference happens. Any help is much appreciated.
> >
> > Stig, did you encounter this? And if you encountered, did you submit a
> > patch for this?
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> > 2018년 1월 30일 (화) 오전 4:45, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
> >
> > > I’m open to synching storm-kafka-client across those releases if that’s
> > > the direction we want to go, but I’d like to hear from others to make
> > sure
> > > there’s consensus.
> > >
> > > -Taylor
> > >
> > > > On Jan 29, 2018, at 4:04 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
> > > >
> > > > I'm waiting for hearing Taylor's opinion regarding replacing
> > > > storm-kafka-client in 1.1.x-branch/1.0.x-branch with 1.x-branch,
> since
> > > he's
> > > > release manager on 1.1.2/1.0.6, and IMHO it's ideal to apply the
> change
> > > > sooner if we agree on this rather than postponing to next release.
> > > >
> > > > I'll continue verifying 1.1.2 RC2 and 1.0.6 RC2 if he would want to
> > take
> > > > only blocker issue for current RCs.
> > > >
> > > > -Jungtaek Lim (HeartSaVioR)
> > > >
> > > > 2018년 1월 27일 (토) 오전 6:11, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
> > > >
> > > >> This is a call to vote on releasing Apache Storm 1.1.2 (rc2)
> > > >>
> > > >> Full list of changes in this release:
> > > >>
> > > >>
> > > >>
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > 1.2-rc2/RELEASE_NOTES.html
> > > >>
> > > >> The tag/commit to be voted upon is v1.1.2:
> > > >>
> > > >>
> > > >>
> > > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=
> >
> 55ab8e9c072b09e634a9eddfc7b1e1e0d5d7c27b;hb=5d2eecf3d282a535541ac7520a88b4
> > 7f01153da1
> > > >>
> > > >> The source archive being voted upon can be found here:
> > > >>
> > > >>
> > > >>
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> > 1.2-rc2/apache-storm-1.1.2-src.tar.gz
> > > >>
> > > >> Other release files, signatures and digests can be found here:
> > > >>
> > > >>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.2-rc2/
> > > >>
> > > >> The release artifacts are signed with the following key:
> > > >>
> > > >>
> > > >>
> > > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_
> > plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > > >>
> > > >> The Nexus staging repository for this release is:
> > > >>
> > > >> https://repository.apache.org/content/repositories/
> > orgapachestorm-1056
> > > >>
> > > >> Please vote on releasing this package as Apache Storm 1.1.2.
> > > >>
> > > >> When voting, please list the actions taken to verify the release.
> > > >>
> > > >> This vote will be open for at least 72 hours.
> > > >>
> > > >> [ ] +1 Release this package as Apache Storm 1.1.2
> > > >> [ ]  0 No opinion
> > > >> [ ] -1 Do not release this package because...
> > > >>
> > > >> Thanks to everyone who contributed to this release.
> > > >>
> > > >> -Taylor
> > > >>
> > >
> > >
> >
>

Re: [VOTE] Release Apache Storm 1.1.2 (rc2)

[Jungtaek Lim]

Build with JDK 7 fails 3 times in same test: "metrics-test" in a row. Looks
like building with JDK 8 helps build to pass, but since we support JDK 7 in
1.x version line, I would like to see build pass with JDK 7. It is hard to
say intermittent failure, yes it may be intermittent, but really high
change.

I'll try to build Storm 1.2.0 RC2 as well as Storm 1.0.6 RC2, and see the
difference. I'll also try to build with Travis CI to double-check my local
dev. for JDK 7 is messed or not. If there's difference in result, I'll
track down why the difference happens. Any help is much appreciated.

Stig, did you encounter this? And if you encountered, did you submit a
patch for this?

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 1월 30일 (화) 오전 4:45, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:

> I’m open to synching storm-kafka-client across those releases if that’s
> the direction we want to go, but I’d like to hear from others to make sure
> there’s consensus.
>
> -Taylor
>
> > On Jan 29, 2018, at 4:04 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
> >
> > I'm waiting for hearing Taylor's opinion regarding replacing
> > storm-kafka-client in 1.1.x-branch/1.0.x-branch with 1.x-branch, since
> he's
> > release manager on 1.1.2/1.0.6, and IMHO it's ideal to apply the change
> > sooner if we agree on this rather than postponing to next release.
> >
> > I'll continue verifying 1.1.2 RC2 and 1.0.6 RC2 if he would want to take
> > only blocker issue for current RCs.
> >
> > -Jungtaek Lim (HeartSaVioR)
> >
> > 2018년 1월 27일 (토) 오전 6:11, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
> >
> >> This is a call to vote on releasing Apache Storm 1.1.2 (rc2)
> >>
> >> Full list of changes in this release:
> >>
> >>
> >>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.2-rc2/RELEASE_NOTES.html
> >>
> >> The tag/commit to be voted upon is v1.1.2:
> >>
> >>
> >>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=55ab8e9c072b09e634a9eddfc7b1e1e0d5d7c27b;hb=5d2eecf3d282a535541ac7520a88b47f01153da1
> >>
> >> The source archive being voted upon can be found here:
> >>
> >>
> >>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.2-rc2/apache-storm-1.1.2-src.tar.gz
> >>
> >> Other release files, signatures and digests can be found here:
> >>
> >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.2-rc2/
> >>
> >> The release artifacts are signed with the following key:
> >>
> >>
> >>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >>
> >> The Nexus staging repository for this release is:
> >>
> >> https://repository.apache.org/content/repositories/orgapachestorm-1056
> >>
> >> Please vote on releasing this package as Apache Storm 1.1.2.
> >>
> >> When voting, please list the actions taken to verify the release.
> >>
> >> This vote will be open for at least 72 hours.
> >>
> >> [ ] +1 Release this package as Apache Storm 1.1.2
> >> [ ]  0 No opinion
> >> [ ] -1 Do not release this package because...
> >>
> >> Thanks to everyone who contributed to this release.
> >>
> >> -Taylor
> >>
>
>

Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch / 1.0.x-branch with 1.x-branch

[Jungtaek Lim]

Bump up this thread so that we could reach consensus earlier. Given that we
got concern related to this, I think it is ideal to release 1.1.x/1.0.x
with making decision and applying the change if we want.

2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> Erik's concern brought from 1.0.6 RC1, because they can't use Storm 1.1.0
> or higher (Storm 1.1.0 broke storm-mesos.). While he could take an
> workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we decide to
> replace) with Storm 1.0.6, it would be better if we don't allow leaving
> storm-kafka-client in 1.0.x in inconsistent state.
>
> IMHO, breaking backward compatibility is worse, but leaving broken thing
> is worst. Hence I'm +1 to replace all, with noticing that it may bring
> backward incompatibility in release announce.
>
> -Jungtaek Lim (HeartSaVioR)
>
> 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>
>> As I mentioned else thread I’m open to this but would defer to community
>> consensus.
>>
>> If there’s concern about doing this for 1.0.x, one option would be skip
>> that version line and only apply it to 1.2.0 and 1.1.x.
>>
>> -Taylor
>>
>> > On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
>> >
>> > Hi devs,
>> >
>> > This is initial post to separate out discussion topic from vote thread,
>> and
>> > continue discussing.
>> >
>> > Background of the topic:
>> > 1. Only 1.x-branch of storm-kafka-client got stabilized. (relatively)
>> > 2. We would avoid to port back patches to 1.1.x and 1.0.x because
>> they're
>> > diverged too much.
>> >
>> > Downside:
>> > Backward compatibility might be broken for 1.1.x and 1.0.x. Not sure for
>> > 1.1.x, but at least 1.0.x, since supported Kafka client version is
>> > different, and if my memory is right, we already applied backward
>> > incompatible change into storm-kafka-client 1.1.0.
>> >
>> > Please put your opinion regarding topic. You're encouraged to copy your
>> > previous post in vote thread which helps to centralize opinions in
>> current
>> > thread.
>> >
>> > Thanks,
>> > Jungtaek Lim (HeartSaVioR)
>>
>>

Re: [DISCUSS] Decouple Storm core and connectors

[Jungtaek Lim]

Agreed for this topic: this is not related to current release candidate and
verifying release candidate is higher priority.
For me I didn't start verifying 1.1.2 / 1.0.6 RC2 because the other topic I
initiated could affect the current release. I'll post a short notice in
that discussion thread.

-Jungtaek Lim (HeartSaVioR)

2018년 1월 31일 (수) 오전 10:58, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:

> Hit send on that too soon...
>
> This is an important discussion topic, but has no effect on the current
> RCs. Id recommend focusing on the current releases and come back to this
> after getting  releases out.
>
> -Taylor
>
> > On Jan 30, 2018, at 8:51 PM, P. Taylor Goetz <ptgo...@gmail.com> wrote:
> >
> > Also, in the interest of getting releases out, we have 3 open RC cycles
> in flight.
> >
> > Discussion energy might be better focused on that.
> >
> > -Taylor
> >
> >> On Jan 30, 2018, at 7:52 PM, P. Taylor Goetz <ptgo...@gmail.com> wrote:
> >>
> >>
> >>
> >>> On Jan 30, 2018, at 7:31 PM, Harsha <st...@harsha.io> wrote:
> >>>
> >>> Hi,
> >>>  In general connectors are independent of Storm run-time for
> most parts. I.e if the APIs are not changed (storm-core or trident haven't
> changed in years except the package re-name). You can take the latest
> connector and run in storm 1.0 or higher. So the users doesn't need to
> upgrade their storm cluster just to get a latest connector upgrade. Which
> they might be doing it but by making the release separate and stating the
> minimum supported storm version for the connectors will help the users.
> >>> This makes it easier for the connectors to be released  independently
> of the core/run-time and makes it easy for them to be fixed and released
> more often. But moving them to Bahir or other external project will make it
> detached from Storm itself that it might not see any co-ordination as
> reviewers from storm  will need to be aware of an external project.
> >>> My proposal would be
> >>> 1. Can we create a sub-project in git under Storm so we can move the
> connectors there and everything else related Storm applies there.
> >>> 2.  Can we keep maintaining storm connectors within same repo but
> different release module for it .
> >>
> >> +1 That’s exactly my point. Just jettisoning connectors to Bahir
> without commitments from the Storm community would be a mistake.
> >>
> >> Releasing connectors independently can be handled easily at the Maven
> level. No need for a separate repo initiaially.
> >>
> >>
> >>>
> >>> This is a separate topic but can improve the release timelines if we
> have multiple release managers that are handling the maint release and also
> main release versions. Its good to have rotation of release managers from
> PMC so that everyone will understand the process and can spread the
> responsibilities. There are threads started before but don't think they are
> addressed or any action item is taken. We should start another thread to
> discuss this process as well.
> >>
> >> Breaking up external modules into separately released versions would be
> a great way to indoctrinate those new to the license grooming and release
> process. Everyone could participate.
> >>
> >>
> >>>
> >>> Thanks,
> >>> Harsha
> >>
> >> -Taylor
> >>
> >>>
> >>>> On Tue, Jan 30, 2018, at 9:49 AM, Hugo Da Cruz Louro wrote:
> >>>> I think that the bahir approach makes sense for connectors that don’t
> >>>> fall into the "first class support” category. I am in favor of moving
> >>>> such lower adoption connectors and have the interested communities
> >>>> support them with the most suitable release cycle. Connectors that are
> >>>> idle, such as some examples that Jungtaek gave, we should consider
> >>>> removing them altogether, especially if they are so outdated that they
> >>>> may not even work.
> >>>>
> >>>> Mainstream connectors such as storm-kafka-client should be kept in the
> >>>> Storm repo. For example, Flink keeps flink-connector-kafka-0.x in the
> >>>> Flink repo.
> >>>>
> >>>> I am in agreement with Jungtaek when he says: "fixing critical bugs in
> >>>> storm-kafka-client should trigger release, instead of waiting for
> Storm
> >>>> core to have some fixes to be worth to release”. Storm’s release
> cadence
> >&g

Re: [DISCUSS] Decouple Storm core and connectors

[Jungtaek Lim]

Let me add a proof of my opinion: major patch of storm-eventhubs hasn't
been getting even a comment over 4 months.
https://github.com/apache/storm/pull/2322

I'd rather want to discuss regarding discontinue supporting officially if
we no longer interest of, or we don't have resource to support, or any
valid reasons. If we agree on discontinue supporting officially, we can
move out to other repo. and let it self maintained. It may be able to get
attention and have enough contributors so that we feel better to get to
Storm core Repository again, or it can be silently forgotten. It shouldn't
affect Storm core repository at any case.

2018년 1월 30일 (화) 오후 2:03, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> If we worry about breaking somethings along with our
> users/consumers/distributors, picking one of less used/updated connector as
> experiment makes more sense to me. It's OK if we want to pick one of most
> active and widely used connector intentionally to accelerate experiment.
>
> Decoupling connectors and moving to other repo. like Bahir will make it
> clear who are having interest of which connectors. storm-eventhubs for
> example, major code contributions were done from MS developers. Now they
> are gone, and I don't know even storm-eventhubs are compatible with recent
> Azure Eventhub. That's just a one of them. I've seen many connectors in
> same, or similar, or possible (say truck number 1) situation.
>
> -Jungtaek Lim (HeartSaVioR)
>
> 2018년 1월 30일 (화) 오후 1:30, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>
>>
>> On Jan 29, 2018, at 8:03 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
>>
>>
>>
>> - Do we ensure they're all maintained?
>> -- Did we exclude inactive committers/PMCs for connector's committer
>>
>> sponsors, and do they have enough committer sponsors after that?
>>
>>
>> Good point. We’ve had some sponsors go silent recently. Maybe ping
>> sponsors and ask if they wish to maintain sponsorship?
>>
>> As a sponsor for a number of connectors, I’ll check on the ones I’ve
>> sponsored.
>>
>> - Do they all worth to keep maintaining in Storm main repository?
>>
>>
>> Again, that’s a question of whether there is user/dev interest.
>>
>>
>> -- Should we trigger release if we find and resolve critical/blocker issue
>> from them? If not, why we allow to leave the thing which is in main
>> repository as inconsistent state?
>>
>>
>> Some are tied to fairly well established protocols, some target really
>> volatile APIs. Bug reports and mailing list activity may not be a good
>> status indicator.
>>
>> Storm’s Kafka integration was the initial model for the “batteries
>> included” impetus behind `external`. If we want to evolve how that works,
>> why not start there, see what works/doesn’t work, and adapt.
>>
>> I don’t want to shock our users/consumers/distributors.
>>
>>
>> -Taylor
>>
>>
>>
>>
>>
>>

Re: [DISCUSS] Decouple Storm core and connectors

[Jungtaek Lim]

If we worry about breaking somethings along with our
users/consumers/distributors, picking one of less used/updated connector as
experiment makes more sense to me. It's OK if we want to pick one of most
active and widely used connector intentionally to accelerate experiment.

Decoupling connectors and moving to other repo. like Bahir will make it
clear who are having interest of which connectors. storm-eventhubs for
example, major code contributions were done from MS developers. Now they
are gone, and I don't know even storm-eventhubs are compatible with recent
Azure Eventhub. That's just a one of them. I've seen many connectors in
same, or similar, or possible (say truck number 1) situation.

-Jungtaek Lim (HeartSaVioR)

2018년 1월 30일 (화) 오후 1:30, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:

>
> On Jan 29, 2018, at 8:03 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
>
>
>
> - Do we ensure they're all maintained?
> -- Did we exclude inactive committers/PMCs for connector's committer
>
> sponsors, and do they have enough committer sponsors after that?
>
>
> Good point. We’ve had some sponsors go silent recently. Maybe ping
> sponsors and ask if they wish to maintain sponsorship?
>
> As a sponsor for a number of connectors, I’ll check on the ones I’ve
> sponsored.
>
> - Do they all worth to keep maintaining in Storm main repository?
>
>
> Again, that’s a question of whether there is user/dev interest.
>
>
> -- Should we trigger release if we find and resolve critical/blocker issue
> from them? If not, why we allow to leave the thing which is in main
> repository as inconsistent state?
>
>
> Some are tied to fairly well established protocols, some target really
> volatile APIs. Bug reports and mailing list activity may not be a good
> status indicator.
>
> Storm’s Kafka integration was the initial model for the “batteries
> included” impetus behind `external`. If we want to evolve how that works,
> why not start there, see what works/doesn’t work, and adapt.
>
> I don’t want to shock our users/consumers/distributors.
>
>
> -Taylor
>
>
>
>
>
>

Re: [DISCUSS] Decouple Storm core and connectors

[Jungtaek Lim]

If we are only considering about storm-kafka-client, I'd rather not
decouple and consider shorter release cycle for Storm itself, since
storm-kafka-client 1.2.0 looks pretty stabilized. (That's more alike
retrospect, like if we were doing it in 1.0.0 or even before 1.1.0.)

Now storm-kafka-client is implicitly considered to support as a first
class, and once we are supporting it as first class, fixing critical bugs
in storm-kafka-client should trigger release, instead of waiting for Storm
core to have some fixes to be worth to release.
We should be able to bring multiple releases even in a month if we have
critical/blocker, but we couldn't. (Storm 1.1.2 was ready to release some
months ago, but waited for 1.2.0 to be ready.) That may be coupled with
this issue.

Btw, my idea mainly includes unmaintained connectors. We have 19
connectors, and for me several questions are brought.

- Do we ensure they're all maintained?
-- Did we exclude inactive committers/PMCs for connector's committer
sponsors, and do they have enough committer sponsors after that?
- Do they all worth to keep maintaining in Storm main repository?
-- Should we trigger release if we find and resolve critical/blocker issue
from them? If not, why we allow to leave the thing which is in main
repository as inconsistent state?

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 1월 30일 (화) 오전 6:59, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:

>
>
> On Jan 29, 2018, at 3:55 PM, Arun Mahadevan <ar...@apache.org> wrote:
>
> When you say storm-kafka-client-vX.X.X, is it going to be completely
> independent of the Storm version and will work across Storm 1.0.x, 1.x and
> 2.0 (future) storm releases?
>
>
>
> It would be completely independent in that it would be released separately
> and not rely on “Storm proper”’s release schedule. As far as
> forward/backward compatibility, I can imagine that compatibility might
> diverge at which point we’d create/maintain some sort of compatibility
> matrix.
>
> -Taylor
>

Re: [DISCUSS] Replace storm-kafka-client on 1.1.x-branch / 1.0.x-branch with 1.x-branch

[Jungtaek Lim]

Erik's concern brought from 1.0.6 RC1, because they can't use Storm 1.1.0
or higher (Storm 1.1.0 broke storm-mesos.). While he could take an
workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we decide to
replace) with Storm 1.0.6, it would be better if we don't allow leaving
storm-kafka-client in 1.0.x in inconsistent state.

IMHO, breaking backward compatibility is worse, but leaving broken thing is
worst. Hence I'm +1 to replace all, with noticing that it may bring
backward incompatibility in release announce.

-Jungtaek Lim (HeartSaVioR)

2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:

> As I mentioned else thread I’m open to this but would defer to community
> consensus.
>
> If there’s concern about doing this for 1.0.x, one option would be skip
> that version line and only apply it to 1.2.0 and 1.1.x.
>
> -Taylor
>
> > On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
> >
> > Hi devs,
> >
> > This is initial post to separate out discussion topic from vote thread,
> and
> > continue discussing.
> >
> > Background of the topic:
> > 1. Only 1.x-branch of storm-kafka-client got stabilized. (relatively)
> > 2. We would avoid to port back patches to 1.1.x and 1.0.x because they're
> > diverged too much.
> >
> > Downside:
> > Backward compatibility might be broken for 1.1.x and 1.0.x. Not sure for
> > 1.1.x, but at least 1.0.x, since supported Kafka client version is
> > different, and if my memory is right, we already applied backward
> > incompatible change into storm-kafka-client 1.1.0.
> >
> > Please put your opinion regarding topic. You're encouraged to copy your
> > previous post in vote thread which helps to centralize opinions in
> current
> > thread.
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
>
>

Re: [VOTE] Release Apache Storm 1.1.2 (rc2)

[Jungtaek Lim]

I'm waiting for hearing Taylor's opinion regarding replacing
storm-kafka-client in 1.1.x-branch/1.0.x-branch with 1.x-branch, since he's
release manager on 1.1.2/1.0.6, and IMHO it's ideal to apply the change
sooner if we agree on this rather than postponing to next release.

I'll continue verifying 1.1.2 RC2 and 1.0.6 RC2 if he would want to take
only blocker issue for current RCs.

-Jungtaek Lim (HeartSaVioR)

2018년 1월 27일 (토) 오전 6:11, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:

> This is a call to vote on releasing Apache Storm 1.1.2 (rc2)
>
> Full list of changes in this release:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.2-rc2/RELEASE_NOTES.html
>
> The tag/commit to be voted upon is v1.1.2:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=55ab8e9c072b09e634a9eddfc7b1e1e0d5d7c27b;hb=5d2eecf3d282a535541ac7520a88b47f01153da1
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.2-rc2/apache-storm-1.1.2-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.2-rc2/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1056
>
> Please vote on releasing this package as Apache Storm 1.1.2.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 1.1.2
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
>

Re: [VOTE] Release Apache Storm 1.2.0 (rc2)

[Jungtaek Lim]

I think we didn't decide STORM-2913 as blocker yet, though I agree it is at
least critical as its priority tells.

Stig, please change priority of issue to 'blocker' and vote to -1 (binding)
if you think the issue is blocker, so that we could decide clearly whether
RC2 should be cancelled or not. Once we started VOTE for RC, I'd like to
see us concerning VOTE first.
The issue should be resolved ASAP (we're only left reviewing) if it is
defined as blocker.

Taylor,
I guess you're closing the gate and only allowing for blocker issues. Do I
understand correctly? If not, I'd like to merge waiting patches into
1.x-branch to be included to 1.2.0.

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 1월 27일 (토) 오후 6:29, Alexandre Vermeerbergen <avermeerber...@gmail.com>님이
작성:

> Hello Stig,
>
> Thank you very much for your answer, I am feeling better now that I
> see that the issue with Storm Kafka Client generating WARN in a loop
> is known and taken as a blocker for 1.2.0 final.
>
> I volunteer to test any binary of storm-kafka-client fixing this
> STORM-2913 issue, if that can hep.
>
> Best regards,
> Alexandre Vermeerbergen
>

Re: [DISCUSS] Decouple Storm core and connectors

[Jungtaek Lim]

My idea is basically came from Apache Bahir. (http://bahir.apache.org/) It
was for Apache Spark, but Flink decided to migrate their connectors to
Bahir so it is also for Apache Flink. They're also maintaining some
connectors (I'd say first class support) in their repositories, but not
all. I think we could select some of connectors to support as first class,
and move out others to Bahir or another storm repository (storm-connectors?
storm-externals?).

- Jungtaek Lim (HeartSavioR)

2018년 1월 29일 (월) 오후 2:30, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> Hi devs,
>
> This is initial post to separate out discussion topic from vote thread,
> and continue discussing.
>
> Background of the topic:
> 1. Releasing Storm requires huge bootstrapping, and normally takes several
> months to release bugfix version. Note that it is not minor version...
> Minor version is released per near a year. Connectors are maintained with
> same release cadence, which makes connectors also long period to release,
> whether it is (implicitly) beta or not.
> 2. Most of the change for connectors are not related to Storm core. It
> tends to be compatible with all release versions with same major version.
> 3. (IMHO) We have too many connectors which we even can't maintain
> actively. For example, ES connector couldn't support ES higher than 1.x.
> 4. Connectors are having same release version for Storm core, hence newly
> added connector will have at least 1.x version which no one would think it
> is beta.
>
> Downside:
> 1. Detached connectors can be easy to be forgotten. (easier than current)
> 2. Connectors may have hard time if we bring backward incompatible change
> to Storm core. We may remedy this with having supported version range for
> specific connector version.
>
> Please put your opinion regarding topic. You're encouraged to copy your
> previous post in vote thread which helps to centralize opinions in current
> thread.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>

[DISCUSS] Replace storm-kafka-client on 1.1.x-branch / 1.0.x-branch with 1.x-branch

[Jungtaek Lim]

Hi devs,

This is initial post to separate out discussion topic from vote thread, and
continue discussing.

Background of the topic:
1. Only 1.x-branch of storm-kafka-client got stabilized. (relatively)
2. We would avoid to port back patches to 1.1.x and 1.0.x because they're
diverged too much.

Downside:
Backward compatibility might be broken for 1.1.x and 1.0.x. Not sure for
1.1.x, but at least 1.0.x, since supported Kafka client version is
different, and if my memory is right, we already applied backward
incompatible change into storm-kafka-client 1.1.0.

Please put your opinion regarding topic. You're encouraged to copy your
previous post in vote thread which helps to centralize opinions in current
thread.

Thanks,
Jungtaek Lim (HeartSaVioR)

Re: [CANCELED] [VOTE] Release Apache Storm 1.0.6 (rc1)

[Jungtaek Lim]

We have two different topics in cancelled vote thread. Let's initiate
another threads to continue discussion.

- Jungtaek Lim (HeartSaVioR)

2018년 1월 27일 (토) 오후 11:56, Stig Rohde Døssing <stigdoess...@gmail.com>님이 작성:

> I don't think storm-kafka-client needs its own project, but I like the idea
> of decoupling storm-kafka-client from the release cadence of Storm.
> Primarily because it would allow us to release more frequently, but also
> because I think most of the time there's no tight coupling between the
> Storm minor/patch version and storm-kafka-client. As far as I know
> storm-kafka-client 1.2.0 runs fine on other Storm 1.x versions. If
> storm-kafka-client released independently, we could probably get away with
> maintaining only 1.x and 2.x compatible versions. I also see some value in
> being able to do breaking changes independently of Storm's major version,
> without breaking semantic versioning. People clearly expect us to not
> introduce breaking changes in minor/patch releases (e.g. Alexandre when
> storm-kafka-client 1.2.0 wouldn't run with his 1.1.1 topology a while
> back), but we've had to do that a few times to avoid postponing fixes until
> Storm 2.0.0. Releasing independently would also address Erik's concern that
> we're releasing a "new" storm-kafka-client 1.1.2 that has known issues.
>
> If we want to decouple storm-kafka-client's release cycle from Storm, I
> don't think we can keep it in the Storm repo. It would get confusing with
> branches and release tags. We might try to decouple it in the same way
> Maven have decoupled their plugins from the main Maven repo, where the code
> for a given plugin is in a separate repository, but the plugin is still
> part of the Maven project and uses the Maven mailing list for discussion
> and release announcements.
>
> My main concern for moving storm-kafka-client to another repository would
> be how we build against unreleased Storm versions, e.g. 2.0.0. I'm
> wondering if anyone has ideas for this? It looks to me like both the Maven
> plugins and Bahir are building against only released versions.
>
> @Hugo
> I think at least a few of your points about storm-kafka-client's implied
> beta status would have been easier to handle if storm-kafka-client were not
> coupled to the Storm release version. It was weird to have a new
> connector's initial release be version 1.0.0, and following Storm's release
> cycle has prevented us from releasing fixes as often as a new connector
> will likely need. I think we could have also saved ourselves a bit of work
> by releasing 0.x versions first, since we then wouldn't have had to worry
> about backward compatibility when doing major API changes like
> https://issues.apache.org/jira/browse/STORM-2548.
>
> Regarding whether it is a concern when we make breaking changes in
> storm-kafka-client if it is due to Kafka breaking something: Kafka was free
> to include breaking changes, because they were still in the pre-1.0.0
> version range which tells the user to expect breaking changes from time to
> time. When we release breaking changes in a 1.0.y version, it defeats the
> purpose of semantic versioning, which is to tell the user upgrading whether
> an upgrade can be done freely, or whether they should expect to have to
> recompile and maybe reserve some time to upgrade.
>
> 2018-01-25 23:44 GMT+01:00 Jungtaek Lim <kabh...@gmail.com>:
>
> > Hugo,
> >
> > My idea is basically came from Apache bahir. (http://bahir.apache.org/)
> It
> > was for Apache Spark, but Flink decided to migrate their connectors to
> > bahir so it is also for Apache Flink.
> > I admit we may want to keep some connectors in core even we split out
> > connectors, since moving to another repo. makes connectors hard to be in
> > sync with core version, and even has a chance to being forgotten. Kafka
> > connector is first class connector, so maybe storm-kafka-client would not
> > take this way even we have similar, but we could incubate it and bring to
> > core once it's stabilized (like storm-kafka-client in 1.2.0).
> >
> > I don't think storm-kafka-client has been technically beta. It might be
> > considered as beta implicitly in 1.0.x, but we had 1.1.0 in 10 months
> ago.
> > storm-kafka-client is introduced over a year being included as Storm
> 1.0.0
> > implicitly announcing we are no longer beta. Explicitly marking as beta
> is
> > very important in practice and that's why InterfaceStability annotation
> > came in and widely used for big project. (We have started to apply it for
> > streams API in 2.0.0: they're marked as @Unstable.)
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> > 2018년 1월 26일 (금) 

Re: [CANCELED] [VOTE] Release Apache Storm 1.0.6 (rc1)

[Jungtaek Lim]

Hugo,

My idea is basically came from Apache bahir. (http://bahir.apache.org/) It
was for Apache Spark, but Flink decided to migrate their connectors to
bahir so it is also for Apache Flink.
I admit we may want to keep some connectors in core even we split out
connectors, since moving to another repo. makes connectors hard to be in
sync with core version, and even has a chance to being forgotten. Kafka
connector is first class connector, so maybe storm-kafka-client would not
take this way even we have similar, but we could incubate it and bring to
core once it's stabilized (like storm-kafka-client in 1.2.0).

I don't think storm-kafka-client has been technically beta. It might be
considered as beta implicitly in 1.0.x, but we had 1.1.0 in 10 months ago.
storm-kafka-client is introduced over a year being included as Storm 1.0.0
implicitly announcing we are no longer beta. Explicitly marking as beta is
very important in practice and that's why InterfaceStability annotation
came in and widely used for big project. (We have started to apply it for
streams API in 2.0.0: they're marked as @Unstable.)

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 1월 26일 (금) 오전 4:16, Hugo Da Cruz Louro <hlo...@hortonworks.com>님이 작성:

> I am in favor of not incubating storm-kafka-client but rather keep it as
> part of the storm project. We can consider supporting a separate branch,
> but before we agree to go that route I would like to hear lessons learned
> from community members that have been part of similar transitions in other
> projects.
>
> As for not back-porting all the storm-kafka-client changes onto 1.0.x and
> 1.1.x branches, I expressed my opinion in the JIRA when the this discussion
> first came up. Basically, I don’t think it is feasible to back-port
> everything. Furthermore, 1.2.0 will be a minor version for which it is
> reasonable to expect users of earlier versions to upgrade to without major
> hassle. In any software it is expected that there will be divergence across
> minor versions. New versions become available because they have
> improvements and and sometimes new, small, features in it. If the user
> wants to benefit from those improvements, she/he will have to upgrade to
> the most recent version. Compatibility should be accounted for across
> versions, but as Stig mentioned, for the most part we believe it has been.
>
> Although it is possible to copy the 1.x storm-kafka-client changes to
> 1.1.x and 1.0.x, that same argument could be made for every other connector
> or storm component, and I am not sure it’s a good principle. I just think
> that it’s natural and beneficial to the community and the product that
> users keep upgrading (especially across backwards compatible versions),
> rather that keeping on patching things up.
>
> A few more opinions on storm-kafka-client
> - Can we identify the pros and cons of keeping Kafka 0.9 dependencies, and
> unless there are very strong reasons not to do so, simply upgrade to a more
> recent version.
> - Although storm-kafka-client was not labeled as beta, technically it was
> beta, and in practice it was used by users as beta. As far as I know, users
> that used it from the beginning were still exploring it, and I don’t think
> that until a lot of the improvements came in it made it to production. So
> Beta is a nice label to have behind a component that can cover for some
> necessary non backwards compatible changes. However, I don’t think it would
> have made a world of difference in practice for storm-kafka-client.
> - The Kaka community has at times made non backward compatible changes. If
> Kafka itself makes such changes, is it that concerning if
> storm-kafka-client has to make or or two such non backward compatible
> changes?
>
> Thanks,
> Hugo
>
> On Jan 25, 2018, at 9:47 AM, P. Taylor Goetz <ptgo...@gmail.com ptgo...@gmail.com>> wrote:
>
>
>
> On Jan 25, 2018, at 12:32 PM, Bobby Evans <ev...@oath.com.INVALID ev...@oath.com.INVALID>> wrote:
>
> To make this happen though someone is going to need to step up and take
> lead on this.
>
> We will need a plan on what to do (is it becoming a separate repo with a
> separate release cycle but still a part of the storm project?)
> Do we plan to spin it off to be an incubator project on its own?
>
> I don’t think it’s necessary to spin it off as a separate incubator
> project, that seems like overkill and would be weird from a community
> perspective.
>
> The path of least resistance might be a separate git repo, or just keeping
> it in the current Storm repo and decoupling it from the main build/release
> process so it can be independently released.
>
> -Taylor
>
>
>

Re: [VOTE] Release Apache Storm 1.1.2

[Jungtaek Lim]

I found blocker issue based on the contributor's report in 1.2.0 RC1, and
found the cause.

Filed https://issues.apache.org/jira/browse/STORM-2912. Will craft patches
for branches shortly.

-1 (binding)

- Jungtaek Lim (HeartSaVioR)

2018년 1월 25일 (목) 오전 2:36, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:

> This is a call to vote on releasing Apache Storm 1.1.2 (rc1)
>
> Full list of changes in this release:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.2-rc1/RELEASE_NOTES.html
>
> The tag/commit to be voted upon is v1.1.2:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=80b885ad20afa61b7c8d5c89eabe1f119df1da15;hb=279f16f2c17eafd0a0ab132bc42d7d3c64fdd3b2;a=tree;h=0071c67836d1fba0eb31e8f980d80459db14efb8;hb=f04b7a3a74580bde2681504313aa158b5956e7de
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.2-rc1/apache-storm-1.1.2-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.2-rc1/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1053
>
> Please vote on releasing this package as Apache Storm 1.1.2.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 1.1.2
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
>

Re: [VOTE] Release Apache Storm 1.0.6 (rc1)

[Jungtaek Lim]

Btw, I found blocker issue based on the contributor's report in 1.2.0 RC1,
and found the cause.

Filed https://issues.apache.org/jira/browse/STORM-2912. Will craft patches
for branches shortly.

-1 (binding)

- Jungtaek Lim (HeartSaVioR)

2018년 1월 25일 (목) 오후 12:43, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> IMHO, I have been wondering that it is still a good strategy to couple
> Storm core and connectors with one version. That has been making painful
> situations in storm-kafka-client, where it could be marked as beta and
> release often with free to break backward compatibility until it is fairly
> stable, but it can't be released often because it requires Storm itself to
> be released which requires huge bootstrap.
>
> 2018년 1월 25일 (목) 오후 12:37, Jungtaek Lim <kabh...@gmail.com>님이 작성:
>
>> Hi Erik,
>>
>> Thanks for bring up issue. Yes you're right that all the patches are
>> going through 1.x-branch (1.2.0), but not 1.1.x-branch because of huge
>> divergence. We have limited resource so I'm afraid to say folks to port
>> back issues with much pain of conflicts.
>>
>> Maybe we could make somewhat crazy but easiest decision: break backward
>> compatibility once again for storm-kafka-client (sad), and just copy the
>> code on 1.x-branch to 1.1.x-branch and also 1.0.x-branch. 1.0.x-branch was
>> support for Kafka 0.9 (odd strategy btw), but once Kafka published
>> 1.0.0, I'd rather not thinking about Kafka 0.9.
>>
>> To. PMCs
>> Makes sense? Need to initiate another discussion thread?
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>> 2018년 1월 25일 (목) 오전 10:48, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>>
>>> Erik,
>>>
>>> Open and honest opinions are always welcome. Even if they may come
>>> across as harsh (and yours did not). We value your opinion, and that of the
>>> mesos community.
>>>
>>> I’d like to get these issues resolved and would ask others to help where
>>> they can.
>>>
>>> You’ve been very forthcoming with information to help resolve this.
>>> Please don’t stop. The more information the better.
>>>
>>> Thanks again,
>>>
>>> -Taylor
>>>
>>>
>>>
>>>
>>>
>>> > On Jan 24, 2018, at 8:12 PM, Erik Weathers
>>> <eweath...@groupon.com.INVALID> wrote:
>>> >
>>> > Hi Taylor,
>>> >
>>> > I apologize that this objection is a bit long, incomplete, and late for
>>> > this release vote.  Also, I by no means intend this as an attack on the
>>> > good folks that develop and maintain storm-kafka-client.  That being
>>> said,
>>> > I'm uncomfortable with the situation of storm-kafka-client in the
>>> > 1.0.x-branch (and possibly 1.1.x-branch too) and I wonder if everyone
>>> is
>>> > aware of the situation.  There seem to be a number of important
>>> > storm-kafka-client changes that haven't been backported to the
>>> > 1.0.x-branch.  e.g.,
>>> >
>>> > (1) We discovered in storm-1.0.3 that a spout can get stuck forever if
>>> its
>>> > stored offsets fall behind the earliest available and
>>> > FirstPollOffsetStrategy is set to UNCOMMITTED_EARLIEST or
>>> > UNCOMMITTED_LATEST.  We believe this behavior to be fixed in newer
>>> > branches, but not in 1.0.x-branch.  The issue is here (note that the
>>> > fetchOffset doesn't get updated in the case that it was out of bounds
>>> for
>>> > the seek):
>>> > * https://github.com/apache/storm/blob/v1.0.6/external/
>>> > storm-kafka-client/src/main/java/org/apache/storm/kafka/
>>> > spout/KafkaSpout.java#L188-L192
>>> >
>>> > (2) storm-1.0.x is using kafka-clients-0.9.0.1, which isn't acceptable
>>> when
>>> > using Kafka 0.10 due to the performance impact on the Kafka cluster.  I
>>> > believe, perhaps naively, that we could use kafka-clients-0.10.x just
>>> fine
>>> > in storm-kafka-client-1.0.x, even when speaking to a Kafka 0.9 cluster.
>>> > (Obviously we'd have to fix any issues with the Kafka API usage from
>>> > storm-kafka-clients.)
>>> >
>>> >
>>> > Notably, this situation is especially problematic for storm-on-mesos,
>>> since
>>> > we *cannot* run anything newer than storm-1.0.x for the daemons
>>> (Nimbus,
>>> > Supervisor, Worker), because of a fundamental change that was made to
>>> the
>>> > s

Re: [VOTE] Release Apache Storm 1.2.0 (rc1)

[Jungtaek Lim]

I found the cause. Filed https://issues.apache.org/jira/browse/STORM-2912.
Will craft patches for branches shortly.

FYI, this issue affects all the RCs being opened now. I'll comment to other
RCs as well.

-Jungtaek Lim (HeartSaVioR)

2018년 1월 25일 (목) 오후 12:09, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> Alexandre,
>
> As Priyank stated we're seeing odd numbers in storm-starter. I also found
> the odd value but it was not 1.2.0 but RC for another version. Maybe I
> missed to check for 1.2.0 here.
> I'm planning to test all the open release candidates as well and test with
> various topologies, but it would be really helpful if you could provide
> information on your test topology to reproduce, like using multilang,
> asynchronous bolt, etc.
>
> Except metrics, other things are still OK.
>
> Anyway, changing my vote to -1 (binding).
>
> Thanks,
> Jungtaek Lim
>
> 2018년 1월 25일 (목) 오후 12:02, Priyank Shah <ps...@hortonworks.com>님이 작성:
>
>> Release looks good overall except for complete latency and capacity.
>>
>> Verified the md5, sha and asc
>> Unzipped and untared the source and binary releases.
>> Built from source and ran daemons from storm-dist binary release after
>> packing from the same.
>> Submitted FastWordCount and RollingTopWords
>> Played around with UI and tried different operations like Deactivate,
>> Activate, Kill, LogViewer, Search log, etc, . Everything works ok.
>>
>> However I noticed complete latency for FastWordCount to be more than 4
>> seconds which seems higher. Also,  capacity for RollingTopWords (was
>> verifying for Jungtaek) seemed to be very high.
>> Running 1.1.1 to compare the same numbers for both the topologies. Will
>> update with my results.
>>
>>
>> On 1/24/18, 12:03 PM, "Alexandre Vermeerbergen" <avermeerber...@gmail.com>
>> wrote:
>>
>> Hello Storm developers,
>>
>> One of my team member (Noureddine Chatti) found that the regression in
>> "Assigned Memory (MB)" columns for our topologies with Storm 1.2.0rc1,
>> where the 65 MB value is displayed regardless of our actual
>> topologies's memory setting:
>>
>> We define our workers's max memory heap using this option (as
>> documented in Hortonworks documentation here:
>>
>> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_storm-component-guide/content/config-storm-settings.html
>> )
>>
>> worker.childopts: "-Xmx2048m"
>>
>> Until Storm 1.1.0, this value is displayed in "Assigned Memory (MB)"
>> columns by Nimbus UI.
>> In Storm 1.2.0 rc1, the 65 MB value is displayed regardless of this
>> setting.
>>
>> He also found that using the following setting:
>>
>> topology.worker.max.heap.size.mb: 2048.
>>
>> then the displayed "Assigned Memory (MB)" columns by Nimbus UI are in
>> line with the value (i.e. they show 2048MB).
>>
>> Questions:
>> 1. This is change of behavior from Storm 1.1.0 to Storm 1.2.0rc1
>> international? any related JIRA who would help understanding the
>> rationale?
>> 2. Is the maximum heap size defined using worker.childopts actually
>> used to setup Worker's max heap size?
>> 3. What's the best practice for setting worker's memory consumption  :
>> is topology.worker.max.heap.size.mb now mandatory, or is the use of
>> -Xmx in worker.childopts still supported?
>> 4. Could be there other differences from Storm 1.1.0 and Storm
>> 1.2.0rc1 which could explain why we get very weird statistics in
>> Nimbus UI for Capacity & Latency for our topologies?
>>
>> Best regards,
>> Alexandre Vermeerbergen
>>
>>
>>
>>

Re: [VOTE] Release Apache Storm 1.0.6 (rc1)

[Jungtaek Lim]

IMHO, I have been wondering that it is still a good strategy to couple
Storm core and connectors with one version. That has been making painful
situations in storm-kafka-client, where it could be marked as beta and
release often with free to break backward compatibility until it is fairly
stable, but it can't be released often because it requires Storm itself to
be released which requires huge bootstrap.

2018년 1월 25일 (목) 오후 12:37, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> Hi Erik,
>
> Thanks for bring up issue. Yes you're right that all the patches are going
> through 1.x-branch (1.2.0), but not 1.1.x-branch because of huge
> divergence. We have limited resource so I'm afraid to say folks to port
> back issues with much pain of conflicts.
>
> Maybe we could make somewhat crazy but easiest decision: break backward
> compatibility once again for storm-kafka-client (sad), and just copy the
> code on 1.x-branch to 1.1.x-branch and also 1.0.x-branch. 1.0.x-branch was
> support for Kafka 0.9 (odd strategy btw), but once Kafka published
> 1.0.0, I'd rather not thinking about Kafka 0.9.
>
> To. PMCs
> Makes sense? Need to initiate another discussion thread?
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2018년 1월 25일 (목) 오전 10:48, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>
>> Erik,
>>
>> Open and honest opinions are always welcome. Even if they may come across
>> as harsh (and yours did not). We value your opinion, and that of the mesos
>> community.
>>
>> I’d like to get these issues resolved and would ask others to help where
>> they can.
>>
>> You’ve been very forthcoming with information to help resolve this.
>> Please don’t stop. The more information the better.
>>
>> Thanks again,
>>
>> -Taylor
>>
>>
>>
>>
>>
>> > On Jan 24, 2018, at 8:12 PM, Erik Weathers
>> <eweath...@groupon.com.INVALID> wrote:
>> >
>> > Hi Taylor,
>> >
>> > I apologize that this objection is a bit long, incomplete, and late for
>> > this release vote.  Also, I by no means intend this as an attack on the
>> > good folks that develop and maintain storm-kafka-client.  That being
>> said,
>> > I'm uncomfortable with the situation of storm-kafka-client in the
>> > 1.0.x-branch (and possibly 1.1.x-branch too) and I wonder if everyone is
>> > aware of the situation.  There seem to be a number of important
>> > storm-kafka-client changes that haven't been backported to the
>> > 1.0.x-branch.  e.g.,
>> >
>> > (1) We discovered in storm-1.0.3 that a spout can get stuck forever if
>> its
>> > stored offsets fall behind the earliest available and
>> > FirstPollOffsetStrategy is set to UNCOMMITTED_EARLIEST or
>> > UNCOMMITTED_LATEST.  We believe this behavior to be fixed in newer
>> > branches, but not in 1.0.x-branch.  The issue is here (note that the
>> > fetchOffset doesn't get updated in the case that it was out of bounds
>> for
>> > the seek):
>> > * https://github.com/apache/storm/blob/v1.0.6/external/
>> > storm-kafka-client/src/main/java/org/apache/storm/kafka/
>> > spout/KafkaSpout.java#L188-L192
>> >
>> > (2) storm-1.0.x is using kafka-clients-0.9.0.1, which isn't acceptable
>> when
>> > using Kafka 0.10 due to the performance impact on the Kafka cluster.  I
>> > believe, perhaps naively, that we could use kafka-clients-0.10.x just
>> fine
>> > in storm-kafka-client-1.0.x, even when speaking to a Kafka 0.9 cluster.
>> > (Obviously we'd have to fix any issues with the Kafka API usage from
>> > storm-kafka-clients.)
>> >
>> >
>> > Notably, this situation is especially problematic for storm-on-mesos,
>> since
>> > we *cannot* run anything newer than storm-1.0.x for the daemons (Nimbus,
>> > Supervisor, Worker), because of a fundamental change that was made to
>> the
>> > storm-core scheduling logic.
>> >
>> > I haven't yet spent the time to fully analyze the set of changes made to
>> > storm-kafka-client in the various active branches, so I apologize for a
>> > lack of links to existing PRs and Jira tickets.  I also need to file
>> > tickets for some of this stuff, as I believe issue (1) above was fixed
>> in
>> > newer branches as part of a large patch for another issue.
>> >
>> > I also suppose that this isn't super actionable, since it might not be
>> fair
>> > to hold back the entire storm release just for these issues.  However,
>> if
>> > we can get these issues fixed 

Re: [VOTE] Release Apache Storm 1.0.6 (rc1)

[Jungtaek Lim]

Hi Erik,

Thanks for bring up issue. Yes you're right that all the patches are going
through 1.x-branch (1.2.0), but not 1.1.x-branch because of huge
divergence. We have limited resource so I'm afraid to say folks to port
back issues with much pain of conflicts.

Maybe we could make somewhat crazy but easiest decision: break backward
compatibility once again for storm-kafka-client (sad), and just copy the
code on 1.x-branch to 1.1.x-branch and also 1.0.x-branch. 1.0.x-branch was
support for Kafka 0.9 (odd strategy btw), but once Kafka published
1.0.0, I'd rather not thinking about Kafka 0.9.

To. PMCs
Makes sense? Need to initiate another discussion thread?

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 1월 25일 (목) 오전 10:48, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:

> Erik,
>
> Open and honest opinions are always welcome. Even if they may come across
> as harsh (and yours did not). We value your opinion, and that of the mesos
> community.
>
> I’d like to get these issues resolved and would ask others to help where
> they can.
>
> You’ve been very forthcoming with information to help resolve this. Please
> don’t stop. The more information the better.
>
> Thanks again,
>
> -Taylor
>
>
>
>
>
> > On Jan 24, 2018, at 8:12 PM, Erik Weathers <eweath...@groupon.com.INVALID>
> wrote:
> >
> > Hi Taylor,
> >
> > I apologize that this objection is a bit long, incomplete, and late for
> > this release vote.  Also, I by no means intend this as an attack on the
> > good folks that develop and maintain storm-kafka-client.  That being
> said,
> > I'm uncomfortable with the situation of storm-kafka-client in the
> > 1.0.x-branch (and possibly 1.1.x-branch too) and I wonder if everyone is
> > aware of the situation.  There seem to be a number of important
> > storm-kafka-client changes that haven't been backported to the
> > 1.0.x-branch.  e.g.,
> >
> > (1) We discovered in storm-1.0.3 that a spout can get stuck forever if
> its
> > stored offsets fall behind the earliest available and
> > FirstPollOffsetStrategy is set to UNCOMMITTED_EARLIEST or
> > UNCOMMITTED_LATEST.  We believe this behavior to be fixed in newer
> > branches, but not in 1.0.x-branch.  The issue is here (note that the
> > fetchOffset doesn't get updated in the case that it was out of bounds for
> > the seek):
> > * https://github.com/apache/storm/blob/v1.0.6/external/
> > storm-kafka-client/src/main/java/org/apache/storm/kafka/
> > spout/KafkaSpout.java#L188-L192
> >
> > (2) storm-1.0.x is using kafka-clients-0.9.0.1, which isn't acceptable
> when
> > using Kafka 0.10 due to the performance impact on the Kafka cluster.  I
> > believe, perhaps naively, that we could use kafka-clients-0.10.x just
> fine
> > in storm-kafka-client-1.0.x, even when speaking to a Kafka 0.9 cluster.
> > (Obviously we'd have to fix any issues with the Kafka API usage from
> > storm-kafka-clients.)
> >
> >
> > Notably, this situation is especially problematic for storm-on-mesos,
> since
> > we *cannot* run anything newer than storm-1.0.x for the daemons (Nimbus,
> > Supervisor, Worker), because of a fundamental change that was made to the
> > storm-core scheduling logic.
> >
> > I haven't yet spent the time to fully analyze the set of changes made to
> > storm-kafka-client in the various active branches, so I apologize for a
> > lack of links to existing PRs and Jira tickets.  I also need to file
> > tickets for some of this stuff, as I believe issue (1) above was fixed in
> > newer branches as part of a large patch for another issue.
> >
> > I also suppose that this isn't super actionable, since it might not be
> fair
> > to hold back the entire storm release just for these issues.  However, if
> > we can get these issues fixed in 1.0.x-branch I hope we can cut the 1.0.7
> > release soon thereafter.
> >
> > Thanks!
> >
> > - Erik
> >
> >> On Wed, Jan 24, 2018 at 10:41 AM, P. Taylor Goetz <ptgo...@gmail.com>
> wrote:
> >>
> >> This is a call to vote on releasing Apache Storm 1.0.6 (rc1)
> >>
> >> Full list of changes in this release:
> >>
> >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.
> >> 0.6-rc1/RELEASE_NOTES.html
> >>
> >> The tag/commit to be voted upon is v1.0.6:
> >>
> >> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h
> >> =24a421e34a71353dc6c750b1f026d06df8ead3f2;hb=bce45993f8622e4
> >> d3e9ccba96cc78e4ef76e48ae
> >>
> >> The source archive being voted upon can be found here:
> >>

Re: [VOTE] Release Apache Storm 1.2.0 (rc1)

[Jungtaek Lim]

Alexandre,

As Priyank stated we're seeing odd numbers in storm-starter. I also found
the odd value but it was not 1.2.0 but RC for another version. Maybe I
missed to check for 1.2.0 here.
I'm planning to test all the open release candidates as well and test with
various topologies, but it would be really helpful if you could provide
information on your test topology to reproduce, like using multilang,
asynchronous bolt, etc.

Except metrics, other things are still OK.

Anyway, changing my vote to -1 (binding).

Thanks,
Jungtaek Lim

2018년 1월 25일 (목) 오후 12:02, Priyank Shah <ps...@hortonworks.com>님이 작성:

> Release looks good overall except for complete latency and capacity.
>
> Verified the md5, sha and asc
> Unzipped and untared the source and binary releases.
> Built from source and ran daemons from storm-dist binary release after
> packing from the same.
> Submitted FastWordCount and RollingTopWords
> Played around with UI and tried different operations like Deactivate,
> Activate, Kill, LogViewer, Search log, etc, . Everything works ok.
>
> However I noticed complete latency for FastWordCount to be more than 4
> seconds which seems higher. Also,  capacity for RollingTopWords (was
> verifying for Jungtaek) seemed to be very high.
> Running 1.1.1 to compare the same numbers for both the topologies. Will
> update with my results.
>
>
> On 1/24/18, 12:03 PM, "Alexandre Vermeerbergen" <avermeerber...@gmail.com>
> wrote:
>
> Hello Storm developers,
>
> One of my team member (Noureddine Chatti) found that the regression in
> "Assigned Memory (MB)" columns for our topologies with Storm 1.2.0rc1,
> where the 65 MB value is displayed regardless of our actual
> topologies's memory setting:
>
> We define our workers's max memory heap using this option (as
> documented in Hortonworks documentation here:
>
> https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/bk_storm-component-guide/content/config-storm-settings.html
> )
>
> worker.childopts: "-Xmx2048m"
>
> Until Storm 1.1.0, this value is displayed in "Assigned Memory (MB)"
> columns by Nimbus UI.
> In Storm 1.2.0 rc1, the 65 MB value is displayed regardless of this
> setting.
>
> He also found that using the following setting:
>
> topology.worker.max.heap.size.mb: 2048.
>
> then the displayed "Assigned Memory (MB)" columns by Nimbus UI are in
> line with the value (i.e. they show 2048MB).
>
> Questions:
> 1. This is change of behavior from Storm 1.1.0 to Storm 1.2.0rc1
> international? any related JIRA who would help understanding the
> rationale?
> 2. Is the maximum heap size defined using worker.childopts actually
> used to setup Worker's max heap size?
> 3. What's the best practice for setting worker's memory consumption  :
> is topology.worker.max.heap.size.mb now mandatory, or is the use of
> -Xmx in worker.childopts still supported?
> 4. Could be there other differences from Storm 1.1.0 and Storm
> 1.2.0rc1 which could explain why we get very weird statistics in
> Nimbus UI for Capacity & Latency for our topologies?
>
> Best regards,
> Alexandre Vermeerbergen
>
>
>
>

Re: [VOTE] Release Apache Storm 1.2.0 (rc1)

[Jungtaek Lim]

Let's back to verify the release and vote.

+1 (binding)

> source

- verify file (signature, MD5, SHA)
-- source, tar.gz : OK
-- source, zip : OK

- extract file
-- source, tar.gz : OK
-- source, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- build source with JDK 7
-- source, tar.gz : integration-test failed, others are OK

- build source dist
-- source, tar.gz : OK

- build binary dist
-- source, tar.gz : OK

> binary

- verify file (signature, MD5, SHA)
-- binary, tar.gz : OK
-- binary, zip : OK

- extract file
-- binary, tar.gz : OK
-- binary, zip : OK

- diff-ing extracted files between tar.gz and zip : OK

- launch daemons : OK

- run RollingTopWords (local) : OK

- run RollingTopWords (remote) : OK
  - activate / deactivate / rebalance / kill : OK
  - logviewer (worker dir, daemon dir) : OK
  - change log level : OK
  - thread dump, heap dump, restart worker : OK
  - log search : OK

I don't see odd numbers while testing, but I don't have stage/production
level of cluster/use case, hence someone might be able to see the behavior
what Alexandre encountered.

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 1월 24일 (수) 오전 10:19, Jungtaek Lim <kabh...@gmail.com>님이 작성:

> Alexandre,
>
> Please file an issue with screenshot and reproducible step (if only
> possible). It would be very appreciated if you could spend time to dive
> into the codebase and find the cause, and fix and submit a patch (only when
> you could get it).
> Open source community can't live without contributors. I think reporting
> issue itself is great contribution, but I feel we don't have enough code
> contributors who could help driving the community.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2018년 1월 24일 (수) 오전 9:57, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>
>> Yes, that’s the same error I got, and I think we both just shaved the
>> same yak. ;)
>>
>> I imagine infra is enforcing TLS > 1.0 now.
>>
>> -Taylor
>>
>> > On Jan 23, 2018, at 7:46 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
>> >
>> > Stig, the script doesn't also work for me, but that's not because of
>> script
>> > or jira module error.
>> > I've encountered TLSV1_ALERT_PROTOCOL_VERSION error and my python2.7 is
>> > unfortunately coupled with OpenSSL 0.9.8zh which doesn't support
>> TLSv1.2.
>> > My python3.6 is coupled with OpenSSL 1.0.2l but the script is not
>> > compatible with python 3. Maybe I need to modify the script to be
>> > compatible with python3.6.
>> >
>> > cc. to Taylor, assuming that we are getting same error.
>> >
>> > - Jungtaek Lim (HeartSaVioR)
>> >
>> > 2018년 1월 24일 (수) 오전 8:21, Stig Rohde Døssing <stigdoess...@gmail.com>님이
>> 작성:
>> >
>> >> Taylor,
>> >>
>> >> The release notes script appears to work fine for me. There are a
>> couple of
>> >> issues with fix version 1.2.0 that are not resolved, which we should
>> fix.
>> >> Note that 2710 is the release 1.2.0 epic, we might want to not mark
>> that
>> >> with a fix version so it isn't included in the release notes.
>> >>
>> >> dev-tools/release_notes.py 1.2.0
>> >> The release is not completed since unresolved issues or improperly
>> resolved
>> >> issues were found still tagged with this release as the fix version:
>> >> Unresolved issue:  STORM-2904 None
>> >> https://issues.apache.org/jira/browse/STORM-2904
>> >> Unresolved issue:  STORM-2710 None
>> >> https://issues.apache.org/jira/browse/STORM-2710
>> >> Unresolved issue:  STORM-2153 None
>> >> https://issues.apache.org/jira/browse/STORM-2153
>> >>
>> >> If I ignore the unresolved issues check, I get the expected release
>> notes
>> >>
>> >> dev-tools/release_notes.py 1.2.0 > release-1.2.0.html produces
>> >> https://pste.eu/p/ZvbF.html
>> >>
>> >> 2018-01-24 0:09 GMT+01:00 Alexandre Vermeerbergen <
>> >> avermeerber...@gmail.com>
>> >> :
>> >>
>> >>> Hello,
>> >>>
>> >>> I'm afraid I my vote in 1.2.0 RC1 is a -1:
>> >>>
>> >>> Indeed metrics displayed in Storm UI from 1.2.0 RC1 are obviously
>> wrong.
>> >>>
>> >>> See for example attached picture showing "Assigned Mem (MB)" for one
>> >>> of our topologies:
>> >>> -  On the left hand side we have Storm 1.1.0 showing 2112 MB on each
>> >>> host, which sounds "normal" to us (in line with what we had with
>> >>> previous Storm 1.0.3 version)
>> >>> -  On the right hand side we have Storm 1.2.0 RC1 showing 65 MB on
>> >>> each host, which sound completely wrong !
>> >>>
>> >>> And I have similar concerns on the statistics on bolts, for example on
>> >>> a bolt of our topology in charge of writing logs into HBase, we have:
>> >>>
>> >>> With Storm 1.1.0, capacity (last 10 min): 0.090 ; Execute Latency
>> (ms):
>> >>> 0.029
>> >>> With Storm 1.2.0, capacity (last 10 min): 438.956 ; Execute Latency
>> >>> (ms): 197.840
>> >>>
>> >>> Am I the only one to find weird numbers in Storm UI 1.2.0 ?
>> >>>
>> >>> Best regards,
>> >>> Alexandre Vermeerbergen
>> >>>
>> >>
>>
>

Re: [VOTE] Release Apache Storm 1.2.0 (rc1)

[Jungtaek Lim]

Alexandre,

Please file an issue with screenshot and reproducible step (if only
possible). It would be very appreciated if you could spend time to dive
into the codebase and find the cause, and fix and submit a patch (only when
you could get it).
Open source community can't live without contributors. I think reporting
issue itself is great contribution, but I feel we don't have enough code
contributors who could help driving the community.

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 1월 24일 (수) 오전 9:57, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:

> Yes, that’s the same error I got, and I think we both just shaved the same
> yak. ;)
>
> I imagine infra is enforcing TLS > 1.0 now.
>
> -Taylor
>
> > On Jan 23, 2018, at 7:46 PM, Jungtaek Lim <kabh...@gmail.com> wrote:
> >
> > Stig, the script doesn't also work for me, but that's not because of
> script
> > or jira module error.
> > I've encountered TLSV1_ALERT_PROTOCOL_VERSION error and my python2.7 is
> > unfortunately coupled with OpenSSL 0.9.8zh which doesn't support TLSv1.2.
> > My python3.6 is coupled with OpenSSL 1.0.2l but the script is not
> > compatible with python 3. Maybe I need to modify the script to be
> > compatible with python3.6.
> >
> > cc. to Taylor, assuming that we are getting same error.
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> > 2018년 1월 24일 (수) 오전 8:21, Stig Rohde Døssing <stigdoess...@gmail.com>님이
> 작성:
> >
> >> Taylor,
> >>
> >> The release notes script appears to work fine for me. There are a
> couple of
> >> issues with fix version 1.2.0 that are not resolved, which we should
> fix.
> >> Note that 2710 is the release 1.2.0 epic, we might want to not mark that
> >> with a fix version so it isn't included in the release notes.
> >>
> >> dev-tools/release_notes.py 1.2.0
> >> The release is not completed since unresolved issues or improperly
> resolved
> >> issues were found still tagged with this release as the fix version:
> >> Unresolved issue:  STORM-2904 None
> >> https://issues.apache.org/jira/browse/STORM-2904
> >> Unresolved issue:  STORM-2710 None
> >> https://issues.apache.org/jira/browse/STORM-2710
> >> Unresolved issue:  STORM-2153 None
> >> https://issues.apache.org/jira/browse/STORM-2153
> >>
> >> If I ignore the unresolved issues check, I get the expected release
> notes
> >>
> >> dev-tools/release_notes.py 1.2.0 > release-1.2.0.html produces
> >> https://pste.eu/p/ZvbF.html
> >>
> >> 2018-01-24 0:09 GMT+01:00 Alexandre Vermeerbergen <
> >> avermeerber...@gmail.com>
> >> :
> >>
> >>> Hello,
> >>>
> >>> I'm afraid I my vote in 1.2.0 RC1 is a -1:
> >>>
> >>> Indeed metrics displayed in Storm UI from 1.2.0 RC1 are obviously
> wrong.
> >>>
> >>> See for example attached picture showing "Assigned Mem (MB)" for one
> >>> of our topologies:
> >>> -  On the left hand side we have Storm 1.1.0 showing 2112 MB on each
> >>> host, which sounds "normal" to us (in line with what we had with
> >>> previous Storm 1.0.3 version)
> >>> -  On the right hand side we have Storm 1.2.0 RC1 showing 65 MB on
> >>> each host, which sound completely wrong !
> >>>
> >>> And I have similar concerns on the statistics on bolts, for example on
> >>> a bolt of our topology in charge of writing logs into HBase, we have:
> >>>
> >>> With Storm 1.1.0, capacity (last 10 min): 0.090 ; Execute Latency (ms):
> >>> 0.029
> >>> With Storm 1.2.0, capacity (last 10 min): 438.956 ; Execute Latency
> >>> (ms): 197.840
> >>>
> >>> Am I the only one to find weird numbers in Storm UI 1.2.0 ?
> >>>
> >>> Best regards,
> >>> Alexandre Vermeerbergen
> >>>
> >>
>

Re: [VOTE] Release Apache Storm 1.2.0 (rc1)

[Jungtaek Lim]

Stig, the script doesn't also work for me, but that's not because of script
or jira module error.
I've encountered TLSV1_ALERT_PROTOCOL_VERSION error and my python2.7 is
unfortunately coupled with OpenSSL 0.9.8zh which doesn't support TLSv1.2.
My python3.6 is coupled with OpenSSL 1.0.2l but the script is not
compatible with python 3. Maybe I need to modify the script to be
compatible with python3.6.

cc. to Taylor, assuming that we are getting same error.

- Jungtaek Lim (HeartSaVioR)

2018년 1월 24일 (수) 오전 8:21, Stig Rohde Døssing <stigdoess...@gmail.com>님이 작성:

> Taylor,
>
> The release notes script appears to work fine for me. There are a couple of
> issues with fix version 1.2.0 that are not resolved, which we should fix.
> Note that 2710 is the release 1.2.0 epic, we might want to not mark that
> with a fix version so it isn't included in the release notes.
>
> dev-tools/release_notes.py 1.2.0
> The release is not completed since unresolved issues or improperly resolved
> issues were found still tagged with this release as the fix version:
> Unresolved issue:  STORM-2904 None
> https://issues.apache.org/jira/browse/STORM-2904
> Unresolved issue:  STORM-2710 None
> https://issues.apache.org/jira/browse/STORM-2710
> Unresolved issue:  STORM-2153 None
> https://issues.apache.org/jira/browse/STORM-2153
>
> If I ignore the unresolved issues check, I get the expected release notes
>
> dev-tools/release_notes.py 1.2.0 > release-1.2.0.html produces
> https://pste.eu/p/ZvbF.html
>
> 2018-01-24 0:09 GMT+01:00 Alexandre Vermeerbergen <
> avermeerber...@gmail.com>
> :
>
> > Hello,
> >
> > I'm afraid I my vote in 1.2.0 RC1 is a -1:
> >
> > Indeed metrics displayed in Storm UI from 1.2.0 RC1 are obviously wrong.
> >
> > See for example attached picture showing "Assigned Mem (MB)" for one
> > of our topologies:
> > -  On the left hand side we have Storm 1.1.0 showing 2112 MB on each
> > host, which sounds "normal" to us (in line with what we had with
> > previous Storm 1.0.3 version)
> > -  On the right hand side we have Storm 1.2.0 RC1 showing 65 MB on
> > each host, which sound completely wrong !
> >
> > And I have similar concerns on the statistics on bolts, for example on
> > a bolt of our topology in charge of writing logs into HBase, we have:
> >
> > With Storm 1.1.0, capacity (last 10 min): 0.090 ; Execute Latency (ms):
> > 0.029
> > With Storm 1.2.0, capacity (last 10 min): 438.956 ; Execute Latency
> > (ms): 197.840
> >
> > Am I the only one to find weird numbers in Storm UI 1.2.0 ?
> >
> > Best regards,
> > Alexandre Vermeerbergen
> >
>