+1
Sorry .. it was a false alarm. It come down to a config issue. The default
configs in Storm 2 are tuned for best latency.
On bumping up the transfer buffer & batch settings for high throughput :
- topology.transfer.batch.size=5k (default=1) and -
topology.transfer.buffer.size=20k (default=1k)
Was able to obtain a very solid ~4 mill/sec inter worker throughput.
FYI: Does not matter now, but i noticed that I accidentally quoted throughput
of 1.6mill/sec but it was actually only ~350 k/sec with default transfer buffer
settings .. which was alarmingly low.
On Saturday, January 19, 2019, 10:45:28 PM PST, Jungtaek Lim
<[email protected]> wrote:
Hello all,
+1 (binding)
> source
- verify file (MD5, SHA)
-- source, tar.gz : OK
-- source, zip : OK
- extract file
-- source, tar.gz : OK
-- source, zip : OK
- diff-ing extracted files between tar.gz and zip : OK
- build source with JDK 8 (-Pall-tests && -Pexternals)
-- source, tar.gz : OK
- build source dist
-- source, tar.gz : OK
- build binary dist
-- source, tar.gz : OK
> binary
- verify file (MD5, SHA)
-- binary, tar.gz : OK
-- binary, zip : OK
- extract file
-- binary, tar.gz : OK
-- binary, zip : OK
- diff-ing extracted files between tar.gz and zip : OK
- launch daemons : OK
- run ExclamationTopology (local) : OK
- run RollingTopWords (remote) : OK
- activate / deactivate / rebalance / kill : OK
- logviewer (worker dir, daemon dir) : OK
- change log level : OK
- thread dump, heap dump, restart worker : OK
- log search : OK
I don't think perf. degrade is a blocker for this release, since we are
still providing far better performance then 1.x. As Roshan is planning to
address this, that could be fixed in 2.0.x hopefully.
Thanks,
Jungtaek Lim (heartSaVioR)
2019년 1월 19일 (토) 오전 5:23, Roshan Naik <[email protected]>님이 작성:
> Was verifying the performance on 2.0 and noticed that the inter worker
> messaging has fallen off a cliff .... With ConstSpoutNullBoltTopo (producer
> batch size=1k, ackers=0, workers=2) it was ~3.2mill/sec down and now its
> down to ~1.6mill/sec. Something has impacted this recently. Will try to
> narrow down the issue by tomorrow hopefully. Single worker numbers look
> good.
> -roshan
> On Friday, January 11, 2019, 2:46:14 AM PST, Stig Rohde Døssing <
> [email protected]> wrote:
>
> Thanks Arun,
>
> I don't have an opinion on it, just wanted to make sure that it wasn't an
> oversight.
>
> +1
>
> Set up Storm from the binary zip file
> Built storm-kafka-client-examples from the zip while pointing Maven at the
> Nexus repo
> Ran one of the example Kafka topologies with a single-node cluster against
> a local Kafka. Verified there were no errors in the logs, clicked around in
> UI.
>
> Den tor. 10. jan. 2019 kl. 22.25 skrev Arun Mahadevan <[email protected]>:
>
> > This is for users to use the "auto credentials" mechanism (delegation
> > tokens) with HDFS/Hive/Hbase.
> >
> > We have been shipping it since 1.x (I think since 1.2.0 release) so that
> > users can just add that directory to class path rather than building it
> > separately to get the right dependencies. We could consider removing it
> > from the main binary and ship it separately but it will need changes to
> the
> > build, release and documentation and users will need to download and
> > install it separately.
> >
> >
> > Thanks,
> > Arun
> >
> > On Thu, 10 Jan 2019 at 10:28, Stig Rohde Døssing <[email protected]
> >
> > wrote:
> >
> > > I think this was remarked on by Roshan in the last RC, but the binary
> > > distribution has become significantly larger since 1.x. It looks like
> > this
> > > is down to storm-autocreds not being added to the exclusion list in
> > > storm-dist/binary/final-package/src/main/assembly/binary.xml.
> > >
> > > Since the module isn't excluded, external/storm-autocreds contains the
> > > module jar, plus all dependency jars. Is this an accident, or do we
> want
> > to
> > > include these jars in the distribution?
> > >
> > > Den ons. 9. jan. 2019 kl. 19.48 skrev Ethan Li <
> > [email protected]
> > > >:
> > >
> > > > +1
> > > >
> > > > - Built from the src, ran all the unit tests and integration tests.
> > > > - Set up a single-node cluster and submit ThroughputVsLatency
> topology.
> > > > - Checked the UI.
> > > > They look good.
> > > >
> > > > Thanks
> > > > Ethan
> > > >
> > > > > On Jan 9, 2019, at 8:48 AM, Bobby Evans <[email protected]> wrote:
> > > > >
> > > > > +1 built from the git tag. Ran all of the unit tests and ran some
> > > manual
> > > > > tests they all passed.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Bobby
> > > > >
> > > > > On Tue, Jan 8, 2019 at 6:30 PM Xin Wang <[email protected]>
> > > wrote:
> > > > >
> > > > >> +1
> > > > >>
> > > > >> Built it and ran all of the tests. Everything passed.
> > > > >>
> > > > >> -Xin
> > > > >>
> > > > >> Kishorkumar Patil <[email protected]> 于2019年1月9日周三
> 上午5:08写道：
> > > > >>
> > > > >>> +1
> > > > >>>
> > > > >>> - built from source code and deployment works.
> > > > >>> - Ran some of the tests for UI, DRPC, ThroughputVsLatency
> > > > >>> - Validated UI bugs reported in the recent past are fixed in
> this
> > > > >> version
> > > > >>>
> > > > >>> -Kishor
> > > > >>>
> > > > >>>
> > > > >>> On Tue, Jan 8, 2019 at 2:29 PM Arun Mahadevan <[email protected]>
> > > > wrote:
> > > > >>>
> > > > >>>> +1
> > > > >>>>
> > > > >>>> - Downloaded the binaries and validated signatures.
> > > > >>>> - Deployed the binaries, ran some sample topologies and checked
> > the
> > > > UI.
> > > > >>>> - Ran top level build using the source zip.
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>> Arun
> > > > >>>>
> > > > >>>>
> > > > >>>> On Tue, 8 Jan 2019 at 11:03, P. Taylor Goetz <[email protected]
> >
> > > > >> wrote:
> > > > >>>>
> > > > >>>>> This is a call to vote on releasing Apache Storm 2.0.0 (rc4)
> > > > >>>>>
> > > > >>>>> Full list of changes in this release:
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc4/RELEASE_NOTES.html
> > > > >>>>>
> > > > >>>>> The tag/commit to be voted upon is v2.0.0:
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=1eece73e8c9ed7f41d2f20f727bc7f644c499360;hb=ddee8decac57d1a4a0aa23cc76066609a2abc8d2
> > > > >>>>>
> > > > >>>>> The source archive being voted upon can be found here:
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc4/apache-storm-2.0.0-src.tar.gz
> > > > >>>>>
> > > > >>>>> Other release files, signatures and digests can be found here:
> > > > >>>>>
> > > > >>>>>
> > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.0.0-rc4/
> > > > >>>>>
> > > > >>>>> The release artifacts are signed with the following key:
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > > > >>>>>
> > > > >>>>> The Nexus staging repository for this release is:
> > > > >>>>>
> > > > >>>>>
> > > > >>
> > > https://repository.apache.org/content/repositories/orgapachestorm-1073
> > > > >>>>>
> > > > >>>>> Please vote on releasing this package as Apache Storm 2.0.0.
> > > > >>>>>
> > > > >>>>> When voting, please list the actions taken to verify the
> release.
> > > > >>>>>
> > > > >>>>> This vote will be open for at least 72 hours.
> > > > >>>>>
> > > > >>>>> [ ] +1 Release this package as Apache Storm 2.0.0
> > > > >>>>> [ ] 0 No opinion
> > > > >>>>> [ ] -1 Do not release this package because...
> > > > >>>>>
> > > > >>>>> Thanks to everyone who contributed to this release.
> > > > >>>>>
> > > > >>>>> -Taylor
> > > > >>>>
> > > > >>>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Thanks,
> > > > >> Xin
> > > > >>
> > > >
> > > >
> > >
> >
