Re: [DISCUSS] Accept JW Player SQE Code Donation

[Aaron Niskodé-Dossett]

+1 (binding)
On Thu, Sep 8, 2016 at 10:51 PM Satish Duggana 
wrote:

> +1 (binding) to accept code SQE code donation.
>
> We can have discussions later on how SQE features can be added in
> storm-sql.
>
> Thanks,
> Satish.
>
>
> On Fri, Sep 9, 2016 at 7:00 AM, Jungtaek Lim  wrote:
>
> > +1 (binding)
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> > On Friday, September 9, 2016, P. Taylor Goetz  wrote:
> >
> > > Following an earlier discussion thread, I’d like to start of VOTE on
> > > whether to accept the SQE donation from JW Player.
> > >
> > > The codebase being donated can be found here [1].
> > >
> > > [ ] +1 Accept the code donation.
> > > [ ] 0 No opinion
> > > [ ] -1 Do not accept the code donation because…
> > >
> > > Everyone is encouraged to vote. PMC member votes are binding.
> > >
> > > -Taylor
> > >
> > > [1] https://github.com/jwplayer/SQE
> > >
> >
> >
> > --
> > Name : Jungtaek Lim
> > Blog : http://medium.com/@heartsavior
> > Twitter : http://twitter.com/heartsavior
> > LinkedIn : http://www.linkedin.com/in/heartsavior
> >
>

[DISCUSS] ElasticSearch 2.x support

[Aaron Niskodé-Dossett]

Hi all,

I started a a discussion about this a while ago, but didn't take it to a
conclusion (my $realjob, etc., etc.).

There are multiple PRs open to provide an Elastic Search 2.x bolt to the
Storm project.  There are two different approaches:

1. Add side-by-side support for 2.x. Example:
https://github.com/apache/storm/pull/1337 (*FULL DISCLOSURE*: this is my
own PR). [I also have some functionality enhancements in this PR, but
that's not relevant to this discussion.]

2. Upgrade existing bolt. Example, https://github.com/apache/storm/pull/1396

The drawback to approach 1 is that it duplicates a lot of code.  The
drawback to approach 2 is that it drops support for ES 1.x.

ES 2.X has been out for a while and if we are serious about supporting it,
we need to have a way to write to ES 2.X.

I believe approach number 1 is ideal (again, it's my own PR) and possibly
deprecating the existing 1.X bolt.

I'd love to hear thoughts from others!

Re: [DISCUSS] ElasticSearch 2.x support

[Aaron Niskodé-Dossett]

Thanks everyone. Could one or more committers +1 the PR that would support
both versions?

https://github.com/apache/storm/pull/1337
On Mon, Sep 19, 2016 at 10:52 AM Satish Duggana <satish.dugg...@gmail.com>
wrote:

> Agree with Bobby, +1 for supporting both versions till EOL and findout how
> many users are really using 1.7.x.
>
> ~Satish.
>
>
> On Mon, Sep 19, 2016 at 7:50 PM, Bobby Evans <ev...@yahoo-inc.com.invalid>
> wrote:
>
> > I am +1 for two modules until EOL.  Jan 2017. - Bobby
> >
> > On Saturday, September 17, 2016 4:19 AM, Jungtaek Lim <
> > kabh...@gmail.com> wrote:
> >
> >
> >  According to the link, last version line of ES1 (1.7.x) will be live to
> > Jan
> > 2017. We need to keep two versions at least EOL of that.
> > I wouldn't mind having two modules and also wouldn't mind having
> duplicate
> > codes, but it would be better if we can extract common parts to parent
> > module.
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> > 2016년 9월 16일 (금) 오후 10:03, Aaron Niskodé-Dossett <doss...@gmail.com>님이
> 작성:
> >
> > > ES1 is close to end of life in terms of commercial support from
> Elastic,
> > > but not quite there (https://www.elastic.co/support/eol).
> Unfortunately
> > > the ES1 and ES2 clients won't interoperate with their opposite
> versions.
> > >
> > > Given that, I take it you would support having ES1 and ES2 bolts
> packaged
> > > separately?  This would somewhat like how we have storm-kafka and
> > > storm-kafka-client for different Kafka versions.
> > >
> > > Thanks! -Aaron
> > >
> > > On Thu, Sep 15, 2016 at 9:12 AM Bobby Evans
> <ev...@yahoo-inc.com.invalid
> > >
> > > wrote:
> > >
> > > > I personally don't use ES as part of my storm work, so I don't
> > > necessarily
> > > > feel qualified to answer this.  In general though I really do like to
> > see
> > > > storm come with batteries included.  If ES1 is not end of life, and
> > there
> > > > is a community of people who want to continue using it/supporting
> it, I
> > > > would say lets continue to do so.  If that is not true, or if ES
> > offers a
> > > > backwards compatible client that could sway things for me to say lets
> > > just
> > > > go forward with ES2. - Bobby
> > > >
> > > >On Wednesday, September 14, 2016 2:47 PM, Aaron Niskodé-Dossett <
> > > > doss...@gmail.com> wrote:
> > > >
> > > >
> > > >  Hi all,
> > > >
> > > > I started a a discussion about this a while ago, but didn't take it
> to
> > a
> > > > conclusion (my $realjob, etc., etc.).
> > > >
> > > > There are multiple PRs open to provide an Elastic Search 2.x bolt to
> > the
> > > > Storm project.  There are two different approaches:
> > > >
> > > > 1. Add side-by-side support for 2.x. Example:
> > > > https://github.com/apache/storm/pull/1337 (*FULL DISCLOSURE*: this
> is
> > my
> > > > own PR). [I also have some functionality enhancements in this PR, but
> > > > that's not relevant to this discussion.]
> > > >
> > > > 2. Upgrade existing bolt. Example,
> > > > https://github.com/apache/storm/pull/1396
> > > >
> > > > The drawback to approach 1 is that it duplicates a lot of code.  The
> > > > drawback to approach 2 is that it drops support for ES 1.x.
> > > >
> > > > ES 2.X has been out for a while and if we are serious about
> supporting
> > > it,
> > > > we need to have a way to write to ES 2.X.
> > > >
> > > > I believe approach number 1 is ideal (again, it's my own PR) and
> > possibly
> > > > deprecating the existing 1.X bolt.
> > > >
> > > > I'd love to hear thoughts from others!
> > > >
> > > >
> > > >
> > >
> >
> >
> >
>

Re: [DISCUSS] Provision for dead-letter topic in storm

[Aaron Niskodé-Dossett]

I like the idea, especially if it can be implemented as generically as
possible. Ideally we could "dead letter" both the original tuple and the
tuple that itself failed. Intervening transformations could have changed
the original tuple. I realize that's adds a lot of complexity to your idea
and may not be feasible.
On Tue, Sep 20, 2016 at 1:15 AM S G  wrote:

> Hi,
>
> I want to gather some thoughts on a suggestion to provide a dead-letter
> functionality common to all spouts/bolts.
>
> Currently, if any spout / bolt reports a failure, it is retried by the
> spout.
> For a single bolt-failure in a large ADG, this retry logic can cause
> several perfectly successful component to replay and yet the Tuple could
> fail exactly at the same bolt on retry.
>
> This is fine usually (if the failure was temporary, say due to a network
> glitch) but sometimes, the message is bad enough such that it should not be
> retried but at the same time important enough that its failure should not
> be ignored.
>
> Example: ElasticSearch-bolt receiving bytes from Kafka-Spout.
>
> Most of the times, it is able to deserialize the bytes correctly but
> sometimes a badly formatted message fails to deserialize. For such cases,
> neither Kafka-Spout should retry nor ES-bolt should report a success. It
> should however be reported to the user somehow that a badly serialized
> message entered the stream.
>
> For cases like temporary network glitch, the Tuple should be retried.
>
> So the proposal is to implement a dead-letter topic as:
>
> 1) Add a new method *failWithoutRetry(Tuple, Exception)* in the collector.
> Bolts will begin using it once its available for use.
>
> 2) Provide the ability to *configure a dead-letter data-store in the
> spout* for
> failed messages reported by #1 above.
>
>
> The configurable data-store should support kafka, solr and redis to
> begin-with (Plus the option to implement one's own and dropping a jar file
> in the classpath).
>
> Such a feature should benefit all the spouts as:
>
> 1) Topologies will not block replaying the same doomed-to-fail tuples.
> 2) Users can set alerts on dead-letters and find out easily actual problems
> in their topologies rather than analyze all failed tuples only to find that
> they failed because of a temporary network glitch.
> 3) Since the entire Tuple is put into the dead-letter, all the data is
> available for retrying after fixing the topology code.
>
> Please share your thoughts if you think it can benefit storm in a generic
> way.
>
> Thx,
> SG
>

Re: [DISCUSS] ElasticSearch 2.x support

[Aaron Niskodé-Dossett]

ES1 is close to end of life in terms of commercial support from Elastic,
but not quite there (https://www.elastic.co/support/eol).  Unfortunately
the ES1 and ES2 clients won't interoperate with their opposite versions.

Given that, I take it you would support having ES1 and ES2 bolts packaged
separately?  This would somewhat like how we have storm-kafka and
storm-kafka-client for different Kafka versions.

Thanks! -Aaron

On Thu, Sep 15, 2016 at 9:12 AM Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:

> I personally don't use ES as part of my storm work, so I don't necessarily
> feel qualified to answer this.  In general though I really do like to see
> storm come with batteries included.  If ES1 is not end of life, and there
> is a community of people who want to continue using it/supporting it, I
> would say lets continue to do so.  If that is not true, or if ES offers a
> backwards compatible client that could sway things for me to say lets just
> go forward with ES2. - Bobby
>
> On Wednesday, September 14, 2016 2:47 PM, Aaron Niskodé-Dossett <
> doss...@gmail.com> wrote:
>
>
>  Hi all,
>
> I started a a discussion about this a while ago, but didn't take it to a
> conclusion (my $realjob, etc., etc.).
>
> There are multiple PRs open to provide an Elastic Search 2.x bolt to the
> Storm project.  There are two different approaches:
>
> 1. Add side-by-side support for 2.x. Example:
> https://github.com/apache/storm/pull/1337 (*FULL DISCLOSURE*: this is my
> own PR). [I also have some functionality enhancements in this PR, but
> that's not relevant to this discussion.]
>
> 2. Upgrade existing bolt. Example,
> https://github.com/apache/storm/pull/1396
>
> The drawback to approach 1 is that it duplicates a lot of code.  The
> drawback to approach 2 is that it drops support for ES 1.x.
>
> ES 2.X has been out for a while and if we are serious about supporting it,
> we need to have a way to write to ES 2.X.
>
> I believe approach number 1 is ideal (again, it's my own PR) and possibly
> deprecating the existing 1.X bolt.
>
> I'd love to hear thoughts from others!
>
>
>

Re: [DISCUSS] Release Storm 1.1.0

[Aaron Niskodé-Dossett]

I would be +1 for including it

On Tue, Nov 15, 2016 at 8:01 AM Xin Wang  wrote:

STORM-2198 ( PR: https://github.com/apache/storm/pull/1773 ) fixes a bug of
storm-hdfs. Do we have a consideration to include this?

Thanks,
Xin Wang (vesense)

2016-11-15 10:03 GMT+08:00 Jungtaek Lim :

> Some issues on Storm SQL are resolved but not documented yet. I'll file an
> issue and assign to 1.1.0 release epic.
> And also I want to address dropping aggregation and join on Storm SQL
> Trident mode before releasing. I'll assign it too.
>
> - Jungtaek Lim (HeartSaVioR)
>
>
> 2016년 11월 15일 (화) 오전 5:55, P. Taylor Goetz 님이 작성:
>
> > I think we’re very close. I would like to confirm that the 1.x-branch is
> > not affected by STORM-2176.
> >
> > The worker lifecycle API was added in 1.0, but doesn’t work in any
> > released version due to STORM-2176.
> >
> > If there are any other open JIRAs that anyone is passionate about, now
> > would be a good time to assign them to the 1.1.0 release epic
> (STORM-1856).
> >
> > -Taylor
> >
> >
> >
> > > On Oct 27, 2016, at 12:19 PM, Jungtaek Lim  wrote:
> > >
> > > Finally Pacemaker H/A, Supervisor V2, and Storm SQL PRs which were
> opened
> > > at the last mail (4 weeks ago) are all merged to 1.x branch.
> > >
> > > There're some more PRs on Storm SQL opened, but given that we can
> release
> > > new minor at any time when we feel it's enough change, I can wait for
> it.
> > > They didn't get reviewed yet indeed.
> > >
> > > Is there something else we would want to include it to 1.1.0?
> > >
> > > Thanks,
> > > Jungtaek Lim (HeartSaVioR)
> > >
> > > 2016년 10월 1일 (토) 오전 9:30, Jungtaek Lim 님이 작성:
> > >
> > >> Personally, merging and porting back to three branches are painful
> > enough,
> > >> especially we don't have merging script and having verbose process (I
> > mean
> > >> CHANGELOG).
> > >> It would be better if merging process is automated (by running script
> or
> > >> so), so I'd +1 to revisit Harsha's suggestion (adopting Kafka merge
> > script)
> > >> and modify script to fit to Storm.
> > >> (It will not work if it's the case we need to handle PRs for each
> > version
> > >> line, since 'Close' in commit log doesn't close the PR if its target
> > branch
> > >> is not master.)
> > >>
> > >> Anyway, without automation I don't want to maintain more version
> lines.
> > >> I'm looking at the announces from other projects, and others are only
> > >> maintaining two version lines.
> > >> Since we maintain 2.0.0 version line we can't reduce version lines to
> 2,
> > >> but hopefully at most 3.
> > >>
> > >> Btw, let's check pending pull requests and enumerate which can be
> > included
> > >> in 1.0.0, and start/finish review and merge them soon.
> > >> For me Supervisor V2 and Pacemaker H/A, and pending Storm SQL PRs can
> be
> > >> included, since they are small or in reviewing and expected to pass
> > review
> > >> phase soon.
> > >> (And some small PRs. There're other valuable PRs in PR list but I'm
> not
> > >> sure we can review them soon. One example is unified API.)
> > >>
> > >> One issue which is not clear is STORM-2006
> > >> . This is a candidate for
> > me,
> > >> but gets blocked while reviewing. If we plan to put great effort to
> > revise
> > >> Metric we can skip this.
> > >>
> > >> Please enumerate other PRs as well if you want to include in 1.1.0.
> > >>
> > >> Thanks,
> > >> Jungtaek Lim
> > >>
> > >> 2016년 9월 30일 (금) 오후 11:09, Bobby Evans 
> 님이
> > 작성:
> > >>
> > >> Sounds good to me.  It would be nice to get some of the new features
> > out.
> > >> Do we expect to maintain both 1.0.x and 1.1.x lines with bug fixes?
> > And if
> > >> so for how long do we want to do this for? - Bobby
> > >>
> > >>On Thursday, September 29, 2016 7:35 PM, Jungtaek Lim <
> > >> kabh...@gmail.com> wrote:
> > >>
> > >>
> > >> Hi devs,
> > >>
> > >> It's been 5 months after releasing Storm 1.0.0, and now 1.x branch
has
> > lots
> > >> of CHANGELOG and also pending reviews.
> > >> It's also been a long time after 1.1.0 RC1 is canceled.
> > >>
> > >> I think it may be good to put some efforts to review and merge
pending
> > pull
> > >> requests (except things which takes time to review and test), and
> > release
> > >> 1.1.0 soon.
> > >>
> > >> What do you think?
> > >>
> > >> I'm also open to volunteer release manager for 1.1.0 after we
document
> > the
> > >> process of official release.
> > >>
> > >> Thanks,
> > >> Jungtaek Lim (HeartSaVioR)
> > >>
> > >>
> > >>
> > >>
> > >>
> >
> >
>

Re: New Committer/PMC Member: Erik Weathers

[Aaron Niskodé-Dossett]

Congrats, Erik, well deserved!
On Sun, Feb 25, 2018 at 10:27 PM Satish Duggana 
wrote:

> Congrats and Welcome Erik!
>
>
>
> On Sat, Feb 24, 2018 at 9:25 PM, Harsha  wrote:
>
> > Congrats Erik.
> > -Harsha
> >
> > On Fri, Feb 23, 2018, at 11:15 AM, Karthick Duraisamy Soundararaj wrote:
> > > Whoa! Congratulations, Erik. Well deserved!
> > >
> > > On Fri, Feb 23, 2018 at 10:37 AM, Erik Weathers <
> > > eweath...@groupon.com.invalid> wrote:
> > >
> > > > Thanks everyone!  And yes Alexandre, the relation of my surname to
> the
> > > > project name was not lost on me. ;-)(Also, my Grandpa went by
> > "Stormy"
> > > > too!)
> > > >
> > > > Also, I must thank my work teammates Srishty Agrawal and Jessica
> > Hartog who
> > > > have contributed greatly to any of the work that I've pushed.
> > > >
> > > > - Erik
> > > >
> > > > On Fri, Feb 23, 2018 at 8:28 AM, Ethan Li  >
> > > > wrote:
> > > >
> > > > > Congratulations! Erik
> > > > >
> > > > > - Ethan
> > > > >
> > > > > > On Feb 22, 2018, at 7:42 PM, Xin Wang 
> > wrote:
> > > > > >
> > > > > > Congrats!
> > > > > >
> > > > > > 2018-02-23 9:41 GMT+08:00 Hugo Da Cruz Louro <
> > hlo...@hortonworks.com>:
> > > > > >
> > > > > >> Congrats & Welcome!
> > > > > >>
> > > > > >>> On Feb 22, 2018, at 2:45 PM, Jungtaek Lim 
> > wrote:
> > > > > >>>
> > > > > >>> Welcome Erik! Congrats!
> > > > > >>>
> > > > > >>> -Jungtaek Lim (HeartSaVioR)
> > > > > >>>
> > > > > >>> 2018년 2월 23일 (금) 오전 7:05, Stig Rohde Døssing <
> > stigdoess...@gmail.com
> > > > > >님이
> > > > > >> 작성:
> > > > > >>>
> > > > >  Congratulations Erik. Happy to see you join.
> > > > > 
> > > > >  2018-02-22 20:53 GMT+01:00 Alexandre Vermeerbergen <
> > > > >  avermeerber...@gmail.com
> > > > > > :
> > > > > 
> > > > > > Hi,
> > > > > >
> > > > > > Welcome to Erik...
> > > > > > ... a Stormy Weather(s) sounds like a fantastic match indeed!
> > > > > >
> > > > > > Alexandre Vermeerbergen (Storm addict)
> > > > > >
> > > > > > 2018-02-22 20:49 GMT+01:00 P. Taylor Goetz <
> ptgo...@apache.org
> > >:
> > > > > >> The Apache Storm PMC has voted to add Erik Weathers as a
> > Committer
> > > > > and
> > > > > > PMC Member.
> > > > > >>
> > > > > >> Please join me in congratulating Erik on his new role!
> > > > > >>
> > > > > >> -Taylor
> > > > > >
> > > > > 
> > > > > >>
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Thanks,
> > > > > > Xin
> > > > >
> > > > >
> > > >
> >
>