Re: Recording - Storm & Kafka Meetup on April 20th 2017

[Harsha Chintalapani]

Hi Aditya,
 Thanks for your interest. We entatively planning one in June
1st week. If you haven't already please register here
https://www.meetup.com/Apache-Storm-Apache-Kafka/ . I'll keep the Storm
lists updated once we finalize the date & location.

Thanks,
Harsha

On Mon, Apr 24, 2017 at 7:02 AM Aditya Desai  wrote:

> Hello Everyone
>
> Can you please let us know when is the next meet up? It would be great if
> we can have in May.
>
> Regards
> Aditya Desai
>
> On Mon, Apr 24, 2017 at 2:16 AM, Xin Wang  wrote:
>
>> How about publishing this to Storm site?
>>
>>  - Xin
>>
>> 2017-04-22 19:27 GMT+08:00 steve tueno :
>>
>>> great
>>>
>>> Thanks
>>>
>>>
>>>
>>> Cordialement,
>>>
>>> TUENO FOTSO Steve Jeffrey
>>> Ingénieur de conception
>>> Génie Informatique
>>> +237 676 57 17 28 <+237%206%2076%2057%2017%2028>
>>> +237 697 86 36 38 <+237%206%2097%2086%2036%2038>
>>>
>>> +33 6 23 71 91 52 <+33%206%2023%2071%2091%2052>
>>>
>>>
>>> https://jobs.jumia.cm/fr/candidats/CVTF1486563.html
>>> 
>>>
>>> __
>>>
>>> https://play.google.com/store/apps/details?id=com.polytech.remotecomputer
>>> 
>>>
>>> https://play.google.com/store/apps/details?id=com.polytech.internetaccesschecker
>>> 
>>> *http://www.traveler.cm/
>>> *
>>> http://remotecomputer.traveler.cm/
>>> 
>>>
>>> https://play.google.com/store/apps/details?id=com.polytech.androidsmssender
>>> 
>>>
>>> https://github.com/stuenofotso/notre-jargon
>>> 
>>> https://play.google.com/store/apps/details?id=com.polytech.welovecameroon
>>> 
>>> https://play.google.com/store/apps/details?id=com.polytech.welovefrance
>>> 
>>>
>>>
>>>
>>> 2017-04-22 3:08 GMT+02:00 Roshan Naik :
>>>
 It was a great meetup and for the benefit of those interested but
 unable to attend it, here is a link to the recording :



 https://www.youtube.com/watch?v=kCRv6iEd7Ow
 




 List of Talks:

 -  *Introduction* –   Suresh Srinivas (Hortonworks)

 -  [4m:31sec] –  *Overview of  Storm 1.1* -  Hugo Louro
 (Hortonworks)

 -  [20m] –  *Rethinking the Storm 2.0 Worker*  - 

Storm Meetup in BayArea on April 20th

[Harsha Chintalapani]

Hi All,
 We are organizing a Storm Meetup at Hortonworks HQ in Santa
Clara,CA. If you are interested in attending please RSVP here
https://www.meetup.com/Apache-Storm-Apache-Kafka/events/238975416/

Thanks,
Harsha

Re: [DISCUSS] Move non-connectors modules to out of external

[Harsha Chintalapani]

We should this in next release of 1.x or 2.0. I am +1 on continue with
current release.
-Harsha
On Fri, Mar 24, 2017 at 8:53 PM P. Taylor Goetz <ptgo...@gmail.com> wrote:

> The question remains if we want to do this in the 1.1.0 release, or later.
>
> If it's the 1.1.0 release we need to make the changes and cut another RC.
> I'm fine with that, but want to make sure we have consensus before going
> down that road.
>
> -Taylor
>
> > On Mar 24, 2017, at 10:57 PM, Harsha Chintalapani <st...@harsha.io>
> wrote:
> >
> > Agree on change like this would be confusing to the users. Lets keep the
> > original plan of moving non-connectors modules of external instead of
> > introducing new changes
> > that are not in scope of this discussion.
> > My +1 still stands on keeping the current external/storm-* in place and
> > move just sql and storm-perf into top-level. We can have discussion for
> > storm 2.0 if we want to do
> > more changes.
> >
> > -Harsha
> >
> >> On Fri, Mar 24, 2017 at 4:31 PM P. Taylor Goetz <ptgo...@gmail.com>
> wrote:
> >>
> >> If we decide to change the structure of the distribution like this, I
> >> think we should do it in masrwe/2.0. If we want this for 1.1.0 we need
> to
> >> cut a new release candidate.
> >>
> >> Changing the structure of the distribution file structure can be
> >> disruptive for users. Even the change to no longer include connector
> >> binaries, as we've learned, will be a headache for some users.
> >>
> >> IMHO, from an ops perspective, changes like this should be handled like
> >> API changes.
> >>
> >> -Taylor
> >>
> >>> On Mar 24, 2017, at 4:07 PM, Hugo Da Cruz Louro <
> hlo...@hortonworks.com>
> >> wrote:
> >>>
> >>> Another possibility is to keep the ‘external' module, and create sub
> >> modules under it. The legacy structure would remain intact, while making
> >> things more modular. An idea would be:
> >>>
> >>> + external
> >>>+ connectors
> >>>+ tools
> >>>+ monitoring
> >>>+ etc
> >>>
> >>> Hugo
> >>>
> >>>> On Mar 24, 2017, at 12:34 PM, P. Taylor Goetz <ptgo...@gmail.com>
> >> wrote:
> >>>>
> >>>> For the background on why “external” was selected, you have to go back
> >> to a lengthy discussion in Feb. 2014.
> >>>>
> >>>> Here’s the start of the thread:
> >>>>
> >>>>
> >>
> http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3e
> >> <
> >>
> http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3E
> >>>
> >>>>
> >>>> It continues into March:
> >>>>
> >>>>
> >>
> http://mail-archives.apache.org/mod_mbox/storm-dev/201403.mbox/%3ccadimvzum1d3om30zayqq4xxe1vjbn7fumqcsgu+524oqgec...@mail.gmail.com%3e
> >>>>
> >>>> I’m -1 on renaming “external”. That’s the name chosen by the community
> >> and it has been the norm for 3 years. Changing it would likely confuse
> >> users.
> >>>>
> >>>> One of the ideas behind “external” was that it would contain
> components
> >> that were not essential to running storm. That line has recently blurred
> >> with some non-connector code sneaking in, so I’m okay with moving
> >> non-connector code out of external. Another point in that thread was a
> >> desire to avoid cluttering up the root directory, so we need to be
> careful
> >> about what the destination for those components is.
> >>>>
> >>>> -Taylor
> >>>>
> >>>>> On Mar 24, 2017, at 3:11 PM, Hugo Da Cruz Louro <
> >> hlo...@hortonworks.com> wrote:
> >>>>>
> >>>>> +1 non-connectors to top level
> >>>>> +1 to renaming external to connectors
> >>>>>
> >>>>> As for storm-kaka, if we are already touching the external modules,
> >> all the modules should be a submodule of a parent module called
> >> storm-kafka. I don’t think we should have 3 parent modules as we
> currently
> >> have (storm-kafka, storm-kafka-client, storm-kafka-monitor)
> >>>>>
> >>>>> The structure should be something along the lines (I 

Re: [DISCUSS] Move non-connectors modules to out of external

[Harsha Chintalapani]

Agree on change like this would be confusing to the users. Lets keep the
original plan of moving non-connectors modules of external instead of
introducing new changes
that are not in scope of this discussion.
My +1 still stands on keeping the current external/storm-* in place and
move just sql and storm-perf into top-level. We can have discussion for
storm 2.0 if we want to do
more changes.

-Harsha

On Fri, Mar 24, 2017 at 4:31 PM P. Taylor Goetz <ptgo...@gmail.com> wrote:

> If we decide to change the structure of the distribution like this, I
> think we should do it in masrwe/2.0. If we want this for 1.1.0 we need to
> cut a new release candidate.
>
> Changing the structure of the distribution file structure can be
> disruptive for users. Even the change to no longer include connector
> binaries, as we've learned, will be a headache for some users.
>
> IMHO, from an ops perspective, changes like this should be handled like
> API changes.
>
> -Taylor
>
> > On Mar 24, 2017, at 4:07 PM, Hugo Da Cruz Louro <hlo...@hortonworks.com>
> wrote:
> >
> > Another possibility is to keep the ‘external' module, and create sub
> modules under it. The legacy structure would remain intact, while making
> things more modular. An idea would be:
> >
> > + external
> > + connectors
> > + tools
> > + monitoring
> > + etc
> >
> > Hugo
> >
> >> On Mar 24, 2017, at 12:34 PM, P. Taylor Goetz <ptgo...@gmail.com>
> wrote:
> >>
> >> For the background on why “external” was selected, you have to go back
> to a lengthy discussion in Feb. 2014.
> >>
> >> Here’s the start of the thread:
> >>
> >>
> http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3e
> <
> http://mail-archives.apache.org/mod_mbox/storm-dev/201402.mbox/%3cee2bd0e2-254c-47af-8a53-257db7f05...@gmail.com%3E
> >
> >>
> >> It continues into March:
> >>
> >>
> http://mail-archives.apache.org/mod_mbox/storm-dev/201403.mbox/%3ccadimvzum1d3om30zayqq4xxe1vjbn7fumqcsgu+524oqgec...@mail.gmail.com%3e
> >>
> >> I’m -1 on renaming “external”. That’s the name chosen by the community
> and it has been the norm for 3 years. Changing it would likely confuse
> users.
> >>
> >> One of the ideas behind “external” was that it would contain components
> that were not essential to running storm. That line has recently blurred
> with some non-connector code sneaking in, so I’m okay with moving
> non-connector code out of external. Another point in that thread was a
> desire to avoid cluttering up the root directory, so we need to be careful
> about what the destination for those components is.
> >>
> >> -Taylor
> >>
> >>> On Mar 24, 2017, at 3:11 PM, Hugo Da Cruz Louro <
> hlo...@hortonworks.com> wrote:
> >>>
> >>> +1 non-connectors to top level
> >>> +1 to renaming external to connectors
> >>>
> >>> As for storm-kaka, if we are already touching the external modules,
> all the modules should be a submodule of a parent module called
> storm-kafka. I don’t think we should have 3 parent modules as we currently
> have (storm-kafka, storm-kafka-client, storm-kafka-monitor)
> >>>
> >>> The structure should be something along the lines (I don’t mean the
> exact names;  we should find better ones. storm-kafka and
> storm-kafka-client are not very self explanatory in my opinion)
> >>>
> >>> + storm-kafka
> >>> + monitoring
> >>> + new-client
> >>> + old-client
> >>>
> >>> If we have to create new modules or submodules (e.g. under utils) so
> be it. The code should be in a module that is named after what its doing.
> >>>
> >>> Hugo
> >>>
> >>>> On Mar 24, 2017, at 11:15 AM, Priyank Shah <ps...@hortonworks.com>
> wrote:
> >>>>
> >>>> +1 to moving non-conncectors to top level. I think we should keep
> stom-kafka-monitor under external or connectors(after renaming).
> >>>>
> >>>> Jungtaek, just to clarify on what you said regarding storm core
> referencing storm-kafka-monitor. Like you said its just calling the script
> from ui jvm. There is no dependency in terms of class files needed to run
> the script from ui. The script itself adds a –cp argument and all it needs
> is storm-kafka-monitor jar in classpath. As far as packaging the script is
> concerned we can do what Satish suggested. i.e. move it to
> storm-kafka-monitor in source and

Re: [DISCUSS] Move non-connectors modules to out of external

[Harsha Chintalapani]

+1 on moving non-connectors to top-level like sql and storm-perf.
Regarding storm-kafka-monitor we can move this into "util" folder or keep
in the external.
-Harsha

On Fri, Mar 24, 2017 at 2:23 AM Satish Duggana 
wrote:

> storm-kafka-monitor is not a connector by itself but it is related to kafka
> connectors. So, any utility related to that connector should be part of
> that connector module(can be a submodule) instead of a top level module.
> core/ui uses this utility referring directly in a hacky way, which we may
> want to fix later. storm-kafka-monitor script exists in bin directory which
> can be moved to storm-kafka-monitor module and the same script can be
> packaged as part of storm/bin directory while packaging the distribution.
>
> Thanks,
> ~Satish.
>
> On Fri, Mar 24, 2017 at 1:07 PM, Jungtaek Lim  wrote:
>
> > storm-kafka-monitor is referred by storm-core, though it's referenced via
> > executing command. Yes it's a bit odd to place it as top directory, but
> > it's not a connector for that reason too. Neither is ideal for me, so
> > ironically, either is fine.
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 3월 24일 (금) 오후 4:19, Satish Duggana 님이
> 작성:
> >
> > > +1 except for storm-kafka-monitor module as this utility is more about
> > > querying topic/partition offsets of kafka spouts in a topology. Do not
> we
> > > want to push this module into connectors/kafka as a submodule along
> with
> > > other submodules including old/new kafka spout modules?
> > >
> > > Thanks,
> > > Satish.
> > >
> > >
> > > On Fri, Mar 24, 2017 at 12:10 PM, Arun Iyer 
> > wrote:
> > >
> > > > +1
> > > >
> > > > Makes sense to move the non-connectors to top level and keep only the
> > > > connectors under “connectors” folder.
> > > >
> > > >
> > > > On 3/24/17, 12:00 PM, "Jungtaek Lim"  wrote:
> > > >
> > > > >(Sent this yesterday but can't find this from storm-dev mbox...
> > sending
> > > it
> > > > >again)
> > > > >
> > > > >Hi dev,
> > > > >
> > > > >I'd like to start discussion regarding moving non-connectors modules
> > out
> > > > of
> > > > >external, maybe top directory.
> > > > >
> > > > >"external" directory has non-connectors (SQL, Flux,
> > storm-kafka-monitor,
> > > > >storm-submit-tools), and except Flux, others should be placed to the
> > > > binary
> > > > >dist. since Storm itself (not from user topology) needs to refer
> them.
> > > > >
> > > > >They're actually tied to the core of Storm, so I feel that it would
> be
> > > > >better to treat them (including Flux) as non-external, maybe same
> > level
> > > as
> > > > >storm-core.
> > > > >(I'm not sure what "external" actually means for Storm project btw.)
> > > > >
> > > > >In addition, after doing that I'd like to change the directory name
> > > > >"external" to "connector" or so, so that the name could be
> > > self-describing
> > > > >and we can only place connectors to that directory.
> > > > >(I know it would be painful for already opened pull requests, so no
> > > strong
> > > > >opinion regarding this.)
> > > > >
> > > > >Looking forward to your opinion!
> > > > >
> > > > >Thanks,
> > > > >Jungtaek Lim (HeartSaVioR)
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Storm 2.0 Roadmap

[Harsha Chintalapani]

Storm 2.0 migration to java in itself is a big win and would attract wider
community and adoption. So my vote would be to resolve the first 3 items to
get a release out.
All the other featured mentioned are great to have but shouldn't be
blockers for 2.0 release.

-Harsha

On Thu, Mar 23, 2017 at 11:51 AM P. Taylor Goetz  wrote:

> With the 1.1.0 release nearing completion, I’d like to turn our attention
> to 2.0 and develop a plan for what features, etc. to include.
>
> The following 3 are what I feel are the minimum for a 2.0 release. These
> could likely be resolved relatively quickly:
>
> * Performance — I’ve not benchmarked the master branch vs. 1.0.x or 1.1.x
> in a while, but I feel it will be important to make sure there are no
> performance regressions, and would hope that we actually have a performance
> improvement over previous versions. To that end (e.g. if there is in fact a
> performance regression), the proposals that Roshan Naik put together for
> revising the threading and execution model (STORM-2307) and replacing
> Disruptor with JCTools (STORM-2306) warrant review and consideration. See
> also STORM-2284 which is the parent JIRA.
>
> * Finish porting Storm UI to java (STORM-1311)
>
> * Finish porting log viewer to java (STORM-1280)
>
> The following are items that are nice to have in 2.0, but I don’t feel are
> absolutely necessary for an initial 2.0 release:
>
> * Beam Runner (I wouldn’t tie this to 2.0, mentioning it because it’s
> relevant) — Initially there seemed to be a lot of interest in this, but
> that seems to have trailed off. I spoke with some Beam developers and there
> seems to be interest from that community as well. Do we want to move that
> effort to the Beam community, or keep it here? Moving it to the Beam
> community might lead to better collaboration between projects.
>
> * Bounded Spouts (needed for Beam Runner implementation) — Currently
> spouts are unbounded, there no end to the stream. Beam has the concept of
> bounded sources (roughly analogous to batch processing). To support that,
> we would need to implement a similar concept in Storm. One benefit of such
> a feature would be the ability to handle both bounded and unbounded
> workflows in Storm.
>
> * Storm-SQL — Jungtaek/Xin: You have been the primary drivers behind this
> effort. What improvements do you envision for 2.0?
>
> * Metrics V2 (STORM-2153: Coda Hale Metrics) — I’ve been targeting this
> for 1.2.0, but it’s designed to be easily portable to master/2.0.
>
> * JStorm Migration — Original outline can be found here [1]. Note a lot of
> the associated JIRAs below are assigned, but there hasn’t been any recent
> activity or pull requests, we should probably consider them unassigned and
> up for grabs.:
>
> * Worker Classloader Isolation (STORM-1338) — Lack of this has been the
> bane of a lot of Storm users almost since day one. We have largely
> addressed it by shading/relocating dependencies. It would be great to see
> this addressed once and for all.
>
> * JStorm back pressure implementation (STORM-1324) — The current back
> pressure implementation leaves a bit to be desired, and the JStorm approach
> looks promising, though it also depends on the JStorm concept of “topology
> master” (STORM-1323), which may have some implications regarding security.
>
> * Dynamic Topology Updates (STORM-1335) — This would provide a command to
> update topology jars and configuration without stopping the topology, and
> is well suited to leverage the blobstore. The restart command (that can
> also update the topology configuration) also looks compelling (STORM-1334).
>
> * Additional Scheduler Implementations (STORM-1320)
>
> * Additional Grouping Implementations (STORM-1328)
>
>
> As always I’m open to any opinions and suggestions.
>
> -Taylor
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61328109
>
>
>

Re: [VOTE] Release Apache Storm 1.1.0 (RC3)

[Harsha Chintalapani]

I propose that we continue with the release and get this patch in next
minor release (probably 1.1.1?). Users doesn't need to upgrade their
cluster to get this change
so I would say this is not a show-stopper for the release.

-Harsha

On Thu, Mar 23, 2017 at 9:18 AM P. Taylor Goetz <ptgo...@gmail.com> wrote:

> Hugo, now that you are a PMC member your vote is binding.
>
> While releases can’t be vetoed (a release vote needs at least three +1s
> and more +1s than -1s), it’s typical to take negative votes under serious
> consideration.
>
> So considering Hugo’s -1 vote, do we want cancel in order to include the
> proposed fix? It would also give us a chance to update the connector
> documentation regarding location of binary artifacts.
>
> -Taylor
>
>
> > On Mar 23, 2017, at 11:58 AM, Hugo Da Cruz Louro <hlo...@hortonworks.com>
> wrote:
> >
> > -1 (non binding)
> >
> > This blocker bug was just reported -
> https://issues.apache.org/jira/browse/STORM-2432
> > Here is the fix: https://github.com/apache/storm/pull/2027
> >
> > I consider that this is a blocker bug because if the user choses
> UNCOMITTED_LATEST first poll strategy, no data gets polled at all.
> >
> > Thanks,
> > Hugo
> >
> > On Mar 23, 2017, at 8:48 AM, Harsha Chintalapani <st...@harsha.io
> <mailto:st...@harsha.io>> wrote:
> >
> > +1 for documenting and releasing.
> > -Harsha
> >
> > On Thu, Mar 23, 2017 at 7:04 AM Jungtaek Lim <kabh...@gmail.com kabh...@gmail.com>> wrote:
> >
> > +1 to the latter.
> >
> > I'm in favor of documenting the change to release note, and also docs so
> > that website can be reflected. The users who are affected to the change
> > wouldn't be much, since using dependency management tool (Maven, Gradle,
> > and so on) has been recommended for creating topology jar.
> >
> > For me it's not a blocker for release.
> >
> > Arun, I initiated another thread to discuss moving non-connectors to the
> > top directory.
> > Could you cast your vote? If you are still not satisfied with excluding
> > jars you can cast -0 or even -1.
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 3월 23일 (목) 오후 10:43, P. Taylor Goetz <ptgo...@gmail.com ptgo...@gmail.com>>님이 작성:
> >
> > Do we want to cancel this RC in order to better document the changes, or
> > will documenting it in the release announcement suffice for now (provided
> > documentation is added for subsequent releases)?
> >
> > I’m partial to the latter, but am open to others’ opinions.
> >
> > -Taylor
> >
> >
> > On Mar 22, 2017, at 9:49 AM, Bobby Evans <ev...@yahoo-inc.com.INVALID
> <mailto:ev...@yahoo-inc.com.INVALID>>
> > wrote:
> >
> > +1 I built form the tag and ran using a single node cluster.
> > The examples and external components are excluded because they are
> > huge.  Because of shading they we distribute the same copy of them
> > multiple
> > times.
> > I agree with Alexandre.  We should document this change better, because
> > it is confusing for people to get a release that used to have these in
> > it,
> > but does not any more.
> >
> >
> > - Bobby
> >
> > On Tuesday, March 21, 2017, 10:46:38 PM CDT, Arun Mahadevan <
> > ar...@apache.org<mailto:ar...@apache.org>> wrote:Verified the
> artifacts. Compiled examples and
> > ran
> > some sample topologies. Looks good.
> >
> > BTW, why are the external modules excluded from the binaries (the .zip
> > and .tar.gz). Isn’t it better if the binary distribution includes them?
> > Maybe it was already discussed but I am missing it. The sql directory
> > however seems to include the jars so it looks inconsistent.
> >
> > - Arun
> >
> >
> > On 3/22/17, 12:56 AM, "P. Taylor Goetz" <ptgo...@apache.org ptgo...@apache.org>> wrote:
> >
> > This is a call to vote on releasing Apache Storm 1.1.0 (rc3)
> >
> > Full list of changes in this release:
> >
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;h=68fbab3c4f91359bd397d93a157830542839b002;hb=e40d213de7067f7d3aa4d4992b81890d8ed6ff31
> >
> > The tag/commit to be voted upon is v1.1.0:
> >
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=7fa62404feb6b86b3143c851b46237580720eb6b;hb=e40d213de7067f7d3aa4d4992b81890d8ed6ff31
> >
> > The source archive being voted upon can be found here:
> >
> >
> >
> >
> 

Re: [VOTE] Release Apache Storm 1.1.0 (RC3)

[Harsha Chintalapani]

+1 for documenting and releasing.
-Harsha

On Thu, Mar 23, 2017 at 7:04 AM Jungtaek Lim  wrote:

> +1 to the latter.
>
> I'm in favor of documenting the change to release note, and also docs so
> that website can be reflected. The users who are affected to the change
> wouldn't be much, since using dependency management tool (Maven, Gradle,
> and so on) has been recommended for creating topology jar.
>
> For me it's not a blocker for release.
>
> Arun, I initiated another thread to discuss moving non-connectors to the
> top directory.
> Could you cast your vote? If you are still not satisfied with excluding
> jars you can cast -0 or even -1.
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2017년 3월 23일 (목) 오후 10:43, P. Taylor Goetz 님이 작성:
>
> > Do we want to cancel this RC in order to better document the changes, or
> > will documenting it in the release announcement suffice for now (provided
> > documentation is added for subsequent releases)?
> >
> > I’m partial to the latter, but am open to others’ opinions.
> >
> > -Taylor
> >
> >
> > > On Mar 22, 2017, at 9:49 AM, Bobby Evans 
> > wrote:
> > >
> > > +1 I built form the tag and ran using a single node cluster.
> > > The examples and external components are excluded because they are
> > huge.  Because of shading they we distribute the same copy of them
> multiple
> > times.
> > > I agree with Alexandre.  We should document this change better, because
> > it is confusing for people to get a release that used to have these in
> it,
> > but does not any more.
> > >
> > >
> > > - Bobby
> > >
> > > On Tuesday, March 21, 2017, 10:46:38 PM CDT, Arun Mahadevan <
> > ar...@apache.org> wrote:Verified the artifacts. Compiled examples and
> ran
> > some sample topologies. Looks good.
> > >
> > > BTW, why are the external modules excluded from the binaries (the .zip
> > and .tar.gz). Isn’t it better if the binary distribution includes them?
> > Maybe it was already discussed but I am missing it. The sql directory
> > however seems to include the jars so it looks inconsistent.
> > >
> > > - Arun
> > >
> > >
> > > On 3/22/17, 12:56 AM, "P. Taylor Goetz"  wrote:
> > >
> > >> This is a call to vote on releasing Apache Storm 1.1.0 (rc3)
> > >>
> > >> Full list of changes in this release:
> > >>
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;h=68fbab3c4f91359bd397d93a157830542839b002;hb=e40d213de7067f7d3aa4d4992b81890d8ed6ff31
> > >>
> > >> The tag/commit to be voted upon is v1.1.0:
> > >>
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=7fa62404feb6b86b3143c851b46237580720eb6b;hb=e40d213de7067f7d3aa4d4992b81890d8ed6ff31
> > >>
> > >> The source archive being voted upon can be found here:
> > >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc3/apache-storm-1.1.0-src.tar.gz
> > >>
> > >> Other release files, signatures and digests can be found here:
> > >>
> > >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.1.0-rc3/
> > >>
> > >> The release artifacts are signed with the following key:
> > >>
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > >>
> > >> The Nexus staging repository for this release is:
> > >>
> > >>
> https://repository.apache.org/content/repositories/orgapachestorm-1047
> > >>
> > >> Please vote on releasing this package as Apache Storm 1.1.0.
> > >>
> > >> When voting, please list the actions taken to verify the release.
> > >>
> > >> This vote will be open for at least 72 hours.
> > >>
> > >> [ ] +1 Release this package as Apache Storm 1.1.0
> > >> [ ]  0 No opinion
> > >> [ ] -1 Do not release this package because...
> > >>
> > >> Thanks to everyone who contributed to this release.
> > >>
> > >> -Taylor
> > >
> >
> >
>

Apache Storm Meetup in BayArea

[Harsha Chintalapani]

Hi All,
   We are planning on scheduling a Storm Meetup in April 1st week.
Here is the meetup link https://www.meetup.com/Apache-Storm-Apache-Kafka/.
If you are interested in talking about your use-cases in storm there is 1
more slot available, please reach out to me.

Thanks,
Harsha

Re: [DISCUSS] Release Storm 1.1.0

[Harsha Chintalapani]

STORM-2340 is more of a feature . Auto-commit mode in storm-kafka used
rarely and most users
run the kafka spout with ackers and get at-least once guarantee.  If its
going to longer to address the PR reviews
I am +1 on moving this out of Storm 1.1.0. We already quite a few patches
storm-kafka-client and 1.1.0 release brings in lot of improvements
and bug-fixes.
-Harsha

On Wed, Feb 8, 2017 at 6:15 PM Jungtaek Lim <kabh...@gmail.com> wrote:

> There seems some pull requests for bugfix/improvement on
> storm-kafka-client, and some authors in PRs are not availble for now.
> (waiting 7 days)
>
> If we plan to get 1.1.1 out soon (say 1 month later or even closer) we can
> postpone, but if not, it might be better to coordinate these things ASAP
> and include to 1.1.0.
>
> There seems to be other small PRs, but nothing seems critical so it would
> be OK to not wait for merging.
>
> - Jungtaek Lim
>
> 2017년 2월 9일 (목) 오전 6:48, P. Taylor Goetz <ptgo...@gmail.com>님이 작성:
>
> Right now we’re down to 1 open issue on the 1.1.0 release epic: STORM-2250
> which is under active review/discussion.
>
> Assuming that is mergeable in the near future, are there any other open
> issues that should be considered for this release?
>
> -Taylor
>
>
> > On Feb 2, 2017, at 4:48 PM, P. Taylor Goetz <ptgo...@gmail.com> wrote:
> >
> > Thanks for putting this list together Jungtaek. I added a few to the 1.1
> release epic that I think are important. Feel free to do the same.
> >
> > Looks like we have a few to go, but there are pull requests for them.
> It’s mostly just a matter of reviews and review responses, so I think we
> are close.
> >
> > -Taylor
> >
> >> On Feb 2, 2017, at 1:41 AM, Jungtaek Lim <kabh...@gmail.com> wrote:
> >>
> >> Seems like there're not blockers for 1.1.0, but some pull requests are
> >> worth to check.
> >> There're pending pull requests for storm-kafka-client waited on
> STORM-2225.
> >> Given that STORM-2225 is now merged, we might need to take a look at.
> >>
> >> *- reviewing*
> >>
> >> [storm-core]
> >>
> >>> STORM-2324 : Fix deployment failure if resources directory is missing
> in
> >> topology jar
> >> (master) https://github.com/apache/storm/pull/1908
> >> (1.x) https://github.com/apache/storm/pull/1898
> >>
> >>> STORM-2321 Handle blobstore zk key deletion in KeySequenceNumber
> >> (master) https://github.com/apache/storm/pull/1904
> >> (1.x) https://github.com/apache/storm/pull/1905
> >>
> >> [storm-kafka]
> >>
> >>> STORM-2270 Kafka spout should consume from latest when ZK partition
> >> commit offset bigger than the latest offset
> >> (1.x) https://github.com/apache/storm/pull/1851
> >>
> >> [storm-kafka-client]
> >>
> >>> STORM-2281: Running Multiple Kafka Spouts (Trident) Throws Illegal
> State
> >> Exception
> >> (1.x) https://github.com/apache/storm/pull/1902
> >>
> >>> STORM-2315 Storm kafka client does not commit offsets when ack is
> disabled
> >> (1.x) https://github.com/apache/storm/pull/1891
> >>
> >>> fix: KafkaSpout is blocked in AutoCommitMode
> >> (master) https://github.com/apache/storm/pull/1863
> >>
> >>> STORM-2250: Kafka Spout Refactoring to Increase Modularity and
> Testability
> >> (master) https://github.com/apache/storm/pull/1832
> >>
> >>> STORM-2014: Put logic around dropping messages into RetryService,
> remove
> >> maxRetry setting from new KafkaSpout
> >> (master) https://github.com/apache/storm/pull/1605
> >>
> >>> fix NullPointException with acked.get(rtp)
> >> (master) https://github.com/apache/storm/pull/1807
> >>
> >> [storm-sql]
> >>
> >>> STORM-1443 [Storm SQL] Support customizing parallelism in StormSQL
> >> https://github.com/apache/storm/pull/1739
> >>
> >> *- pending*
> >>
> >> [storm-kafka-client]
> >>
> >>> STORM-2296 Kafka spout no dup on leader changes
> >> (1.0.x) https://github.com/apache/storm/pull/1873
> >> (1.x) https://github.com/apache/storm/pull/1888
> >>
> >> [storm-sql]
> >>
> >>> STORM-2148 [Storm SQL] Trident mode: back to code generate and compile
> >> Trident topology
> >> https://github.com/apache/storm/pull/1743
> >>
> >> Thanks,
> >> Jungtaek Lim (HeartSaVioR)
> >>
> >> 2017년 2월 2일 (목) 오전 8:14, Harsha Chintala

Re: [VOTE] Release Apache Storm 1.0.3 (rc1)

[Harsha Chintalapani]

+1
Tested in 3 node vagant cluster and ran few example topologies. Looks good
and verified the signature of artifacts.

On Wed, Feb 1, 2017 at 7:13 AM Bobby Evans 
wrote:

> +1 Ran some simple tests in cluster mode.
>
>
> - Bobby
>
> On Wednesday, February 1, 2017, 2:58:44 AM CST, Jungtaek Lim <
> kabh...@gmail.com> wrote:FYI: both issues are filed to STORM-2335
>  and STORM-2336
> , and patches are
> available for each issue.
>
> Please go ahead verifying RC1. They're not blocker and even we might want
> to include them to 1.0.3, we can verify RC1 first and test only two things
> for RC2 later.
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2017년 2월 1일 (수) 오후 3:45, Jungtaek Lim 님이 작성:
>
> > > 2. running topology with local cluster doesn't terminate after cluster
> > > shutdown: Took stack trace, and found some non-daemon threads are live:
> > > Async Localizer, Localizer Cache Cleanup x 2.
> >
> > Local mode or a single node cluster?
> >
> > In local mode. Cluster mode is OK.
> >
> > Btw, thanks for the kind words. Hope this helps reducing others load for
> > participating RC verification.
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 2월 1일 (수) 오후 2:43, P. Taylor Goetz 님이 작성:
> >
> > Thanks for thorough review. It's a great example of what a good review
> > looks like.
> >
> > Comments in line below.
> >
> > -Taylor
> >
> > > On Jan 31, 2017, at 11:33 PM, Jungtaek Lim  wrote:
> > >
> > > +1 (binding)
> > >
> > >> source
> > >
> > > - verify file (signature, MD5, SHA)
> > > -- source, tar.gz : OK
> > > -- source, zip : OK
> > >
> > > - extract file
> > > -- source, tar.gz : OK
> > > -- source, zip : OK
> > >
> > > - diff-ing extracted files between tar.gz and zip : OK
> > >
> > > - build source with JDK 7
> > > -- source, tar.gz : OK
> > >
> > > - build source dist
> > > -- source, tar.gz : OK
> > >
> > > - build binary dist
> > > -- source, tar.gz : OK
> > >
> > >> binary
> > >
> > > - verify file (signature, MD5, SHA)
> > > -- binary, tar.gz : OK
> > > -- binary, zip : OK
> > >
> > > - extract file
> > > -- source, tar.gz : OK
> > > -- source, zip : OK
> > >
> > > - diff-ing extracted files between tar.gz and zip : OK
> > >
> > > - launch daemons : OK
> > > - run RollingTopWords (local) : OK
> > > - run RollingTopWords (remote) : OK
> > > - activate / deactivate / rebalance / kill : OK
> > > - logviewer (worker dir, daemon dir) : OK
> > > - change log level : OK
> > >  - thread dump, heap dump, restart worker : OK
> > > - log search : OK
> > > - run WordCountTopology (multilang, remote) : OK
> > > - run StatefulTopology (local) : OK
> > > - run StatefulTopology (remote) : OK
> > > - run StatefulWindowingTopology (remote) : OK
> > >
> > > Found two bugs here which are not blocker:
> > >
> > > 1. visualization: it doesn't show graph correctly while worker is being
> > > initializing, and below error message is printed on console, which I
> > think
> > > is long-lasting bug.
> > >
> >
> > I think this has been around for a while. I agree that it's not a
> blocker.
> >
> > > 2. running topology with local cluster doesn't terminate after cluster
> > > shutdown: Took stack trace, and found some non-daemon threads are live:
> > > Async Localizer, Localizer Cache Cleanup x 2.
> >
> > Local mode or a single node cluster?
> >
> > I saw this when testing the worker shutdown hook bug in local mode. I
> > don't think this is a blocker, but we should definitely follow up with a
> > JIRA. This could affect users who rely on local mode for integration
> tests.
> >
> > >
> > > I'll file two issues afterwards.
> > >
> > > Thanks,
> > > Jungtaek Lim (HeartSaVioR)
> > >
> > > 2017년 2월 1일 (수) 오전 5:55, P. Taylor Goetz 님이 작성:
> > >
> > >> This is a call to vote on releasing Apache Storm 1.0.3 (rc1)
> > >>
> > >> Full list of changes in this release:
> > >>
> > >>
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=6cb735d18ec11eccd3e80bab8f879c989a9b3967;hb=a81ec2580fce1f2ee6349a9028dcb75763798bec
> > >>
> > >> The tag/commit to be voted upon is v1.0.3:
> > >>
> > >>
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;h=b8adddc6aa20107288e59cba6a2976c0951742fb;hb=a81ec2580fce1f2ee6349a9028dcb75763798bec
> > >>
> > >> The source archive being voted upon can be found here:
> > >>
> > >>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.3-rc1/apache-storm-1.0.3-src.tar.gz
> > >>
> > >> Other release files, signatures and digests can be found here:
> > >>
> > >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.3-rc1/
> > >>
> > >> The release artifacts are signed with the following key:
> > >>
> > >>
> > >>
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> > >>

Re: [DISCUSS] Release Storm 1.1.0

[Harsha Chintalapani]

Trying to check the status on this release of 1.1.0. Are we going to do
this release anytime soon?


On Fri, Jan 13, 2017 at 7:50 PM S G  wrote:

> Not sure if its a little late to include for the 1.1.0 and 1.0.3 releases
> now, but can we consider using zookeeper 3.4.9 for the future versions as
> 3.4.9 brings in a lot of stability improvements (
> http://zookeeper.apache.org/releases.html) and storm is still using 3.4.6
> (
> https://github.com/apache/storm/blob/master/pom.xml)
>
> On Fri, Jan 6, 2017 at 11:54 AM, P. Taylor Goetz 
> wrote:
>
> > Thanks for the update Jungtaek.
> >
> > I’m verifying the patches now. And they should be mergeable shortly.
> >
> > I think we’ll likely be ready for 1.1.0 and 1.0.3 releases next week.
> >
> > -Taylor
> >
> > > On Jan 6, 2017, at 3:40 AM, Jungtaek Lim  wrote:
> > >
> > > I just submitted a patch for STORM-2176
> > > . Since it's a small
> > fix,
> > > we just need to handle STORM-2228
> > >  to release.
> > >
> > > - Jungtaek Lim (HeartSaVioR)
> > >
> > > 2017년 1월 5일 (목) 오후 1:43, Jungtaek Lim 님이 작성:
> > >
> > >> Recently we receive some requests regarding release Storm 1.0.3, so
> > would
> > >> like to bump this again.
> > >>
> > >> Given that blocker issues for Storm 1.1.0 are also blocker for 1.0.3,
> > I'd
> > >> like to ask a favor of taking care of 'open' / 'in progress' issues on
> > >> 1.1.0 epic.
> > >> https://issues.apache.org/jira/browse/STORM-1856
> > >>
> > >> There're one 'open' issue and three 'in progress' issues. Two of three
> > 'in
> > >> progress' issues are tiny fix so easy to be handled, so actual
> blockers
> > are
> > >> STORM-2176  and
> > >> STORM-2228 .
> > >>
> > >> Thanks,
> > >> Jungtaek Lim (HeartSaVioR)
> > >>
> > >> 2016년 11월 17일 (목) 오후 11:41, Satish Duggana  >님이
> > >> 작성:
> > >>
> > >>STORM-2205: Race condition in getting nimbus summaries while ZK
> > >> connections are reconnected.
> > >>
> > >> This issue seems to occur in our environments and I would like this to
> > be
> > >> part of 1.1.0.
> > >>
> > >> Thanks,
> > >> Satish.
> > >>
> > >> On Thu, Nov 17, 2016 at 9:36 AM, Jungtaek Lim 
> > wrote:
> > >>
> > >>> I have no idea on storm-kafka-client, but some bugfix issues for
> > >>> storm-kafka-client are waiting for reviewing / merging.
> > >>>
> > >>> STORM-2014 
> > >>> STORM-2087 
> > >>> STORM-2104 
> > >>>
> > >>> If someone can review them in several days it would be great.
> > >>>
> > >>> I hope that we include currently opened pull requests for Storm SQL
> so
> > >> that
> > >>> we can release 'usable Storm SQL' more usable, but I'm also OK to
> > >> postpone
> > >>> them to be included to next release if they drag the release.
> > >>>
> > >>> STORM-1446 
> > >>> STORM-1443 
> > >>> STORM-2148 
> > >>> STORM-2170 
> > >>>
> > >>> I can see some pull requests which address Trident implementations
> for
> > >>> storm-kafka-client, storm-mongodb, storm-cassandra.
> > >>>
> > >>> storm-kafka-client: STORM-1694
> > >>>  (patch for 2.0 is
> > >>> merged, patch for 1.x is ready for reviewing)
> > >>> storm-cassandra: STORM-1369
> > >>> 
> > >>> storm-mongodb: STORM-1607  > >>> jira/browse/STORM-1607>
> > >>>
> > >>> If we want to cut the release now, we could include only bugfix
> issues
> > >> and
> > >>> postpone others. Otherwise we could discuss and include some or all
> of
> > >> the
> > >>> above.
> > >>>
> > >>> What do you think? When we want to start the release process for
> 1.1.0?
> > >>>
> > >>> - Jungtaek Lim (HeartSaVioR)
> > >>>
> > >>> 2016년 11월 16일 (수) 오전 4:11, P. Taylor Goetz 님이 작성:
> > >>>
> > >>> Thanks Xin, I added it to the 1.1.0 epic.
> > >>>
> > >>> -Taylor
> > >>>
> >  On Nov 15, 2016, at 9:01 AM, Xin Wang 
> wrote:
> > 
> >  STORM-2198 ( PR: https://github.com/apache/storm/pull/1773 ) fixes
> a
> > >> bug
> > >>> of
> >  storm-hdfs. Do we have a consideration to include this?
> > 
> >  Thanks,
> >  Xin Wang (vesense)
> > 
> >  2016-11-15 10:03 GMT+08:00 Jungtaek Lim :
> > 
> > > Some issues on Storm SQL are resolved but not documented yet. I'll
> > >> file
> > >>> an
> > > issue and 

Re: [DISCUSS] Adopt pull request template

[Harsha Chintalapani]

   I am ok with adopting templates but lets keep this process simpler.
We've new contributors coming in and they probably didn't have chance to go
through process.
We can guide them through the process or have strict template that everyone
needs to adopt to.
   I don't think having guidelines such as having exactly 1 space after
the description is helpful to anyone. Its just a PR as long as the title
reflects the JIRA title and
JIRA is descriptive enough we all understand whats the patch intends do.
   We shouldn't be going through loops for opening a PR with
restrictions such as above.
Hugo,
  "STORM-XXX: Title matching exactly the JIRA Summary (title)" this
looks good to me. But having description details is hard to follow and
unnecessary.
regarding Squashing guidelines, we've gone through this discussion and I
think this should be left to author discretion as they would understand
what commits are more important.
Also as reviewers one can suggest to squash the commits if additional
commits doesn't make sense.

Thanks,
Harsha

On Thu, Jan 19, 2017 at 10:29 AM Hugo Da Cruz Louro 
wrote:

> I am strongly +1. In addition to the templates suggested by Jungtaek, in
> the past I suggested the template bellow to a different project. It is a
> bit simpler than the ones listed, as I think it may be too demanding to ask
> people to categorize in the git message the type of contribution the patch
> is addressing (BUG, Feature, …), if it has docs, if it has tests, etc. I
> also think that it may make the git message too verbose. That info is
> contained in the JIRA, and if there is a 1 to 1 mapping between Git commit
> and JIRA, it’s fairly easy to track down.
>
> The PR template is important, but more important, in my opinion, is to
> have strong guidelines requiring that people squash commits within
> functional units/features the code is addressing. Furthermore, if the
> commits are for one JIRA/change (when no JIRA), and there are more than one
> commit, all should have the same title (STORM-XXX) in it. We should at all
> costs avoid merging into master extreme cases like 10 commits related to
> one patch, where 5 or 6 of them are commits along the lines, addressed code
> review, added tests, cleaned up tests, addressed more code reviews, etc...
> without even having the STORM-XXX prefix in them.
>
> Squashing commits not only will make cherry-pick much easier, but more
> importantly, if there is a regression, it is very easy to track down which
> change is causing the regression. Not doing this makes it very hard to
> track down which change is causing problems.
>
> If we agree on some guidelines, I volunteer to update the documentation.
>
> === GIT TEMPLATE GUIDELINES ===
>
> STORM-XXX: Title matching exactly the JIRA Summary (title)
>
> - An optional, bulleted (+, -, ., *), description of the contents of
> - the patch. The goal is not to describe the contents of every file,
> - but rather give a quick overview of the main functional areas
> - addressed by the patch.
>
> The text immediately following the JIRA number (STORM-XXX: ) must be an
> exact transcription of the JIRA summary (title), not the a summary of the
> contents of the patch. If the JIRA summary does not accurately describe
> what the patch is addressing, the JIRA summary must be fixed, and then
> copied to the Git commit message.
>
> A description with the contents of the patch is optional but strongly
> encouraged if the patch is large and/or the JIRA title is not expressive
> enough to highlight what the patch is doing. This text must be added
> bellow, with one empty line of space in between, and be bulleted using one
> of the following bullet points (+, -, ., *). There must be at last a 1
> space indent before the bullet char, and exactly one space between bullet
> char and the first letter of the text. Bullets are not optional but
> required.
>
> === GIT TEMPLATE GUIDELINES FOR COMMITS WITHOUT JIRA ===
>
> For the sake of having a pattern one could grep, or exclude from grep, I
> suggest that we prefix every commit that does not have a JIRA associated
> with it with something along the lines:
>
> STORM-MINOR: Title summarizing what the patch is addressing.
>
> If a description of the change is necessary, it should follow the
> guidelines above. The squashing guidelines also apply.
>
> === SQUASHING GUIDELINES ===
>
> - Each commit message should have the first line matching the associated
> JIRA (STORM-XXX, or STORM-MINOR) as described
> - If possible, one JIRA/change should be one commit.
> - If the patch is large, and it makes sense to break it into multiple
> commits to make it easier to understand and cherry-pick (not easy to
> merge), each commit should represent the functional units/features the code
> is addressing
> - During code review it is OK to have multiple commits addressing the
> changes proposed by the reviewers (also because it makes it easy to track
> those changes), but once the 

Re: [DISCUSSION] Disable integration test

[Harsha Chintalapani]

Having integration tests run as part of the travis helps finding any
run-time issues that we might not be able to catch otherwise. Can you
please file a JIRA, Raghav who put this together might help.

Thanks,
Harsha

On Tue, Dec 27, 2016 at 6:15 AM Xin Wang  wrote:

> Hi Jungtaek,
>
> I agree with you. If we can't fix the issue in short time, we'd better to
> disable integration test.
>
> - Xin
>
> 2016-12-27 16:34 GMT+08:00 Jungtaek Lim :
>
> > Hi devs,
> >
> > Storm build result on Travis CI is failing most of time and it's because
> > integration test. Due to this, we don't see Travis CI result, and miss
> > other things like JDK compatibility (JDK 8 vs JDK 7 on between branches),
> > or RAT failing.
> >
> > Unless there's no volunteer to fix integration test in short time, I
> think
> > we would be better to disable integration test.
> >
> > What do you think?
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
>

Re: [DISCUSS] breaking changes in 2.x

[Harsha Chintalapani]

Why don't we start introducing protocol versions in thrift API. This way a
old supervisor can still communicate by providing a older version and new
nimbus can handle both protocols. API changes doesn't necessarily mean
we've to break the older API. If the existing users doesn't have good
upgrade path there will less uses on the new version.
Critical API changes can still be possible by versioning the API.
-Harsha

On Thu, Nov 10, 2016 at 7:06 AM Kyle Nusbaum <knusb...@yahoo-inc.com.invalid>
wrote:

> On Wednesday, November 9, 2016, 7:23:09 AM CST, Harsha Chintalapani <
> st...@harsha.io> wrote:> If we want users to upgrade to new version, the
> rolling upgrade is a major
> > decision factor. As a community, we need to look API updates or breaking
> > changes much more diligently.
> Within a major version, I agree. APIs should be as stable as possible
> within a version release.
>
> > I agree to an extent we shouldn't limiting ourselves with rolling
> upgrade.
> > But having announced rolling-upgrade in 0.10 and then not supporting it
> in
> > 1.x and now in 2.x. In User's point of view, Storm is not rolling
> > upgradable although we shipped a release stating that rolling upgrade is
> > supported and in follow-up release we taken that off.
> The user would be correct. Storm would not be rolling-upgradable *between
> major versions.*I don't see how it's possible to develop and improve a
> project if it must remain perpetually backwards compatible, so I think it's
> necessary to reject compatibility as a *primary* goal.
> Eventually (hopefully) we'll arrive at an API that we're happy with that
> we don't feel like we need to change.Then we can claim rolling upgrades
> across major version numbers.
>
> > Does these API changes are critical and worth breaking rolling upgrade?
> My position is that we don't want to limit ourselves to "critical" API
> changes. This will stick us with an inferior API that we can't evolve.It's
> accepting the long-term pain of inconsistent API or old baggage to avoid
> the short-term pain of relaunching or updating topologies when you do a
> major version upgrade.
> Storm is not at the place in its life where it has stopped evolving, and I
> don't want to stifle its development.
>

Re: [DISCUSS] breaking changes in 2.x

[Harsha Chintalapani]

If we want users to upgrade to new version, the rolling upgrade is a major
decision factor. As a community, we need to look API updates or breaking
changes much more diligently.
I agree to an extent we shouldn't limiting ourselves with rolling upgrade.
But having announced rolling-upgrade in 0.10 and then not supporting it in
1.x and now in 2.x. In User's point of view, Storm is not rolling
upgradable although we shipped a release stating that rolling upgrade is
supported and in follow-up release we taken that off.
Does these API changes are critical and worth breaking rolling upgrade?

-Harsha

On Mon, Nov 7, 2016 at 9:16 AM Kyle Nusbaum 
wrote:

I worry that making it a priority to have rolling upgrades between major
versions significantly restricts the kinds of changes that we can make,
including some kinds of changes that a major version increment is supposed
to mark. I'm not really in support of trying to do that.
If we can't make changes that break compatibility now, when can we make
them? Can changes like that ever be made? I don't know that it's good to
limit ourselves like that.
Trying to accommodate rolling upgrades is a good idea, but I'm not sure
about rejecting code changes across major versions to support them. 2.x
represents a large shift in the project, and I expect once the translation,
etc. is done, things will calm down and APIs will become more stable,
allowing more of our future releases to be rolling even across major
versions. I'd rather get these kinds of changes out of the way now in the
2.x release than cart along the vestiges of 1.x from now on.
What do others think about this?
-- Kyle

On Monday, November 7, 2016, 3:10:08 AM CST, Bobby Evans
 wrote:Let's distinguish between wire
compatibility changes and API compatibility changes, along with impact to
workers vs impact to clients.
For 3) splitting the classpath up for each daemon wire compatibility is not
impacted, but we are potentially removing a bunch of APIs from the worker
and client classpath.  Most of these where shaded and users should not be
impacted by them being removed, but a few like servlet-api-2.5.jar are
likely to be removed.  So yes the impact here would likely be very small.
However on the client side if a topology wants to include LocalCluster
(like we do in a lot of examples) the topology jar will get a lot bigger.
LocalCluster needs access to nimbus, supervisor, and drpc server.  These
would not be on the worker classpath any more and would then need to be
packaged into the topology jar to make LocalCluster work.  In production I
would expect LocalCluster to be used by tests and not be included like we
do in a lot of examples.  This is more of a shift for how we expect users
to interact with the LocalCluster.
For 1) NodeInfo.port depending on how we do it, it could break wire
compatibility and API compatibility (which is what I would prefer).  We
could play some games to maintain compatibility, but for me it is enough of
a pain that I am not sure it is worth it.  However this is not likely to
impact workers because they don't use these APIs directly.  It might impact
clients but only if they have custom code to profile their topologies.  IF
they use the build in CLI/UI it would not be impacted.
For 2) nocamel would break API compatibility, but not wire compatibility.
This is not likely to impact workers because like with 1 workers don't
really interact with nimbus directly so it would not be a problem.  Old
clients running with older versions of storm would continue to work, but
any custom client code (think what gets run by storm jar) would need to be
recompiled/modified to be able to run on against a storm-2.x client.

- Bobby

Re: Fw: Re: [DISCUSS] breaking changes in 2.x

[Harsha Chintalapani]

My only concern here is the rolling upgrade of storm cluster. We supported
the rolling upgrade going to 0.10 and broke it because of storm 1.x
release. Users are not inclined to upgrade to a new release if it's not
rolling upgradable. In this case, it looks like we are going to break this.
Correct me if I am wrong here.

Thanks,
Harsha

On Mon, Nov 7, 2016 at 7:43 AM Bobby Evans 
wrote:

> Made a mistake and put something on private that never should have been
> there.  Here is the discussion in full so far.
> In response to Jungtake removing the nocamel option will change
> set_bar/get_bar in the generated thrift code to setBar/getBar.  So any
> thrift object that clients interact with will need to make similar changes.
> - Bobby
>
>  - Forwarded Message -From: Bobby Evans To: "
> priv...@storm.apache.org" Sent: Monday,
> November 7, 2016, 3:37:50 AM CST Subject: Re: [DISCUSS] breaking changes in
> 2.x
>  Sorry you are right, I put in the wrong mailing list.  I feel dumb.  Will
> move it to dev.
> - Bobby
>
> On Sunday, November 6, 2016, 2:18:33 AM CST, P. Taylor Goetz <
> ptgo...@gmail.com> wrote:Can we move this discussion to dev@? There
> doesn't seem to be anything sensitive about the subject.
> -Taylor
> On Nov 6, 2016, at 1:28 AM, Jungtaek Lim  wrote:
>
>
> Regarding breaking change, I'm OK to break backward UX compatibility if
> we're sure that it gives better UX. And since we're talking about next
> major, if we don't do it now, we might have no chance to do the near future.
> For 1) this should be corrected, and for 2) I'm not sure how it affects to
> end-users, but it would be better to address if it doesn't hurt user codes
> much. 3) I think expected user impact is similar to what we did it for
> adding dependencies to shade list. Then it doesn't look matter to me.
> Thanks,Jungtaek Lim (HeartSaVioR)
> 2016년 11월 5일 (토) 오후 11:01, Bobby Evans 님이 작성:
>
> I have been working on getting some of clojure code translated to java,
> and I have run into a few issues that I really would like to address, but
> will impact users.  Because 2.x has not been released yet technically these
> should be OK, but I want to get feedback from others before I spend a lot
> of time on them or even file a JIRA.
> 1)WHAT: The thrift NodeInfo.port should be a single long, not a set of
> longs.USER IMPACT: it is exposed to end users through the profiling request
> API. BACK COMPAT: it is possible, but I would prefer not to.
>
> 2)WHAT: Using 'nocamel' when generating thrift makes generated exceptions
> less useful and the API less java like.USER IMPACT: users when upgrading
> would need to modify some API calls when setting/reading thrift
> objects.BACK COMPAT: most older clients should still work for a lot of
> things
> 3) WHAT: split up storm-core so we have class path separation between
> daemons and a true client.USER IMPACT: Users may need to include new
> dependencies to get what they want.  Old topologies may not get access to
> everything that they want.BACK COMPAT:  we could leave some old
> dependencies in place that are not needed, but why?
> If we do all of these it is likely that profiling requests from an old
> client will not work, but for the most part a 1.x client would still work
> with 2.x.
> - Bobby
>
>

Re: [DISCUSS] Accept JW Player SQE Code Donation

[Harsha Chintalapani]

>From the looks of it the benefit accepting this code donation is to attract
more developers on storm-sql . What is stopping SQE developers to come and
contribute to JIRAs that are open on storm-sql?.  I don't see just
accepting code donation and setting aside is not much of a
incentive/motivation for someone to contribute.

-Harsha

On Thu, Aug 25, 2016 at 12:14 PM P. Taylor Goetz  wrote:

> I didn’t mean to imply that the code donation would come with automatic
> committership. I would expect the SQE developers to engage with the Storm
> community and show merit before being invited to become committers. If for
> some reason that didn’t happen, we would always have the option of not
> doing anything with the code donation.
>
> With regard to it recently being open sourced, yes that is the case. JW
> Player open sourced it in order to make the code donation, which explains
> the number of commits/contributors. I’m not sure if it’s due to a github
> auto-watch setting, but the project has a lot of watchers from JW Player
> (which might be an indicator of potential contributor pool):
>
> https://github.com/jwplayer/SQE/watchers
>
> When they approached me about the code contribution, I mentioned the
> existing SQL work that’s been/being done. They are aware of that effort,
> and see this as a collaboration opportunity to improve SQL support in Storm
> (i.e. not a competitor to the existing SQL work).
>
> Personally I would prefer a single SQL API/implementation for Storm. If
> that involves cherry-picking/merging two projects into one, I’m okay with
> that.
>
> -Taylor
>
>
>
> On Aug 25, 2016, at 9:30 AM, Bobby Evans 
> wrote:
>
> I agree that committership should not come form a code donation, but on
> the other hand a code donation needs to have a community with it to be
> viable.
>
> That is the one thing that makes slightly nervous here.
> https://github.com/jwplayer/SQE/pulse
>
> https://github.com/jwplayer/SQE/graphs/contributors
>
> show that the code was very recently open sourced and only one person has
> made two commits.  Which totally makes since from the perspective of a
> corporation that just released the code.  But by the same right I would
> like to hear from others to see how many people are currently interested in
> helping to maintain this new code base, and also what is each person's long
> term plan/vision for this code.
> HeartSavior has indicated his intention is to try and combine the back-end
> with SQL which seems like a good idea, but at the same time others have
> expressed concern that we have multiple different APIs and execution
> environments for Storm.  Adding another without any plan on where we want
> to go long term gives me pause.
>  - Bobby
>
>On Thursday, August 25, 2016 5:07 AM, Jungtaek Lim 
> wrote:
>
>
> I'd postpone to talk about committership a bit later after they show active
> contributions. Personally I don't like associating code donation and
> committership but it's just me.
> Storm SQL (or SQL feature for all projects) has lots of places for
> contributions so they can be eventually invited like other commiters/PMCs,
> similar to John Fang.
>
> Regarding future of storm-sql after SQE, it would be better to discuss
> again when the process of code donation is in place, so that we can discuss
> with comparison between storm-sql and SQE at that time.
>
> Thanks,
> Jungtaek Lim (HeartSaVIoR)
>
> 2016년 8월 25일 (목) 오후 3:44, Abhishek Agarwal 님이 작성:
>
> what happens to current storm-sql once SQE is merged into apache
> repository?
>
> On Thu, Aug 25, 2016 at 10:26 AM, P. Taylor Goetz 
> wrote:
>
>
> On Aug 24, 2016, at 5:37 PM, Jungtaek Lim  wrote:
>
> While I feel Storm SQL covers (and will cover in near future) all of
>
> the
>
> features what SQE has, I'd like to consider this as not only code
>
> donation
>
> but also having more contributors on Storm SQL.
>
>
> +1
>
> This is a great opportunity to grow the community interested in streaming
> SQL.
>
>
> As the one working on storm-sql now, I think Storm SQL should have
> more contributors who are experienced or expert on this area, or at
>
> least
>
> have a mind to contribute heavily.
>
>
> +1 again.
>
> In my conversations with JW Player, they indicated that their developers
> are very interested in joining the Storm community.
>
> There're still many features to implement (join and windowing,
>
> parallelism,
>
> and so on) which needs some efforts so we can boost up if more
>
> contributors
>
> would play with Storm SQL.
> And "optimization" is at the heart of SQL and we haven't had time, and
> knowledge, and experience to address that.
>
>
> SQE seems kind of ahead to a certain degree in this respect.
>
>
> IMHO, Storm SQL seems to be going on the right track so I don't think
>
> we
>
> need to replace the codebase or merge. We can still apply the advanced
> areas of SQE to 

Re: Rolling upgrades for a topology using kafka spout

[Harsha Chintalapani]

Abhishek,
Are you looking rolling upgrade kafka cluster or storm?
Harsha

On Fri, Aug 26, 2016 at 6:18 AM Abhishek Agarwal 
wrote:

>
> On Aug 26, 2016 2:50 PM, "Abhishek Agarwal"  wrote:
>
> >
>
> > Here is an interesting use case - To upgrade a topology without any
> downtime. Let's say, the topology has only Kafka as a source and two
> versions of it are running (different topology names of course) in parallel
> and sharing the kafka input load.
> >
> > In old kafka spout, rolling upgrade is not possible, partition
> assignment is derived from the number of tasks in the topology.
> >
> > In new kafka spout, partition assignment is done externally by Kafka
> server. If I deploy two different topologies with same* kafka consumer
> group id*, is it fair to assume that load will be automatically
> distributed across topologies? Are there any corner cases to consider?
> >
> > --
> > Regards,
> > Abhishek Agarwal
> >
>

Re: [VOTE] Release Apache Storm 0.10.2 (RC1)

[Harsha Chintalapani]

We need to get this patch in for 0.10 release
https://github.com/apache/storm/pull/1645

Thanks,
Harsha

On Fri, Aug 26, 2016 at 7:28 AM Bobby Evans 
wrote:

> +1 - Bobby
>
> On Friday, August 26, 2016 9:03 AM, Jungtaek Lim 
> wrote:
>
>
>  +1 (binding)
>
> - testing with source distribution : OK
>   - verify signature (zip, tar.gz): OK
>   - uncompress : OK
>   - building from source dist : OK
>   - how to build: running `mvn clean install` on unzipped source dist.
>
> - testing with binary distribution (one machine) : OK
>   - verify signature (zip, tar.gz): OK
>  - launch daemons : OK
>  - run RollingTopWords (local) : OK
>  - run RollingTopWords (remote) : OK
>   - activate / deactivate / rebalance / kill : OK
>   - logviewer (worker) : OK
>   - run WordCountTopology (multilang, local) : OK
>   - run WordCountTopology (multilang, remote) : OK
>
> Btw, would we want to include STORM-2042 to 0.10.2?
> I agree this is definitely not a blocker, but we have been releasing 0.10.x
> line slowly so might be worth to consider.
> As I voted +1 I'm also OK to not include it.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 8월 25일 (목) 오전 5:22, P. Taylor Goetz 님이 작성:
>
> > This is a call to vote on releasing Apache Storm 0.10.2 (rc1)
> >
> > Full list of changes in this release:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=3c24dce8b35527b409cae68a1ebc0e25e5a0b03f
> >
> > The tag/commit to be voted upon is v0.10.2:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=2c356b9c1085d35cfc6d0869fcca1d570910053e;hb=3c24dce8b35527b409cae68a1ebc0e25e5a0b03f
> >
> > The source archive being voted upon can be found here:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.2-rc1/apache-storm-0.10.2-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.10.2-rc1/
> >
> > The release artifacts are signed with the following key:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1040
> >
> > Please vote on releasing this package as Apache Storm 0.10.2.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 0.10.2
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
> >
>
>

Re: [Vote] Make Java 8 as minimum requirement for Storm 2.0

[Harsha Chintalapani]

I guess everyone has different interpretation of what Bylaws means . More
context

https://github.com/apache/storm/pull/1628

anything wrong with Vote thread?


On Tue, Aug 16, 2016 at 8:04 PM P. Taylor Goetz <ptgo...@gmail.com> wrote:

> Why is this a VOTE?
>
>
> > On Aug 16, 2016, at 9:18 PM, Harsha Chintalapani <st...@harsha.io>
> wrote:
> >
> > Hi All,
> >We had a discussion thread for removing Java 7 support for Storm
> > 2.0.
> > Here is a formal voting thread and the JIRA
> > https://issues.apache.org/jira/browse/STORM-2041.
> >
> > Thanks,
> > Harsha
>
>

[Vote] Make Java 8 as minimum requirement for Storm 2.0

[Harsha Chintalapani]

Hi All,
We had a discussion thread for removing Java 7 support for Storm
2.0.
Here is a formal voting thread and the JIRA
https://issues.apache.org/jira/browse/STORM-2041.

Thanks,
Harsha

Re: [VOTE] Release Apache Storm 0.9.7 (RC1)

[Harsha Chintalapani]

Do we have any user requests on releasing 0.9.x . I rather propose them to
move on to 0.10.x line and retire 0.9.x branches. There are quite few
issues that got fixed in 0.10.x release line and keep maintaining 0.9.x
line wouldn't be beneficial.

Thanks,
Harsha

On Mon, Aug 15, 2016 at 4:48 PM Jungtaek Lim  wrote:

> Would it be better to have consensus for this?
>
> In bylaw we have this sentence 'In particular all releases must be approved
> by the PMC.'
> One of PMC can call out the discussion for releasing specific version, but
> general consensus should be made before starting prepare release.
>
> I'd like to see this starting from there. If we just want to also address
> opinions (not verification) of release from VOTE thread, I'll do that.
>
> Btw, personally I'm -1 for this release, since STORM-1928 is even not
> backported to 0.10.x version line.
> I'm even not supportive to maintain 0.x line (because we did the best
> effort for 0.x line users to move on 1.x), but if we really want to ship
> this bugfix to 0.x version line, for me it's more making sense to release
> 0.10.2.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 8월 16일 (화) 오전 5:10, P. Taylor Goetz 님이 작성:
>
> > This is a call to vote on releasing Apache Storm 0.9.7 (rc1).
> >
> > This release candidate addresses a critical issue with Storm’s multi-lang
> > component where an improperly formed multi-lang spout message would cause
> > the spout to hang indefinitely.
> >
> > Full list of changes in this release:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=b75d9b2b6dbfd0fda0b114feafec1384bbfa30aa
> >
> > The tag/commit to be voted upon is v0.9.7:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h=3d7c7be37ecc1dcc25223fde670d8185a6afbf00;hb=b75d9b2b6dbfd0fda0b114feafec1384bbfa30aa
> >
> > The source archive being voted upon can be found here:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.9.7-rc1//apache-storm-0.9.7-src.tar.gz
> >
> > Other release files, signatures and digests can be found here:
> >
> > https://dist.apache.org/repos/dist/dev/storm/apache-storm-0.9.7-rc1/
> >
> > The release artifacts are signed with the following key:
> >
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
> >
> > The Nexus staging repository for this release is:
> >
> > https://repository.apache.org/content/repositories/orgapachestorm-1039
> >
> > Please vote on releasing this package as Apache Storm 0.9.7.
> >
> > When voting, please list the actions taken to verify the release.
> >
> > This vote will be open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache Storm 0.9.7
> > [ ]  0 No opinion
> > [ ] -1 Do not release this package because...
> >
> > Thanks to everyone who contributed to this release.
> >
> > -Taylor
> >
>

[Discussion] Dropping Java 7 support on master

[Harsha Chintalapani]

Hi All,
  Dropping java 7 support on master will allow us to use the new api in
Java 8 and since the master is being used for java migration its good to
make the decision now. Let me know your thoughts.

Thanks,
Harsha

Re: [VOTE] Release Apache Storm 1.0.2 (rc4)

[Harsha Chintalapani]

+1 (binding)
Ran a single node cluster and tried example topologies. Verified signatures
on the artifcats.
-Harsha

On Fri, Aug 5, 2016 at 8:02 AM P. Taylor Goetz  wrote:

> +1 (binding)
>
> Ran on a 3-node cluster and verified fixes.
>
> -Taylor
>
> On Jul 26, 2016, at 4:04 PM, P. Taylor Goetz  wrote:
>
> This is a call to vote on releasing Apache Storm 1.0.2 (rc4)
>
> Full list of changes in this release:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=c9768c154166b6217ec7bffc4a9aa73e90f2339d
>
> The tag/commit to be voted upon is v1.0.2:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=CHANGELOG.md;hb=54f319fd56c33437364c550a800ac9e6fe058b95
>
> The source archive being voted upon can be found here:
>
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.2-rc4/apache-storm-1.0.2-src.tar.gz
>
> Other release files, signatures and digests can be found here:
>
> https://dist.apache.org/repos/dist/dev/storm/apache-storm-1.0.2-rc4/
>
> The release artifacts are signed with the following key:
>
>
> https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd
>
> The Nexus staging repository for this release is:
>
> https://repository.apache.org/content/repositories/orgapachestorm-1038
>
> Please vote on releasing this package as Apache Storm 1.0.2.
>
> When voting, please list the actions taken to verify the release.
>
> This vote will be open for at least 72 hours.
>
> [ ] +1 Release this package as Apache Storm 1.0.2
> [ ]  0 No opinion
> [ ] -1 Do not release this package because...
>
> Thanks to everyone who contributed to this release.
>
> -Taylor
>
>
>