Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[Jungtaek Lim]

+1 from me, too.

On Sat, 11 Mar 2017 at 5:05 AM P. Taylor Goetz  wrote:

> Yes, that's the plan. All examples would be source only with poms that
> allow the user to compile.
>
> -Taylor
>
> > On Mar 10, 2017, at 10:28 AM, Bobby Evans 
> wrote:
> >
> > +1 for that, although I think it is the examples that ends up being the
> heavy part.  The examples are all uber jars and pull in lots of
> dependencies.  Some of them multiple times over.  Do we want the examples
> to only be a source release so before running an example they have to build
> it?
> >
> >
> > - Bobby
> >
> > On Thursday, March 9, 2017, 2:22:42 PM CST, P. Taylor Goetz <
> ptgo...@gmail.com> wrote:I’m fine with dropping connectors from the
> binary distribution in favor having users use Maven, gradle, etc. handle
> user dependencies. Building topology jars manually (vs. letting Maven do
> the heavy lifting) is painful and something to be avoided, specifically for
> new users.
> >
> > So the “binary release” for connectors, etc. would essentially become
> just a README that documents how to use them, and what the Maven
> coordinates are for each component.
> >
> > So with the connector jars just hosted in the Maven repo, and all
> examples source-only (to be built by users) the size of the binary
> distribution should be reduced dramatically.
> >
> > Any other opinions?
> >
> > -Taylor
> >
> >
> >> On Mar 8, 2017, at 4:26 PM, Jungtaek Lim  wrote:
> >>
> >> Regarding 3, if we don't make uber jar for connectors, adding jar file
> in
> >> binary dist. to the classpath will not work, which means that including
> >> connector jar libraries in binary dist. is no effect.
> >> (Someone proposed shading storm-redis because of dependencies... We can
> >> cover it by various way like adding to pom, or using '--artifact' so no
> >> issue at all for now.)
> >>
> >> So if we want to go with 3, I'd also like to propose excluding connector
> >> libraries in binary dist. at all.
> >>
> >> Others look great to me.
> >>
> >> - Jungtaek Lim (HeartSaVioR)
> >>
> >> 2017년 3월 9일 (목) 오전 6:12, P. Taylor Goetz 님이 작성:
> >>
> >>> I’d like to propose the following to address the binary distribution
> size:
> >>>
> >>> 1. Finish moving connector examples to the “examples” directory.
> >>> 2. Make all examples source-only in the binary distribution. This will
> >>> mean users will have to build the examples themselves, but that’s
> typically
> >>> what one does when looking at example code.
> >>> 3. Stop unnecessarily shading connector libraries (i.e. let the user’s
> pom
> >>> pull in the transitive dependencies when building a topology jar).
> >>>
> >>> If we agree on this approach, I’ll take up the necessary changes for
> the
> >>> 1.1.0 release.
> >>>
> >>> -Taylor
> >>>
> >>>
> >>>
>  On Mar 8, 2017, at 9:16 AM, Xin Wang  wrote:
> 
>  Agree with Jungtaek, STORM-2390 should not block the 1.1.0 release.
> 
>  - Xin
> 
>  2017-03-07 9:16 GMT+08:00 Jungtaek Lim :
> 
> > Sorry Roshan but I don't get it.
> > We're already having separate modules for each of the connectors, and
> >>> even
> > examples. We also don't have any common modules for external
> >>> connectors, so
> > those are completely individual.
> >
> > As I reported earlier, binary dist. will go up more than 500 MB when
> we
> > address STORM-2390(STORM-2249 actually), which the community seemed
> not
> > want to.
> >
> > Since STORM-2390 is linked to epic for release Storm 1.1.0, I'd
> rather
> > unlink from epic to get out of release, and discuss more.
> > Any objections?
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> > 2017년 3월 4일 (토) 오전 10:02, Roshan Naik 님이 작성:
> >
> >> As having a separate module for each of the connectors will cause a
> lot
> > of
> >> bloat … may be good to club them into one.
> >>
> >> -roshan
> >>
> >>
> >> On 3/2/17, 10:39 AM, "Sree V" 
> wrote:
> >>
> >>   +1separate main binaries, connectors and samples/examples.
> >>
> >>   Thanking you.
> >>   With Regards
> >>   Sree
> >>
> >>   On Thursday, March 2, 2017 8:02 AM, Satish Duggana <
> >> satish.dugg...@gmail.com> wrote:
> >>
> >>
> >> Agree that such huge binaries may not be acceptable. We should
> > really
> >> think
> >>   about the options proposed earlier like excluding some of the
> > external
> >>   connectors from binary and update the documentation respectively.
> >>
> >>   Thanks,
> >>   Satish.
> >>
> >>   On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim 
> >> wrote:
> >>
> >>> Adding my observation to my last mail:
> >>> I just give STORM-2249 a try (against master branch) and compare
> 

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[P. Taylor Goetz]

Yes, that's the plan. All examples would be source only with poms that allow 
the user to compile.

-Taylor

> On Mar 10, 2017, at 10:28 AM, Bobby Evans  wrote:
> 
> +1 for that, although I think it is the examples that ends up being the heavy 
> part.  The examples are all uber jars and pull in lots of dependencies.  Some 
> of them multiple times over.  Do we want the examples to only be a source 
> release so before running an example they have to build it?
> 
> 
> - Bobby
> 
> On Thursday, March 9, 2017, 2:22:42 PM CST, P. Taylor Goetz 
>  wrote:I’m fine with dropping connectors from the binary 
> distribution in favor having users use Maven, gradle, etc. handle user 
> dependencies. Building topology jars manually (vs. letting Maven do the heavy 
> lifting) is painful and something to be avoided, specifically for new users.
> 
> So the “binary release” for connectors, etc. would essentially become just a 
> README that documents how to use them, and what the Maven coordinates are for 
> each component.
> 
> So with the connector jars just hosted in the Maven repo, and all examples 
> source-only (to be built by users) the size of the binary distribution should 
> be reduced dramatically.
> 
> Any other opinions?
> 
> -Taylor
> 
> 
>> On Mar 8, 2017, at 4:26 PM, Jungtaek Lim  wrote:
>> 
>> Regarding 3, if we don't make uber jar for connectors, adding jar file in
>> binary dist. to the classpath will not work, which means that including
>> connector jar libraries in binary dist. is no effect.
>> (Someone proposed shading storm-redis because of dependencies... We can
>> cover it by various way like adding to pom, or using '--artifact' so no
>> issue at all for now.)
>> 
>> So if we want to go with 3, I'd also like to propose excluding connector
>> libraries in binary dist. at all.
>> 
>> Others look great to me.
>> 
>> - Jungtaek Lim (HeartSaVioR)
>> 
>> 2017년 3월 9일 (목) 오전 6:12, P. Taylor Goetz 님이 작성:
>> 
>>> I’d like to propose the following to address the binary distribution size:
>>> 
>>> 1. Finish moving connector examples to the “examples” directory.
>>> 2. Make all examples source-only in the binary distribution. This will
>>> mean users will have to build the examples themselves, but that’s typically
>>> what one does when looking at example code.
>>> 3. Stop unnecessarily shading connector libraries (i.e. let the user’s pom
>>> pull in the transitive dependencies when building a topology jar).
>>> 
>>> If we agree on this approach, I’ll take up the necessary changes for the
>>> 1.1.0 release.
>>> 
>>> -Taylor
>>> 
>>> 
>>> 
 On Mar 8, 2017, at 9:16 AM, Xin Wang  wrote:
 
 Agree with Jungtaek, STORM-2390 should not block the 1.1.0 release.
 
 - Xin
 
 2017-03-07 9:16 GMT+08:00 Jungtaek Lim :
 
> Sorry Roshan but I don't get it.
> We're already having separate modules for each of the connectors, and
>>> even
> examples. We also don't have any common modules for external
>>> connectors, so
> those are completely individual.
> 
> As I reported earlier, binary dist. will go up more than 500 MB when we
> address STORM-2390(STORM-2249 actually), which the community seemed not
> want to.
> 
> Since STORM-2390 is linked to epic for release Storm 1.1.0, I'd rather
> unlink from epic to get out of release, and discuss more.
> Any objections?
> 
> - Jungtaek Lim (HeartSaVioR)
> 
> 2017년 3월 4일 (토) 오전 10:02, Roshan Naik 님이 작성:
> 
>> As having a separate module for each of the connectors will cause a lot
> of
>> bloat … may be good to club them into one.
>> 
>> -roshan
>> 
>> 
>> On 3/2/17, 10:39 AM, "Sree V"  wrote:
>> 
>>   +1separate main binaries, connectors and samples/examples.
>> 
>>   Thanking you.
>>   With Regards
>>   Sree
>> 
>>   On Thursday, March 2, 2017 8:02 AM, Satish Duggana <
>> satish.dugg...@gmail.com> wrote:
>> 
>> 
>> Agree that such huge binaries may not be acceptable. We should
> really
>> think
>>   about the options proposed earlier like excluding some of the
> external
>>   connectors from binary and update the documentation respectively.
>> 
>>   Thanks,
>>   Satish.
>> 
>>   On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim 
>> wrote:
>> 
>>> Adding my observation to my last mail:
>>> I just give STORM-2249 a try (against master branch) and compare
>> before vs
>>> after.
>>> It adds more than 300 MB, and the final archive is more than 550
> MB.
>> I
>>> guess it would be similar for 1.x branch.
>>> 
>>> Before --
>>> -rw-r--r--  1 jlim  staff  3.3K  3  2 19:41
>> apache-storm-2.0.0-SNAPSHOT.

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[Bobby Evans]

+1 for that, although I think it is the examples that ends up being the heavy 
part.  The examples are all uber jars and pull in lots of dependencies.  Some 
of them multiple times over.  Do we want the examples to only be a source 
release so before running an example they have to build it?


- Bobby

On Thursday, March 9, 2017, 2:22:42 PM CST, P. Taylor Goetz  
wrote:I’m fine with dropping connectors from the binary distribution in favor 
having users use Maven, gradle, etc. handle user dependencies. Building 
topology jars manually (vs. letting Maven do the heavy lifting) is painful and 
something to be avoided, specifically for new users.

So the “binary release” for connectors, etc. would essentially become just a 
README that documents how to use them, and what the Maven coordinates are for 
each component.

So with the connector jars just hosted in the Maven repo, and all examples 
source-only (to be built by users) the size of the binary distribution should 
be reduced dramatically.

Any other opinions?

-Taylor


> On Mar 8, 2017, at 4:26 PM, Jungtaek Lim  wrote:
> 
> Regarding 3, if we don't make uber jar for connectors, adding jar file in
> binary dist. to the classpath will not work, which means that including
> connector jar libraries in binary dist. is no effect.
> (Someone proposed shading storm-redis because of dependencies... We can
> cover it by various way like adding to pom, or using '--artifact' so no
> issue at all for now.)
> 
> So if we want to go with 3, I'd also like to propose excluding connector
> libraries in binary dist. at all.
> 
> Others look great to me.
> 
> - Jungtaek Lim (HeartSaVioR)
> 
> 2017년 3월 9일 (목) 오전 6:12, P. Taylor Goetz 님이 작성:
> 
>> I’d like to propose the following to address the binary distribution size:
>> 
>> 1. Finish moving connector examples to the “examples” directory.
>> 2. Make all examples source-only in the binary distribution. This will
>> mean users will have to build the examples themselves, but that’s typically
>> what one does when looking at example code.
>> 3. Stop unnecessarily shading connector libraries (i.e. let the user’s pom
>> pull in the transitive dependencies when building a topology jar).
>> 
>> If we agree on this approach, I’ll take up the necessary changes for the
>> 1.1.0 release.
>> 
>> -Taylor
>> 
>> 
>> 
>>> On Mar 8, 2017, at 9:16 AM, Xin Wang  wrote:
>>> 
>>> Agree with Jungtaek, STORM-2390 should not block the 1.1.0 release.
>>> 
>>> - Xin
>>> 
>>> 2017-03-07 9:16 GMT+08:00 Jungtaek Lim :
>>> 
 Sorry Roshan but I don't get it.
 We're already having separate modules for each of the connectors, and
>> even
 examples. We also don't have any common modules for external
>> connectors, so
 those are completely individual.
 
 As I reported earlier, binary dist. will go up more than 500 MB when we
 address STORM-2390(STORM-2249 actually), which the community seemed not
 want to.
 
 Since STORM-2390 is linked to epic for release Storm 1.1.0, I'd rather
 unlink from epic to get out of release, and discuss more.
 Any objections?
 
 - Jungtaek Lim (HeartSaVioR)
 
 2017년 3월 4일 (토) 오전 10:02, Roshan Naik 님이 작성:
 
> As having a separate module for each of the connectors will cause a lot
 of
> bloat … may be good to club them into one.
> 
> -roshan
> 
> 
> On 3/2/17, 10:39 AM, "Sree V"  wrote:
> 
>  +1separate main binaries, connectors and samples/examples.
> 
>  Thanking you.
>  With Regards
>  Sree
> 
>      On Thursday, March 2, 2017 8:02 AM, Satish Duggana <
> satish.dugg...@gmail.com> wrote:
> 
> 
>    Agree that such huge binaries may not be acceptable. We should
 really
> think
>  about the options proposed earlier like excluding some of the
 external
>  connectors from binary and update the documentation respectively.
> 
>  Thanks,
>  Satish.
> 
>  On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim 
> wrote:
> 
>> Adding my observation to my last mail:
>> I just give STORM-2249 a try (against master branch) and compare
> before vs
>> after.
>> It adds more than 300 MB, and the final archive is more than 550
 MB.
> I
>> guess it would be similar for 1.x branch.
>> 
>> Before --
>> -rw-r--r--  1 jlim  staff  3.3K  3  2 19:41
> apache-storm-2.0.0-SNAPSHOT.
>> pom
>> -rw-r--r--  1 jlim  staff  473B  3  2 19:41
> apache-storm-2.0.0-SNAPSHOT.
>> pom.asc
>> -rw-r--r--  1 jlim  staff  264M  3  2 19:41
> apache-storm-2.0.0-SNAPSHOT.
>> tar.gz
>> -rw-r--r--  1 jlim  staff  473B  3  2 19:41
> apache-storm-2.0.0-SNAPSHOT.
>> tar.gz.asc
>> -rw-r--r--  1 jlim  staff  264M  3  2 19:41

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[P. Taylor Goetz]

I’m fine with dropping connectors from the binary distribution in favor having 
users use Maven, gradle, etc. handle user dependencies. Building topology jars 
manually (vs. letting Maven do the heavy lifting) is painful and something to 
be avoided, specifically for new users.

So the “binary release” for connectors, etc. would essentially become just a 
README that documents how to use them, and what the Maven coordinates are for 
each component.

So with the connector jars just hosted in the Maven repo, and all examples 
source-only (to be built by users) the size of the binary distribution should 
be reduced dramatically.

Any other opinions?

-Taylor


> On Mar 8, 2017, at 4:26 PM, Jungtaek Lim  wrote:
> 
> Regarding 3, if we don't make uber jar for connectors, adding jar file in
> binary dist. to the classpath will not work, which means that including
> connector jar libraries in binary dist. is no effect.
> (Someone proposed shading storm-redis because of dependencies... We can
> cover it by various way like adding to pom, or using '--artifact' so no
> issue at all for now.)
> 
> So if we want to go with 3, I'd also like to propose excluding connector
> libraries in binary dist. at all.
> 
> Others look great to me.
> 
> - Jungtaek Lim (HeartSaVioR)
> 
> 2017년 3월 9일 (목) 오전 6:12, P. Taylor Goetz 님이 작성:
> 
>> I’d like to propose the following to address the binary distribution size:
>> 
>> 1. Finish moving connector examples to the “examples” directory.
>> 2. Make all examples source-only in the binary distribution. This will
>> mean users will have to build the examples themselves, but that’s typically
>> what one does when looking at example code.
>> 3. Stop unnecessarily shading connector libraries (i.e. let the user’s pom
>> pull in the transitive dependencies when building a topology jar).
>> 
>> If we agree on this approach, I’ll take up the necessary changes for the
>> 1.1.0 release.
>> 
>> -Taylor
>> 
>> 
>> 
>>> On Mar 8, 2017, at 9:16 AM, Xin Wang  wrote:
>>> 
>>> Agree with Jungtaek, STORM-2390 should not block the 1.1.0 release.
>>> 
>>> - Xin
>>> 
>>> 2017-03-07 9:16 GMT+08:00 Jungtaek Lim :
>>> 
 Sorry Roshan but I don't get it.
 We're already having separate modules for each of the connectors, and
>> even
 examples. We also don't have any common modules for external
>> connectors, so
 those are completely individual.
 
 As I reported earlier, binary dist. will go up more than 500 MB when we
 address STORM-2390(STORM-2249 actually), which the community seemed not
 want to.
 
 Since STORM-2390 is linked to epic for release Storm 1.1.0, I'd rather
 unlink from epic to get out of release, and discuss more.
 Any objections?
 
 - Jungtaek Lim (HeartSaVioR)
 
 2017년 3월 4일 (토) 오전 10:02, Roshan Naik 님이 작성:
 
> As having a separate module for each of the connectors will cause a lot
 of
> bloat … may be good to club them into one.
> 
> -roshan
> 
> 
> On 3/2/17, 10:39 AM, "Sree V"  wrote:
> 
>   +1separate main binaries, connectors and samples/examples.
> 
>   Thanking you.
>   With Regards
>   Sree
> 
>   On Thursday, March 2, 2017 8:02 AM, Satish Duggana <
> satish.dugg...@gmail.com> wrote:
> 
> 
>Agree that such huge binaries may not be acceptable. We should
 really
> think
>   about the options proposed earlier like excluding some of the
 external
>   connectors from binary and update the documentation respectively.
> 
>   Thanks,
>   Satish.
> 
>   On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim 
> wrote:
> 
>> Adding my observation to my last mail:
>> I just give STORM-2249 a try (against master branch) and compare
> before vs
>> after.
>> It adds more than 300 MB, and the final archive is more than 550
 MB.
> I
>> guess it would be similar for 1.x branch.
>> 
>> Before --
>> -rw-r--r--  1 jlim  staff  3.3K  3  2 19:41
> apache-storm-2.0.0-SNAPSHOT.
>> pom
>> -rw-r--r--  1 jlim  staff  473B  3  2 19:41
> apache-storm-2.0.0-SNAPSHOT.
>> pom.asc
>> -rw-r--r--  1 jlim  staff  264M  3  2 19:41
> apache-storm-2.0.0-SNAPSHOT.
>> tar.gz
>> -rw-r--r--  1 jlim  staff  473B  3  2 19:41
> apache-storm-2.0.0-SNAPSHOT.
>> tar.gz.asc
>> -rw-r--r--  1 jlim  staff  264M  3  2 19:41
> apache-storm-2.0.0-SNAPSHOT.
>> zip
>> -rw-r--r--  1 jlim  staff  473B  3  2 19:41
> apache-storm-2.0.0-SNAPSHOT.
>> zip.asc
>> 
>> After ---
>> -rw-r--r--  1 jlim  staff  3.3K  3  2 19:46
> apache-storm-2.0.0-SNAPSHOT.
>> pom
>> -rw-r--r--  1 jlim  staff  473B  3  2 19:46
> apache-storm-2.0.0-SNAPSHOT.
>> pom.asc

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[Jungtaek Lim]

Regarding 3, if we don't make uber jar for connectors, adding jar file in
binary dist. to the classpath will not work, which means that including
connector jar libraries in binary dist. is no effect.
(Someone proposed shading storm-redis because of dependencies... We can
cover it by various way like adding to pom, or using '--artifact' so no
issue at all for now.)

So if we want to go with 3, I'd also like to propose excluding connector
libraries in binary dist. at all.

Others look great to me.

- Jungtaek Lim (HeartSaVioR)

2017년 3월 9일 (목) 오전 6:12, P. Taylor Goetz 님이 작성:

> I’d like to propose the following to address the binary distribution size:
>
> 1. Finish moving connector examples to the “examples” directory.
> 2. Make all examples source-only in the binary distribution. This will
> mean users will have to build the examples themselves, but that’s typically
> what one does when looking at example code.
> 3. Stop unnecessarily shading connector libraries (i.e. let the user’s pom
> pull in the transitive dependencies when building a topology jar).
>
> If we agree on this approach, I’ll take up the necessary changes for the
> 1.1.0 release.
>
> -Taylor
>
>
>
> > On Mar 8, 2017, at 9:16 AM, Xin Wang  wrote:
> >
> > Agree with Jungtaek, STORM-2390 should not block the 1.1.0 release.
> >
> > - Xin
> >
> > 2017-03-07 9:16 GMT+08:00 Jungtaek Lim :
> >
> >> Sorry Roshan but I don't get it.
> >> We're already having separate modules for each of the connectors, and
> even
> >> examples. We also don't have any common modules for external
> connectors, so
> >> those are completely individual.
> >>
> >> As I reported earlier, binary dist. will go up more than 500 MB when we
> >> address STORM-2390(STORM-2249 actually), which the community seemed not
> >> want to.
> >>
> >> Since STORM-2390 is linked to epic for release Storm 1.1.0, I'd rather
> >> unlink from epic to get out of release, and discuss more.
> >> Any objections?
> >>
> >> - Jungtaek Lim (HeartSaVioR)
> >>
> >> 2017년 3월 4일 (토) 오전 10:02, Roshan Naik 님이 작성:
> >>
> >>> As having a separate module for each of the connectors will cause a lot
> >> of
> >>> bloat … may be good to club them into one.
> >>>
> >>> -roshan
> >>>
> >>>
> >>> On 3/2/17, 10:39 AM, "Sree V"  wrote:
> >>>
> >>>+1separate main binaries, connectors and samples/examples.
> >>>
> >>>Thanking you.
> >>>With Regards
> >>>Sree
> >>>
> >>>On Thursday, March 2, 2017 8:02 AM, Satish Duggana <
> >>> satish.dugg...@gmail.com> wrote:
> >>>
> >>>
> >>> Agree that such huge binaries may not be acceptable. We should
> >> really
> >>> think
> >>>about the options proposed earlier like excluding some of the
> >> external
> >>>connectors from binary and update the documentation respectively.
> >>>
> >>>Thanks,
> >>>Satish.
> >>>
> >>>On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim 
> >>> wrote:
> >>>
>  Adding my observation to my last mail:
>  I just give STORM-2249 a try (against master branch) and compare
> >>> before vs
>  after.
>  It adds more than 300 MB, and the final archive is more than 550
> >> MB.
> >>> I
>  guess it would be similar for 1.x branch.
> 
>  Before --
>  -rw-r--r--  1 jlim  staff  3.3K  3  2 19:41
> >>> apache-storm-2.0.0-SNAPSHOT.
>  pom
>  -rw-r--r--  1 jlim  staff  473B  3  2 19:41
> >>> apache-storm-2.0.0-SNAPSHOT.
>  pom.asc
>  -rw-r--r--  1 jlim  staff  264M  3  2 19:41
> >>> apache-storm-2.0.0-SNAPSHOT.
>  tar.gz
>  -rw-r--r--  1 jlim  staff  473B  3  2 19:41
> >>> apache-storm-2.0.0-SNAPSHOT.
>  tar.gz.asc
>  -rw-r--r--  1 jlim  staff  264M  3  2 19:41
> >>> apache-storm-2.0.0-SNAPSHOT.
>  zip
>  -rw-r--r--  1 jlim  staff  473B  3  2 19:41
> >>> apache-storm-2.0.0-SNAPSHOT.
>  zip.asc
> 
>  After ---
>  -rw-r--r--  1 jlim  staff  3.3K  3  2 19:46
> >>> apache-storm-2.0.0-SNAPSHOT.
>  pom
>  -rw-r--r--  1 jlim  staff  473B  3  2 19:46
> >>> apache-storm-2.0.0-SNAPSHOT.
>  pom.asc
>  -rw-r--r--  1 jlim  staff  564M  3  2 19:46
> >>> apache-storm-2.0.0-SNAPSHOT.
>  tar.gz
>  -rw-r--r--  1 jlim  staff  473B  3  2 19:46
> >>> apache-storm-2.0.0-SNAPSHOT.
>  tar.gz.asc
>  -rw-r--r--  1 jlim  staff  565M  3  2 19:46
> >>> apache-storm-2.0.0-SNAPSHOT.
>  zip
>  -rw-r--r--  1 jlim  staff  473B  3  2 19:47
> >>> apache-storm-2.0.0-SNAPSHOT.
>  zip.asc
> 
>  While 264M is already a bit huge for me, 564M is not that I can
> >>> accept.
>  (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka
> >>> 0.10.2 is
>  37M.)
> 
>  Btw, we're including source code of examples, and "mvn clean
> >>> package" will
>  work for every example modules.
> 
>  2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim 님이 작성:
> 

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[P. Taylor Goetz]

I’d like to propose the following to address the binary distribution size:

1. Finish moving connector examples to the “examples” directory.
2. Make all examples source-only in the binary distribution. This will mean 
users will have to build the examples themselves, but that’s typically what one 
does when looking at example code.
3. Stop unnecessarily shading connector libraries (i.e. let the user’s pom pull 
in the transitive dependencies when building a topology jar).

If we agree on this approach, I’ll take up the necessary changes for the 1.1.0 
release.

-Taylor



> On Mar 8, 2017, at 9:16 AM, Xin Wang  wrote:
> 
> Agree with Jungtaek, STORM-2390 should not block the 1.1.0 release.
> 
> - Xin
> 
> 2017-03-07 9:16 GMT+08:00 Jungtaek Lim :
> 
>> Sorry Roshan but I don't get it.
>> We're already having separate modules for each of the connectors, and even
>> examples. We also don't have any common modules for external connectors, so
>> those are completely individual.
>> 
>> As I reported earlier, binary dist. will go up more than 500 MB when we
>> address STORM-2390(STORM-2249 actually), which the community seemed not
>> want to.
>> 
>> Since STORM-2390 is linked to epic for release Storm 1.1.0, I'd rather
>> unlink from epic to get out of release, and discuss more.
>> Any objections?
>> 
>> - Jungtaek Lim (HeartSaVioR)
>> 
>> 2017년 3월 4일 (토) 오전 10:02, Roshan Naik 님이 작성:
>> 
>>> As having a separate module for each of the connectors will cause a lot
>> of
>>> bloat … may be good to club them into one.
>>> 
>>> -roshan
>>> 
>>> 
>>> On 3/2/17, 10:39 AM, "Sree V"  wrote:
>>> 
>>>+1separate main binaries, connectors and samples/examples.
>>> 
>>>Thanking you.
>>>With Regards
>>>Sree
>>> 
>>>On Thursday, March 2, 2017 8:02 AM, Satish Duggana <
>>> satish.dugg...@gmail.com> wrote:
>>> 
>>> 
>>> Agree that such huge binaries may not be acceptable. We should
>> really
>>> think
>>>about the options proposed earlier like excluding some of the
>> external
>>>connectors from binary and update the documentation respectively.
>>> 
>>>Thanks,
>>>Satish.
>>> 
>>>On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim 
>>> wrote:
>>> 
 Adding my observation to my last mail:
 I just give STORM-2249 a try (against master branch) and compare
>>> before vs
 after.
 It adds more than 300 MB, and the final archive is more than 550
>> MB.
>>> I
 guess it would be similar for 1.x branch.
 
 Before --
 -rw-r--r--  1 jlim  staff  3.3K  3  2 19:41
>>> apache-storm-2.0.0-SNAPSHOT.
 pom
 -rw-r--r--  1 jlim  staff  473B  3  2 19:41
>>> apache-storm-2.0.0-SNAPSHOT.
 pom.asc
 -rw-r--r--  1 jlim  staff  264M  3  2 19:41
>>> apache-storm-2.0.0-SNAPSHOT.
 tar.gz
 -rw-r--r--  1 jlim  staff  473B  3  2 19:41
>>> apache-storm-2.0.0-SNAPSHOT.
 tar.gz.asc
 -rw-r--r--  1 jlim  staff  264M  3  2 19:41
>>> apache-storm-2.0.0-SNAPSHOT.
 zip
 -rw-r--r--  1 jlim  staff  473B  3  2 19:41
>>> apache-storm-2.0.0-SNAPSHOT.
 zip.asc
 
 After ---
 -rw-r--r--  1 jlim  staff  3.3K  3  2 19:46
>>> apache-storm-2.0.0-SNAPSHOT.
 pom
 -rw-r--r--  1 jlim  staff  473B  3  2 19:46
>>> apache-storm-2.0.0-SNAPSHOT.
 pom.asc
 -rw-r--r--  1 jlim  staff  564M  3  2 19:46
>>> apache-storm-2.0.0-SNAPSHOT.
 tar.gz
 -rw-r--r--  1 jlim  staff  473B  3  2 19:46
>>> apache-storm-2.0.0-SNAPSHOT.
 tar.gz.asc
 -rw-r--r--  1 jlim  staff  565M  3  2 19:46
>>> apache-storm-2.0.0-SNAPSHOT.
 zip
 -rw-r--r--  1 jlim  staff  473B  3  2 19:47
>>> apache-storm-2.0.0-SNAPSHOT.
 zip.asc
 
 While 264M is already a bit huge for me, 564M is not that I can
>>> accept.
 (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka
>>> 0.10.2 is
 37M.)
 
 Btw, we're including source code of examples, and "mvn clean
>>> package" will
 work for every example modules.
 
 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim 님이 작성:
 
> I guess it might be good time to think why we add all connectors
>>> to the
> binary distribution.
> 
> Spark and Flink don't include them to binary dist. They even
>> moved
>>> some
 or
> most of connectors out of repo, have been maintaining them in
>>> Apache
 Bahir.
> (Personally this is something I'm in favor of. We have lots of
>>> connectors
> and many of them are outdated - clear example is
>>> storm-elasticsearch.)
> 
> If we are assuming online then we don't even need to think about
>>> users
> touching binary dist. version of connectors. Users have been
>>> including
 them
> via build tools' dependency management, or even starting 1.1.0,
>>> users can
> include them via '--artifact' option.
> 
> I also was just one of users for Storm, and I 

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[Xin Wang]

Agree with Jungtaek, STORM-2390 should not block the 1.1.0 release.

- Xin

2017-03-07 9:16 GMT+08:00 Jungtaek Lim :

> Sorry Roshan but I don't get it.
> We're already having separate modules for each of the connectors, and even
> examples. We also don't have any common modules for external connectors, so
> those are completely individual.
>
> As I reported earlier, binary dist. will go up more than 500 MB when we
> address STORM-2390(STORM-2249 actually), which the community seemed not
> want to.
>
> Since STORM-2390 is linked to epic for release Storm 1.1.0, I'd rather
> unlink from epic to get out of release, and discuss more.
> Any objections?
>
> - Jungtaek Lim (HeartSaVioR)
>
> 2017년 3월 4일 (토) 오전 10:02, Roshan Naik 님이 작성:
>
> > As having a separate module for each of the connectors will cause a lot
> of
> > bloat … may be good to club them into one.
> >
> > -roshan
> >
> >
> > On 3/2/17, 10:39 AM, "Sree V"  wrote:
> >
> > +1separate main binaries, connectors and samples/examples.
> >
> > Thanking you.
> > With Regards
> > Sree
> >
> > On Thursday, March 2, 2017 8:02 AM, Satish Duggana <
> > satish.dugg...@gmail.com> wrote:
> >
> >
> >  Agree that such huge binaries may not be acceptable. We should
> really
> > think
> > about the options proposed earlier like excluding some of the
> external
> > connectors from binary and update the documentation respectively.
> >
> > Thanks,
> > Satish.
> >
> > On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim 
> > wrote:
> >
> > > Adding my observation to my last mail:
> > > I just give STORM-2249 a try (against master branch) and compare
> > before vs
> > > after.
> > > It adds more than 300 MB, and the final archive is more than 550
> MB.
> > I
> > > guess it would be similar for 1.x branch.
> > >
> > > Before --
> > > -rw-r--r--  1 jlim  staff  3.3K  3  2 19:41
> > apache-storm-2.0.0-SNAPSHOT.
> > > pom
> > > -rw-r--r--  1 jlim  staff  473B  3  2 19:41
> > apache-storm-2.0.0-SNAPSHOT.
> > > pom.asc
> > > -rw-r--r--  1 jlim  staff  264M  3  2 19:41
> > apache-storm-2.0.0-SNAPSHOT.
> > > tar.gz
> > > -rw-r--r--  1 jlim  staff  473B  3  2 19:41
> > apache-storm-2.0.0-SNAPSHOT.
> > > tar.gz.asc
> > > -rw-r--r--  1 jlim  staff  264M  3  2 19:41
> > apache-storm-2.0.0-SNAPSHOT.
> > > zip
> > > -rw-r--r--  1 jlim  staff  473B  3  2 19:41
> > apache-storm-2.0.0-SNAPSHOT.
> > > zip.asc
> > >
> > > After ---
> > > -rw-r--r--  1 jlim  staff  3.3K  3  2 19:46
> > apache-storm-2.0.0-SNAPSHOT.
> > > pom
> > > -rw-r--r--  1 jlim  staff  473B  3  2 19:46
> > apache-storm-2.0.0-SNAPSHOT.
> > > pom.asc
> > > -rw-r--r--  1 jlim  staff  564M  3  2 19:46
> > apache-storm-2.0.0-SNAPSHOT.
> > > tar.gz
> > > -rw-r--r--  1 jlim  staff  473B  3  2 19:46
> > apache-storm-2.0.0-SNAPSHOT.
> > > tar.gz.asc
> > > -rw-r--r--  1 jlim  staff  565M  3  2 19:46
> > apache-storm-2.0.0-SNAPSHOT.
> > > zip
> > > -rw-r--r--  1 jlim  staff  473B  3  2 19:47
> > apache-storm-2.0.0-SNAPSHOT.
> > > zip.asc
> > >
> > > While 264M is already a bit huge for me, 564M is not that I can
> > accept.
> > > (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka
> > 0.10.2 is
> > > 37M.)
> > >
> > > Btw, we're including source code of examples, and "mvn clean
> > package" will
> > > work for every example modules.
> > >
> > > 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim 님이 작성:
> > >
> > > > I guess it might be good time to think why we add all connectors
> > to the
> > > > binary distribution.
> > > >
> > > > Spark and Flink don't include them to binary dist. They even
> moved
> > some
> > > or
> > > > most of connectors out of repo, have been maintaining them in
> > Apache
> > > Bahir.
> > > > (Personally this is something I'm in favor of. We have lots of
> > connectors
> > > > and many of them are outdated - clear example is
> > storm-elasticsearch.)
> > > >
> > > > If we are assuming online then we don't even need to think about
> > users
> > > > touching binary dist. version of connectors. Users have been
> > including
> > > them
> > > > via build tools' dependency management, or even starting 1.1.0,
> > users can
> > > > include them via '--artifact' option.
> > > >
> > > > I also was just one of users for Storm, and I haven't use them
> > directly.
> > > > How much UX gets worse when we remove connectors to binary dist?
> > It only
> > > > helps some users who are not connected to the internet, and IMHO
> > it's a
> > > > rare case.
> > > >
> > > > I would like to see the opposite approach, removing all
> connectors
> > (or
> > > > just keeping storm-kafka/storm-kafka-client 

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[Jungtaek Lim]

Sorry Roshan but I don't get it.
We're already having separate modules for each of the connectors, and even
examples. We also don't have any common modules for external connectors, so
those are completely individual.

As I reported earlier, binary dist. will go up more than 500 MB when we
address STORM-2390(STORM-2249 actually), which the community seemed not
want to.

Since STORM-2390 is linked to epic for release Storm 1.1.0, I'd rather
unlink from epic to get out of release, and discuss more.
Any objections?

- Jungtaek Lim (HeartSaVioR)

2017년 3월 4일 (토) 오전 10:02, Roshan Naik 님이 작성:

> As having a separate module for each of the connectors will cause a lot of
> bloat … may be good to club them into one.
>
> -roshan
>
>
> On 3/2/17, 10:39 AM, "Sree V"  wrote:
>
> +1separate main binaries, connectors and samples/examples.
>
> Thanking you.
> With Regards
> Sree
>
> On Thursday, March 2, 2017 8:02 AM, Satish Duggana <
> satish.dugg...@gmail.com> wrote:
>
>
>  Agree that such huge binaries may not be acceptable. We should really
> think
> about the options proposed earlier like excluding some of the external
> connectors from binary and update the documentation respectively.
>
> Thanks,
> Satish.
>
> On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim 
> wrote:
>
> > Adding my observation to my last mail:
> > I just give STORM-2249 a try (against master branch) and compare
> before vs
> > after.
> > It adds more than 300 MB, and the final archive is more than 550 MB.
> I
> > guess it would be similar for 1.x branch.
> >
> > Before --
> > -rw-r--r--  1 jlim  staff  3.3K  3  2 19:41
> apache-storm-2.0.0-SNAPSHOT.
> > pom
> > -rw-r--r--  1 jlim  staff  473B  3  2 19:41
> apache-storm-2.0.0-SNAPSHOT.
> > pom.asc
> > -rw-r--r--  1 jlim  staff  264M  3  2 19:41
> apache-storm-2.0.0-SNAPSHOT.
> > tar.gz
> > -rw-r--r--  1 jlim  staff  473B  3  2 19:41
> apache-storm-2.0.0-SNAPSHOT.
> > tar.gz.asc
> > -rw-r--r--  1 jlim  staff  264M  3  2 19:41
> apache-storm-2.0.0-SNAPSHOT.
> > zip
> > -rw-r--r--  1 jlim  staff  473B  3  2 19:41
> apache-storm-2.0.0-SNAPSHOT.
> > zip.asc
> >
> > After ---
> > -rw-r--r--  1 jlim  staff  3.3K  3  2 19:46
> apache-storm-2.0.0-SNAPSHOT.
> > pom
> > -rw-r--r--  1 jlim  staff  473B  3  2 19:46
> apache-storm-2.0.0-SNAPSHOT.
> > pom.asc
> > -rw-r--r--  1 jlim  staff  564M  3  2 19:46
> apache-storm-2.0.0-SNAPSHOT.
> > tar.gz
> > -rw-r--r--  1 jlim  staff  473B  3  2 19:46
> apache-storm-2.0.0-SNAPSHOT.
> > tar.gz.asc
> > -rw-r--r--  1 jlim  staff  565M  3  2 19:46
> apache-storm-2.0.0-SNAPSHOT.
> > zip
> > -rw-r--r--  1 jlim  staff  473B  3  2 19:47
> apache-storm-2.0.0-SNAPSHOT.
> > zip.asc
> >
> > While 264M is already a bit huge for me, 564M is not that I can
> accept.
> > (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka
> 0.10.2 is
> > 37M.)
> >
> > Btw, we're including source code of examples, and "mvn clean
> package" will
> > work for every example modules.
> >
> > 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim 님이 작성:
> >
> > > I guess it might be good time to think why we add all connectors
> to the
> > > binary distribution.
> > >
> > > Spark and Flink don't include them to binary dist. They even moved
> some
> > or
> > > most of connectors out of repo, have been maintaining them in
> Apache
> > Bahir.
> > > (Personally this is something I'm in favor of. We have lots of
> connectors
> > > and many of them are outdated - clear example is
> storm-elasticsearch.)
> > >
> > > If we are assuming online then we don't even need to think about
> users
> > > touching binary dist. version of connectors. Users have been
> including
> > them
> > > via build tools' dependency management, or even starting 1.1.0,
> users can
> > > include them via '--artifact' option.
> > >
> > > I also was just one of users for Storm, and I haven't use them
> directly.
> > > How much UX gets worse when we remove connectors to binary dist?
> It only
> > > helps some users who are not connected to the internet, and IMHO
> it's a
> > > rare case.
> > >
> > > I would like to see the opposite approach, removing all connectors
> (or
> > > just keeping storm-kafka/storm-kafka-client and some more preferred
> > things)
> > > and its relevant examples from binary dist.
> > >
> > > What do you think about it?
> > >
> > > - Jungtaek Lim (HeartSaVioR)
> > >
> > >
> > > 2017년 3월 2일 (목) 오전 10:23, Roshan Naik 님이
> 작성:
> > >
> > > Once all of the shaded examples are included the size will go up
> further.
> > >
> > > But currently as 

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[Roshan Naik]

As having a separate module for each of the connectors will cause a lot of 
bloat … may be good to club them into one.

-roshan


On 3/2/17, 10:39 AM, "Sree V"  wrote:

+1separate main binaries, connectors and samples/examples.
 
Thanking you.
With Regards
Sree 

On Thursday, March 2, 2017 8:02 AM, Satish Duggana 
 wrote:
 

 Agree that such huge binaries may not be acceptable. We should really think
about the options proposed earlier like excluding some of the external
connectors from binary and update the documentation respectively.

Thanks,
Satish.

On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim  wrote:

> Adding my observation to my last mail:
> I just give STORM-2249 a try (against master branch) and compare before vs
> after.
> It adds more than 300 MB, and the final archive is more than 550 MB. I
> guess it would be similar for 1.x branch.
>
> Before --
> -rw-r--r--  1 jlim  staff  3.3K  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> pom
> -rw-r--r--  1 jlim  staff  473B  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> pom.asc
> -rw-r--r--  1 jlim  staff  264M  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> tar.gz
> -rw-r--r--  1 jlim  staff  473B  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> tar.gz.asc
> -rw-r--r--  1 jlim  staff  264M  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> zip
> -rw-r--r--  1 jlim  staff  473B  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> zip.asc
>
> After ---
> -rw-r--r--  1 jlim  staff  3.3K  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
> pom
> -rw-r--r--  1 jlim  staff  473B  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
> pom.asc
> -rw-r--r--  1 jlim  staff  564M  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
> tar.gz
> -rw-r--r--  1 jlim  staff  473B  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
> tar.gz.asc
> -rw-r--r--  1 jlim  staff  565M  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
> zip
> -rw-r--r--  1 jlim  staff  473B  3  2 19:47 apache-storm-2.0.0-SNAPSHOT.
> zip.asc
>
> While 264M is already a bit huge for me, 564M is not that I can accept.
> (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka 0.10.2 is
> 37M.)
>
> Btw, we're including source code of examples, and "mvn clean package" will
> work for every example modules.
>
> 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim 님이 작성:
>
> > I guess it might be good time to think why we add all connectors to the
> > binary distribution.
> >
> > Spark and Flink don't include them to binary dist. They even moved some
> or
> > most of connectors out of repo, have been maintaining them in Apache
> Bahir.
> > (Personally this is something I'm in favor of. We have lots of 
connectors
> > and many of them are outdated - clear example is storm-elasticsearch.)
> >
> > If we are assuming online then we don't even need to think about users
> > touching binary dist. version of connectors. Users have been including
> them
> > via build tools' dependency management, or even starting 1.1.0, users 
can
> > include them via '--artifact' option.
> >
> > I also was just one of users for Storm, and I haven't use them directly.
> > How much UX gets worse when we remove connectors to binary dist? It only
> > helps some users who are not connected to the internet, and IMHO it's a
> > rare case.
> >
> > I would like to see the opposite approach, removing all connectors (or
> > just keeping storm-kafka/storm-kafka-client and some more preferred
> things)
> > and its relevant examples from binary dist.
> >
> > What do you think about it?
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> >
> > 2017년 3월 2일 (목) 오전 10:23, Roshan Naik 님이 작성:
> >
> > Once all of the shaded examples are included the size will go up 
further.
> >
> > But currently as they are not part of the tar.gz … something else is the
> > culprit for the bloat.
> >
> >
> >
> > Below is a comparative listing of 1.0.3 vs 1.1.0  binary releases .. of
> > files that are larger than 4MB.
> >
> >
> >
> >
> >
> > @Jungtaek Lim :  I am thinking, since the
> > code for the examples can be easily viewed online … it would be valuable
> to
> > have the executable topologies made available to the user as part of the
> > binary release … rather have them figure out how to build it correctly
> > before trying them out.
> >
> >
> >
> > -roshan
> >
> >
> >
> >
> >
> > ➜  apache-storm-1.0.3 >  find . -type f -size +4096 -exec ls -lh {} \;
> >
> > -rw-r--r--@ 1 roshan  staff70M Feb  7 

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[Sree V]

+1separate main binaries, connectors and samples/examples.
 
Thanking you.
With Regards
Sree 

On Thursday, March 2, 2017 8:02 AM, Satish Duggana 
 wrote:
 

 Agree that such huge binaries may not be acceptable. We should really think
about the options proposed earlier like excluding some of the external
connectors from binary and update the documentation respectively.

Thanks,
Satish.

On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim  wrote:

> Adding my observation to my last mail:
> I just give STORM-2249 a try (against master branch) and compare before vs
> after.
> It adds more than 300 MB, and the final archive is more than 550 MB. I
> guess it would be similar for 1.x branch.
>
> Before --
> -rw-r--r--  1 jlim  staff  3.3K  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> pom
> -rw-r--r--  1 jlim  staff  473B  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> pom.asc
> -rw-r--r--  1 jlim  staff  264M  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> tar.gz
> -rw-r--r--  1 jlim  staff  473B  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> tar.gz.asc
> -rw-r--r--  1 jlim  staff  264M  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> zip
> -rw-r--r--  1 jlim  staff  473B  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> zip.asc
>
> After ---
> -rw-r--r--  1 jlim  staff  3.3K  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
> pom
> -rw-r--r--  1 jlim  staff  473B  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
> pom.asc
> -rw-r--r--  1 jlim  staff  564M  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
> tar.gz
> -rw-r--r--  1 jlim  staff  473B  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
> tar.gz.asc
> -rw-r--r--  1 jlim  staff  565M  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
> zip
> -rw-r--r--  1 jlim  staff  473B  3  2 19:47 apache-storm-2.0.0-SNAPSHOT.
> zip.asc
>
> While 264M is already a bit huge for me, 564M is not that I can accept.
> (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka 0.10.2 is
> 37M.)
>
> Btw, we're including source code of examples, and "mvn clean package" will
> work for every example modules.
>
> 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim 님이 작성:
>
> > I guess it might be good time to think why we add all connectors to the
> > binary distribution.
> >
> > Spark and Flink don't include them to binary dist. They even moved some
> or
> > most of connectors out of repo, have been maintaining them in Apache
> Bahir.
> > (Personally this is something I'm in favor of. We have lots of connectors
> > and many of them are outdated - clear example is storm-elasticsearch.)
> >
> > If we are assuming online then we don't even need to think about users
> > touching binary dist. version of connectors. Users have been including
> them
> > via build tools' dependency management, or even starting 1.1.0, users can
> > include them via '--artifact' option.
> >
> > I also was just one of users for Storm, and I haven't use them directly.
> > How much UX gets worse when we remove connectors to binary dist? It only
> > helps some users who are not connected to the internet, and IMHO it's a
> > rare case.
> >
> > I would like to see the opposite approach, removing all connectors (or
> > just keeping storm-kafka/storm-kafka-client and some more preferred
> things)
> > and its relevant examples from binary dist.
> >
> > What do you think about it?
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> >
> > 2017년 3월 2일 (목) 오전 10:23, Roshan Naik 님이 작성:
> >
> > Once all of the shaded examples are included the size will go up further.
> >
> > But currently as they are not part of the tar.gz … something else is the
> > culprit for the bloat.
> >
> >
> >
> > Below is a comparative listing of 1.0.3 vs 1.1.0  binary releases .. of
> > files that are larger than 4MB.
> >
> >
> >
> >
> >
> > @Jungtaek Lim :  I am thinking, since the
> > code for the examples can be easily viewed online … it would be valuable
> to
> > have the executable topologies made available to the user as part of the
> > binary release … rather have them figure out how to build it correctly
> > before trying them out.
> >
> >
> >
> > -roshan
> >
> >
> >
> >
> >
> > ➜  apache-storm-1.0.3 >  find . -type f -size +4096 -exec ls -lh {} \;
> >
> > -rw-r--r--@ 1 roshan  staff    70M Feb  7 12:33
> ./examples/storm-starter/
> > storm-starter-topologies-1.0.3.jar
> >
> > -rwxr-xr-x@ 1 roshan  staff    65M Feb  7 12:30
> > ./external/flux/flux-examples-1.0.3.jar
> >
> > -rwxr-xr-x@ 1 roshan  staff  3.5M Feb  7 12:32
> > ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
> >
> > -rwxr-xr-x@ 1 roshan  staff  2.1M Feb  7 12:32
> > ./external/sql/storm-sql-core/guava-16.0.1.jar
> >
> > -rwxr-xr-x@ 1 roshan  staff  7.3M Feb  7 12:30
> > ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-
> with-dependencies.jar
> >
> > -rwxr-xr-x@ 1 roshan  staff  5.6M Feb  7 12:33
> > ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
> >
> > -rwxr-xr-x@ 1 roshan  staff  

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[Satish Duggana]

Agree that such huge binaries may not be acceptable. We should really think
about the options proposed earlier like excluding some of the external
connectors from binary and update the documentation respectively.

Thanks,
Satish.

On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim  wrote:

> Adding my observation to my last mail:
> I just give STORM-2249 a try (against master branch) and compare before vs
> after.
> It adds more than 300 MB, and the final archive is more than 550 MB. I
> guess it would be similar for 1.x branch.
>
> Before --
> -rw-r--r--   1 jlim  staff   3.3K  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> pom
> -rw-r--r--   1 jlim  staff   473B  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> pom.asc
> -rw-r--r--   1 jlim  staff   264M  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> tar.gz
> -rw-r--r--   1 jlim  staff   473B  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> tar.gz.asc
> -rw-r--r--   1 jlim  staff   264M  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> zip
> -rw-r--r--   1 jlim  staff   473B  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
> zip.asc
>
> After ---
> -rw-r--r--   1 jlim  staff   3.3K  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
> pom
> -rw-r--r--   1 jlim  staff   473B  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
> pom.asc
> -rw-r--r--   1 jlim  staff   564M  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
> tar.gz
> -rw-r--r--   1 jlim  staff   473B  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
> tar.gz.asc
> -rw-r--r--   1 jlim  staff   565M  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
> zip
> -rw-r--r--   1 jlim  staff   473B  3  2 19:47 apache-storm-2.0.0-SNAPSHOT.
> zip.asc
>
> While 264M is already a bit huge for me, 564M is not that I can accept.
> (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka 0.10.2 is
> 37M.)
>
> Btw, we're including source code of examples, and "mvn clean package" will
> work for every example modules.
>
> 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim 님이 작성:
>
> > I guess it might be good time to think why we add all connectors to the
> > binary distribution.
> >
> > Spark and Flink don't include them to binary dist. They even moved some
> or
> > most of connectors out of repo, have been maintaining them in Apache
> Bahir.
> > (Personally this is something I'm in favor of. We have lots of connectors
> > and many of them are outdated - clear example is storm-elasticsearch.)
> >
> > If we are assuming online then we don't even need to think about users
> > touching binary dist. version of connectors. Users have been including
> them
> > via build tools' dependency management, or even starting 1.1.0, users can
> > include them via '--artifact' option.
> >
> > I also was just one of users for Storm, and I haven't use them directly.
> > How much UX gets worse when we remove connectors to binary dist? It only
> > helps some users who are not connected to the internet, and IMHO it's a
> > rare case.
> >
> > I would like to see the opposite approach, removing all connectors (or
> > just keeping storm-kafka/storm-kafka-client and some more preferred
> things)
> > and its relevant examples from binary dist.
> >
> > What do you think about it?
> >
> > - Jungtaek Lim (HeartSaVioR)
> >
> >
> > 2017년 3월 2일 (목) 오전 10:23, Roshan Naik 님이 작성:
> >
> > Once all of the shaded examples are included the size will go up further.
> >
> > But currently as they are not part of the tar.gz … something else is the
> > culprit for the bloat.
> >
> >
> >
> > Below is a comparative listing of 1.0.3 vs 1.1.0  binary releases .. of
> > files that are larger than 4MB.
> >
> >
> >
> >
> >
> > @Jungtaek Lim :  I am thinking, since the
> > code for the examples can be easily viewed online … it would be valuable
> to
> > have the executable topologies made available to the user as part of the
> > binary release … rather have them figure out how to build it correctly
> > before trying them out.
> >
> >
> >
> > -roshan
> >
> >
> >
> >
> >
> > ➜  apache-storm-1.0.3 >  find . -type f -size +4096 -exec ls -lh {} \;
> >
> > -rw-r--r--@ 1 roshan  staff70M Feb  7 12:33
> ./examples/storm-starter/
> > storm-starter-topologies-1.0.3.jar
> >
> > -rwxr-xr-x@ 1 roshan  staff65M Feb  7 12:30
> > ./external/flux/flux-examples-1.0.3.jar
> >
> > -rwxr-xr-x@ 1 roshan  staff   3.5M Feb  7 12:32
> > ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
> >
> > -rwxr-xr-x@ 1 roshan  staff   2.1M Feb  7 12:32
> > ./external/sql/storm-sql-core/guava-16.0.1.jar
> >
> > -rwxr-xr-x@ 1 roshan  staff   7.3M Feb  7 12:30
> > ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-
> with-dependencies.jar
> >
> > -rwxr-xr-x@ 1 roshan  staff   5.6M Feb  7 12:33
> > ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
> >
> > -rwxr-xr-x@ 1 roshan  staff   9.9M Feb  7 12:33
> > ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar
> >
> > -rw-r--r--@ 1 roshan  staff   3.7M Nov  4 10:02 ./lib/clojure-1.7.0.jar
> >
> > 

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[Jungtaek Lim]

Adding my observation to my last mail:
I just give STORM-2249 a try (against master branch) and compare before vs
after.
It adds more than 300 MB, and the final archive is more than 550 MB. I
guess it would be similar for 1.x branch.

Before --
-rw-r--r--   1 jlim  staff   3.3K  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
pom
-rw-r--r--   1 jlim  staff   473B  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
pom.asc
-rw-r--r--   1 jlim  staff   264M  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
tar.gz
-rw-r--r--   1 jlim  staff   473B  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
tar.gz.asc
-rw-r--r--   1 jlim  staff   264M  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
zip
-rw-r--r--   1 jlim  staff   473B  3  2 19:41 apache-storm-2.0.0-SNAPSHOT.
zip.asc

After ---
-rw-r--r--   1 jlim  staff   3.3K  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
pom
-rw-r--r--   1 jlim  staff   473B  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
pom.asc
-rw-r--r--   1 jlim  staff   564M  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
tar.gz
-rw-r--r--   1 jlim  staff   473B  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
tar.gz.asc
-rw-r--r--   1 jlim  staff   565M  3  2 19:46 apache-storm-2.0.0-SNAPSHOT.
zip
-rw-r--r--   1 jlim  staff   473B  3  2 19:47 apache-storm-2.0.0-SNAPSHOT.
zip.asc

While 264M is already a bit huge for me, 564M is not that I can accept.
(Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka 0.10.2 is
37M.)

Btw, we're including source code of examples, and "mvn clean package" will
work for every example modules.

2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim 님이 작성:

> I guess it might be good time to think why we add all connectors to the
> binary distribution.
>
> Spark and Flink don't include them to binary dist. They even moved some or
> most of connectors out of repo, have been maintaining them in Apache Bahir.
> (Personally this is something I'm in favor of. We have lots of connectors
> and many of them are outdated - clear example is storm-elasticsearch.)
>
> If we are assuming online then we don't even need to think about users
> touching binary dist. version of connectors. Users have been including them
> via build tools' dependency management, or even starting 1.1.0, users can
> include them via '--artifact' option.
>
> I also was just one of users for Storm, and I haven't use them directly.
> How much UX gets worse when we remove connectors to binary dist? It only
> helps some users who are not connected to the internet, and IMHO it's a
> rare case.
>
> I would like to see the opposite approach, removing all connectors (or
> just keeping storm-kafka/storm-kafka-client and some more preferred things)
> and its relevant examples from binary dist.
>
> What do you think about it?
>
> - Jungtaek Lim (HeartSaVioR)
>
>
> 2017년 3월 2일 (목) 오전 10:23, Roshan Naik 님이 작성:
>
> Once all of the shaded examples are included the size will go up further.
>
> But currently as they are not part of the tar.gz … something else is the
> culprit for the bloat.
>
>
>
> Below is a comparative listing of 1.0.3 vs 1.1.0  binary releases .. of
> files that are larger than 4MB.
>
>
>
>
>
> @Jungtaek Lim :  I am thinking, since the
> code for the examples can be easily viewed online … it would be valuable to
> have the executable topologies made available to the user as part of the
> binary release … rather have them figure out how to build it correctly
> before trying them out.
>
>
>
> -roshan
>
>
>
>
>
> ➜  apache-storm-1.0.3 >  find . -type f -size +4096 -exec ls -lh {} \;
>
> -rw-r--r--@ 1 roshan  staff70M Feb  7 12:33 ./examples/storm-starter/
> storm-starter-topologies-1.0.3.jar
>
> -rwxr-xr-x@ 1 roshan  staff65M Feb  7 12:30
> ./external/flux/flux-examples-1.0.3.jar
>
> -rwxr-xr-x@ 1 roshan  staff   3.5M Feb  7 12:32
> ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar
>
> -rwxr-xr-x@ 1 roshan  staff   2.1M Feb  7 12:32
> ./external/sql/storm-sql-core/guava-16.0.1.jar
>
> -rwxr-xr-x@ 1 roshan  staff   7.3M Feb  7 12:30
> ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-with-dependencies.jar
>
> -rwxr-xr-x@ 1 roshan  staff   5.6M Feb  7 12:33
> ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar
>
> -rwxr-xr-x@ 1 roshan  staff   9.9M Feb  7 12:33
> ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar
>
> -rw-r--r--@ 1 roshan  staff   3.7M Nov  4 10:02 ./lib/clojure-1.7.0.jar
>
> -rw-r--r--@ 1 roshan  staff19M Feb  7 12:26 ./lib/storm-core-1.0.3.jar
>
> -rw-r--r--@ 1 roshan  staff   2.4M Feb  7 12:26
> ./lib/storm-rename-hack-1.0.3.jar
>
>
>
> ➜  apache-storm-1.1.0 >  find . -type f -size +4096 -exec ls -lh {} \;
>
> -rwxr-xr-x@ 1 roshan  staff   8.0M Feb 24 12:23 ./examples/storm-pmml-
> examples/storm-pmml-examples-1.1.0.jar
>
> -rwxr-xr-x@ 1 roshan  staff60M Feb 24 12:20 ./examples/storm-starter/
> storm-starter-topologies-1.1.0.jar
>
> -rwxr-xr-x@ 1 roshan  staff66M Feb 24 12:11
> ./external/flux/flux-examples-1.1.0.jar

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[Jungtaek Lim]

I guess it might be good time to think why we add all connectors to the
binary distribution.

Spark and Flink don't include them to binary dist. They even moved some or
most of connectors out of repo, have been maintaining them in Apache Bahir.
(Personally this is something I'm in favor of. We have lots of connectors
and many of them are outdated - clear example is storm-elasticsearch.)

If we are assuming online then we don't even need to think about users
touching binary dist. version of connectors. Users have been including them
via build tools' dependency management, or even starting 1.1.0, users can
include them via '--artifact' option.

I also was just one of users for Storm, and I haven't use them directly.
How much UX gets worse when we remove connectors to binary dist? It only
helps some users who are not connected to the internet, and IMHO it's a
rare case.

I would like to see the opposite approach, removing all connectors (or just
keeping storm-kafka/storm-kafka-client and some more preferred things) and
its relevant examples from binary dist.

What do you think about it?

- Jungtaek Lim (HeartSaVioR)


2017년 3월 2일 (목) 오전 10:23, Roshan Naik 님이 작성:

Once all of the shaded examples are included the size will go up further.

But currently as they are not part of the tar.gz … something else is the
culprit for the bloat.



Below is a comparative listing of 1.0.3 vs 1.1.0  binary releases .. of
files that are larger than 4MB.





@Jungtaek Lim :  I am thinking, since the code
for the examples can be easily viewed online … it would be valuable to have
the executable topologies made available to the user as part of the binary
release … rather have them figure out how to build it correctly before
trying them out.



-roshan





➜  apache-storm-1.0.3 >  find . -type f -size +4096 -exec ls -lh {} \;

-rw-r--r--@ 1 roshan  staff70M Feb  7 12:33
./examples/storm-starter/storm-starter-topologies-1.0.3.jar

-rwxr-xr-x@ 1 roshan  staff65M Feb  7 12:30
./external/flux/flux-examples-1.0.3.jar

-rwxr-xr-x@ 1 roshan  staff   3.5M Feb  7 12:32
./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar

-rwxr-xr-x@ 1 roshan  staff   2.1M Feb  7 12:32
./external/sql/storm-sql-core/guava-16.0.1.jar

-rwxr-xr-x@ 1 roshan  staff   7.3M Feb  7 12:30
./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-with-dependencies.jar

-rwxr-xr-x@ 1 roshan  staff   5.6M Feb  7 12:33
./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar

-rwxr-xr-x@ 1 roshan  staff   9.9M Feb  7 12:33
./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar

-rw-r--r--@ 1 roshan  staff   3.7M Nov  4 10:02 ./lib/clojure-1.7.0.jar

-rw-r--r--@ 1 roshan  staff19M Feb  7 12:26 ./lib/storm-core-1.0.3.jar

-rw-r--r--@ 1 roshan  staff   2.4M Feb  7 12:26
./lib/storm-rename-hack-1.0.3.jar



➜  apache-storm-1.1.0 >  find . -type f -size +4096 -exec ls -lh {} \;

-rwxr-xr-x@ 1 roshan  staff   8.0M Feb 24 12:23
./examples/storm-pmml-examples/storm-pmml-examples-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff60M Feb 24 12:20
./examples/storm-starter/storm-starter-topologies-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff66M Feb 24 12:11
./external/flux/flux-examples-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff   4.0M Feb 24 12:16
./external/sql/storm-sql-core/calcite-core-1.11.0.jar

-rwxr-xr-x@ 1 roshan  staff   2.1M Feb 24 12:16
./external/sql/storm-sql-core/guava-16.0.1.jar

-rwxr-xr-x@ 1 roshan  staff   4.0M Feb 24 12:12
./external/sql/storm-sql-runtime/calcite-core-1.11.0.jar

-rwxr-xr-x@ 1 roshan  staff   2.1M Feb 24 12:12
./external/sql/storm-sql-runtime/guava-16.0.1.jar

-rwxr-xr-x@ 1 roshan  staff78M Feb 24 12:18
./external/storm-druid/storm-druid-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff   7.3M Feb 24 12:11
./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-with-dependencies.jar

-rwxr-xr-x@ 1 roshan  staff   5.6M Feb 24 12:20
./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar

-rwxr-xr-x@ 1 roshan  staff   6.7M Feb 24 12:18
./external/storm-submit-tools/storm-submit-tools-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff   3.7M Nov  4 10:02 ./lib/clojure-1.7.0.jar

-rwxr-xr-x@ 1 roshan  staff20M Feb 24 12:07 ./lib/storm-core-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff   2.4M Feb 24 12:07
./lib/storm-rename-hack-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff18M Feb 24 12:19
./toollib/storm-kafka-monitor-1.1.0.jar













On 3/1/17, 4:43 PM, "Jungtaek Lim"  wrote:



About STORM-2249, since examples are shading their dependencies, binary

dist will grow much bigger. I've left some comments regarding that.

Btw, I have another view of this. Showing example codes is more
important

than just let users execute some topologies. That's what example modules

are for. We need to include source as well. If we need to pick one,
source

code would be better.



STORM-2343 seems better to add to 1.1.0. I just am not enough familiar

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[Roshan Naik]

Looks like the formatting  on my prev email got dropped. But observation is 
that storm-kafka-monitor  and storm-druid are the biggest new contributors.
-roshan


On 3/1/17, 5:23 PM, "Roshan Naik"  wrote:

Once all of the shaded examples are included the size will go up further.

But currently as they are not part of the tar.gz … something else is the 
culprit for the bloat.

Below is a comparative listing of 1.0.3 vs 1.1.0  binary releases .. of 
files that are larger than 4MB.


@Jungtaek Lim :  I am thinking, since the code 
for the examples can be easily viewed online … it would be valuable to have the 
executable topologies made available to the user as part of the binary release 
… rather have them figure out how to build it correctly before trying them out.

-roshan





➜  apache-storm-1.0.3 >  find . -type f -size +4096 -exec ls -lh {} \;

-rw-r--r--@ 1 roshan  staff70M Feb  7 12:33 
./examples/storm-starter/storm-starter-topologies-1.0.3.jar

-rwxr-xr-x@ 1 roshan  staff65M Feb  7 12:30 
./external/flux/flux-examples-1.0.3.jar

-rwxr-xr-x@ 1 roshan  staff   3.5M Feb  7 12:32 
./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar

-rwxr-xr-x@ 1 roshan  staff   2.1M Feb  7 12:32 
./external/sql/storm-sql-core/guava-16.0.1.jar

-rwxr-xr-x@ 1 roshan  staff   7.3M Feb  7 12:30 
./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-with-dependencies.jar

-rwxr-xr-x@ 1 roshan  staff   5.6M Feb  7 12:33 
./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar

-rwxr-xr-x@ 1 roshan  staff   9.9M Feb  7 12:33 
./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar

-rw-r--r--@ 1 roshan  staff   3.7M Nov  4 10:02 ./lib/clojure-1.7.0.jar

-rw-r--r--@ 1 roshan  staff19M Feb  7 12:26 ./lib/storm-core-1.0.3.jar

-rw-r--r--@ 1 roshan  staff   2.4M Feb  7 12:26 
./lib/storm-rename-hack-1.0.3.jar



➜  apache-storm-1.1.0 >  find . -type f -size +4096 -exec ls -lh {} \;

-rwxr-xr-x@ 1 roshan  staff   8.0M Feb 24 12:23 
./examples/storm-pmml-examples/storm-pmml-examples-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff60M Feb 24 12:20 
./examples/storm-starter/storm-starter-topologies-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff66M Feb 24 12:11 
./external/flux/flux-examples-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff   4.0M Feb 24 12:16 
./external/sql/storm-sql-core/calcite-core-1.11.0.jar

-rwxr-xr-x@ 1 roshan  staff   2.1M Feb 24 12:16 
./external/sql/storm-sql-core/guava-16.0.1.jar

-rwxr-xr-x@ 1 roshan  staff   4.0M Feb 24 12:12 
./external/sql/storm-sql-runtime/calcite-core-1.11.0.jar

-rwxr-xr-x@ 1 roshan  staff   2.1M Feb 24 12:12 
./external/sql/storm-sql-runtime/guava-16.0.1.jar

-rwxr-xr-x@ 1 roshan  staff78M Feb 24 12:18 
./external/storm-druid/storm-druid-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff   7.3M Feb 24 12:11 
./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-with-dependencies.jar

-rwxr-xr-x@ 1 roshan  staff   5.6M Feb 24 12:20 
./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar

-rwxr-xr-x@ 1 roshan  staff   6.7M Feb 24 12:18 
./external/storm-submit-tools/storm-submit-tools-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff   3.7M Nov  4 10:02 ./lib/clojure-1.7.0.jar

-rwxr-xr-x@ 1 roshan  staff20M Feb 24 12:07 ./lib/storm-core-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff   2.4M Feb 24 12:07 
./lib/storm-rename-hack-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff18M Feb 24 12:19 
./toollib/storm-kafka-monitor-1.1.0.jar













On 3/1/17, 4:43 PM, "Jungtaek Lim"  wrote:



About STORM-2249, since examples are shading their dependencies, binary

dist will grow much bigger. I've left some comments regarding that.

Btw, I have another view of this. Showing example codes is more 
important

than just let users execute some topologies. That's what example modules

are for. We need to include source as well. If we need to pick one, 
source

code would be better.



STORM-2343 seems better to add to 1.1.0. I just am not enough familiar 
with

storm-kafka-client so not sure I can review that, but I'll try to. I 
feel

it's not that make release dragged. Let's add to 1.1.0 epic.



Let's make minimum merge before another RC vote. Addressing STORM-2389 
(and

maybe STORM-2343) is enough for me. Others are not that critical.



Thanks,

Jungtaek Lim (HeartSaVioR)





On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro 


wrote:
   

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[Roshan Naik]

Once all of the shaded examples are included the size will go up further.

But currently as they are not part of the tar.gz … something else is the 
culprit for the bloat.



Below is a comparative listing of 1.0.3 vs 1.1.0  binary releases .. of files 
that are larger than 4MB.





@Jungtaek Lim :  I am thinking, since the code for 
the examples can be easily viewed online … it would be valuable to have the 
executable topologies made available to the user as part of the binary release 
… rather have them figure out how to build it correctly before trying them out.



-roshan





➜  apache-storm-1.0.3 >  find . -type f -size +4096 -exec ls -lh {} \;

-rw-r--r--@ 1 roshan  staff70M Feb  7 12:33 
./examples/storm-starter/storm-starter-topologies-1.0.3.jar

-rwxr-xr-x@ 1 roshan  staff65M Feb  7 12:30 
./external/flux/flux-examples-1.0.3.jar

-rwxr-xr-x@ 1 roshan  staff   3.5M Feb  7 12:32 
./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar

-rwxr-xr-x@ 1 roshan  staff   2.1M Feb  7 12:32 
./external/sql/storm-sql-core/guava-16.0.1.jar

-rwxr-xr-x@ 1 roshan  staff   7.3M Feb  7 12:30 
./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-with-dependencies.jar

-rwxr-xr-x@ 1 roshan  staff   5.6M Feb  7 12:33 
./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar

-rwxr-xr-x@ 1 roshan  staff   9.9M Feb  7 12:33 
./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar

-rw-r--r--@ 1 roshan  staff   3.7M Nov  4 10:02 ./lib/clojure-1.7.0.jar

-rw-r--r--@ 1 roshan  staff19M Feb  7 12:26 ./lib/storm-core-1.0.3.jar

-rw-r--r--@ 1 roshan  staff   2.4M Feb  7 12:26 
./lib/storm-rename-hack-1.0.3.jar



➜  apache-storm-1.1.0 >  find . -type f -size +4096 -exec ls -lh {} \;

-rwxr-xr-x@ 1 roshan  staff   8.0M Feb 24 12:23 
./examples/storm-pmml-examples/storm-pmml-examples-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff60M Feb 24 12:20 
./examples/storm-starter/storm-starter-topologies-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff66M Feb 24 12:11 
./external/flux/flux-examples-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff   4.0M Feb 24 12:16 
./external/sql/storm-sql-core/calcite-core-1.11.0.jar

-rwxr-xr-x@ 1 roshan  staff   2.1M Feb 24 12:16 
./external/sql/storm-sql-core/guava-16.0.1.jar

-rwxr-xr-x@ 1 roshan  staff   4.0M Feb 24 12:12 
./external/sql/storm-sql-runtime/calcite-core-1.11.0.jar

-rwxr-xr-x@ 1 roshan  staff   2.1M Feb 24 12:12 
./external/sql/storm-sql-runtime/guava-16.0.1.jar

-rwxr-xr-x@ 1 roshan  staff78M Feb 24 12:18 
./external/storm-druid/storm-druid-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff   7.3M Feb 24 12:11 
./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-with-dependencies.jar

-rwxr-xr-x@ 1 roshan  staff   5.6M Feb 24 12:20 
./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar

-rwxr-xr-x@ 1 roshan  staff   6.7M Feb 24 12:18 
./external/storm-submit-tools/storm-submit-tools-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff   3.7M Nov  4 10:02 ./lib/clojure-1.7.0.jar

-rwxr-xr-x@ 1 roshan  staff20M Feb 24 12:07 ./lib/storm-core-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff   2.4M Feb 24 12:07 
./lib/storm-rename-hack-1.1.0.jar

-rwxr-xr-x@ 1 roshan  staff18M Feb 24 12:19 
./toollib/storm-kafka-monitor-1.1.0.jar













On 3/1/17, 4:43 PM, "Jungtaek Lim"  wrote:



About STORM-2249, since examples are shading their dependencies, binary

dist will grow much bigger. I've left some comments regarding that.

Btw, I have another view of this. Showing example codes is more important

than just let users execute some topologies. That's what example modules

are for. We need to include source as well. If we need to pick one, source

code would be better.



STORM-2343 seems better to add to 1.1.0. I just am not enough familiar with

storm-kafka-client so not sure I can review that, but I'll try to. I feel

it's not that make release dragged. Let's add to 1.1.0 epic.



Let's make minimum merge before another RC vote. Addressing STORM-2389 (and

maybe STORM-2343) is enough for me. Others are not that critical.



Thanks,

Jungtaek Lim (HeartSaVioR)





On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro 

wrote:



> Roshan, does this PR and JIRA<

> https://issues.apache.org/jira/browse/STORM-2249> address the missing

> jars problem that you mentioned. I had created it in December 2016, but

> there is an ongoing discussion if we should indeed put the jars in the

> examples location or not.

>

> On a different note, this storm-kafka-client/KafkaSpout PR<

> https://github.com/apache/storm/pull/1924> fixes a bug with the number of

> uncommitted offsets that is quite important. It is not a blocker, but it 
is

> quite critical. I am going to do one last pass reviewing today. It would 
be

> good if we could have this PR included with the 

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[Jungtaek Lim]

About STORM-2249, since examples are shading their dependencies, binary
dist will grow much bigger. I've left some comments regarding that.
Btw, I have another view of this. Showing example codes is more important
than just let users execute some topologies. That's what example modules
are for. We need to include source as well. If we need to pick one, source
code would be better.

STORM-2343 seems better to add to 1.1.0. I just am not enough familiar with
storm-kafka-client so not sure I can review that, but I'll try to. I feel
it's not that make release dragged. Let's add to 1.1.0 epic.

Let's make minimum merge before another RC vote. Addressing STORM-2389 (and
maybe STORM-2343) is enough for me. Others are not that critical.

Thanks,
Jungtaek Lim (HeartSaVioR)


On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro 
wrote:

> Roshan, does this PR and JIRA<
> https://issues.apache.org/jira/browse/STORM-2249> address the missing
> jars problem that you mentioned. I had created it in December 2016, but
> there is an ongoing discussion if we should indeed put the jars in the
> examples location or not.
>
> On a different note, this storm-kafka-client/KafkaSpout PR<
> https://github.com/apache/storm/pull/1924> fixes a bug with the number of
> uncommitted offsets that is quite important. It is not a blocker, but it is
> quite critical. I am going to do one last pass reviewing today. It would be
> good if we could have this PR included with the release. Can anyone else
> review it as well ?
>
> Thanks,
> Hugo
>
> On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz  e...@gmail.com>> wrote:
>
> Yeah, I don’t think the file size is a killer/blocker. It’s largely due to
> shaded examples, etc. But it’s something to keep an eye on. Our binary
> releases shouldn’t have to be that big.
>
> -Taylor
>
> On Mar 1, 2017, at 12:09 PM, Roshan Naik > wrote:
>
> Have filed Jiras so for the 3 issues mentioned. Not sure if we need a JIRA
> for the file size getting bloated by that much.
> Somebody better familiar with the matter may want to take about that?
> -roshan
>
>
> On 3/1/17, 8:13 AM, "P. Taylor Goetz"  e...@gmail.com>> wrote:
>
>   Thanks for bringing these up Roshan. Feel free to file JIRA tickets for
> these issues and assign the “Release Apache Storm 1.1.0” epic so they can
> be tracked for this release.
>
>   -Taylor
>
> On Mar 1, 2017, at 9:27 AM, Roshan Naik > wrote:
>
> Found these additional issues:
>
>
>
> 1- BUG: Even if topology.eventlogger.executors=0,  the event_logger bolt
> is instantiated … previously observed to cause ~10% degradation in perf
> even with logging disabled.
>
> 2- Missing Jars: The storm-*-examples jars are missing in the binary
> distro (other than a storm-pmml-examples.jar, storm-jms-examples.jar &
> flux-examples.jar).
>
> 3- Minor: HdfsSpoutTopology example has not been moved into
> storm-hdfs-examples from storm-starter
>
>
>
> Another side observation …  v1.0.3 tar.gz downloadable was 190MB.   This
> v1.1.0 tar.gz  downloadable is 297MB !! …. even though some of the example
> topologies didn’t make it.
>
>
>
> -roshan
>
>
>
>
>
>
>
>


-- 
Name : Jungtaek Lim
Blog : http://medium.com/@heartsavior
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[Hugo Da Cruz Louro]

Roshan, does this PR and 
JIRA address the missing jars 
problem that you mentioned. I had created it in December 2016, but there is an 
ongoing discussion if we should indeed put the jars in the examples location or 
not.

On a different note, this storm-kafka-client/KafkaSpout 
PR fixes a bug with the number of 
uncommitted offsets that is quite important. It is not a blocker, but it is 
quite critical. I am going to do one last pass reviewing today. It would be 
good if we could have this PR included with the release. Can anyone else review 
it as well ?

Thanks,
Hugo

On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz 
> wrote:

Yeah, I don’t think the file size is a killer/blocker. It’s largely due to 
shaded examples, etc. But it’s something to keep an eye on. Our binary releases 
shouldn’t have to be that big.

-Taylor

On Mar 1, 2017, at 12:09 PM, Roshan Naik 
> wrote:

Have filed Jiras so for the 3 issues mentioned. Not sure if we need a JIRA for 
the file size getting bloated by that much.
Somebody better familiar with the matter may want to take about that?
-roshan


On 3/1/17, 8:13 AM, "P. Taylor Goetz" 
> wrote:

  Thanks for bringing these up Roshan. Feel free to file JIRA tickets for these 
issues and assign the “Release Apache Storm 1.1.0” epic so they can be tracked 
for this release.

  -Taylor

On Mar 1, 2017, at 9:27 AM, Roshan Naik 
> wrote:

Found these additional issues:



1- BUG: Even if topology.eventlogger.executors=0,  the event_logger bolt is 
instantiated … previously observed to cause ~10% degradation in perf even with 
logging disabled.

2- Missing Jars: The storm-*-examples jars are missing in the binary distro 
(other than a storm-pmml-examples.jar, storm-jms-examples.jar & 
flux-examples.jar).

3- Minor: HdfsSpoutTopology example has not been moved into storm-hdfs-examples 
from storm-starter



Another side observation …  v1.0.3 tar.gz downloadable was 190MB.   This v1.1.0 
tar.gz  downloadable is 297MB !! …. even though some of the example topologies 
didn’t make it.



-roshan

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[P. Taylor Goetz]

Yeah, I don’t think the file size is a killer/blocker. It’s largely due to 
shaded examples, etc. But it’s something to keep an eye on. Our binary releases 
shouldn’t have to be that big.

-Taylor

> On Mar 1, 2017, at 12:09 PM, Roshan Naik  wrote:
> 
> Have filed Jiras so for the 3 issues mentioned. Not sure if we need a JIRA 
> for the file size getting bloated by that much. 
> Somebody better familiar with the matter may want to take about that?
> -roshan
> 
> 
> On 3/1/17, 8:13 AM, "P. Taylor Goetz"  wrote:
> 
>Thanks for bringing these up Roshan. Feel free to file JIRA tickets for 
> these issues and assign the “Release Apache Storm 1.1.0” epic so they can be 
> tracked for this release.
> 
>-Taylor
> 
>> On Mar 1, 2017, at 9:27 AM, Roshan Naik  wrote:
>> 
>> Found these additional issues:
>> 
>> 
>> 
>> 1- BUG: Even if topology.eventlogger.executors=0,  the event_logger bolt is 
>> instantiated … previously observed to cause ~10% degradation in perf even 
>> with logging disabled.
>> 
>> 2- Missing Jars: The storm-*-examples jars are missing in the binary distro 
>> (other than a storm-pmml-examples.jar, storm-jms-examples.jar & 
>> flux-examples.jar).
>> 
>> 3- Minor: HdfsSpoutTopology example has not been moved into 
>> storm-hdfs-examples from storm-starter
>> 
>> 
>> 
>> Another side observation …  v1.0.3 tar.gz downloadable was 190MB.   This 
>> v1.1.0 tar.gz  downloadable is 297MB !! …. even though some of the example 
>> topologies didn’t make it.
>> 
>> 
>> 
>> -roshan
> 
> 
> 
> 

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[Roshan Naik]

Have filed Jiras so for the 3 issues mentioned. Not sure if we need a JIRA for 
the file size getting bloated by that much. 
Somebody better familiar with the matter may want to take about that?
-roshan


On 3/1/17, 8:13 AM, "P. Taylor Goetz"  wrote:

Thanks for bringing these up Roshan. Feel free to file JIRA tickets for 
these issues and assign the “Release Apache Storm 1.1.0” epic so they can be 
tracked for this release.

-Taylor

> On Mar 1, 2017, at 9:27 AM, Roshan Naik  wrote:
> 
> Found these additional issues:
> 
> 
> 
> 1- BUG: Even if topology.eventlogger.executors=0,  the event_logger bolt 
is instantiated … previously observed to cause ~10% degradation in perf even 
with logging disabled.
> 
> 2- Missing Jars: The storm-*-examples jars are missing in the binary 
distro (other than a storm-pmml-examples.jar, storm-jms-examples.jar & 
flux-examples.jar).
> 
> 3- Minor: HdfsSpoutTopology example has not been moved into 
storm-hdfs-examples from storm-starter
> 
> 
> 
> Another side observation …  v1.0.3 tar.gz downloadable was 190MB.   This 
v1.1.0 tar.gz  downloadable is 297MB !! …. even though some of the example 
topologies didn’t make it.
> 
> 
> 
> -roshan

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[P. Taylor Goetz]

Thanks for bringing these up Roshan. Feel free to file JIRA tickets for these 
issues and assign the “Release Apache Storm 1.1.0” epic so they can be tracked 
for this release.

-Taylor

> On Mar 1, 2017, at 9:27 AM, Roshan Naik  wrote:
> 
> Found these additional issues:
> 
> 
> 
> 1- BUG: Even if topology.eventlogger.executors=0,  the event_logger bolt is 
> instantiated … previously observed to cause ~10% degradation in perf even 
> with logging disabled.
> 
> 2- Missing Jars: The storm-*-examples jars are missing in the binary distro 
> (other than a storm-pmml-examples.jar, storm-jms-examples.jar & 
> flux-examples.jar).
> 
> 3- Minor: HdfsSpoutTopology example has not been moved into 
> storm-hdfs-examples from storm-starter
> 
> 
> 
> Another side observation …  v1.0.3 tar.gz downloadable was 190MB.   This 
> v1.1.0 tar.gz  downloadable is 297MB !! …. even though some of the example 
> topologies didn’t make it.
> 
> 
> 
> -roshan

Re: [CANCELED] [VOTE] Release Apache Storm 1.1.0 (RC2)

[Roshan Naik]

Found these additional issues:



1- BUG: Even if topology.eventlogger.executors=0,  the event_logger bolt is 
instantiated … previously observed to cause ~10% degradation in perf even with 
logging disabled.

2- Missing Jars: The storm-*-examples jars are missing in the binary distro 
(other than a storm-pmml-examples.jar, storm-jms-examples.jar & 
flux-examples.jar).

3- Minor: HdfsSpoutTopology example has not been moved into storm-hdfs-examples 
from storm-starter



Another side observation …  v1.0.3 tar.gz downloadable was 190MB.   This v1.1.0 
tar.gz  downloadable is 297MB !! …. even though some of the example topologies 
didn’t make it.



-roshan