I’m fine with dropping connectors from the binary distribution in favor having users use Maven, gradle, etc. handle user dependencies. Building topology jars manually (vs. letting Maven do the heavy lifting) is painful and something to be avoided, specifically for new users.
So the “binary release” for connectors, etc. would essentially become just a README that documents how to use them, and what the Maven coordinates are for each component. So with the connector jars just hosted in the Maven repo, and all examples source-only (to be built by users) the size of the binary distribution should be reduced dramatically. Any other opinions? -Taylor > On Mar 8, 2017, at 4:26 PM, Jungtaek Lim <kabh...@gmail.com> wrote: > > Regarding 3, if we don't make uber jar for connectors, adding jar file in > binary dist. to the classpath will not work, which means that including > connector jar libraries in binary dist. is no effect. > (Someone proposed shading storm-redis because of dependencies... We can > cover it by various way like adding to pom, or using '--artifact' so no > issue at all for now.) > > So if we want to go with 3, I'd also like to propose excluding connector > libraries in binary dist. at all. > > Others look great to me. > > - Jungtaek Lim (HeartSaVioR) > > 2017년 3월 9일 (목) 오전 6:12, P. Taylor Goetz <ptgo...@gmail.com>님이 작성: > >> I’d like to propose the following to address the binary distribution size: >> >> 1. Finish moving connector examples to the “examples” directory. >> 2. Make all examples source-only in the binary distribution. This will >> mean users will have to build the examples themselves, but that’s typically >> what one does when looking at example code. >> 3. Stop unnecessarily shading connector libraries (i.e. let the user’s pom >> pull in the transitive dependencies when building a topology jar). >> >> If we agree on this approach, I’ll take up the necessary changes for the >> 1.1.0 release. >> >> -Taylor >> >> >> >>> On Mar 8, 2017, at 9:16 AM, Xin Wang <data.xinw...@gmail.com> wrote: >>> >>> Agree with Jungtaek, STORM-2390 should not block the 1.1.0 release. >>> >>> - Xin >>> >>> 2017-03-07 9:16 GMT+08:00 Jungtaek Lim <kabh...@gmail.com>: >>> >>>> Sorry Roshan but I don't get it. >>>> We're already having separate modules for each of the connectors, and >> even >>>> examples. We also don't have any common modules for external >> connectors, so >>>> those are completely individual. >>>> >>>> As I reported earlier, binary dist. will go up more than 500 MB when we >>>> address STORM-2390(STORM-2249 actually), which the community seemed not >>>> want to. >>>> >>>> Since STORM-2390 is linked to epic for release Storm 1.1.0, I'd rather >>>> unlink from epic to get out of release, and discuss more. >>>> Any objections? >>>> >>>> - Jungtaek Lim (HeartSaVioR) >>>> >>>> 2017년 3월 4일 (토) 오전 10:02, Roshan Naik <ros...@hortonworks.com>님이 작성: >>>> >>>>> As having a separate module for each of the connectors will cause a lot >>>> of >>>>> bloat … may be good to club them into one. >>>>> >>>>> -roshan >>>>> >>>>> >>>>> On 3/2/17, 10:39 AM, "Sree V" <sree_at_ch...@yahoo.com.INVALID> wrote: >>>>> >>>>> +1separate main binaries, connectors and samples/examples. >>>>> >>>>> Thanking you. >>>>> With Regards >>>>> Sree >>>>> >>>>> On Thursday, March 2, 2017 8:02 AM, Satish Duggana < >>>>> satish.dugg...@gmail.com> wrote: >>>>> >>>>> >>>>> Agree that such huge binaries may not be acceptable. We should >>>> really >>>>> think >>>>> about the options proposed earlier like excluding some of the >>>> external >>>>> connectors from binary and update the documentation respectively. >>>>> >>>>> Thanks, >>>>> Satish. >>>>> >>>>> On Thu, Mar 2, 2017 at 4:41 PM, Jungtaek Lim <kabh...@gmail.com> >>>>> wrote: >>>>> >>>>>> Adding my observation to my last mail: >>>>>> I just give STORM-2249 a try (against master branch) and compare >>>>> before vs >>>>>> after. >>>>>> It adds more than 300 MB, and the final archive is more than 550 >>>> MB. >>>>> I >>>>>> guess it would be similar for 1.x branch. >>>>>> >>>>>> Before ---------- >>>>>> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:41 >>>>> apache-storm-2.0.0-SNAPSHOT. >>>>>> pom >>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41 >>>>> apache-storm-2.0.0-SNAPSHOT. >>>>>> pom.asc >>>>>> -rw-r--r-- 1 jlim staff 264M 3 2 19:41 >>>>> apache-storm-2.0.0-SNAPSHOT. >>>>>> tar.gz >>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41 >>>>> apache-storm-2.0.0-SNAPSHOT. >>>>>> tar.gz.asc >>>>>> -rw-r--r-- 1 jlim staff 264M 3 2 19:41 >>>>> apache-storm-2.0.0-SNAPSHOT. >>>>>> zip >>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:41 >>>>> apache-storm-2.0.0-SNAPSHOT. >>>>>> zip.asc >>>>>> >>>>>> After ----------- >>>>>> -rw-r--r-- 1 jlim staff 3.3K 3 2 19:46 >>>>> apache-storm-2.0.0-SNAPSHOT. >>>>>> pom >>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:46 >>>>> apache-storm-2.0.0-SNAPSHOT. >>>>>> pom.asc >>>>>> -rw-r--r-- 1 jlim staff 564M 3 2 19:46 >>>>> apache-storm-2.0.0-SNAPSHOT. >>>>>> tar.gz >>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:46 >>>>> apache-storm-2.0.0-SNAPSHOT. >>>>>> tar.gz.asc >>>>>> -rw-r--r-- 1 jlim staff 565M 3 2 19:46 >>>>> apache-storm-2.0.0-SNAPSHOT. >>>>>> zip >>>>>> -rw-r--r-- 1 jlim staff 473B 3 2 19:47 >>>>> apache-storm-2.0.0-SNAPSHOT. >>>>>> zip.asc >>>>>> >>>>>> While 264M is already a bit huge for me, 564M is not that I can >>>>> accept. >>>>>> (Binary dist. of Flink 1.2.0 is 127M, Spark 2.1.0 is 195M, Kafka >>>>> 0.10.2 is >>>>>> 37M.) >>>>>> >>>>>> Btw, we're including source code of examples, and "mvn clean >>>>> package" will >>>>>> work for every example modules. >>>>>> >>>>>> 2017년 3월 2일 (목) 오전 11:29, Jungtaek Lim <kabh...@gmail.com>님이 작성: >>>>>> >>>>>>> I guess it might be good time to think why we add all connectors >>>>> to the >>>>>>> binary distribution. >>>>>>> >>>>>>> Spark and Flink don't include them to binary dist. They even >>>> moved >>>>> some >>>>>> or >>>>>>> most of connectors out of repo, have been maintaining them in >>>>> Apache >>>>>> Bahir. >>>>>>> (Personally this is something I'm in favor of. We have lots of >>>>> connectors >>>>>>> and many of them are outdated - clear example is >>>>> storm-elasticsearch.) >>>>>>> >>>>>>> If we are assuming online then we don't even need to think about >>>>> users >>>>>>> touching binary dist. version of connectors. Users have been >>>>> including >>>>>> them >>>>>>> via build tools' dependency management, or even starting 1.1.0, >>>>> users can >>>>>>> include them via '--artifact' option. >>>>>>> >>>>>>> I also was just one of users for Storm, and I haven't use them >>>>> directly. >>>>>>> How much UX gets worse when we remove connectors to binary dist? >>>>> It only >>>>>>> helps some users who are not connected to the internet, and IMHO >>>>> it's a >>>>>>> rare case. >>>>>>> >>>>>>> I would like to see the opposite approach, removing all >>>> connectors >>>>> (or >>>>>>> just keeping storm-kafka/storm-kafka-client and some more >>>> preferred >>>>>> things) >>>>>>> and its relevant examples from binary dist. >>>>>>> >>>>>>> What do you think about it? >>>>>>> >>>>>>> - Jungtaek Lim (HeartSaVioR) >>>>>>> >>>>>>> >>>>>>> 2017년 3월 2일 (목) 오전 10:23, Roshan Naik <ros...@hortonworks.com>님이 >>>>> 작성: >>>>>>> >>>>>>> Once all of the shaded examples are included the size will go up >>>>> further. >>>>>>> >>>>>>> But currently as they are not part of the tar.gz … something else >>>>> is the >>>>>>> culprit for the bloat. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases >>>>> .. of >>>>>>> files that are larger than 4MB. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> @Jungtaek Lim<mailto:j...@hortonworks.com> : I am thinking, >>>>> since the >>>>>>> code for the examples can be easily viewed online … it would be >>>>> valuable >>>>>> to >>>>>>> have the executable topologies made available to the user as part >>>>> of the >>>>>>> binary release … rather have them figure out how to build it >>>>> correctly >>>>>>> before trying them out. >>>>>>> >>>>>>> >>>>>>> >>>>>>> -roshan >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> ➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh >>>>> {} \; >>>>>>> >>>>>>> -rw-r--r--@ 1 roshan staff 70M Feb 7 12:33 >>>>>> ./examples/storm-starter/ >>>>>>> storm-starter-topologies-1.0.3.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30 >>>>>>> ./external/flux/flux-examples-1.0.3.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32 >>>>>>> ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32 >>>>>>> ./external/sql/storm-sql-core/guava-16.0.1.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30 >>>>>>> ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar- >>>>>> with-dependencies.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33 >>>>>>> >>>>> ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33 >>>>>>> ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar >>>>>>> >>>>>>> -rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02 >>>>> ./lib/clojure-1.7.0.jar >>>>>>> >>>>>>> -rw-r--r--@ 1 roshan staff 19M Feb 7 12:26 >>>>>> ./lib/storm-core-1.0.3.jar >>>>>>> >>>>>>> -rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26 >>>>>>> ./lib/storm-rename-hack-1.0.3.jar >>>>>>> >>>>>>> >>>>>>> >>>>>>> ➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh >>>>> {} \; >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23 >>>>> ./examples/storm-pmml- >>>>>>> examples/storm-pmml-examples-1.1.0.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20 >>>>>> ./examples/storm-starter/ >>>>>>> storm-starter-topologies-1.1.0.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11 >>>>>>> ./external/flux/flux-examples-1.1.0.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16 >>>>>>> ./external/sql/storm-sql-core/calcite-core-1.11.0.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16 >>>>>>> ./external/sql/storm-sql-core/guava-16.0.1.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12 >>>>>> ./external/sql/storm-sql- >>>>>>> runtime/calcite-core-1.11.0.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12 >>>>>> ./external/sql/storm-sql- >>>>>>> runtime/guava-16.0.1.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18 >>>>>>> ./external/storm-druid/storm-druid-1.1.0.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11 >>>>>>> ./external/storm-eventhubs/storm-eventhubs-1.1.0-jar- >>>>>> with-dependencies.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20 >>>>>>> >>>>> ./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18 >>>>>>> ./external/storm-submit-tools/storm-submit-tools-1.1.0.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02 >>>>> ./lib/clojure-1.7.0.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07 >>>>>> ./lib/storm-core-1.1.0.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07 >>>>>>> ./lib/storm-rename-hack-1.1.0.jar >>>>>>> >>>>>>> -rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19 >>>>>>> ./toollib/storm-kafka-monitor-1.1.0.jar >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 3/1/17, 4:43 PM, "Jungtaek Lim" <kabh...@gmail.com> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> About STORM-2249, since examples are shading their >>>> dependencies, >>>>>> binary >>>>>>> >>>>>>> dist will grow much bigger. I've left some comments regarding >>>>> that. >>>>>>> >>>>>>> Btw, I have another view of this. Showing example codes is >>>> more >>>>>>> important >>>>>>> >>>>>>> than just let users execute some topologies. That's what >>>> example >>>>>>> modules >>>>>>> >>>>>>> are for. We need to include source as well. If we need to pick >>>>> one, >>>>>>> source >>>>>>> >>>>>>> code would be better. >>>>>>> >>>>>>> >>>>>>> >>>>>>> STORM-2343 seems better to add to 1.1.0. I just am not enough >>>>>> familiar >>>>>>> with >>>>>>> >>>>>>> storm-kafka-client so not sure I can review that, but I'll try >>>>> to. I >>>>>>> feel >>>>>>> >>>>>>> it's not that make release dragged. Let's add to 1.1.0 epic. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Let's make minimum merge before another RC vote. Addressing >>>>>> STORM-2389 >>>>>>> (and >>>>>>> >>>>>>> maybe STORM-2343) is enough for me. Others are not that >>>>> critical. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Jungtaek Lim (HeartSaVioR) >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro < >>>>>>> hlo...@hortonworks.com> >>>>>>> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Roshan, does this PR< >>>>> https://github.com/apache/storm/pull/1831> >>>>>> and >>>>>>> JIRA< >>>>>>> >>>>>>>> https://issues.apache.org/jira/browse/STORM-2249> address >>>> the >>>>>>> missing >>>>>>> >>>>>>>> jars problem that you mentioned. I had created it in >>>> December >>>>> 2016, >>>>>>> but >>>>>>> >>>>>>>> there is an ongoing discussion if we should indeed put the >>>>> jars in >>>>>>> the >>>>>>> >>>>>>>> examples location or not. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> On a different note, this storm-kafka-client/KafkaSpout PR< >>>>>>> >>>>>>>> https://github.com/apache/storm/pull/1924> fixes a bug with >>>>> the >>>>>>> number of >>>>>>> >>>>>>>> uncommitted offsets that is quite important. It is not a >>>>> blocker, >>>>>>> but it is >>>>>>> >>>>>>>> quite critical. I am going to do one last pass reviewing >>>>> today. It >>>>>>> would be >>>>>>> >>>>>>>> good if we could have this PR included with the release. Can >>>>> anyone >>>>>>> else >>>>>>> >>>>>>>> review it as well ? >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> Thanks, >>>>>>> >>>>>>>> Hugo >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz < >>>>> ptgo...@gmail.com >>>>>>> <mailto:ptgo >>>>>>> >>>>>>>> e...@gmail.com>> wrote: >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> Yeah, I don’t think the file size is a killer/blocker. It’s >>>>> largely >>>>>>> due to >>>>>>> >>>>>>>> shaded examples, etc. But it’s something to keep an eye on. >>>>> Our >>>>>>> binary >>>>>>> >>>>>>>> releases shouldn’t have to be that big. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> -Taylor >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> On Mar 1, 2017, at 12:09 PM, Roshan Naik < >>>>> ros...@hortonworks.com< >>>>>>> mailto: >>>>>>> >>>>>>>> ros...@hortonworks.com>> wrote: >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> Have filed Jiras so for the 3 issues mentioned. Not sure if >>>>> we need >>>>>>> a JIRA >>>>>>> >>>>>>>> for the file size getting bloated by that much. >>>>>>> >>>>>>>> Somebody better familiar with the matter may want to take >>>>> about >>>>>> that? >>>>>>> >>>>>>>> -roshan >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgo...@gmail.com >>>>> <mailto: >>>>>> ptgo >>>>>>> >>>>>>>> e...@gmail.com>> wrote: >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> Thanks for bringing these up Roshan. Feel free to file JIRA >>>>>>> tickets for >>>>>>> >>>>>>>> these issues and assign the “Release Apache Storm 1.1.0” >>>> epic >>>>> so >>>>>>> they can >>>>>>> >>>>>>>> be tracked for this release. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> -Taylor >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> On Mar 1, 2017, at 9:27 AM, Roshan Naik < >>>>> ros...@hortonworks.com< >>>>>>> mailto: >>>>>>> >>>>>>>> ros...@hortonworks.com>> wrote: >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> Found these additional issues: >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> 1- BUG: Even if topology.eventlogger.executors=0, the >>>>>> event_logger >>>>>>> bolt >>>>>>> >>>>>>>> is instantiated … previously observed to cause ~10% >>>>> degradation in >>>>>>> perf >>>>>>> >>>>>>>> even with logging disabled. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> 2- Missing Jars: The storm-*-examples jars are missing in >>>> the >>>>>> binary >>>>>>> >>>>>>>> distro (other than a storm-pmml-examples.jar, >>>>>> storm-jms-examples.jar >>>>>>> & >>>>>>> >>>>>>>> flux-examples.jar). >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> 3- Minor: HdfsSpoutTopology example has not been moved into >>>>>>> >>>>>>>> storm-hdfs-examples from storm-starter >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> Another side observation … v1.0.3 tar.gz downloadable was >>>>> 190MB. >>>>>>> This >>>>>>> >>>>>>>> v1.1.0 tar.gz downloadable is 297MB !! …. even though some >>>>> of the >>>>>>> example >>>>>>> >>>>>>>> topologies didn’t make it. >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> -roshan >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Name : Jungtaek Lim >>>>>>> >>>>>>> Blog : http://medium.com/@heartsavior >>>>>>> >>>>>>> Twitter : http://twitter.com/heartsavior >>>>>>> >>>>>>> LinkedIn : http://www.linkedin.com/in/heartsavior >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>> >> >>
