Yeah I'm not sure why it increased, dependency management is a difficult task if you have a good tool, and an almost impossible task if done manually. Just type ./gradlew dependencies and observe the absolutely insane dependency tree. So I can't tell if the size increased because we pulled more libraries unnecessarily while converting or if the reason is wrong dependencies in the past, maybe probably both!
Anyway, whatever the correct size is, it is still "too much". We depend way on too many libraries, which adds complexity and this scary dependency tree, and so I think the right solution is to start cutting out unneeded libraries. I already did multiple rounds in build.gradle and identified libraries that I think are not needed. Also, if we decide to disable the plugins by default (especially BIRT) then that would also substantially reduce the size. This is one area that needs a lot of cleanup. On Tue, Aug 30, 2016 at 3:51 PM, Jacques Le Roux < [email protected]> wrote: > I double checked all these jars comes with the birt component. I was also > surprised that we had eclipse jars in the Gradle caches! > > So not a question, and we have effectively 350MB of dependencies instead > of 150MB before. > > Jacques > > > > Le 30/08/2016 à 13:36, Taher Alkhateeb a écrit : > >> Be careful, those could be just regular OFBiz requirements. For example, >> take a look at the below outputs from ./gradlew dependencies. As you can >> see there is the eclipse compiler as a requirement from tomcat. Just >> because it has the word eclipse does not mean it came from eclipse. >> >> +--- org.apache.tomcat:tomcat-jasper:8.0.36 >> | +--- org.apache.tomcat:tomcat-servlet-api:8.0.36 >> | +--- org.apache.tomcat:tomcat-juli:8.0.36 >> | +--- org.apache.tomcat:tomcat-jsp-api:8.0.36 (*) >> | +--- org.apache.tomcat:tomcat-el-api:8.0.36 >> | +--- org.eclipse.jdt.core.compiler:ecj:4.5 >> >> >> On Tue, Aug 30, 2016 at 2:30 PM, Jacques Le Roux < >> [email protected]> wrote: >> >> I mean those using Intellij don't run gradlew eclipse, so have less stuff >>> in cache, no? >>> >>> OK, I just answered myself :) There are 30 MB of *eclipse*.jar in the >>> Gradle caches, so it's only 320 MB of dependencies related to only OFBiz >>> >>> Jacques >>> >>> >>> >>> Le 30/08/2016 à 13:18, Taher Alkhateeb a écrit : >>> >>> Hmmm, not sure if Intellij or eclipse would make a difference in the >>>> cache >>>> size? It is gradle that is downloading, not eclipse nor intellij. >>>> >>>> On Tue, Aug 30, 2016 at 2:02 PM, Jacques Le Roux < >>>> [email protected]> wrote: >>>> >>>> Le 30/08/2016 à 12:43, Taher Alkhateeb a écrit : >>>> >>>>> Hi Jacques, >>>>> >>>>>> I think to get the absolute minimum cache size you do the following: >>>>>> - delete .gradle >>>>>> - ./gradlew cleanAll build (do not run eclipse so that you do not >>>>>> download >>>>>> the source dependencies) >>>>>> >>>>>> This would give you the minimum cache needed for OFBiz to run. >>>>>> >>>>>> Yep, sure if someone using IntelIJ could share the burden that would >>>>>> be >>>>>> >>>>> easier for me ;) >>>>> Not a big deal anyway >>>>> >>>>> Jacques >>>>> >>>>> >>>>> Taher Alkhateeb >>>>> >>>>> On Tue, Aug 30, 2016 at 1:38 PM, Jacques Le Roux < >>>>>> [email protected]> wrote: >>>>>> >>>>>> Le 30/08/2016 à 11:25, Taher Alkhateeb a écrit : >>>>>> >>>>>> Hi Jacques, >>>>>>> >>>>>>> I know you probably meant this as a friendly joke but this is our >>>>>>>> official >>>>>>>> ML, so I have to state for the record that I did not play a "trick", >>>>>>>> my >>>>>>>> objective was to say that you changed the topic and therefore we >>>>>>>> need >>>>>>>> to >>>>>>>> start a new thread. Choosing the new topic is entirely within your >>>>>>>> control. >>>>>>>> >>>>>>>> Yes of course, only a friendly joke :) >>>>>>>> >>>>>>>> Now as you probably already know Gradle has different dependencies >>>>>>> such >>>>>>> as >>>>>>> >>>>>>> compile and runtime. Part of the difference in size could be due to >>>>>>> >>>>>>>> copying >>>>>>>> only one of these dependencies and not all of them. For example the >>>>>>>> copy >>>>>>>> libs task (discussed earlier) only copied runtime dependencies. But >>>>>>>> is >>>>>>>> this >>>>>>>> the right thing to do? are you not going to compile anything in the >>>>>>>> future >>>>>>>> in the production environment? Maybe yes maybe no it depends doesn't >>>>>>>> it? >>>>>>>> It's different from one deployment to another. Therefore it is >>>>>>>> specific >>>>>>>> to >>>>>>>> each user in their own environment. >>>>>>>> >>>>>>>> Yes, that's why I did not continue this way. I have though still to >>>>>>>> >>>>>>>> find a >>>>>>> right solution for OWASP-DC >>>>>>> I mean https://issues.apache.org/jira/browse/OFBIZ-7930 >>>>>>> >>>>>>> Another reason could be that the development machine contains >>>>>>> additional >>>>>>> >>>>>>> unneeded dependencies. So if you try to delete the cache folder and >>>>>>> run >>>>>>> >>>>>>>> the >>>>>>>> build again you might get a smaller size upon copying. >>>>>>>> >>>>>>>> I just tried, after downloading the Internet again (kiiiidiiing ;)) >>>>>>>> it's >>>>>>>> >>>>>>>> indeed much smaller (Eclipse included) it's only 350 MB, a good >>>>>>> news! >>>>>>> >>>>>>> BTW handling (copying, deleting, moving) the caches on Windows is "a >>>>>>> bit" >>>>>>> long. Because Windows does not handle well a folder with plenty of >>>>>>> files >>>>>>> (I >>>>>>> guess some are small did not check). >>>>>>> Not a big deal since most of the time (if not all the time) Windows >>>>>>> is >>>>>>> not >>>>>>> used as a server. >>>>>>> >>>>>>> Also for the record, if no internet connection is a substantial >>>>>>> enough >>>>>>> >>>>>>> problem and multiple people are facing it then it makes sense that we >>>>>>> >>>>>>>> help >>>>>>>> our users, but this needs to be discussed thoroughly in ML to come >>>>>>>> up >>>>>>>> with >>>>>>>> a good clean solution before starting multiple jiras like the ones >>>>>>>> mentioned earlier. To me personally I don't think this is a big >>>>>>>> issue >>>>>>>> but >>>>>>>> I >>>>>>>> could be wrong so I leave it to others to have their say. >>>>>>>> >>>>>>>> I agree a plethora of Jira is not good. I think we have discussed >>>>>>>> this >>>>>>>> >>>>>>>> enough, now we need to continue to update the documentation. >>>>>>> For that you need 1st to know what you are talking about, hence this >>>>>>> discussion indeed. >>>>>>> >>>>>>> I wonder about the dependencies introduced in Gradle cache by Eclipse >>>>>>> Could someone using IntelliJ confirm it has much less than 350 MB in >>>>>>> cache? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Jacques >>>>>>> >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Taher Alkhateeb >>>>>>>> >>>>>>>> On Aug 30, 2016 11:50 AM, "Jacques Le Roux" < >>>>>>>> [email protected] >>>>>>>> wrote: >>>>>>>> >>>>>>>> Le 30/08/2016 à 09:21, Taher Alkhateeb a écrit : >>>>>>>> >>>>>>>> Hi Jacques, All, >>>>>>>> >>>>>>>>> Okay just to put things into context, the reason that kick-started >>>>>>>>> >>>>>>>>>> this >>>>>>>>>> discussion (multiple times so far) is that Pierre Smits had >>>>>>>>>> deployment >>>>>>>>>> requirements in which Gradle is not allowed as detailed in this >>>>>>>>>> thread: >>>>>>>>>> http://markmail.org/message/dzq3e55n6z4cwmre >>>>>>>>>> >>>>>>>>>> To make things short, I was of the opinion that Pierre's request >>>>>>>>>> represents >>>>>>>>>> unusual and specific deployment requirements and that it should >>>>>>>>>> not >>>>>>>>>> be >>>>>>>>>> the >>>>>>>>>> default way to deploy OFBiz because people would lose all the >>>>>>>>>> power >>>>>>>>>> and >>>>>>>>>> advantages from having a powerful build system automating >>>>>>>>>> everything. >>>>>>>>>> The >>>>>>>>>> discussions were had between myself, Pierre, Jacques with >>>>>>>>>> occasional >>>>>>>>>> input >>>>>>>>>> from others. >>>>>>>>>> >>>>>>>>>> Despite my above position, the following JIRAs were created and >>>>>>>>>> had >>>>>>>>>> patches >>>>>>>>>> / initiatives, all of which were very specific to the deployment >>>>>>>>>> requirements of Pierre: >>>>>>>>>> >>>>>>>>>> - https://issues.apache.org/jira/browse/OFBIZ-7782 -> The JIRA >>>>>>>>>> had >>>>>>>>>> a >>>>>>>>>> patch >>>>>>>>>> to create a task to copy all OFBiz runtime libraries to a certain >>>>>>>>>> folder. >>>>>>>>>> I >>>>>>>>>> objected to it as being too specific to the deployment >>>>>>>>>> requirements >>>>>>>>>> of >>>>>>>>>> Pierre. >>>>>>>>>> - https://issues.apache.org/jira/browse/OFBIZ-7893 -> >>>>>>>>>> Reintroduce a >>>>>>>>>> task >>>>>>>>>> to >>>>>>>>>> copy all OFBiz runtime libraries to a certain folder and remove >>>>>>>>>> gradle >>>>>>>>>> from >>>>>>>>>> the deployment scripts in /tools. I objected saying this is the >>>>>>>>>> wrong >>>>>>>>>> this >>>>>>>>>> to do and that I already objected in OFBIZ-7782 >>>>>>>>>> - https://issues.apache.org/jira/browse/OFBIZ-7796 -> Attempted >>>>>>>>>> to >>>>>>>>>> strip >>>>>>>>>> away gradle from the deployment scripts and replace it with Java. >>>>>>>>>> Again >>>>>>>>>> I >>>>>>>>>> objected to it for being too specific to the deployment >>>>>>>>>> requirements >>>>>>>>>> of >>>>>>>>>> Pierre. >>>>>>>>>> >>>>>>>>>> Sorry for the long introduction but the context is important for >>>>>>>>>> people >>>>>>>>>> to >>>>>>>>>> know where we stand exactly. Now to your question of how to deploy >>>>>>>>>> OFBiz >>>>>>>>>> without Gradle, I would like to reply in two parts: >>>>>>>>>> >>>>>>>>>> OK you played a trick here Taher by suggesting me the title :D >>>>>>>>>> >>>>>>>>>> The problem I tried to tackle is not "How to deploy OFBiz without >>>>>>>>>> >>>>>>>>> Gradle" >>>>>>>>> but "How to use Gradle w/o Internet connection" >>>>>>>>> >>>>>>>>> 1- Why without Gradle? >>>>>>>>> >>>>>>>>> 2- Available Options >>>>>>>>> >>>>>>>>> Why without Gradle? >>>>>>>>>> --------------------------- >>>>>>>>>> Imagine if I tell you that I want to deploy a ruby-on-rails >>>>>>>>>> application >>>>>>>>>> without rails. Or I want to deploy OFBiz without the widget >>>>>>>>>> engine. >>>>>>>>>> Is >>>>>>>>>> that >>>>>>>>>> a normal requirement? No, it is not a normal requirement, it is a >>>>>>>>>> requirement specific to my needs. >>>>>>>>>> >>>>>>>>>> Gradle is a fundamental part of the OFBiz solution. Stripping it >>>>>>>>>> away >>>>>>>>>> is >>>>>>>>>> like stripping away a core component. It is already holding a lot >>>>>>>>>> of >>>>>>>>>> responsibilities and expected to hold a lot more. You must have a >>>>>>>>>> good >>>>>>>>>> reason to want to remove it, and you should have the technical >>>>>>>>>> capabilities >>>>>>>>>> to do such a thing. >>>>>>>>>> >>>>>>>>>> So, it is in my opinion unreasonable to require the code base to >>>>>>>>>> have >>>>>>>>>> a >>>>>>>>>> solution to remove Gradle as an option (A task defined inside >>>>>>>>>> Gradle >>>>>>>>>> to >>>>>>>>>> copy libs so you can later run java -jar). Because if that is >>>>>>>>>> proper >>>>>>>>>> then >>>>>>>>>> we should have such options in the system to remove the service >>>>>>>>>> engine, >>>>>>>>>> the >>>>>>>>>> entity engine, the widgets, and many other fundamental components. >>>>>>>>>> >>>>>>>>>> Available Options >>>>>>>>>> ----------------------- >>>>>>>>>> Taking into consideration that this is an _advanced_ topic and >>>>>>>>>> those >>>>>>>>>> involved _should_ have the technical capacity to do it themselves; >>>>>>>>>> there >>>>>>>>>> are different scenarios we are discussing here and I provide a >>>>>>>>>> sample >>>>>>>>>> solution for each: >>>>>>>>>> >>>>>>>>>> 1- No internet connection: One solution is to copy the .gradle >>>>>>>>>> directory >>>>>>>>>> to >>>>>>>>>> the deployment server and run all gradle commands with --offline >>>>>>>>>> >>>>>>>>>> That's what I did above but only with the caches (and the needed >>>>>>>>>> >>>>>>>>>> wrapper). >>>>>>>>>> >>>>>>>>> It works but needs 10 times more disk space than before. >>>>>>>>> Since it's local and servers are supposed to have sufficient disk >>>>>>>>> spaces, >>>>>>>>> it's only a problem of bandwidth to copy the Gradle cache each time >>>>>>>>> it's >>>>>>>>> changed >>>>>>>>> To clarify the OFBiz compiled running code is all >>>>>>>>> build\libs\ofbiz.jar >>>>>>>>> and >>>>>>>>> only the external dependencies are in the the Gradle cache, right? >>>>>>>>> >>>>>>>>> BTW I also tried with the whole .gradle directory (the one in your >>>>>>>>> user >>>>>>>>> folder) but got issues because I guess I tried only on Windows (too >>>>>>>>> long >>>>>>>>> file paths). >>>>>>>>> I see no reasons it would not work, since by simply copying the >>>>>>>>> caches >>>>>>>>> and >>>>>>>>> the wrapper it worked for me (only slighter less space: 10MB) >>>>>>>>> >>>>>>>>> 2- Gradle not allowed: One solution is to copy the .gradle >>>>>>>>> directory >>>>>>>>> to >>>>>>>>> the >>>>>>>>> >>>>>>>>> deployment server and run java -jar build/ofbiz.jar (make sure to >>>>>>>>> put >>>>>>>>> >>>>>>>>> the >>>>>>>>>> correct JVM arguments) >>>>>>>>>> >>>>>>>>>> Yep, that's also the reason I kept and updated this information in >>>>>>>>>> >>>>>>>>>> https://cwiki.apache.org/confluence/display/OFBIZ/Apache+ >>>>>>>>>> >>>>>>>>> OFBiz+Technical+Production+Setup+Guide, despite your reluctance ;) >>>>>>>>> I still need to complete it... >>>>>>>>> >>>>>>>>> Of course the option proposed by Pierre and yourself by having a >>>>>>>>> gradle >>>>>>>>> >>>>>>>>> task to copy the runtime libraries to some location and then run >>>>>>>>> java >>>>>>>>> >>>>>>>>> -jar >>>>>>>>>> adding that folder to the classpath works. But it is too specific >>>>>>>>>> and >>>>>>>>>> awkward. >>>>>>>>>> >>>>>>>>>> The only reason is you have 10(!) times less data to move between >>>>>>>>>> >>>>>>>>>> machines... >>>>>>>>>> >>>>>>>>> If you want to customize things to your liking, it is extremely >>>>>>>>> >>>>>>>>> easy to create a gradle subproject (a file somewhere) and put all >>>>>>>>> the >>>>>>>>> >>>>>>>>> custom deployment logic in it, and then just create a patch that >>>>>>>>>> adds >>>>>>>>>> " >>>>>>>>>> apply from: 'foo/bar/my-custom-build-script.gradle' " >>>>>>>>>> >>>>>>>>>> OK, I'll consider it (as I did with the 1st post of this commit) >>>>>>>>>> before >>>>>>>>>> >>>>>>>>>> adding it in the doc >>>>>>>>>> >>>>>>>>> As you can see, I do that to document the possibilities for our >>>>>>>>> users. >>>>>>>>> Because I know it will happen and it's good for adoption. >>>>>>>>> >>>>>>>>> So to conclude, if we succumb to every deployment scenario for >>>>>>>>> every >>>>>>>>> >>>>>>>>> person, we would indeed have a very big code base. I suggest to >>>>>>>>> keep >>>>>>>>> >>>>>>>>> things >>>>>>>>>> lean and clean, and to avoid adding anything to the code base for >>>>>>>>>> the >>>>>>>>>> specific exceptional deployment requirements of some individuals. >>>>>>>>>> >>>>>>>>>> I totally agree on that! >>>>>>>>>> >>>>>>>>>> Jacques >>>>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Taher Alkhateeb. >>>>>>>>> >>>>>>>>>> In case of no internet connection, there are multiple options: >>>>>>>>>> - Copy the .gradle directory to the server and run all gradle >>>>>>>>>> commands >>>>>>>>>> with >>>>>>>>>> --offline >>>>>>>>>> - >>>>>>>>>> >>>>>>>>>> On Tue, Aug 30, 2016 at 9:21 AM, Jacques Le Roux < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I just tried something which seemed very simple after reading >>>>>>>>>> Taher's >>>>>>>>>> >>>>>>>>>> suggestion >>>>>>>>>>> >>>>>>>>>>> in OFBIZ-7783 >>>>>>>>>>> "For example if your problem is simply that you cannot build on a >>>>>>>>>>> disconnected computer even though the gradle cache is available >>>>>>>>>>> then >>>>>>>>>>> the >>>>>>>>>>> solution is to issue the command ./gradlew --offline >>>>>>>>>>> tasks-in-here. >>>>>>>>>>> So >>>>>>>>>>> one >>>>>>>>>>> solution is to simply archive the gradle cache and restore it." >>>>>>>>>>> >>>>>>>>>>> in the "Should we do binary releases?" thread >>>>>>>>>>> "You can also copy the .gradle cache from another computer and >>>>>>>>>>> start >>>>>>>>>>> using >>>>>>>>>>> it with the --offline flag" >>>>>>>>>>> >>>>>>>>>>> My idea was to mix --offline with --gradle-user-home Gradle >>>>>>>>>>> commands, >>>>>>>>>>> because I wanted to do this on the same machine and did not know >>>>>>>>>>> where >>>>>>>>>>> to >>>>>>>>>>> copy the .gradle cache (where the dependencies are) >>>>>>>>>>> -g, --gradle-user-home <-> Specifies the Gradle user home >>>>>>>>>>> directory. >>>>>>>>>>> The default is the .gradle directory in the user's home >>>>>>>>>>> directory. >>>>>>>>>>> >>>>>>>>>>> So, *on the same machine*, I copied the cache (830 MB) from >>>>>>>>>>> .gradle >>>>>>>>>>> directory to another place (I picked the shortest one, ie >>>>>>>>>>> c:\gradle). I >>>>>>>>>>> got >>>>>>>>>>> 9 issues with file names too long (was surprised about that since >>>>>>>>>>> from >>>>>>>>>>> the >>>>>>>>>>> user's home directory they should be longer) >>>>>>>>>>> Then tried to use "gradlew --offline -g c:\gradle ofbiz" but got >>>>>>>>>>> a >>>>>>>>>>> syntax >>>>>>>>>>> error and Gradle began to download the Internet again: >>>>>>>>>>> La syntaxe du nom de fichier, de répertoire ou de volume est >>>>>>>>>>> incorrecte. >>>>>>>>>>> "Downloading https://services.gradle.org/di >>>>>>>>>>> stributions/gradle-2.13-bin.zip >>>>>>>>>>> " >>>>>>>>>>> >>>>>>>>>>> I stopped. I guess I missed something, and rather decided to set >>>>>>>>>>> the >>>>>>>>>>> set >>>>>>>>>>> GRADLE_USER_HOME to c:\gradle in the gradlew.bat script. but then >>>>>>>>>>> got >>>>>>>>>>> >>>>>>>>>>> Exception in thread "main" java.io.FileNotFoundException: >>>>>>>>>>> "c:\gradle"\wrapper\dists\gradle-2.13-bin\4xsgxlfjcxvrea7akf >>>>>>>>>>> 941nvc7\ >>>>>>>>>>> gradle-2.13-bin.zip.lck (La syntaxe du nom de fichier, de >>>>>>>>>>> répertoire >>>>>>>>>>> ou >>>>>>>>>>> de volumeest incorrecte) >>>>>>>>>>> >>>>>>>>>>> I then copied the missing wrapper folder in c:\gradle (400 MB). >>>>>>>>>>> Despite >>>>>>>>>>> Windows and its damned limitation on paths names, it then worked >>>>>>>>>>> perfectly >>>>>>>>>>> well. >>>>>>>>>>> >>>>>>>>>>> But it's still disappointing because you have to copy 1,2 GB >>>>>>>>>>> instead >>>>>>>>>>> of >>>>>>>>>>> 150 MB (160MB including OFBiz jars) >>>>>>>>>>> >>>>>>>>>>> So my next question: should we use that as an advice to our users >>>>>>>>>>> who >>>>>>>>>>> can't use Gradle on their production and alike machines, or >>>>>>>>>>> should >>>>>>>>>>> we >>>>>>>>>>> try >>>>>>>>>>> to refine and find a better option? >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>> Jacques >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >
