Saturday, February 25, 2017, 8:02:06 PM, Taher Alkhateeb wrote: > Hello Folks, > > Okay I've been trying to migrate the compile task to gradle, but did not > expect so many hacks and quirks in this target. I can copy instructions > blindly but then that wouldn't make much sense since gradle operates very > differently with its declarative approach. Examples of where it was > confusing: > > - boot classpath checks? Why do they exist
To ensure that you don't link to things (in rt.jar) that don't exist in the officially supported lowest Java SE version. People often just hastily use Java version what they have already installed, rather than hunting down the specific (and usually ancient) Java version needed. This obviously can't be allowed when building a release (that's why we check in during "dist"). This has some extra importance as far as we have no "modules", because within the only "module" we have, different classes are linked against different Java SE versions, but IDE-s don't support that, so usually you work with the highest supported version there. Thus only the build will catch if you accidentally use a too modern API somewhere (it used to happen in reality). > - regex replacements of Java source files! What's the story behind that As the comment says: "<!-- Comment out @SuppressFBWarnings, as it causes compilation warnings in dependent Gradle projects -->" See also: http://stackoverflow.com/questions/37422191/how-to-remove-suppressfbwarnings-annotations-from-released-jar-s > - why is the rmic compiler used? To generate the RMI stubs of course, but what do you mean? > - why do we have many separate ivy:cachepath declarations? do we have > different artifacts? Because different classes have different *allowed* dependencies. Like Configuration can't accidentally depend on the Servlet API, only the Servlet support classes can, or else FreeMarker will crash in an environment without servlets despite that you don't try to use its servlet support (already happened when back then the build script was sloppy). The build has to ensure that no such human error slips in, because the IDE won't point it out. This is though yet again because of the monolithic (one jar) approach. That's why I sad the we can split FM2 to multiple modules (requires some Java expertise and understanding FM internals), as far as at the end we can re-unite it in a single jar. Can Gradle do that? For FM3, we don't do the last, we actually will have multiple artifacts. > - Why are we manually copying files? At various places for various reasons... Which one you don't understand? > I think if we can have some interactive session or something to get my head > around the logic of this build then this would help me in translating it. There's no way around reading and thus understanding build.xml, as you intend to rewrite the build process. I mean, everything is written down there. If you don't get the reason of something, ask, as you did. > My conclusion so far is that either the product is inherently complex, or > the build logic is painfully manual. It's low level library that has to work in various environments, has optional dependencies, and dependencies on different versions of the same library. (Less so than even in few years ago, but it's still like that.) What makes the build more trickier than it could be is that FM2 is monolithic. We can't change it in FM2, as far as the final artifact is concerned. (I think it was the right compromise back then, but now that we have dependency management, it's not anymore.) But really it's not a big deal (with Ant). You have multiple classpaths, and multiple subsets of classes paired with them. That's it. > Getting the big picture of what is happening is lost in the midst of > these unending build instructions. > > I also am under the impression that the code base should be a multi-project > architecture given this constant shoving of files around. That's what were talking about in the past, right? It should be modular, in FM3 it truly will be, in FM2 it can be *if* we can then unite it into a signle jar that's equivalent with what we have now. If that's not feasible with Gradle (which would be strange), then only FM3 will switch to Gradle. > Am I correct in my conclusions so far? > > On Tue, Feb 14, 2017 at 11:00 PM, Daniel Dekany <[email protected]> wrote: > >> Tuesday, February 14, 2017, 7:53:36 PM, Mauricio Nuñez wrote: >> >> > Hi, >> > >> > I have a question about this. With FM2, the ant script allows differents >> > combinations. I think this is not needed for FM3. Or we need to be aware >> > about specifics combinations of dependencies? This is related to drop >> > support to older dependencies in FM3. >> >> We certainly will need to be able to support multiple versions of the >> same library in FM3 too, though such cases will be much fewer at the >> beginning. However, it's possible that in FM3 we solve that with >> modularization (where each module is a separate jar and a separate >> Maven artifact, and so can be built separately). It was already >> decided that we will modularize FM3 to freemarker-core, >> freemarker-sevlet, etc. So that's modularization by functionality. >> Perhaps we should modularize further by dependency version. For >> example, we could have freemarker-core-java8, which adds java.time >> support to freemarker-core. So then, freemarker-core can remain purely >> Java 7 (the minimal required version of FM3), and >> freemarker-core-java-8 contains all that depends on Java 8, and >> freemarker-core finds freemarker-core-java-8 via the SPI mechanism of >> the Java platform (if that works reliably in all environments...). >> But, while that's the cleanest solution as far as the build goes, it's >> not necessary the most convenient solution for the user. He now has to >> remember adding a dependency on freemarker-core-java-8 manually (if he >> wants time API support). Things can get worse if this happens with >> some other modules as well, like we have freemarker-serlet, and later >> we realize that we will need a freemarker-servlet-5 as well. We can >> hope that such things won't happen in reality, but the history of FM2 >> shows otherwise. (And the absolute horror is if you have to support >> permutations of the versions of multiple dependencies for the same >> functional module... but let's assume we won't be so unlucky.) Compare >> this to FM2, where the user didn't have to care... you drop in >> freemarker.jar, and it discovers its environment automatically. >> >> I wonder if we could modularize FM2 on the source code level too, but >> then smash all the output together into a single jar (and that's the >> published artifact), while in FM3 they would remain separately >> published artifacts. Thus the FM2 and FM3 would be more similar. >> >> > Regards, >> > >> > Mauricio >> > >> > >> > >> > 2017-02-14 15:39 GMT-03:00 Taher Alkhateeb <[email protected]>: >> > >> >> Yeah that would be my bad :) We've been doing heart surgery in the OFBiz >> >> project that got me delayed a bit. >> >> >> >> Anyway, to share what I've worked on so far, I realized most of the >> >> complexity in the script is in the "compile" target. There is _lots_ of >> >> shuffling files around and a good amount of ivy:cachepath directives. I >> did >> >> not expect the logic to be so customized. >> >> >> >> I would like to add a build.gradle file that for now only handles this >> >> target, I will try to deliver something soon but this work should >> really go >> >> in baby steps ... it was overwhelming to try to do it all in one shot. >> >> Perhaps I can put a skeleton and we can work together to improve it. >> >> >> >> Anyway, sit tight, I'll work on something in the next few days. >> >> >> >> On Tue, Feb 14, 2017 at 5:23 PM, Daniel Dekany <[email protected]> >> >> wrote: >> >> >> >> > Any update regarding the Gadle-isation? >> >> > >> >> > >> >> > Monday, January 2, 2017, 9:36:03 AM, Taher Alkhateeb wrote: >> >> > >> >> > > Hi Daniel, >> >> > > >> >> > > Okay things are much more clear thank you! >> >> > > >> >> > > With respect to the repository, I guess there are two uses of the >> word >> >> > > repository. On one hand it is what you meant (a place to publish >> your >> >> > > artifacts) and on another hand it is where you pull your >> dependencies >> >> > from >> >> > > (libraries). >> >> > > >> >> > > What I mean is the second form (pulling dependencies). Do you want >> to >> >> > pull >> >> > > from MavenCentral, or do you prefer to pull from JCenter? To give >> you >> >> > some >> >> > > perspective on the difference this might be helpful -> >> >> > > http://stackoverflow.com/questions/24852219/android- >> >> > buildscript-repositories-jcenter-vs-mavencentral#28457914 >> >> > > >> >> > > Cheers, >> >> > > >> >> > > Taher Alkhateeb >> >> > > >> >> > > On Mon, Jan 2, 2017 at 11:29 AM, Daniel Dekany <[email protected] >> > >> >> > wrote: >> >> > > >> >> > >> Monday, January 2, 2017, 8:14:25 AM, Taher Alkhateeb wrote: >> >> > >> >> >> > >> > Hi Daniel, >> >> > >> > >> >> > >> > Okay thank you for all the feedback. I will start to experiment >> in >> >> > code a >> >> > >> > little bit, but a few more lingering comments / questions: >> >> > >> > >> >> > >> > # Comments >> >> > >> > - Every project optionally ends with a jar, but again, you can >> tweak >> >> > that >> >> > >> > to your liking. >> >> > >> > - Embedded Ant is pretty clean code (we use it a lot). Ant is a >> >> first >> >> > >> class >> >> > >> > citizen in Gradle and fully integrated syntax wise (it's not XML) >> >> > >> > - Your comment on JavaCC is similar to multi-project architecture >> >> > >> > - Gradle has built-in plugins for both Eclipse and Intellij with >> >> quite >> >> > >> > powerful DSL for customization >> >> > >> > >> >> > >> > # Questions >> >> > >> > - The code directory structure is strange to me. By convention >> src >> >> > means >> >> > >> a >> >> > >> > directory for all source code in multiple different languages. >> So it >> >> > >> could >> >> > >> > be something like src/main/java src/test/java src/main/groovy etc >> >> ... >> >> > >> >> >> > >> I have a "javacc" directory on that level, which is a language so >> it >> >> > >> fits into this, and "misc" which are the things I don't know here >> else >> >> > >> to put, but they still belong to the sources of "main"... >> >> > >> >> >> > >> > However, you have other directories that sound to me like >> >> "resources" >> >> > or >> >> > >> > such rather than src like "ide-settings" and "manual". >> >> > >> >> >> > >> They aren't part of "main", nor of "test" (which is just the test >> of >> >> > >> main in the Maven concentions), but they are sources. They aren't >> the >> >> > >> "resources" of "main" for sure (they would be part of >> freemarker.jar >> >> > >> otherwise). >> >> > >> >> >> > >> Where should we put them? I believe the current layout follows the >> >> > >> Maven principles, but I can be wrong. >> >> > >> >> >> > >> > Furthermore, the dist directory seems to hold generated >> artifacts. >> >> > >> > This should go in somewhere under /build no? >> >> > >> >> >> > >> No, src/dist doesn't hold generated artifacts. If you run `ant >> dist`, >> >> > >> there will be a build/dist, for which it's used as a source. >> >> > >> >> >> > >> > For example in Gradle the directory structure >> >> > >> > under build contains among other directories things like >> "classes", >> >> > >> "libs", >> >> > >> > "reports", "test-results", and so on. >> >> > >> >> >> > >> Here too (though the subdirectory names differ somewhat). >> >> > >> >> >> > >> > - I got confused by having src inside src, what does that mean? >> For >> >> > >> example >> >> > >> > .. src/main/misc/identifierChars/src/main/freemarker/adhoc >> >> > >> >> >> > >> It's just a Java class used for generating a piece of the actual >> >> > >> source code. It can only be ran manually, and then you just copy >> paste >> >> > >> over its output. It's not used by the build process, but I wanted >> the >> >> > >> source be in the VCS. Because it's java, it uses the usual >> structure >> >> > >> inside src/main/misc/identifierChars... I'm not sure where to put >> it. >> >> > >> >> >> > >> > - Are you open / wanting to restructure the directory layout to >> >> > something >> >> > >> > more conventional, or do you prefer maintaining it? I ask >> because I >> >> > think >> >> > >> > the structure seems a bit unconventional and surprised me when I >> >> > started >> >> > >> to >> >> > >> > look around. >> >> > >> >> >> > >> I'm open to improve the layout, but I'm not sure how to make it >> more >> >> > >> standard. I mean I think it pretty much is standard. To my >> >> > >> understanding, things that don't belong to the sources used for >> >> > >> generating the main Maven artifact (freemarker.jar), should be in >> >> > >> src/<something>, where <something> can be anything that doesn't >> clash >> >> > >> with "main", "test" and maybe some more common stuff. We might have >> >> > >> some such subdirectories that aren't common, but they have to be >> >> > >> somewhere after all. >> >> > >> >> >> > >> > - Are you folks actually using OSGI? I can see the file >> "osgi.bnd" >> >> in >> >> > top >> >> > >> > level directory. Is this something to worry about? >> >> > >> >> >> > >> Yes, many use FreeMarker under OSGi. >> >> > >> >> >> > >> > - What is the purpose of build.properties.sample? >> >> > >> >> >> > >> To copy it to build.properties and modify it to fit your >> environment. >> >> > >> See also Building in the README. I don't know Gradle much, but I >> think >> >> > >> gradle.properties used have the same use case (and some more, like >> >> > >> JVM-level setup). >> >> > >> >> >> > >> > - Do you have a preference for a remote repo? Jcenter is the >> >> biggest ( >> >> > >> > https://bintray.com/bintray/jcenter), but you might have >> specific >> >> > >> > requirements from MavenCentral? >> >> > >> >> >> > >> I'm not sure I understand the question. We release to the Maven >> >> > >> Central Repository (through Apache's Nexus since we are here at >> ASF), >> >> > >> just like everybody else. Are you talking about some non-release >> >> > >> artifact maybe? >> >> > >> >> >> > >> > Too many questions, I know, but I'm having a bit of a challenge >> >> > trying to >> >> > >> > grok the structure of the application because this is the first >> >> thing >> >> > to >> >> > >> > define in Gradle (where things are). >> >> > >> > >> >> > >> > Cheers, >> >> > >> > >> >> > >> > Taher Alkhateeb >> >> > >> > >> >> > >> > On Sun, Jan 1, 2017 at 9:40 PM, Daniel Dekany < >> [email protected]> >> >> > >> wrote: >> >> > >> > >> >> > >> >> Sunday, January 1, 2017, 12:32:22 PM, Daniel Dekany wrote: >> >> > >> >> >> >> > >> >> [snip] >> >> > >> >> > Only the final results matter, which are: >> >> > >> >> > >> >> > >> >> > - freemarker.jar - monolithic (in 2.x.x at least), and note >> the >> >> > things >> >> > >> >> > under the META-INF, and the substitutions in >> >> version.properties. >> >> > >> >> > >> >> > >> >> > - Maven source and javadoc artifacts >> >> > >> >> > >> >> > >> >> > - Manual (HTML genereated via docgen from the XDocBook file) >> >> > >> >> > >> >> > >> >> > - JavaDoc (with some class exclusions and with Java 8's >> >> typographic >> >> > >> >> > nonsense fixed) >> >> > >> >> > >> >> > >> >> > - Apache release artifacts (src and bin, signed and all, as >> >> usual). >> >> > >> >> > Note that the bin release contains the generated Manual and >> >> > JavaDoc. >> >> > >> >> > >> >> > >> >> > - Uploading release into the Maven staging repository >> >> > >> >> > >> >> > >> >> > - Rat report >> >> > >> >> > >> >> > >> >> > - Ensure that the project works with Eclipse fine (and with >> >> > IntelliJ >> >> > >> >> > ideally, but frankly I don't even know if the current one >> works >> >> > with >> >> > >> >> > that well). In the README there's a detailed description of >> how >> >> > to >> >> > >> >> > set up Eclipse, where you can describe the necessary steps. >> >> > >> >> [snip] >> >> > >> >> >> >> > >> >> And have forgotten one: >> >> > >> >> >> >> > >> >> - Running the tests >> >> > >> >> >> >> > >> >> -- >> >> > >> >> Thanks, >> >> > >> >> Daniel Dekany >> >> > >> >> >> >> > >> >> >> >> > >> >> >> > >> -- >> >> > >> Thanks, >> >> > >> Daniel Dekany >> >> > >> >> >> > >> >> >> > >> >> > -- >> >> > Thanks, >> >> > Daniel Dekany >> >> > >> >> > >> >> >> >> -- >> Thanks, >> Daniel Dekany >> >> -- Thanks, Daniel Dekany
