Hi Daniel, So there is nothing that "cannot" be done with Gradle, it's just a matter of how many hacks and imperative code sequences. I was trying to convert the code into a declarative style but my eyes were hurting from this logic :) I'm sure you guys are used to it more than me so thats why it's taking time on my side.
Thank you for all the above answers. Based on your feedback I think we definitely need a collaboration platform (JIRA okay?) to take this issue in bite sizes. I need to write one section at a time and each time ask questions like .. what is this? Is it important? Many of the checks happening in place can be just other tasks with dependencies to them (called hidden tasks). This would simplify and clarify the logic a bit. Also I feel strongly that this build script requires multiple different sourcesets with custom classpath settings in addition to the multiple Java versions required "phew amazing!". So my next question .. for the very first step, if I cut all the fat away, then the main focus of the compile target is the javac commands .. is this correct? Yeah I know you're patching and whatnot, but let's do this in baby steps if I may? Cheers, Taher Alkhateeb On Feb 26, 2017 1:27 AM, "Daniel Dekany" <[email protected]> wrote: > Saturday, February 25, 2017, 8:02:06 PM, Taher Alkhateeb wrote: > > > Hello Folks, > > > > Okay I've been trying to migrate the compile task to gradle, but did not > > expect so many hacks and quirks in this target. I can copy instructions > > blindly but then that wouldn't make much sense since gradle operates very > > differently with its declarative approach. Examples of where it was > > confusing: > > > > - boot classpath checks? Why do they exist > > To ensure that you don't link to things (in rt.jar) that don't exist > in the officially supported lowest Java SE version. People often just > hastily use Java version what they have already installed, rather than > hunting down the specific (and usually ancient) Java version needed. > This obviously can't be allowed when building a release (that's why we > check in during "dist"). > > This has some extra importance as far as we have no "modules", because > within the only "module" we have, different classes are linked against > different Java SE versions, but IDE-s don't support that, so usually > you work with the highest supported version there. Thus only the build > will catch if you accidentally use a too modern API somewhere (it used > to happen in reality). > > > - regex replacements of Java source files! What's the story behind that > > As the comment says: "<!-- Comment out @SuppressFBWarnings, as it > causes compilation warnings in dependent Gradle projects -->" > > See also: http://stackoverflow.com/questions/37422191/how-to- > remove-suppressfbwarnings-annotations-from-released-jar-s > > > - why is the rmic compiler used? > > To generate the RMI stubs of course, but what do you mean? > > > - why do we have many separate ivy:cachepath declarations? do we have > > different artifacts? > > Because different classes have different *allowed* dependencies. Like > Configuration can't accidentally depend on the Servlet API, only the > Servlet support classes can, or else FreeMarker will crash in an > environment without servlets despite that you don't try to use its > servlet support (already happened when back then the build script was > sloppy). The build has to ensure that no such human error slips in, > because the IDE won't point it out. > > This is though yet again because of the monolithic (one jar) approach. > That's why I sad the we can split FM2 to multiple modules (requires > some Java expertise and understanding FM internals), as far as at the > end we can re-unite it in a single jar. Can Gradle do that? For FM3, > we don't do the last, we actually will have multiple artifacts. > > > - Why are we manually copying files? > > At various places for various reasons... Which one you don't > understand? > > > I think if we can have some interactive session or something to get my > head > > around the logic of this build then this would help me in translating it. > > There's no way around reading and thus understanding build.xml, as you > intend to rewrite the build process. I mean, everything is written > down there. If you don't get the reason of something, ask, as you did. > > > My conclusion so far is that either the product is inherently complex, or > > the build logic is painfully manual. > > It's low level library that has to work in various environments, has > optional dependencies, and dependencies on different versions of the > same library. (Less so than even in few years ago, but it's still like > that.) > > What makes the build more trickier than it could be is that FM2 is > monolithic. We can't change it in FM2, as far as the final artifact is > concerned. (I think it was the right compromise back then, but now > that we have dependency management, it's not anymore.) But really it's > not a big deal (with Ant). You have multiple classpaths, and multiple > subsets of classes paired with them. That's it. > > > Getting the big picture of what is happening is lost in the midst of > > these unending build instructions. > > > > I also am under the impression that the code base should be a > multi-project > > architecture given this constant shoving of files around. > > That's what were talking about in the past, right? It should be > modular, in FM3 it truly will be, in FM2 it can be *if* we can then > unite it into a signle jar that's equivalent with what we have now. If > that's not feasible with Gradle (which would be strange), then only > FM3 will switch to Gradle. > > > Am I correct in my conclusions so far? > > > > On Tue, Feb 14, 2017 at 11:00 PM, Daniel Dekany <[email protected]> > wrote: > > > >> Tuesday, February 14, 2017, 7:53:36 PM, Mauricio Nuñez wrote: > >> > >> > Hi, > >> > > >> > I have a question about this. With FM2, the ant script allows > differents > >> > combinations. I think this is not needed for FM3. Or we need to be > aware > >> > about specifics combinations of dependencies? This is related to drop > >> > support to older dependencies in FM3. > >> > >> We certainly will need to be able to support multiple versions of the > >> same library in FM3 too, though such cases will be much fewer at the > >> beginning. However, it's possible that in FM3 we solve that with > >> modularization (where each module is a separate jar and a separate > >> Maven artifact, and so can be built separately). It was already > >> decided that we will modularize FM3 to freemarker-core, > >> freemarker-sevlet, etc. So that's modularization by functionality. > >> Perhaps we should modularize further by dependency version. For > >> example, we could have freemarker-core-java8, which adds java.time > >> support to freemarker-core. So then, freemarker-core can remain purely > >> Java 7 (the minimal required version of FM3), and > >> freemarker-core-java-8 contains all that depends on Java 8, and > >> freemarker-core finds freemarker-core-java-8 via the SPI mechanism of > >> the Java platform (if that works reliably in all environments...). > >> But, while that's the cleanest solution as far as the build goes, it's > >> not necessary the most convenient solution for the user. He now has to > >> remember adding a dependency on freemarker-core-java-8 manually (if he > >> wants time API support). Things can get worse if this happens with > >> some other modules as well, like we have freemarker-serlet, and later > >> we realize that we will need a freemarker-servlet-5 as well. We can > >> hope that such things won't happen in reality, but the history of FM2 > >> shows otherwise. (And the absolute horror is if you have to support > >> permutations of the versions of multiple dependencies for the same > >> functional module... but let's assume we won't be so unlucky.) Compare > >> this to FM2, where the user didn't have to care... you drop in > >> freemarker.jar, and it discovers its environment automatically. > >> > >> I wonder if we could modularize FM2 on the source code level too, but > >> then smash all the output together into a single jar (and that's the > >> published artifact), while in FM3 they would remain separately > >> published artifacts. Thus the FM2 and FM3 would be more similar. > >> > >> > Regards, > >> > > >> > Mauricio > >> > > >> > > >> > > >> > 2017-02-14 15:39 GMT-03:00 Taher Alkhateeb < > [email protected]>: > >> > > >> >> Yeah that would be my bad :) We've been doing heart surgery in the > OFBiz > >> >> project that got me delayed a bit. > >> >> > >> >> Anyway, to share what I've worked on so far, I realized most of the > >> >> complexity in the script is in the "compile" target. There is _lots_ > of > >> >> shuffling files around and a good amount of ivy:cachepath > directives. I > >> did > >> >> not expect the logic to be so customized. > >> >> > >> >> I would like to add a build.gradle file that for now only handles > this > >> >> target, I will try to deliver something soon but this work should > >> really go > >> >> in baby steps ... it was overwhelming to try to do it all in one > shot. > >> >> Perhaps I can put a skeleton and we can work together to improve it. > >> >> > >> >> Anyway, sit tight, I'll work on something in the next few days. > >> >> > >> >> On Tue, Feb 14, 2017 at 5:23 PM, Daniel Dekany <[email protected]> > >> >> wrote: > >> >> > >> >> > Any update regarding the Gadle-isation? > >> >> > > >> >> > > >> >> > Monday, January 2, 2017, 9:36:03 AM, Taher Alkhateeb wrote: > >> >> > > >> >> > > Hi Daniel, > >> >> > > > >> >> > > Okay things are much more clear thank you! > >> >> > > > >> >> > > With respect to the repository, I guess there are two uses of the > >> word > >> >> > > repository. On one hand it is what you meant (a place to publish > >> your > >> >> > > artifacts) and on another hand it is where you pull your > >> dependencies > >> >> > from > >> >> > > (libraries). > >> >> > > > >> >> > > What I mean is the second form (pulling dependencies). Do you > want > >> to > >> >> > pull > >> >> > > from MavenCentral, or do you prefer to pull from JCenter? To give > >> you > >> >> > some > >> >> > > perspective on the difference this might be helpful -> > >> >> > > http://stackoverflow.com/questions/24852219/android- > >> >> > buildscript-repositories-jcenter-vs-mavencentral#28457914 > >> >> > > > >> >> > > Cheers, > >> >> > > > >> >> > > Taher Alkhateeb > >> >> > > > >> >> > > On Mon, Jan 2, 2017 at 11:29 AM, Daniel Dekany < > [email protected] > >> > > >> >> > wrote: > >> >> > > > >> >> > >> Monday, January 2, 2017, 8:14:25 AM, Taher Alkhateeb wrote: > >> >> > >> > >> >> > >> > Hi Daniel, > >> >> > >> > > >> >> > >> > Okay thank you for all the feedback. I will start to > experiment > >> in > >> >> > code a > >> >> > >> > little bit, but a few more lingering comments / questions: > >> >> > >> > > >> >> > >> > # Comments > >> >> > >> > - Every project optionally ends with a jar, but again, you can > >> tweak > >> >> > that > >> >> > >> > to your liking. > >> >> > >> > - Embedded Ant is pretty clean code (we use it a lot). Ant is > a > >> >> first > >> >> > >> class > >> >> > >> > citizen in Gradle and fully integrated syntax wise (it's not > XML) > >> >> > >> > - Your comment on JavaCC is similar to multi-project > architecture > >> >> > >> > - Gradle has built-in plugins for both Eclipse and Intellij > with > >> >> quite > >> >> > >> > powerful DSL for customization > >> >> > >> > > >> >> > >> > # Questions > >> >> > >> > - The code directory structure is strange to me. By convention > >> src > >> >> > means > >> >> > >> a > >> >> > >> > directory for all source code in multiple different languages. > >> So it > >> >> > >> could > >> >> > >> > be something like src/main/java src/test/java src/main/groovy > etc > >> >> ... > >> >> > >> > >> >> > >> I have a "javacc" directory on that level, which is a language > so > >> it > >> >> > >> fits into this, and "misc" which are the things I don't know > here > >> else > >> >> > >> to put, but they still belong to the sources of "main"... > >> >> > >> > >> >> > >> > However, you have other directories that sound to me like > >> >> "resources" > >> >> > or > >> >> > >> > such rather than src like "ide-settings" and "manual". > >> >> > >> > >> >> > >> They aren't part of "main", nor of "test" (which is just the > test > >> of > >> >> > >> main in the Maven concentions), but they are sources. They > aren't > >> the > >> >> > >> "resources" of "main" for sure (they would be part of > >> freemarker.jar > >> >> > >> otherwise). > >> >> > >> > >> >> > >> Where should we put them? I believe the current layout follows > the > >> >> > >> Maven principles, but I can be wrong. > >> >> > >> > >> >> > >> > Furthermore, the dist directory seems to hold generated > >> artifacts. > >> >> > >> > This should go in somewhere under /build no? > >> >> > >> > >> >> > >> No, src/dist doesn't hold generated artifacts. If you run `ant > >> dist`, > >> >> > >> there will be a build/dist, for which it's used as a source. > >> >> > >> > >> >> > >> > For example in Gradle the directory structure > >> >> > >> > under build contains among other directories things like > >> "classes", > >> >> > >> "libs", > >> >> > >> > "reports", "test-results", and so on. > >> >> > >> > >> >> > >> Here too (though the subdirectory names differ somewhat). > >> >> > >> > >> >> > >> > - I got confused by having src inside src, what does that > mean? > >> For > >> >> > >> example > >> >> > >> > .. src/main/misc/identifierChars/src/main/freemarker/adhoc > >> >> > >> > >> >> > >> It's just a Java class used for generating a piece of the actual > >> >> > >> source code. It can only be ran manually, and then you just copy > >> paste > >> >> > >> over its output. It's not used by the build process, but I > wanted > >> the > >> >> > >> source be in the VCS. Because it's java, it uses the usual > >> structure > >> >> > >> inside src/main/misc/identifierChars... I'm not sure where to > put > >> it. > >> >> > >> > >> >> > >> > - Are you open / wanting to restructure the directory layout > to > >> >> > something > >> >> > >> > more conventional, or do you prefer maintaining it? I ask > >> because I > >> >> > think > >> >> > >> > the structure seems a bit unconventional and surprised me > when I > >> >> > started > >> >> > >> to > >> >> > >> > look around. > >> >> > >> > >> >> > >> I'm open to improve the layout, but I'm not sure how to make it > >> more > >> >> > >> standard. I mean I think it pretty much is standard. To my > >> >> > >> understanding, things that don't belong to the sources used for > >> >> > >> generating the main Maven artifact (freemarker.jar), should be > in > >> >> > >> src/<something>, where <something> can be anything that doesn't > >> clash > >> >> > >> with "main", "test" and maybe some more common stuff. We might > have > >> >> > >> some such subdirectories that aren't common, but they have to be > >> >> > >> somewhere after all. > >> >> > >> > >> >> > >> > - Are you folks actually using OSGI? I can see the file > >> "osgi.bnd" > >> >> in > >> >> > top > >> >> > >> > level directory. Is this something to worry about? > >> >> > >> > >> >> > >> Yes, many use FreeMarker under OSGi. > >> >> > >> > >> >> > >> > - What is the purpose of build.properties.sample? > >> >> > >> > >> >> > >> To copy it to build.properties and modify it to fit your > >> environment. > >> >> > >> See also Building in the README. I don't know Gradle much, but I > >> think > >> >> > >> gradle.properties used have the same use case (and some more, > like > >> >> > >> JVM-level setup). > >> >> > >> > >> >> > >> > - Do you have a preference for a remote repo? Jcenter is the > >> >> biggest ( > >> >> > >> > https://bintray.com/bintray/jcenter), but you might have > >> specific > >> >> > >> > requirements from MavenCentral? > >> >> > >> > >> >> > >> I'm not sure I understand the question. We release to the Maven > >> >> > >> Central Repository (through Apache's Nexus since we are here at > >> ASF), > >> >> > >> just like everybody else. Are you talking about some non-release > >> >> > >> artifact maybe? > >> >> > >> > >> >> > >> > Too many questions, I know, but I'm having a bit of a > challenge > >> >> > trying to > >> >> > >> > grok the structure of the application because this is the > first > >> >> thing > >> >> > to > >> >> > >> > define in Gradle (where things are). > >> >> > >> > > >> >> > >> > Cheers, > >> >> > >> > > >> >> > >> > Taher Alkhateeb > >> >> > >> > > >> >> > >> > On Sun, Jan 1, 2017 at 9:40 PM, Daniel Dekany < > >> [email protected]> > >> >> > >> wrote: > >> >> > >> > > >> >> > >> >> Sunday, January 1, 2017, 12:32:22 PM, Daniel Dekany wrote: > >> >> > >> >> > >> >> > >> >> [snip] > >> >> > >> >> > Only the final results matter, which are: > >> >> > >> >> > > >> >> > >> >> > - freemarker.jar - monolithic (in 2.x.x at least), and note > >> the > >> >> > things > >> >> > >> >> > under the META-INF, and the substitutions in > >> >> version.properties. > >> >> > >> >> > > >> >> > >> >> > - Maven source and javadoc artifacts > >> >> > >> >> > > >> >> > >> >> > - Manual (HTML genereated via docgen from the XDocBook > file) > >> >> > >> >> > > >> >> > >> >> > - JavaDoc (with some class exclusions and with Java 8's > >> >> typographic > >> >> > >> >> > nonsense fixed) > >> >> > >> >> > > >> >> > >> >> > - Apache release artifacts (src and bin, signed and all, as > >> >> usual). > >> >> > >> >> > Note that the bin release contains the generated Manual > and > >> >> > JavaDoc. > >> >> > >> >> > > >> >> > >> >> > - Uploading release into the Maven staging repository > >> >> > >> >> > > >> >> > >> >> > - Rat report > >> >> > >> >> > > >> >> > >> >> > - Ensure that the project works with Eclipse fine (and with > >> >> > IntelliJ > >> >> > >> >> > ideally, but frankly I don't even know if the current one > >> works > >> >> > with > >> >> > >> >> > that well). In the README there's a detailed description > of > >> how > >> >> > to > >> >> > >> >> > set up Eclipse, where you can describe the necessary > steps. > >> >> > >> >> [snip] > >> >> > >> >> > >> >> > >> >> And have forgotten one: > >> >> > >> >> > >> >> > >> >> - Running the tests > >> >> > >> >> > >> >> > >> >> -- > >> >> > >> >> Thanks, > >> >> > >> >> Daniel Dekany > >> >> > >> >> > >> >> > >> >> > >> >> > >> > >> >> > >> -- > >> >> > >> Thanks, > >> >> > >> Daniel Dekany > >> >> > >> > >> >> > >> > >> >> > > >> >> > -- > >> >> > Thanks, > >> >> > Daniel Dekany > >> >> > > >> >> > > >> >> > >> > >> -- > >> Thanks, > >> Daniel Dekany > >> > >> > > -- > Thanks, > Daniel Dekany > >
