Hi Tim, On Sat, 26 May 2018 02:44:37 -0400 Tim Boudreau <[email protected]> wrote:
> > Guess you missed that my Netbeans ebuilds wrap nothing... They call > > javac directly. NO ant, maven, gradle etc. It CAN wrap other build > > systems, or it can BE the build system :) > > > > Great. Got a link to this work? Yes, its still a work in progress, feel free to open issues for any problems encountered. https://github.com/Obsidian-StudiosInc/os-xtoo/ > > > In my case portage IS the build system for Netbeans. Its very > > trivial. Probably the most simplistic way to build Netbeans. > > > > Gentoo-prefix (portable gentoo build system that uses a custom > $PREFIX - https://wiki.gentoo.org/wiki/Project:Prefix might be useful > for running on other OS's [but probably not Windows]). It pretty > well as long as you're not trying to install something that needs to > create OS-level users or things like that. Actually Windows is on there. No clue as to the support, I would assume paths or such would need to be different. Unless cygwin handles that stuff. I am not very familiar with all that. I would hope any next generation portage would have better support there. Not that I care much about Windows, but being python and what not. Most all of it should in theory work on Windows. Just need refinement and Windows specific changes and conditionals. Not sure as to the effort put forth for making things happen on Windows. The compiling from source and all might get a big interesting for other stuff. For java it would be pretty basic. But I would need to adapt jem or something like it for Windows. Its used heavily. https://github.com/Obsidian-StudiosInc/jem > > Ant and Gradle are trivial to build compared to Maven, its junk > > IMHO. > > We'll have to agree to disagree there. In the comments on my blog > linked earlier, you'll find a comment added be me a couple of years > ago: Sure, but I can build ant from source, I can build Gradle from source. I am totally stuck for months on Maven. Which maybe a blocker for my netbeans work. At least 1 project encountered relies on Maven classes. I need to confirm which its been sometime. It was not maven related but used classes from Maven. I have only been able to package 3 maven sub projects because I cannot process their mdo files to java. Stuck on this one which is core... https://github.com/apache/maven/tree/master/maven-core/src/main/mdo Packaged these, no mdo files to process https://github.com/Obsidian-StudiosInc/os-xtoo/tree/master/dev-java/maven-artifact https://github.com/Obsidian-StudiosInc/os-xtoo/tree/master/dev-java/maven-builder-support https://github.com/Obsidian-StudiosInc/os-xtoo/tree/master/dev-java/maven-settings If anyone is familiar I would like to hear from you. Otherwise I likely need to post to a maven development list. To learn about this mdo format no one uses other than Maven it seems. > I spent a few hours the other day in Gradleware's offices, debating > build system philosophy with Hans Doktor, the father of Gradle, > before we moved on to more productive topics. Interesting, if only to be a fly on the wall then.... Their enterprise stuff is interesting. Not sure how beneficial. > To me the bottom line is this: There is a lot of badly engineered > stuff out there. If your goal is to have the net amount of badly > engineered stuff go down, you find out what are common ailments, and > try to design systems that make those ailments an impossibility. Maven maybe good once built, but I feel plexus and modello fall into that badly engineered aspect. Or something as like no one uses it but Maven to build Maven. That does not seem to good to me. > If your goal is to make money, you write something that mitigates the > harm that bad habits do, so people can keep practicing them. Nobody > actually wants to hear *You are thinking about your own software in a > way that does not reflect reality* - they'd much rather hear *Poor > baby, did the mean engineer say you got things wrong? We'll make it > so you can go right on doing what you've been doing*. > > Bottom line: People do dumb things when building software. And > they'll pay money to someone who will facilitate letting them keep > doing dumb things, because they imagine that is cheaper than ripping > the dumb things out. If you would prefer not to do dumb things, > don't let something in the door that's designed to make that easy. > > I get that there's a lot of money to be made helping people shoot > themselves in the foot and feel less pain. But I do not want to shoot > myself in the foot, or anyone else to - the world is a better place > with less of that. > > These are my basic problems with Gradle: > > 1. Scriptable build systems increase the probability that your build > will, over time May not be the best habit, but it is the norm with make, cmake, meson, etc. You can script all of them in ways. The reverse argument there is plugins... They are voodoo, and you end up having to build them as well... Like this just to build native code https://github.com/gentoo/gentoo/tree/master/dev-java/maven-hawtjni-plugin After packaging that, just for one package this is the result https://github.com/gentoo/gentoo/blob/master/dev-java/netty-tcnative/netty-tcnative-1.1.33.15.ebuild Or my version, tossing that maven plugin upstream netty uses https://github.com/Obsidian-StudiosInc/os-xtoo/blob/master/dev-java/netty-tcnative/netty-tcnative-9999.ebuild Similar but easier to maintain as no maven plugin used. Others in Gentoo wanted to use the plugin. I did not want to package. Just an example. I do not see the Maven plugin doing anything that special. Surely not to justify time to package and maintain that plugin. > - Slowly become more and more unmaintainable I see some pretty complex make, cmake, and now meson build systems. For some they maybe hard, but for others familiar not so much. The goal with each is to make it more maintainable. That is the biggest argument for Meson over others. > - Require the dreaded four hours of "how to set your machine up so > builds work" that, pre-Maven, was the norm That last one I have to call out. I fail to see how installing a jdk and running gradlew is harder than Maven. Maven hasn't a wrapper. Which means I need to download and do something there. You can just toss in a gradlew into a project, and it passes all commands to gradle. After it downloads and installs it etc. Not sure on this 4 hour thing. That part is pretty nifty. Though likely could do something along those lines for Maven. But they do not presently. No maven wrapper that I am aware of that will download itself and run commands as if it was mvn. > 2. The ability to script, or have ad-hoc conditionals in a build > script means the only way a tool can figure out what it does is to > run it and hope it will do the same thing the next time I have seen other times where it breaks, like Cmake works with make, but switching to Ninja it breaks. Due to conditionals and what not in the CMakeLists.txt > One of the things an IDE needs to be able to do is to look at a > description of what to build, *reason* about it, and *make the > minimal modification that will change its behavior in some way the > user wants.* That is impossible with Ant (and is the reason NetBeans > Ant projects create the dreaded inscrutable > nbproject/build-impl.xml). It is impossible with Gradle. It *is* > possible with Maven - precisely because if you are compiling Java, > you are using the compiler plugin; if you're running tests, you're > using the surefire plugin; etc. That just sounds like the difference of maven using plugins for various things. Plugins that also have to be packaged after packaging Maven. > The key difference is that a build system should *describe what to > do*, not *describe how to do it.* I guess I am failing to see how Maven does that and others do not. Then again I am not super familiar with Maven. I was not keen on its concepts since inceptions. I never used it for any projects. > It's the same difference as the difference between > SQL and looking things up in a wad of memory. The SQL can be > optimized, because it doesn't say "read five bytes at offset 3027, > another five bytes at offset 4052", it says "find the last names of > people named Joe" and lets the back end decide how to do it, and > anything in between it and the back end can do transforms on that SQL > and actually know *exactly how it's changing the semantics.* Yes and in SQL you can get widely different results from different RDBMS. Why there is a SQL spec, even than many deviate like MySQL and others. If its always doing the same operation, then the outcome should be more consistent. It is a different approach. I do see your point just the same. Like cmake and meson, both just spit out makefiles or ninja.build. Rather than doing that stuff directly without some intermediary format. I suspect building a C project with gradle is along those lines. Maven seems to wrap make... Previous example shows its still using configure, etc. The maven-hawtjni-plugin isn't compiling that native code directly. Plus I think both Maven and Gradle fail at install. Actually installing the package it built into the system. Maybe possible but most I have seen omit that. The jar ends up in the repo. Which would not work for say Tomcat. Where you need scripts to start it and other stuff on the system. Not in a maven or gradle repo/cache. > If the build system *describes what to do*, you can build tools that > reason about that - that is why there is so much excellent tooling > around SQL, or HTTP - that's what lets you build higher level tools, > such as IDEs that, say, can add building a new kind of artifact, or > signing JARs or whatever into a build and *not break anything*, or > things that convert from one kind of project to another, or adapt > builds into a packaging system such as Portage or Pkgsrc. I fail to see how Maven describes stuff. Likely missing something due to not being familiar with Maven. > Anyway, unless someone here is going to actually do the work of > converting the NetBeans build system to Gradle, none of this is > terribly relevant. Yes sure I agree. It just came up per discussing Gentoo build, and then the Bazel, Buck thing. After looking at those for C purpose. Gradle seemed to shine more and more is all. > > XML is dead, and we should kill it ASAP. Ant and Maven continue this > > XML legacy, most the rest of the world has moved passed horrendous > > XML. > > I'll agree that the industry had an unhealthy fetish with XML, and > both Ant and Maven are victims of it. I know Maven did add some sort > of Groovy thing a few years back - and anyway, there are an infinity > of config file markup languages (YAML, etc.) that could be adapted to > replace XML there. Yes, and there are SOOOO many xml libraries that I wish would die and go away. Even the jakarata ant ones are odd. No longer developed, in attic, but maybe still used in Ant... Seems we can never move passed xml. Plus there are super odd circular deps with stuff. Like this mess https://github.com/jaxen-xpath/jaxen/issues/2 I just remove jdom and jaxen https://github.com/Obsidian-StudiosInc/os-xtoo/blob/master/dev-java/jaxen/jaxen-9999.ebuild#L39 Then from jdom2 remove jaxen... https://github.com/Obsidian-StudiosInc/os-xtoo/blob/master/dev-java/jdom/jdom-9999.ebuild#L43 So much legacy XML cruft, and other stuff drives me crazy... Die die die > > Netbeans ant build can be fragile. It was next to impossible to wrap > > its ant build system in Gentoo's portage. > > Agreed, the Ant setup is pretty bespoke. I imagine you ran into it > wanting to build all of its own dependencies. Not building them, but mixing downloaded dependencies with ones already built on system. It was a bit to much work. Easier to just replace them all and not deal with the hashes and what not. This is just one and its a considerable amount https://github.com/gentoo/gentoo/blob/master/dev-java/netbeans-java/netbeans-java-8.2.ebuild > > > Such that it was easier to > > bypass it entirely. I believe the result of what I am working with > > is considerably less complex. No java code, no extending a build > > system, no plugins etc. > > > > Interesting. Given the way Portage builds each package under > /var/tmp/portage/$TYPE/$PACKAGE, did you avoid having a separate > netbeans source checkout for each module you build? NO!!! That is the Achilles heal. Portage keeps unpacking the same tarball. Though I have not tried live from Git. That would use the same repo after checkout. That would likely speed up the build considerably. But portage itself is a slug in general. I suspect there are other delays that will come from it. Even once the repeat unpacking issue is overcome. Unpacking the same archive for each module is highly inefficient. I reduced it to just what was needed. But still takes time to extract that. Just reduces disk I/O as its not all unpacked. Here I extract to disk just the sub project/module being built. https://github.com/Obsidian-StudiosInc/os-xtoo/blob/master/eclass/java-netbeans.eclass#L89 That was about as efficient as I can make things using a tarball. Git likely is a bit faster, but I haven't tried that. > In theory, you *should* be able to get portage to parallelize building > multiple modules for free, since it's managing the build order based > on the dependency graph. With a shared checkout, you'd have to be > very careful to ensure two concurrent builds don't ever alter the > same files at the same time, though. Without it, yeah, performance > is going to be awful. Yes the issue there is portage bad coding. You cannot say I want 2 parallel builds or any exact number. It some crazy voodoo You pass a --load-avg, and a --jobs. Which it uses the load average to spawn jobs up to the point of the load average threshold. When I want say just 2, sometimes I have 10... Its insane.... I can never get an exact amount of parallel builds. Such that it spawns so many it makes it slower than if just using one.... Portage python coding is likely some of the worst in existence. Its horrible slow and does things in very strange ways... > > > IMHO faster hardware is not always the answer to a speed solution. > > Code reduction and efficiency etc could be more of a factor. > > Combine that with faster hardware. You have a real winner :) > > > > You're not thinking closely about what building software is really > *doing*. By a huge margin, the majority of the work is reading and > writing files - that's literally what a compiler does. > Computationally, what compilers do is fairly cheap (though it can be > memory intensive). > > Test it yourself - efficiency is great, but speeding up I/O or > minimizing it is going to swamp microoptimizations by orders of > magnitude. If you want to optimize a build, start by looking for I/O > that is done multiple times identically. Sorry I was referring more to RAD. To many focus on writing code quickly, without regard to the after effects. How much time it will take others to run the code. The system resources etc. Companies and others like to save developers time, but then end up costing the rest of the world. Like say coding in Python vs C or Java. I rarely if ever see enterprise stuff in Python. Most enterprise has some Java in some form. Look at Buck and Bazel, both mostly Java. Likely for speed and other purposes. Python may have been easier to code in, or faster. But then it may make the end result slower. This is the portage issue. Easier to develop, saved developers time, but then costs everyone else time in running it.... > > > Take portage, systems have gotten faster, but Gentoos portage slower > > IMHO due to bad coding and maybe use of Python vs C or other. > > Interesting that others stuff like Bazel and Buck also use mostly > > Java and some python. Rather than all python. > > > > The main thing I've noticed slowing down Portage is the size of the > things commonly in use, and the build systems those things use. Its dependency calculation is horrendously slow. Gradle, Maven, and other things can do similar calculations in fractions of the time. > Chromium is a monster; electron builds most of chromium as well. > Seems like the mainstreaming of things that embed v8 is the biggest > culprit; and build/dependency-management systems like Maven, Gradle, > Cargo, facilitate people writing things that have vast dependency > trees. Great for reuse, sucks if you build it from source. The build time is not so much of my concern. Its the time to get the build started. I lose WAY to much time waiting for portage to do something. I tend to use --oneshot and --nodeps to speed it along. Even then its not super quick. Why I cannot recommend it to others. Portage is super slow. > > For that matter, there is probably a lot of low hanging fruit in > > > improving performance of the existing Ant build - I don't know > > > that much effort has ever been put into that. > > > > Netbeans build is not that slow IMHO. Though could be faster. I > > think the main reason to move past Ant is its mostly yesterdays > > technology. IMHO Ant was Java's 1st generation build system. Maven > > 2nd generation, and Gradle is 3rd and current. > > > > I'd say Gradle is more like 1.5 :-) Likely it has like 1 year on Maven, 03 vs 04 :) > > > Given the rise of Meson, that may pave the way for more Gradle > > usage. The one thing Gradle has almost nothing else does is gradle > > wrapper, gradlew. The ability to install and run itself from a > > minimal script. That alone is super useful for an environment that > > is not ready for a project build. Get your JDK, run gradlew, and > > your off to the races. > > Or: Gradle devs can't be bothered with keeping compatibility with > themselves, and gradlew is the hack to work around that :-) I haven't experienced that, but I am aware. Others slotted gradle on Gentoo for this purpose. But that could also be them not keeping up. Not sure their exact issues but does seem like things changed and broke stuff for others. I hate that and ran into it a few times with meson. Having your build system break for some reason does not help with application development that is for sure!!! > > > But the work to switch Netbeans to Gradle would be considerable. > > Hard to say if thats worth it. Other than moving things forward vs > > remaining the same. But it is some what moot. > > > > I'd say keep looking. I haven't really switched over any of my ant stuff. The reward has to be worth the effort, and not sure its there or justified for many things. Tomcat is also still ant as an example. > Your ebuild stuff sounds neat, though. It's probably too esoteric a > toolchain to consider as a replacement For sure, at least for now. Not sure i would recommend it for the project to replace its build system. Its mostly just a theoretical experiment. But maybe it leads to other changes in other stuff. > - you'd like it, I'd like it and the rest of the world would scream > "why do I have to install the bones of another operating system under > my real one just to build this thing?" Yes for sure. I do not see people being eager to install portage in preffix just to bulid netbeans. Kind of my same gripe with Meson. You need python installed, and also pip, so you can install Meson.... Guess its the same with Java and Ant, Gradle, Maven, etc. But people are ok with installing a language. Portage is not that and would also need python... Why I mostly keep to myself. I do not see the larger world having a need or interest in such. Then again things can change in time. If the next generation portage comes with an installer and preffix ready to go. Such that you install that, it installs python or what ever, and all you do is unpack portage and add 3rd party repos. Then maybe closer to the other stuff. For now just some mad scientist in a lab doing crazy stuff :) -- William L. Thomson Jr.
pgpnsZVQwrW2d.pgp
Description: OpenPGP digital signature
