Re: Modular build
Thanks Dan, your help and guidance will definitely be appreciated! Cheers, Peter. On 1/02/2018 11:52 AM, Dan Rollo wrote: Hi Peter, I’ll try to pitch in where I can. Dan From: Peter<j...@zeus.net.au> Subject: Modular build Date: January 30, 2018 at 5:11:08 AM EST To: "<dev@river.apache.org>"<dev@river.apache.org> Hello fellow River folks! How do we want to start the modular build, should we create a new svn branch and replace trunk when it's complete? I'm just initially looking at creating the build structure and copying across the relevant packages. This will be based on the JGDMS build structure, but with River code from trunk as it is now. Apart from modularity, the JGDMS build also checks for known security bugs and uses findbugs to perform static analysis. Once we've got a modular build working, we can compare the JGDMS code, module by module. Anyone have any spare cycles to help? Regards, Peter.
Re: Modular build
Hi Peter, I’ll try to pitch in where I can. Dan From: Peter <j...@zeus.net.au> Subject: Modular build Date: January 30, 2018 at 5:11:08 AM EST To: "<dev@river.apache.org>" <dev@river.apache.org> Hello fellow River folks! How do we want to start the modular build, should we create a new svn branch and replace trunk when it's complete? I'm just initially looking at creating the build structure and copying across the relevant packages. This will be based on the JGDMS build structure, but with River code from trunk as it is now. Apart from modularity, the JGDMS build also checks for known security bugs and uses findbugs to perform static analysis. Once we've got a modular build working, we can compare the JGDMS code, module by module. Anyone have any spare cycles to help? Regards, Peter.
Modular build
Hello fellow River folks! How do we want to start the modular build, should we create a new svn branch and replace trunk when it's complete? I'm just initially looking at creating the build structure and copying across the relevant packages. This will be based on the JGDMS build structure, but with River code from trunk as it is now. Apart from modularity, the JGDMS build also checks for known security bugs and uses findbugs to perform static analysis. Once we've got a modular build working, we can compare the JGDMS code, module by module. Anyone have any spare cycles to help? Regards, Peter.
Re: Decision process for a Modular build tool
On Apr 23, 2014, at 8:01 AM, Peter j...@zeus.net.au wrote: - Original message - On Apr 22, 2014, at 7:47 PM, Peter Firmstone peter.firmst...@zeus.net.au wrote: From: Peter Firmstone peter.firmst...@zeus.net.au Subject: Decision process for a Modular build tool Date: April 22, 2014 at 7:40:29 PM EDT To: d...@apache.river.org Reply-To: Peter Firmstone peter.firmst...@zeus.net.au I started qa-refactor with the intent of fixing latent bugs, an unintentional benefit is significantly reduced processing times, contention and increased scalability. Changes in timing exposed more bugs. Up until recently an occassional build failure would be experienced due to classdep only partially writing a dep file, resulting in ClassNotFoundException during testing. Knowing that RFC3986URLClassLoader is much faster resolving classes than URLClassLoader, I thought, I'd try using it in ClassDep. Guess what the result was? That's right lot's more ClassNotFoundException's That seems kind of odd. Since ClassDep is single-threaded (it’s basically a command line utility after all), how would faster class path resolution have any impact on the output file? Ok, fair call, ClassDep has a bug, I'm not sure of the exact cause. I would suggest just adding a call to System.out.flush() at the end of main() and even System.out.close() just because you may be using a broken library that is not successfully flushing and closing. Look at the size of the files that are output. Are they multiples of some power of 2 that would be like a block write size? That would indicate that blocks are being written as a blocks worth of output is created. Gregg Wonderly
Re: Decision process for a Modular build tool
Thanks for the tip. Cheers, Peter. - Original message - On Apr 23, 2014, at 8:01 AM, Peter j...@zeus.net.au wrote: - Original message - On Apr 22, 2014, at 7:47 PM, Peter Firmstone peter.firmst...@zeus.net.au wrote: From: Peter Firmstone peter.firmst...@zeus.net.au Subject: Decision process for a Modular build tool Date: April 22, 2014 at 7:40:29 PM EDT To: d...@apache.river.org Reply-To: Peter Firmstone peter.firmst...@zeus.net.au I started qa-refactor with the intent of fixing latent bugs, an unintentional benefit is significantly reduced processing times, contention and increased scalability. Changes in timing exposed more bugs. Up until recently an occassional build failure would be experienced due to classdep only partially writing a dep file, resulting in ClassNotFoundException during testing. Knowing that RFC3986URLClassLoader is much faster resolving classes than URLClassLoader, I thought, I'd try using it in ClassDep. Guess what the result was? That's right lot's more ClassNotFoundException's That seems kind of odd. Since ClassDep is single-threaded (it’s basically a command line utility after all), how would faster class path resolution have any impact on the output file? Ok, fair call, ClassDep has a bug, I'm not sure of the exact cause. I would suggest just adding a call to System.out.flush() at the end of main() and even System.out.close() just because you may be using a broken library that is not successfully flushing and closing. Look at the size of the files that are output. Are they multiples of some power of 2 that would be like a block write size? That would indicate that blocks are being written as a blocks worth of output is created. Gregg Wonderly
Re: Decision process for a Modular build tool
On Apr 22, 2014, at 7:47 PM, Peter Firmstone peter.firmst...@zeus.net.au wrote: From: Peter Firmstone peter.firmst...@zeus.net.au Subject: Decision process for a Modular build tool Date: April 22, 2014 at 7:40:29 PM EDT To: d...@apache.river.org Reply-To: Peter Firmstone peter.firmst...@zeus.net.au I started qa-refactor with the intent of fixing latent bugs, an unintentional benefit is significantly reduced processing times, contention and increased scalability. Changes in timing exposed more bugs. Up until recently an occassional build failure would be experienced due to classdep only partially writing a dep file, resulting in ClassNotFoundException during testing. Knowing that RFC3986URLClassLoader is much faster resolving classes than URLClassLoader, I thought, I'd try using it in ClassDep. Guess what the result was? That's right lot's more ClassNotFoundException's That seems kind of odd. Since ClassDep is single-threaded (it’s basically a command line utility after all), how would faster class path resolution have any impact on the output file? What exactly do you mean by “only partially writing a dep file”? Does it not get closed off? Are characters omitted? Do you have examples of the incomplete “del” files? The once or twice I’ve looked at it in the past, the classes in question were actually in the jar files on the class path, which wouldn’t seem to point to ClassDep. Is it possible that there’s a problem with the new ClassLoader (RFC3986URLClassLoader)? Qa-refactor is stable, but only when the build process succeeds without error. I now must choose, do I waste time fixing a legacy build tool, or is it time for a modular build to reduce long term maintenance and simplify, lowering the bar for new users? The question of modular builds is orthogonal to problems with either ClassDep or the new ClassLoader you’re talking about. It makes the most sense to get a solid diagnosis (and probably fix) for the problem as it has presented itself, rather than diving into a new source code layout. Not to mention that there may actually be users of ClassDepAndJar, apart from the River build. Cheers, Greg Trasuk. To me, going modular is a no brainer, but what tool should we use and why? * Maven * Gradle * Ivy ant Please keep responses to the point, with any pro's and cons you have, eg: Maven: +widely adopted +well structured dependency relationships -dependency downloads. Gradle: -adoption +Groovy build scripts Ivy: +uses ant, our existing build tool -adoption When complete, I suggest, we all can rank the relevance or importance of each point raised (on a scale of -10 to 10) then add up for the final score to decide the tool. Regards, Peter.
Re: Decision process for a Modular build tool
On Wed, Apr 23, 2014 at 1:47 AM, Peter Firmstone peter.firmst...@zeus.net.au wrote: To me, going modular is a no brainer, but what tool should we use and why? * Maven * Gradle * Ivy ant Please keep responses to the point, with any pro's and cons you have, eg: While deciding on a possible build system is certainly the business of who participate in development, I would add some thoughts to gradle: - how to judge adoption - gradle dependency handling is outstanding - transiting from ant can first be done by using what we already have, ant is a first class citizen - gradle folks are certainly willing to help in switching a project of this caliber - for more thoughts please check this out: http://www.gradleware.com/resources/tech/java/maven - mainstream IDEs support is not going to be a shortcoming: http://www.gradle.org/tooling (I know only IDEA where it works nice) Zsolt
Re: Decision process for a Modular build tool
- Original message - On Apr 22, 2014, at 7:47 PM, Peter Firmstone peter.firmst...@zeus.net.au wrote: From: Peter Firmstone peter.firmst...@zeus.net.au Subject: Decision process for a Modular build tool Date: April 22, 2014 at 7:40:29 PM EDT To: d...@apache.river.org Reply-To: Peter Firmstone peter.firmst...@zeus.net.au I started qa-refactor with the intent of fixing latent bugs, an unintentional benefit is significantly reduced processing times, contention and increased scalability. Changes in timing exposed more bugs. Up until recently an occassional build failure would be experienced due to classdep only partially writing a dep file, resulting in ClassNotFoundException during testing. Knowing that RFC3986URLClassLoader is much faster resolving classes than URLClassLoader, I thought, I'd try using it in ClassDep. Guess what the result was? That's right lot's more ClassNotFoundException's That seems kind of odd. Since ClassDep is single-threaded (it’s basically a command line utility after all), how would faster class path resolution have any impact on the output file? Ok, fair call, ClassDep has a bug, I'm not sure of the exact cause. What exactly do you mean by “only partially writing a dep file”? Does it not get closed off? Are characters omitted? Yes, the dep list is incomplete and appears to be cut short. Do you have examples of the incomplete “del” files? The once or twice I’ve looked at it in the past, the classes in question were actually in the jar files on the class path, which wouldn’t seem to point to ClassDep. Have a look at the latest jdk7 build, while it still exists, the jar files for reggie's smart proxy are missing class files, reggie-dl.dep has been truncated and is incomplete. When I have previously thought a class was present on the classpath, I was looking in the wrong jar file. The class files were compiled, just not included in the right jar. Is it possible that there’s a problem with the new ClassLoader (RFC3986URLClassLoader)? The bug is still present with URLClassLoader, but occurs less often. The good news is failures are more prevalent on Jenkins, I seldom see it elsewhere. Qa-refactor is stable, but only when the build process succeeds without error. I now must choose, do I waste time fixing a legacy build tool, or is it time for a modular build to reduce long term maintenance and simplify, lowering the bar for new users? The question of modular builds is orthogonal to problems with either ClassDep or the new ClassLoader you’re talking about. It makes the most sense to get a solid diagnosis (and probably fix) for the problem as it has presented itself, rather than diving into a new source code layout. Not to mention that there may actually be users of ClassDepAndJar, apart from the River build. ClassDep presents a significant maintenance burden for the project, OSGi offers a similar tool for dependency analysis. I upgraded ClassDep to dependency analyse Java 5 language features, that bought time and allowed people to continue using it. I don't have time to maintain it and enable support of Java 8 language features. The use of other tools for River's build process won't prevent others using or maintaining ClassDep or its continued use in the 2.2 maintenance branch. ClassDep is blocking reliable test runs and progress on the qa-refactor branch. If you feel strongly about class dep, add it to the list with pro's and cons and we can score it against modular build options. Cheers, Greg Trasuk. To me, going modular is a no brainer, but what tool should we use and why? * Maven * Gradle * Ivy ant Please keep responses to the point, with any pro's and cons you have, eg: Maven: +widely adopted +well structured dependency relationships -dependency downloads. Gradle: -adoption +Groovy build scripts Ivy: +uses ant, our existing build tool -adoption When complete, I suggest, we all can rank the relevance or importance of each point raised (on a scale of -10 to 10) then add up for the final score to decide the tool. Regards, Peter.
Re: Modular Build
If there's something else you want to learn, then I don't see why not picking up gradle now would be a problem. The problem I've outlined is equally applicable to Ant, Maven or any other build tool you could name. On 19 May 2011 23:31, Patricia Shanahan p...@acm.org wrote: On 5/19/2011 12:29 PM, Tom Hobbs wrote: ... Thoughts anyone? ... I have a question. Should I start studying gradle at this time? There are arguments both ways: Pro: After learning about it, I might be able to contribute to discussions of how to set up to use it. Con: I have other stuff I want to study, and if I remain gradle-ignorant I might be able to better test build instructions. Patricia
Re: Modular Build
On 19-05-11 21:29, Tom Hobbs wrote: So I got to thinking about a way around that. My first reaction was to create a special gradle build would use all the source directories in one go and then use those same source directories as a template for JAR-ing the class files. (Example at end). But that feels more like a fudge than a fix. To prototype the build, a favorable option though. My proposal is to take the current source tree, and as a first step in the build copy the source files to a new directory structure, and build with gradle from there. If it works to our satisfaction we can then rearrange the current source tree into the prefered tree layout for the gradle build. Gr. Sim -- QCG, Software voor het MKB, 071-5890970, http://www.qcg.nl Quality Consultancy Group b.v., Leiderdorp, Kvk Den Haag: 28088397
Re: Modular Build
I think that's largely what Dennis has done. The problem has come from the dependencies between source files and how he and Peter have considered how they should be spit up. Maybe that split is wrong, I don't have the whole context or something else. I don't know. It's good to hear from you again, Sim! On 20 May 2011 10:44, Sim IJskes - QCG s...@qcg.nl wrote: On 19-05-11 21:29, Tom Hobbs wrote: So I got to thinking about a way around that. My first reac... To prototype the build, a favorable option though. My proposal is to take the current source tree, and as a first step in the build copy the source files to a new directory structure, and build with gradle from there. If it works to our satisfaction we can then rearrange the current source tree into the prefered tree layout for the gradle build. Gr. Sim -- QCG, Software voor het MKB, 071-5890970, http://www.qcg.nl Quality Consultancy Group b.v., Leiderdorp, Kvk Den Haag: 28088397
Re: Modular Build
On 20-05-11 11:51, Tom Hobbs wrote: I think that's largely what Dennis has done. The problem has come from the dependencies between source files and how he and Peter have considered how they should be spit up. Maybe that split is wrong, I don't have the whole context or something else. I don't know. If the gradle build can coexist next to the current build, why not move it to the trunk? We can tune the module splits by modifying the copy process, and if we svn:ignore the the 'new' source tree, and make sure it doesn't get checked in we can experiment until everybody is happy. And if we make the copy process based on a macro copy src='src/blah' dest='gradle/mod1/blah' /, we can later change that macro to modify the subversion source tree, when we definitively step over. But i'm repeating myself now, and last time i proposed this it didn't make it. Gr. Sim -- QCG, Software voor het MKB, 071-5890970, http://www.qcg.nl Quality Consultancy Group b.v., Leiderdorp, Kvk Den Haag: 28088397
Re: Modular Build
On 5/19/2011 12:29 PM, Tom Hobbs wrote: ... Thoughts anyone? ... I have a question. Should I start studying gradle at this time? There are arguments both ways: Pro: After learning about it, I might be able to contribute to discussions of how to set up to use it. Con: I have other stuff I want to study, and if I remain gradle-ignorant I might be able to better test build instructions. Patricia
Re: River-300 | Modular Build
On Apr 10, 2011, at 444PM, Tom Hobbs wrote: Sorry, I've made no attempt to debug this *at all*, but if I get a couple It works on my machines I'll assume the problem is my end. Given that we want to make the download and build process as easy as possible, I've just checked out all of skunk/modules. Installed Gradle 1.0 Milestone 2 and ran the commands: $ cd ~/workspace/River_River300 $ gradle build As per instructions here https://issues.apache.org/jira/browse/RIVER-300. Obviously, River_River300 is the root of where I checked out the branch. I then get a bunch of happy messages, but then some compiler errors, starting with river-platform/src/main/java/net/jini/core/discovery/LookupLocator.java:38: complaining that it can't find net.jini.discovery.LookupLocatorDiscovery. I can debug these errors as well as the next fella, I'm just wondering, are those commands supposed to work straight out of the box or did I miss a step? Should work right out of the box (just downloaded the apache-river-gradle.zip and ran okay for me). Did you download apache-river-gradle.zip referenced on the issue? The compiler error you do see is because LookupLocator has a reference to LookupLocatorDiscovery (@see LookupLocatorDiscovery), and LookupLocatorDiscovery is not in the platform (jsk-platform.jar). We either move net.jini.discovery.LookupLocatorDiscovery into the platform or remove the javadoc reference. I havent worked on this issue since January, I planned on getting back into it in a few weeks. Dennis
River-300 Modular Build - Preferred Classes.
Preferred classes are used for the proxy's currently, the jsk-dl.jar in our current build is included in proxy codebase annotations, this is to ensure that com.sun.* namespace implementation classes are loaded by the proxy ClassLoader. In the modular build, we have a dependency relationship where river-lib.jar depends on river-dl.jar to avoid duplication of source files. This means that river-lib.jar and river-dl.jar (it has been suggested to rename it to river-api.jar), will be on the classpath. However river-dl.jar must continue to be part of the codebase annotation for proxy's to ensure that proxy's load the com.sun.* namespace (except for classes which are not preferred), into the proxy classloader to prevent codebase annotation loss and to allow those implementation classes to change, such that the proxy's don't have to share the same implementation version as the platform. Non preferred classes however are loaded from the classpath if available, or loaded into the proxy classloader if they are unavailable on the classpath. This means we are free to re-implement preferred classes in river-lib.jar and river-dl.jar over time, without worrying much about backward compatibility, since different versions can be selected by service implementers. There are classes that are currently not preferred (which will be loaded by the application classloader if available) we should investigate moving these to the platform.jar or another aptly named jar, then we won't have to list them as not preferred and simply list the preferred namespaces in our proxy jar files. Regards, Peter.
