+1 Greg Trasuk
On Feb 10, 2014, at 2:50 PM, Greg Trasuk <tras...@stratuscom.com> wrote: > > As discussion has settled somewhat, I would like to call another vote to > accept the latest patch described in > https://issues.apache.org/jira/browse/RIVER-432 > > The patch removes the archived jar files for Velocity and ASM and replaces > them with Apache Ivy scripts that download the jars from Maven Central the > first time a build is done. From then on, the jar files are in Ivy’s > repository (for more info, see http://ant.apache.org/ivy). > > Voting will remain open at least until 2000 UTC Feb 13, 2014. > > Cheers, > > Greg. > > > On Jan 3, 2014, at 12:57 PM, Greg Trasuk <tras...@stratuscom.com> wrote: > >> >> On Jan 3, 2014, at 5:25 AM, Simon IJskes - QCG <si...@qcg.nl> wrote: >> >>> In order to gain some time to discuss this first i will vote -1. >>> >>> First, we decided to NOT remove velocity builder. >> >> When I read the email chain, my impression was that we wanted to remove it >> (to quote you Sim, “To be honest, I hate it”), but there was a dependency on >> it in the ‘extras’ folder that was added in the trunk branch. As there is >> no ‘extras’ in the 2.2 branch, and that is what this patch applies to, I >> thought it was fair to remove VelocityConfigurationBuilder from the 2.2 >> branch. Perhaps we should revisit the ConfigurationBuilder approach in >> another thread. For now I’ll spin another patch that doesn’t remove >> VelocityConfigurationBuilder. >> >>> >>> Second, no need to remove the jars as specified in your own comments on >>> RIVER-432. >>> >>> Pulling in external jars at compile time, shall we start here? >>> >>> They are already in the svn. They are already in the build scripts. What >>> does this patch fix? No legal problems? >>> >> >> Apache policy is somewhat unclear on this point. One needs to examine the >> mailing lists for clues on what we should really do. I will argue that: >> >> 1 - The fundamental distribution model of Apache is source code, not >> binaries. >> 2 - Distributing binaries is tolerated but not encouraged. Since the svn >> repository can be seen as a distribution point, binaries in svn are also >> tolerated but not encouraged. >> 3 - Downloading dependency binaries at build time is technologically easy, >> provides the same guarantees as putting them in cvs, and avoids the question >> of effectively distributing someone else’s code. >> >> All these together suggest that although we’re technically OK to put >> dependency jars in a “-deps” package (note that the status quo _is_ >> unacceptable - at the very least, we need to restructure the dependencies >> into a “-deps” binary package), there is some policy uncertainty which we >> avoid totally by having dependencies downloaded from a known-good source at >> build time. >> >> Let’s examine these points: >> >> 1 - The fundamental distribution model of Apache is source code, not >> binaries. Apache began with httpd. Back in those days, “Open Source” >> software was distributed in source form only, simply because it was mostly >> intended for Unix systems (then later Linux). I recall the first release of >> Perl coming down as a ten-part uunet news message. Part of this >> distribution model was practical necessity - System differences made it >> necessary to compile your software on the hardware it was going to run on. >> Part of it was open-source philosophy. Having the source code meant that >> you could take a look at it and verify that it wasn’t hazardous to your >> operations. >> >> In any case, the way we use to use open source software was (“./configure; >> make; make install”). If the software had dependencies, you built them from >> source, for the same reasons. >> >> Now, as time has gone on, we’ve gotten used to having the JVM as a common >> runtime, and jar files as a common binary distribution medium. But the >> Apache Foundation’s mandate is to produce open source software that is >> freely usable under the Apache License. That means source code is at the >> heart of Apache, despite the rest of the world’s comfort with binaries. >> Hence Roy’s statements in (1): >> >>> Class files are not open source. Jar files filled with class files >>> are not open source. The fact that they are derived from open source >>> is applicable only to what we allow projects to be dependent upon, >>> not what we vote on as a release package. Release votes are on verified >>> open source artifacts. Binary packages are separate from source packages. >>> One cannot vote to approve a release containing a mix of source and >>> binary code because the binary is not open source and cannot be verified >>> to be safe for release (even if it was derived from open source). >>> >>> I thought that was frigging obvious. Why do I need to write documentation >>> to explain something that is fundamental to the open source definition? >> He’s talking about binary packages, not jar files in svn, but I read that >> (and many other emails) as a distaste for binary distributions. >> >> In fact, if you look at Apache httpd’s download page, it doesn’t appear that >> the Apache project publishes any Unix or Linux binaries. They leave that to >> other organizations. >> >> 2 - Distributing binaries is tolerated but not encouraged. Since the svn >> repository can be seen as a distribution point, binaries in svn are also >> tolerated but not encouraged. >> >> It’s hard to find a single reference that encapsulates this outlook, but >> that’s the impression I get from reading the various mailing lists. For >> instance, Sam Ruby says (2): >>> IMO, our projects release source. So, our projects should not maintain >>> object/binary artifacts >>> in their svn release tree, regardless of license (category a or b). >> There is some debate on whether the svn tree should be considered a >> distribution point. Incubator releases are regularly called out for not >> having “NOTICE” and “RELEASE” files at all reasonable checkout points in >> svn. [LEGAL-26] (https://issues.apache.org/jira/browse/LEGAL-26) concerns >> this and remains unresolved. >> >> Doug Cutting (3) says: >>> On Mon, Sep 16, 2013 at 2:50 AM, Stephen Connolly >>> <stephen.alan.conno...@gmail.com> wrote: >>>> * Source control is not an Apache distribution and hence we do not need to >>>> have LICENSE and NOTICE files in source control, it can be a nice >>>> convenience, but there is no *requirement*. >>> >>> I think perhaps you're looking for clear lines where things are >>> actually a bit fuzzy. Certainly releases are official distributions >>> and need LICENSE and NOTICE files. That line is clear. On the other >>> hand, we try to discourage folks from thinking that source control is >>> a distribution. Rather we wish it to be considered our shared >>> workspace, containing works in progress, not yet always ready for >>> distribution to folks outside the foundation. But, since we work in >>> public, folks from outside the foundation can see our shared workspace >>> and might occasionally mistake it for an official distribution. We'd >>> like them to still see a LICENSE and NOTICE file. So it's not a >>> hard-and-fast requirement that every tree that can possibly be checked >>> out have a LICENSE and NOTICE file at its root, but it's a good >>> practice for those trees that are likely to be checked out have them, >>> so that folks who might consume them are well informed. >> Again, he’s not talking directly about jar files in svn, however I think his >> statement that “since we work in public, folks from outside the foundation >> can see our shared workspace and might occasionally mistake it for an >> official distribution” applies here. Fundamentally, the decision on how and >> where to distribute ‘velocity.jar’ rightly belongs with the Velocity group >> and I don’t think we ought to redistribute it. >> >> 3 - Downloading dependency binaries at build time is technologically easy, >> provides the same guarantees as putting them in cvs, and avoids the question >> of effectively distributing someone else’s code. >> >> There doesn’t seem to be clear policy in the ASF on this, as evidenced by >> the frequent debates on it, and the lack of documentation. I’ve tried to >> lay out an argument that having jars in svn is not encouraged by the ASF >> (really, it’s not in line with the ASF’s charter), even if it isn’t >> disallowed. You may disagree, and I won’t claim I’ve made a strong >> argument, simply because the policy isn’t clear. So instead of going >> through arguments that amount to differences of opinion on Apache policy, >> let’s use a technological solution that is simple, common, and avoids the >> question altogether, by automatically downloading the dependencies at build >> time. >> >> Projects that use Maven do this automatic download as standard practice >> (that’s what Maven does, and that’s what the Maven Central infrastructure is >> there to support). We don’t use Maven, which is fine (our customers have >> asked us to make our binaries available in Maven Central, and we’ve done >> that). Apache Ivy is a popular add-on to Apache Ant that provides similar >> dependency resolution to an Ant-based build. >> >> I was a little surprised how easy it was to persuade Ivy to get the required >> dependencies at build time. The “ivy.xml” file is 39 lines including the >> ASL header (which by the way I forgot to include in the patch - I’ll fix >> that). There are about 50 lines added to ‘build.xml’ to download Ivy and >> then download the required jar files >> >> So, given that the status-quo seems to be unacceptable (Roy talks about not >> having jar files in the open-source trees, only in “-deps” and “tools” >> trees), we have two options: >> >> (a) - restructure the svn repository and the build to allow a separate >> “-deps” distribution. This wouldn’t affect our binary distributions (note >> that I’m no longer using the term “binary release”), but to build from >> source, a user would have to download a separate archive, unpack it, and >> then copy those files into the directory that was unpacked from the source >> release. This option effectively still has us distributing dependent >> binaries, which is not the goal of the ASF, just with a disclaimer that says >> “this isn’t an ASF release, its just a binary distribution put together by a >> committer for your convenience, so don’t feel you should trust it too much”. >> >> (b) - use Ivy to get the jars from Maven Central automatically as part of >> the build. >> >> I think (b) is the option that causes the least hassle for our downstream >> consumers, and not much hassle for us. >> >> >>> Pulling external jars at compile time also makes it more difficult to >>> certify the software. In order to certify the software you need to >>> establish baseline that will be garanteed the same, even if you pull it >>> from the archive 10 years later. >> >> As I said above, Apache’s focus is creating software that can be built from >> source, not distributing binaries (note that QCG or another company might >> have a different focus, and is perfectly able to distribute binaries under >> the Apache license). I think a reasonably prudent user would ask “How can I >> trust the ‘velocity.jar’ that’s included in this binary?” And the answer >> would be either “because I built it from source and installed it in my >> corporate repository” (very cautious, but not unheard-of) or “It was >> published by the Velocity group to a trusted repository, Maven Central” >> (more common). >> >> If you look in the “ivy.xml” file you’ll see that the dependencies are >> specified using Maven-style “group-artifact-version” coordinates. Old >> versions are maintained in Maven Central forever. I suppose it’s possible >> that a publisher could convince Maven Central to remove a version for some >> reason (security or licensing problems perhaps), but then, would we want to >> be distributing that version in a “-deps” package? >> >> I agree that it’s not enough to just say “you need to download such-and-such >> jar”, hence the automatic download managed by “Ivy” from Maven Central. >> >>> It is not a high level project that builds on several frameworks. It is a >>> lowlevel system library. The stuff below the stack is minimal. The number >>> of jars we use is limited. Why bother? >>> >> >> In the currently released branches, the dependencies are limited to ASM and >> Velocity. Looking forward to the trunk branch and the qa_refactor branch, >> the number of external dependencies seem to be increasing (IMO I don’t like >> that, because I also view River as a low level system library, but I’m only >> one PMC member). We need to get in front of the problem before we start >> distributing large numbers of dependencies. >> >> This point rolls in with the general question of jar files in version >> control. I was always taught that version control was for source code, and >> putting binaries into version control was a bad idea. In addition, there >> are practical problems - with older systems like cvs, even doing an update >> or commit effectively downloads the binaries, which slows things down if >> there are large binary files. On newer distributed version control systems >> like git or Mercurial, the entire repository, including all versions of >> binary artifacts, comes down with the project checkout. Currently, we have >> one version of relatively few jar files in our repository, so it’s not a >> major issue. But it gets worse as time goes on. So I suggest we work out >> the technology now to avoid the problem. >> >>> Gr. Simon >>> >> >> Thanks for the questions, Sim. I hope you’ll come around to removing your >> ‘-1’. >> >> Cheers, >> >> Greg >> >> Footnotes >> —————— >> >> (1) - Roy Fielding - http://s.apache.org/roy-binary-deps-1 >> (2) - Sam Ruby - http://s.apache.org/r5 >> (3) - Doug Cutting - http://s.apache.org/GNP >> >>> On 02-01-14 18:22, Greg Trasuk wrote: >>>> >>>> Hello all: >>>> >>>> Please have a look at the patch mentioned below and cast a vote on it. >>>> >>>> The main idea is to remove the dependency jar files from the source >>>> distribution. As a side effect of using Ivy, it becomes reasonable to >>>> remove them from the svn archive as well. Also, the Velocity dependency >>>> was there to support the VelocityConfigurationBuilder. We had discussed >>>> removing that component, so rather than move that dependency to Ivy, I’ve >>>> removed VelocityConfigurationBuilder. >>>> >>>> It’s arguable whether the VelocityConfigurationBuider was part of the >>>> official Jini API (I see it as a utility, not API), so I don’t think this >>>> commit actually requires a vote. However, it does seem like a significant >>>> change to the build process that ought to be reviewed. So I propose to >>>> treat this as a “lazy consensus” vote, and will commit the change to the >>>> 2.2 branch if there are no objections in 72 hours (i.e. 1730UTC 20140105). >>>> >>>> At the same time, based on discussions over on >>>> gene...@incubator.apache.org, I’ll withdraw my assertion that we can’t >>>> have jars in svn. Those interested may want to check out the thread at >>>> http://mail-archives.apache.org/mod_mbox/incubator-general/201312.mbox/%3C01B04CC4-95B8-4A39-BC16-04BAA4269B65%40stratuscom.com%3E >>>> >>>> Cheers, >>>> >>>> Greg. >>>> >>>> On Jan 2, 2014, at 12:05 PM, Greg Trasuk (JIRA) <j...@apache.org> wrote: >>>> >>>>> >>>>> [ >>>>> https://issues.apache.org/jira/browse/RIVER-432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel >>>>> ] >>>>> >>>>> Greg Trasuk updated RIVER-432: >>>>> ------------------------------ >>>>> >>>>> Attachment: river-2_2_remove_jars.diff >>>>> >>>>> The attached patch for the 2.2 branch does the following: >>>>> - removes the 'asm' directory and 'tests/lib' directories which currently >>>>> contain the asm library, mockito, and junit jars. >>>>> - Modifies 'build.xml', 'common.xml', and adds 'ivy.xml' so that the >>>>> Apache Ivy ant plugin is downloaded at build time, and then used to >>>>> retrieve the libraries mentioned above from Maven Central. This removes >>>>> the need to have the jar files in svn. >>>>> - Removes (as per discussion >>>>> http://mail-archives.apache.org/mod_mbox/river-dev/201211.mbox/%3C509B99E3.6080800%40qcg.nl%3E) >>>>> the VelocityConfigBuilder, and associated Velocity jars. Note that the >>>>> 'extras' folder is not present in the 2.2 branch, so Sim's last comments >>>>> in the thread do not apply. >>>>> >>>>>> Jar files in svn and src distributions >>>>>> -------------------------------------- >>>>>> >>>>>> Key: RIVER-432 >>>>>> URL: https://issues.apache.org/jira/browse/RIVER-432 >>>>>> Project: River >>>>>> Issue Type: Bug >>>>>> Reporter: Greg Trasuk >>>>>> Attachments: river-2_2_remove_jars.diff >>>>>> >>>>>> >>>>>> Recent traffic on the incubator lists has pointed out that including jar >>>>>> files for dependencies in the subversion repository and the source >>>>>> distributions is against Apache policy. >>>>>> In River, the following libraries appear in the Subversion repository >>>>>> and the source distributions (these are from trunk, a smaller set appear >>>>>> in the 2.2 branch): >>>>>> animal-sniffer >>>>>> asm >>>>>> bouncy-castle >>>>>> dnsjava >>>>>> high-scale-lib >>>>>> rc-libs >>>>>> velocity >>>>>> They all have to go. What are we using them for? As I understand it, >>>>>> we were going to remove the VelocityConfigurationBuilder, so that's not >>>>>> a problem. Some of the others are available from Maven Central, so we >>>>>> can get them at build time using Ivy or another build tool. Which ones >>>>>> are actually required? And where did they come from? >>>>> >>>>> >>>>> >>>>> -- >>>>> This message was sent by Atlassian JIRA >>>>> (v6.1.5#6160) >>>> >>> >>> >>> -- >>> QCG, Software voor het MKB, 071-5890970, http://www.qcg.nl >>> Quality Consultancy Group b.v., Leiderdorp, Kvk Den Haag: 28088397 >> >
