Re: [freenet-dev] Maven revisited
I meant that more as a question than a statement. But if the horse is starting to die I'll avoid beating it. On Sat, 02 Feb 2013 07:23:42 + Ximin Luo infini...@gmx.com wrote: Agreed; maven is better than our current build scripts in every other respect, but we MUST NOT use it until secure downloads are implemented. On 02/02/13 05:12, Travis Wellman wrote: How is maven different than ruby gems? http://venturebeat.com/2013/01/30/rubygems-org-hacked-interrupting-heroku-services-and-putting-millions-of-sites-using-rails-at-risk/ http://traviswellman.com/ ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Maven revisited
It's not implemented on Rails. On Fri, Feb 1, 2013 at 11:12 PM, Travis Wellman tra...@traviswellman.comwrote: How is maven different than ruby gems? http://venturebeat.com/2013/01/30/rubygems-org-hacked-interrupting-heroku-services-and-putting-millions-of-sites-using-rails-at-risk/ -- Travis Wellman tra...@traviswellman.com http://traviswellman.com/ On Thu, 31 Jan 2013 10:36:09 -0600 Ian Clarke i...@freenetproject.org wrote: I recall that the reason for not using Maven is that it doesn't operate over a secure connection, and it leaves us open to the compromise of any of Freenet's dependencies Maven repositories. This is despite the fact that no such compromise as ever occurred on any project that I'm aware of, and since we don't do code audits of Freenet's current dependencies, our current approach doesn't immunize us against it anyway. ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl -- Ian Clarke Personal blog: http://blog.locut.us/ ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Maven revisited
On Thursday 31 Jan 2013 22:22:06 Ian Clarke wrote: On Thu, Jan 31, 2013 at 3:34 PM, Matthew Toseland t...@amphibian.dyndns.org wrote: But most importantly, we need it to be reasonably easy to *develop Freenet anonymously*. This is not a theoretical aspiration. There are anonymous developers today, and some of them are extremely productive at times. They can use a Tor proxy. IMHO we should not force that on them. Tor has a different threat model, and is much easier to block. Whereas developing over Freenet, without using Tor at all, is quite possible right now, or would be if we maintained an official on-freenet git/hg repo (using tools that already exist). To be fair, existing anonymous devs do pull from the main repo via Tor, but IMHO we should not require them to do so. Now that I think about it, it may be possible to host a Maven repository in Freenet… AFAIK it's just straight-up HTTP GETs. No server side scripting? Interesting in principle. I wonder how it would deal with failures and updates. In general HTTP-based services over Freenet (like apt-get) won't work well unless they are rewritten to be more robust, to use Freenet's updating mechanisms, and to use parallel connections (and possibly avoiding some freenet antipatterns such as giant directories). Certainly the uploading side would need to be written. I'm trying to bring us into 2013, Maven is virtually a standard Java tool these days. freenet-ext.jar has to be built, has to be kept up-to-date. It's basically an ugly home-grown dependency management solution. Originally there were no alternatives, but now there are, and there are easy solutions to the problems that you've outlined with it. No, Maven does not help with freenet-ext.jar at all. The end-user does not use Maven. Using Maven's assembly plugin - it's trivially easy to compile your code, together with all dependencies, into a single .jar. This is not a Maven feature, it's an ant feature. But it does not work if one of the jars is a JCA crypto provider. These jars have to be signed, and therefore have to be separate. At an absolute minimum we need to ship TWO jars, and we need to be able to update both of them. However there are good reasons for splitting freenet-ext.jar up in a more fine grained way, and the auto-updater infrastructure is quite capable of dealing with this. For example, reducing the size of the download when updating; using the system jars on packages on certain operating systems; etc etc. There are workarounds, but they involve significant cost in cash, maintenance or performance: We could buy a Java code signing certificate and rebuild and re-sign it ourselves. Which doesn't fit cleanly with Maven, though I'm sure it's possible. Or we could give up on using the standard Java encryption API (JCA) and use the Bouncycastle lightweight API for everything, and have to ship our own native JNI libraries for optimised ECC (native gives a 4x speedup IIRC even on x86, granted it's only used for connection setup and SSKs). We can't rely on Oracle's built in crypto either, as some versions of Java 1.6 don't support modern (ECC) crypto at all. Maven does not help us with deployment. The major problems with deployment have already been solved, apart from updating the wrapper (which again requires a separate jar so that we can update both the jar and the native binaries at the same time). Just because something is new does not mean it is appropriate for us. I have been trying to use modern components where possible, e.g. JCA. Now can we please shut up about Maven? It will not save us any time in the short term, or in the long term, and we have a huge amount of stuff to deal with already! On Thursday 31 Jan 2013 21:34:49 Matthew Toseland wrote: Including the dependency jars in the main freenet jar shipped is possible with or without Maven - except that it isn't for at least one jar, the Bouncycastle crypto provider, which needs to be bundled separately as it is signed. I'm not sure whether we could combine it if we signed the whole file, but even then we'd need a code signing cert for Java. We do need one for Windows, but IIRC you mostly have to pay separately for Java vs for Windows. For linux installs it's good for packages to be able to use the system version of bouncycastle (and other libraries), which is what originally motivated infinity0's work on splitting up freenet-ext.jar. What does make a difference is the changes made to the auto-updater I made last year. These allow us to ship the bouncycastle jar, to update it, and to ship whatever other jars we need, updating them when we need to. We can split up freenet-ext.jar however we want (including using infinity0's branch). But given that freenet-ext.jar changes *very* slowly, I don't see an urgent issue. The most urgent issue related to this area is updating the wrapper, which can
Re: [freenet-dev] Maven revisited
How is maven different than ruby gems? http://venturebeat.com/2013/01/30/rubygems-org-hacked-interrupting-heroku-services-and-putting-millions-of-sites-using-rails-at-risk/ -- Travis Wellman tra...@traviswellman.com http://traviswellman.com/ On Thu, 31 Jan 2013 10:36:09 -0600 Ian Clarke i...@freenetproject.org wrote: I recall that the reason for not using Maven is that it doesn't operate over a secure connection, and it leaves us open to the compromise of any of Freenet's dependencies Maven repositories. This is despite the fact that no such compromise as ever occurred on any project that I'm aware of, and since we don't do code audits of Freenet's current dependencies, our current approach doesn't immunize us against it anyway. ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Maven revisited
Agreed; maven is better than our current build scripts in every other respect, but we MUST NOT use it until secure downloads are implemented. On 02/02/13 05:12, Travis Wellman wrote: How is maven different than ruby gems? http://venturebeat.com/2013/01/30/rubygems-org-hacked-interrupting-heroku-services-and-putting-millions-of-sites-using-rails-at-risk/ -- GPG: 4096R/5FBBDBCE https://github.com/infinity0 https://bitbucket.org/infinity0 https://launchpad.net/~infinity0 signature.asc Description: OpenPGP digital signature ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
[freenet-dev] Maven revisited
I was thinking about the fact that we still build Freenet using the tools that were available to us a decade ago, while the Java world has moved on to more sophisticated dependency management tools like Maven. I recall that the reason for not using Maven is that it doesn't operate over a secure connection, and it leaves us open to the compromise of any of Freenet's dependencies Maven repositories. This is despite the fact that no such compromise as ever occurred on any project that I'm aware of, and since we don't do code audits of Freenet's current dependencies, our current approach doesn't immunize us against it anyway. However, one approach that might alleviate this concern is that we run our own Maven repository which will host any dependencies we need, and then configure Maven not to pull from the central Maven repos. There is the other issue that Maven can be a PITA to use, however there are similar alternatives: http://www.streamhead.com/maven-alternatives/ Thoughts? Ian. -- Ian Clarke Founder, The Freenet Project Email: i...@freenetproject.org ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Maven revisited
On Thu, Jan 31, 2013 at 11:36 AM, Ian Clarke i...@freenetproject.org wrote: I was thinking about the fact that we still build Freenet using the tools that were available to us a decade ago, while the Java world has moved on to more sophisticated dependency management tools like Maven. I recall that the reason for not using Maven is that it doesn't operate over a secure connection, and it leaves us open to the compromise of any of Freenet's dependencies Maven repositories. This is despite the fact that no such compromise as ever occurred on any project that I'm aware of, and since we don't do code audits of Freenet's current dependencies, our current approach doesn't immunize us against it anyway. However, one approach that might alleviate this concern is that we run our own Maven repository which will host any dependencies we need, and then configure Maven not to pull from the central Maven repos. There is the other issue that Maven can be a PITA to use, however there are similar alternatives: http://www.streamhead.com/maven-alternatives/ Thoughts? Maven's really not that bad. If people are absolutely terrified about depedencies being compromised, maybe make a quick script to do a checksum on the dependencies once they're donwloaded. Ian. -- Ian Clarke Founder, The Freenet Project Email: i...@freenetproject.org ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Maven revisited
On Thursday 31 Jan 2013 17:50:32 Michael Grube wrote: On Thu, Jan 31, 2013 at 11:36 AM, Ian Clarke i...@freenetproject.org wrote: I was thinking about the fact that we still build Freenet using the tools that were available to us a decade ago, while the Java world has moved on to more sophisticated dependency management tools like Maven. I recall that the reason for not using Maven is that it doesn't operate over a secure connection, and it leaves us open to the compromise of any of Freenet's dependencies Maven repositories. This is despite the fact that no such compromise as ever occurred on any project that I'm aware of, and since we don't do code audits of Freenet's current dependencies, our current approach doesn't immunize us against it anyway. Have you actually tried to find out? However, one approach that might alleviate this concern is that we run our own Maven repository which will host any dependencies we need, and then configure Maven not to pull from the central Maven repos. There is the other issue that Maven can be a PITA to use, however there are similar alternatives: http://www.streamhead.com/maven-alternatives/ Thoughts? Maven's really not that bad. If people are absolutely terrified about depedencies being compromised, maybe make a quick script to do a checksum on the dependencies once they're donwloaded. Maven does not do any sort of signature checking. Maven's own repository doesn't even do SSL IIRC. It is therefore not suitable for building binaries that will be distributed. In my view this is true of any binaries that will be distributed to anyone, but it certainly isn't true of building binaries for an auto-updater capable of deploying 5,000 nodes within an hour - a significant target for conventional malware even if it wasn't for the fact that some of these people really do need their privacy. If we run our own repository: - We need to maintain it. This is more unnecessary work. - We need to host it. This is more CPU usage on the small, cheap, rather limited VM that runs the website etc. But most importantly, we need it to be reasonably easy to *develop Freenet anonymously*. This is not a theoretical aspiration. There are anonymous developers today, and some of them are extremely productive at times. Exactly what problem are you trying to solve here? It's really not that hard to build Freenet. Granted it should be easier; the immediate problem is you need not only freenet-ext.jar (which the build scripts will fetch for you if you set one line in a config file; the first time you run ant it will tell you this), but also the bouncycastle jar, which isn't auto-fetched. If you really want security advice ask nextgens. But it looks to me like Maven is hopeless for our purposes. For a non-security-related project, for a single developer who doesn't distribute the resulting binaries, fine. For a corporate setup where both the developers and the server are inside the firewall, fine. But for us, it does not make sense. Regarding not auditing dependencies, we do try to obtain clean copies of our dependencies. Also most of them aren't security critical, and so aren't updated regularly. Ordinarily this would be a bad thing - but it does reduce the number of opportunities for malware to slip in. The biggest dependency is db4o, and IMHO we should get rid of it soon, it's been nothing but a nightmare. Whenever we have looked into updating it we have found new and wonderful bugs, and so haven't bothered... In any case, the fact that we haven't audited every line of some of our dependencies is not an excuse for failing to perform basic due dilligence on our build process. Freenet is security sensitive, it has an auto-updater, it's not safe for us to just grab jars from wherever and hope for the best, which seems to be what most of the Java community do. And it's what Maven does too, without any form of authentication. The best person to ask for security advice on this sort of issue is Nextgens anyway. He's been around lately. signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Maven revisited
On Thu, Jan 31, 2013 at 2:31 PM, Michael Grube michael.gr...@gmail.comwrote: On Thu, Jan 31, 2013 at 1:59 PM, Matthew Toseland t...@amphibian.dyndns.org wrote: On Thursday 31 Jan 2013 17:50:32 Michael Grube wrote: On Thu, Jan 31, 2013 at 11:36 AM, Ian Clarke i...@freenetproject.org wrote: I was thinking about the fact that we still build Freenet using the tools that were available to us a decade ago, while the Java world has moved on to more sophisticated dependency management tools like Maven. I recall that the reason for not using Maven is that it doesn't operate over a secure connection, and it leaves us open to the compromise of any of Freenet's dependencies Maven repositories. This is despite the fact that no such compromise as ever occurred on any project that I'm aware of, and since we don't do code audits of Freenet's current dependencies, our current approach doesn't immunize us against it anyway. Have you actually tried to find out? However, one approach that might alleviate this concern is that we run our own Maven repository which will host any dependencies we need, and then configure Maven not to pull from the central Maven repos. There is the other issue that Maven can be a PITA to use, however there are similar alternatives: http://www.streamhead.com/maven-alternatives/ Thoughts? Maven's really not that bad. If people are absolutely terrified about depedencies being compromised, maybe make a quick script to do a checksum on the dependencies once they're donwloaded. Maven does not do any sort of signature checking. Maven's own repository doesn't even do SSL IIRC. http://maven.apache.org/guides/mini/guide-repository-ssl.html It is therefore not suitable for building binaries that will be distributed. In my view this is true of any binaries that will be distributed to anyone, but it certainly isn't true of building binaries for an auto-updater capable of deploying 5,000 nodes within an hour - a significant target for conventional malware even if it wasn't for the fact that some of these people really do need their privacy. If we run our own repository: - We need to maintain it. This is more unnecessary work. - We need to host it. This is more CPU usage on the small, cheap, rather limited VM that runs the website etc. But most importantly, we need it to be reasonably easy to *develop Freenet anonymously*. This is not a theoretical aspiration. There are anonymous developers today, and some of them are extremely productive at times. Some kind of Infocalypse bridge? Exactly what problem are you trying to solve here? It's really not that hard to build Freenet. Granted it should be easier; the immediate problem is you need not only freenet-ext.jar (which the build scripts will fetch for you if you set one line in a config file; the first time you run ant it will tell you this), but also the bouncycastle jar, which isn't auto-fetched. If you really want security advice ask nextgens. But it looks to me like Maven is hopeless for our purposes. For a non-security-related project, for a single developer who doesn't distribute the resulting binaries, fine. For a corporate setup where both the developers and the server are inside the firewall, fine. But for us, it does not make sense. Regarding not auditing dependencies, we do try to obtain clean copies of our dependencies. Also most of them aren't security critical, and so aren't updated regularly. Ordinarily this would be a bad thing - but it does reduce the number of opportunities for malware to slip in. The biggest dependency is db4o, and IMHO we should get rid of it soon, it's been nothing but a nightmare. Whenever we have looked into updating it we have found new and wonderful bugs, and so haven't bothered... In any case, the fact that we haven't audited every line of some of our dependencies is not an excuse for failing to perform basic due dilligence on our build process. Freenet is security sensitive, it has an auto-updater, it's not safe for us to just grab jars from wherever and hope for the best, which seems to be what most of the Java community do. And it's what Maven does too, without any form of authentication. The best person to ask for security advice on this sort of issue is Nextgens anyway. He's been around lately. ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Maven revisited
On Thu, Jan 31, 2013 at 12:59 PM, Matthew Toseland t...@amphibian.dyndns.org wrote: On Thursday 31 Jan 2013 17:50:32 Michael Grube wrote: On Thu, Jan 31, 2013 at 11:36 AM, Ian Clarke i...@freenetproject.org wrote: This is despite the fact that no such compromise as ever occurred on any project that I'm aware of, and since we don't do code audits of Freenet's current dependencies, our current approach doesn't immunize us against it anyway. Have you actually tried to find out? If by try you mean a quick Googlehttps://www.google.com/webhp?sourceid=chrome-instantion=1ie=UTF-8#hl=entbo=dsclient=psy-abq=maven%20repository%20compromiseoq=gs_l=pbx=1fp=eba5ecb19bdd79c3ion=1bav=on.2,or.r_gc.r_pw.r_cp.r_qf.bvm=bv.41642243,d.b2Ibiw=1371bih=983search, then yes. If we run our own repository: - We need to maintain it. This is more unnecessary work. Not a lot, probably less than dealing with the freenet-ext.jar mess. - We need to host it. This is more CPU usage on the small, cheap, rather limited VM that runs the website etc. It won't use significant CPU or bandwidth, only developers will access it, and Maven caches dependencies locally. But most importantly, we need it to be reasonably easy to *develop Freenet anonymously*. This is not a theoretical aspiration. There are anonymous developers today, and some of them are extremely productive at times. They can use a Tor proxy. Exactly what problem are you trying to solve here? It's really not that hard to build Freenet. Granted it should be easier; the immediate problem is you need not only freenet-ext.jar (which the build scripts will fetch for you if you set one line in a config file; the first time you run ant it will tell you this), but also the bouncycastle jar, which isn't auto-fetched. I'm trying to bring us into 2013, Maven is virtually a standard Java tool these days. freenet-ext.jar has to be built, has to be kept up-to-date. It's basically an ugly home-grown dependency management solution. Originally there were no alternatives, but now there are, and there are easy solutions to the problems that you've outlined with it. Ian. -- Ian Clarke Personal blog: http://blog.locut.us/ ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Maven revisited
On Thursday 31 Jan 2013 20:37:43 Ian Clarke wrote: On Thu, Jan 31, 2013 at 12:59 PM, Matthew Toseland t...@amphibian.dyndns.org wrote: On Thursday 31 Jan 2013 17:50:32 Michael Grube wrote: On Thu, Jan 31, 2013 at 11:36 AM, Ian Clarke i...@freenetproject.org wrote: This is despite the fact that no such compromise as ever occurred on any project that I'm aware of, and since we don't do code audits of Freenet's current dependencies, our current approach doesn't immunize us against it anyway. Have you actually tried to find out? If by try you mean a quick Googlehttps://www.google.com/webhp?sourceid=chrome-instantion=1ie=UTF-8#hl=entbo=dsclient=psy-abq=maven%20repository%20compromiseoq=gs_l=pbx=1fp=eba5ecb19bdd79c3ion=1bav=on.2,or.r_gc.r_pw.r_cp.r_qf.bvm=bv.41642243,d.b2Ibiw=1371bih=983search, then yes. They don't have to compromise the repository. All they have to do is spoof it, if we're using HTTP. Although as Michael pointed out it is possible to use HTTPS. If we run our own repository: - We need to maintain it. This is more unnecessary work. Not a lot, probably less than dealing with the freenet-ext.jar mess. See below. - We need to host it. This is more CPU usage on the small, cheap, rather limited VM that runs the website etc. It won't use significant CPU or bandwidth, only developers will access it, and Maven caches dependencies locally. Ok. But most importantly, we need it to be reasonably easy to *develop Freenet anonymously*. This is not a theoretical aspiration. There are anonymous developers today, and some of them are extremely productive at times. They can use a Tor proxy. IMHO we should not force that on them. Tor has a different threat model, and is much easier to block. Whereas developing over Freenet, without using Tor at all, is quite possible right now, or would be if we maintained an official on-freenet git/hg repo (using tools that already exist). To be fair, existing anonymous devs do pull from the main repo via Tor, but IMHO we should not require them to do so. Exactly what problem are you trying to solve here? It's really not that hard to build Freenet. Granted it should be easier; the immediate problem is you need not only freenet-ext.jar (which the build scripts will fetch for you if you set one line in a config file; the first time you run ant it will tell you this), but also the bouncycastle jar, which isn't auto-fetched. I'm trying to bring us into 2013, Maven is virtually a standard Java tool these days. freenet-ext.jar has to be built, has to be kept up-to-date. It's basically an ugly home-grown dependency management solution. Originally there were no alternatives, but now there are, and there are easy solutions to the problems that you've outlined with it. No, Maven does not help with freenet-ext.jar at all. The end-user does not use Maven. Including the dependency jars in the main freenet jar shipped is possible with or without Maven - except that it isn't for at least one jar, the Bouncycastle crypto provider, which needs to be bundled separately as it is signed. I'm not sure whether we could combine it if we signed the whole file, but even then we'd need a code signing cert for Java. We do need one for Windows, but IIRC you mostly have to pay separately for Java vs for Windows. For linux installs it's good for packages to be able to use the system version of bouncycastle (and other libraries), which is what originally motivated infinity0's work on splitting up freenet-ext.jar. What does make a difference is the changes made to the auto-updater I made last year. These allow us to ship the bouncycastle jar, to update it, and to ship whatever other jars we need, updating them when we need to. We can split up freenet-ext.jar however we want (including using infinity0's branch). But given that freenet-ext.jar changes *very* slowly, I don't see an urgent issue. The most urgent issue related to this area is updating the wrapper, which can cause problems on Windows, but which is tricky to update because wrapper.jar is included in freenet-ext.jar, and needs to be compatible with the native binaries. Maven does not help here either. signature.asc Description: This is a digitally signed message part. ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
Re: [freenet-dev] Maven revisited
On Thu, Jan 31, 2013 at 3:34 PM, Matthew Toseland t...@amphibian.dyndns.org wrote: But most importantly, we need it to be reasonably easy to *develop Freenet anonymously*. This is not a theoretical aspiration. There are anonymous developers today, and some of them are extremely productive at times. They can use a Tor proxy. IMHO we should not force that on them. Tor has a different threat model, and is much easier to block. Whereas developing over Freenet, without using Tor at all, is quite possible right now, or would be if we maintained an official on-freenet git/hg repo (using tools that already exist). To be fair, existing anonymous devs do pull from the main repo via Tor, but IMHO we should not require them to do so. Now that I think about it, it may be possible to host a Maven repository in Freenet… AFAIK it's just straight-up HTTP GETs. I'm trying to bring us into 2013, Maven is virtually a standard Java tool these days. freenet-ext.jar has to be built, has to be kept up-to-date. It's basically an ugly home-grown dependency management solution. Originally there were no alternatives, but now there are, and there are easy solutions to the problems that you've outlined with it. No, Maven does not help with freenet-ext.jar at all. The end-user does not use Maven. Using Maven's assembly plugin - it's trivially easy to compile your code, together with all dependencies, into a single .jar. Ian. -- Ian Clarke Personal blog: http://blog.locut.us/ ___ Devl mailing list Devl@freenetproject.org https://emu.freenetproject.org/cgi-bin/mailman/listinfo/devl
