Re: armel boxes for Debian
Joey Hess wrote: If we had an armel buildd that used ccache and had pre-built versions of all the security sensitive packages in its cache, updates for most packages could probably be built in a timeframe that compares with other architectures. Aside from the complexity of setting this up and desire for KISS, is there any reason not to consider doing this? I bet the ccache would be volatile enough that you wouldn't be able to exploit it repeatably. But you could task that maintenance work to the machine itself, so there's no reason not configure it that way. I think the reality is that ARM machines just can't compete with the high-horsepower machines in x86 and PPC worlds. If that makes us "second-class citizens" to the Security team, there's no point in denying it. I like the idea that Security patches come out as quickly as they can, without being gated by the performance of a slow architecture. Compared to x86, ARM isn't a very inviting exploit target so if we're 12 hours behind them, I really don't see the problem. Far better that we tune for consistency configuration-wise with the rest of Debian, methinks, and just accept that our CPUs are slower. Over time, the performance gap may close without us doing anything special, but if we produce a headache-inducing setup in an attempt to improve performance in the near-term, then we have to go through more work to undo that setup later when we get faster chips. I don't like to do work twice! Just my (non-DD) opinion... BTW, I've got my n4100 running armel now, and even with 512MB the performance is ... underwhelming. And by ARM standards, this machine is big-iron! b.g. -- Bill Gatliff [EMAIL PROTECTED] -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: armel boxes for Debian
If we had an armel buildd that used ccache and had pre-built versions of all the security sensitive packages in its cache, updates for most packages could probably be built in a timeframe that compares with other architectures. Aside from the complexity of setting this up and desire for KISS, is there any reason not to consider doing this? -- see shy jo signature.asc Description: Digital signature
Re: armel boxes for Debian
On Wed, Feb 20, 2008 at 07:44:30PM -0500, Joey Hess wrote: > Moritz Muehlenhoff wrote: > > We could just declare arm a second-class architecture for security updates, > > i.e. DSAs being released once all archs are available except arm and arm > > updates being released once available. For small to medium packages most > > updates would still be released in sync, since we're not available to > > release updates 24/7. > > Yeah, that's what I was suggesting. Ok, if that's an agreeable consensus to arm porters, we should add it to the Lenny release notes. > > > I'm also unsure based on Moritz's mail exactly what kind of speed > > > they're looking for from an arm security buildd. He mentioned something > > > on the order of 14 hours to build xulrunner -- would an arm box that > > > builds it in 9 hours[1] be a worthwhile improvement, or will that still > > > leave the security team waiting until the next day for arm to catch up? > > > > 9 instead of 14 would still help. I also think a second security buildd > > would help: It wouldn't address the spikes of giga packages like xulrunner, > > but it would help for cases, where several updates are building in > > parallel. > > The benefits I see from such a speedup are that it would let the arm > advisory arrive 5 hours faster (but still 7 hours after everything > else), and that it would increase the number of packages that wouldn't > need a delayed advisory for arm. Accurate? Yes, that's correct. Cheers, Moritz -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: armel boxes for Debian
Am Donnerstag, 21. Februar 2008 09:18:35 schrieb Florian Weimer: > Based on the numbers, I'd guess that the box hasn't got enough RAM. Are > you sure it's CPU-bound? > > > Based on these xulrunner build times, we'd need an arm box that's 4x as > > fast as the current one to be "competitive": > > > > i3860:42 (amd64 is of course approx. the same) > > alpha 1:02 > > s3901:10 > > sparc 1:59 > > ia641:19 > > hppa2:39 > > powerpc 2:45 > > mips3:09 > > mipsel 3:19 > > arm 13:01 It is definitely CPU-bound. Keep in mind, current arm cpus are neither superscalar nor do they have out-of-order execution. They can not even do loads in parallel to arithmetic instructions, which means a 600MHz Xscale is equivalent to an Athlon64 with 200...400MHz from terms of raw CPU power. If it comes to floating point (which is required for many script languages even if they deal with integer numbers), it is much worse, as we do not have a FPU. But what is probably even more important for compilation, current arm cpus do not have L2 caches, so if data does not fit into the L1 cache (32kBytes in the XScale) they have to access the RAM. Unfortunately, the XScale's RAM interface is crippled by some serious design bugs. We only get a memcopy speed of around 40MB/s and a memory latency of 200ns. An Athlon64 can get a memcopy speed of 600MB/s and a latency of 50ns out of the same memory module, and if you put in two of them you get even a memcopy speed of 1130MB/s (latency does not improve of course). If you take into account these facts, you must admit that the Thecus just does a good fight ;-) Peter -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: armel boxes for Debian
Florian Weimer wrote: > Based on the numbers, I'd guess that the box hasn't got enough RAM. Are > you sure it's CPU-bound? The only place it seemed significantly ram-bound was during linking, which used more than 128 mb of swap. There might be other places where more ram than I had would help a xulrunner build. -- see shy jo signature.asc Description: Digital signature
Re: armel boxes for Debian
Florian Weimer wrote: > * Joey Hess: > > >> Riku Voipio wrote: >> >>> The security buildd is a different story. Parallell buildd's compiling >>> several packages at time don't help[1], they want single builds >>> completed fast, so they can release security advisories with minimal >>> delay. For this reason, Moritz from the security team expressed >>> being unpleased with the speed of a Thecus and want something faster >>> as buildd. >>> > > In the future, we might regularly have more than one package in the > build queue, so more buildds might help. This was not true in the past, > and as Moritz explained, latency matters more than throughput to us. > The parallel builds was an afterthought. My main point was that scratchbox could reduce latency. Jason -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: armel boxes for Debian
On Thu, February 21, 2008 14:28, Riku Voipio wrote: > Sounds reasonable proposal to me. What kind of insight do you need? Any insight is already better than the current situation, where the only thing we can see about the buildd's are build logs that are being mailed to us. Things like the queue or what is currently building, for every arch, would be very useful in my opinion. Of course it would need to be ensured that it doesn't leak relevant information to the world. > IJ's udpate-buildd.net scripts provide some insight (see the bottom of > page for example on of two of the three buildd's we have:) Yes, such a thing would be great. Thijs -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: armel boxes for Debian
On Thu, Feb 21, 2008 at 12:44:40PM +0100, Thijs Kinkhorst wrote: > I agree with both of the following proposals: > * There's a defined set of archs that we will not delay DSA's for, and them >becoming available later will lead to them being installed into the >archive without a -2 followup mail issued. > * There should be more insight into the queues and functioning of the >buildd's by the security team, similar to that available for the regular >archive. > That would resolve the most imporant problems and therefore make it much > easier to decide to include a new slower architecture. Sounds reasonable proposal to me. What kind of insight do you need? IJ's udpate-buildd.net scripts provide some insight (see the bottom of page for example on of two of the three buildd's we have:) http://unstable.buildd.net/index-armel.html It shows that the buildd is up, running and conntected to internet, and what it's building and for how long. It does not tell if the build is stuck or spinning to infinity, but that should be quite rare for packages that managed to get to stable (and thus needing security uploads). -- "rm -rf" only sounds scary if you don't have backups signature.asc Description: Digital signature
Re: armel boxes for Debian
On Thu, February 21, 2008 09:18, Florian Weimer wrote: > And what do you do when you haven't got arm build logs after 14 hours? > Will they arrive eventually? Is the box down? > > > I understand that this is only partly arm's fault, but low buildd speeds > exacerbate the issues. Yes, the main problem with buildd's and DSA's is that the first 90% arrives the same day, but the others either arrive some days later or never, and there's no clear view on which one is going to happen. I agree with both of the following proposals: * There's a defined set of archs that we will not delay DSA's for, and them becoming available later will lead to them being installed into the archive without a -2 followup mail issued. * There should be more insight into the queues and functioning of the buildd's by the security team, similar to that available for the regular archive. That would resolve the most imporant problems and therefore make it much easier to decide to include a new slower architecture. Thijs -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: armel boxes for Debian
* Joey Hess: > Riku Voipio wrote: >> The security buildd is a different story. Parallell buildd's compiling >> several packages at time don't help[1], they want single builds >> completed fast, so they can release security advisories with minimal >> delay. For this reason, Moritz from the security team expressed >> being unpleased with the speed of a Thecus and want something faster >> as buildd. In the future, we might regularly have more than one package in the build queue, so more buildds might help. This was not true in the past, and as Moritz explained, latency matters more than throughput to us. > I wonder if the security team is being entirely reasonable with their > expectations for speed. The team typically wants to get a package built > on all architectures before releasing an advisory, but since we don't > have a rule that supported architectures all need to have hardware > that's even within an order of magnitude of the speed of other > architectures[2], isn't that a fundamentally unreasonable expectation for > the security team to have? It's a practical requirement. Keep in mind that we've got near zero visibility into the buildd network, so the logs are our only clue to what's going on. > I'm also unsure based on Moritz's mail exactly what kind of speed > they're looking for from an arm security buildd. He mentioned something > on the order of 14 hours to build xulrunner -- would an arm box that > builds it in 9 hours[1] be a worthwhile improvement, or will that still > leave the security team waiting until the next day for arm to catch up? Based on the numbers, I'd guess that the box hasn't got enough RAM. Are you sure it's CPU-bound? > Based on these xulrunner build times, we'd need an arm box that's 4x as > fast as the current one to be "competitive": > > i3860:42 (amd64 is of course approx. the same) > alpha 1:02 > s3901:10 > sparc 1:59 > ia641:19 > hppa2:39 > powerpc 2:45 > mips3:09 > mipsel 3:19 > arm 13:01 And what do you do when you haven't got arm build logs after 14 hours? Will they arrive eventually? Is the box down? I understand that this is only partly arm's fault, but low buildd speeds exacerbate the issues. -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: armel boxes for Debian
On Wed, Feb 20, 2008 at 07:44:30PM -0500, Joey Hess wrote: > Moritz Muehlenhoff wrote: > > We could just declare arm a second-class architecture for security updates, > > i.e. DSAs being released once all archs are available except arm and arm > > updates being released once available. For small to medium packages most > > updates would still be released in sync, since we're not available to > > release updates 24/7. > > Yeah, that's what I was suggesting. > > > But we should stop withholding security updates because we wait on slow > > archs. Debian was the first to fix the vmsplice root exploit. If we > > had released all archs in sync we would have left most of our users > > unprotected for two more days. > > And that advisory was initially only released for alpha, i386, amd64, ia64, > and s390 -- so it wasn't only arm being too slow in that case. Perhaps > arm delayed the follow-up advisory though? In this case I released builds as soon as I saw them come available, but waited until they were all released before sending an update to Florian's DSA. sparc took longer than arm to send a build log, which is abnormal in my experience. I actually ended up releasing a manual build under the assumption the sparc buildd was having problems. -- dann frazier -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: armel boxes for Debian
Moritz Muehlenhoff wrote: > We could just declare arm a second-class architecture for security updates, > i.e. DSAs being released once all archs are available except arm and arm > updates being released once available. For small to medium packages most > updates would still be released in sync, since we're not available to > release updates 24/7. Yeah, that's what I was suggesting. > But we should stop withholding security updates because we wait on slow > archs. Debian was the first to fix the vmsplice root exploit. If we > had released all archs in sync we would have left most of our users > unprotected for two more days. And that advisory was initially only released for alpha, i386, amd64, ia64, and s390 -- so it wasn't only arm being too slow in that case. Perhaps arm delayed the follow-up advisory though? > > I'm also unsure based on Moritz's mail exactly what kind of speed > > they're looking for from an arm security buildd. He mentioned something > > on the order of 14 hours to build xulrunner -- would an arm box that > > builds it in 9 hours[1] be a worthwhile improvement, or will that still > > leave the security team waiting until the next day for arm to catch up? > > 9 instead of 14 would still help. I also think a second security buildd > would help: It wouldn't address the spikes of giga packages like xulrunner, > but it would help for cases, where several updates are building in > parallel. The benefits I see from such a speedup are that it would let the arm advisory arrive 5 hours faster (but still 7 hours after everything else), and that it would increase the number of packages that wouldn't need a delayed advisory for arm. Accurate? -- see shy jo signature.asc Description: Digital signature
Re: armel boxes for Debian
Joey Hess wrote: Riku Voipio wrote: The security buildd is a different story. Parallell buildd's compiling several packages at time don't help[1], they want single builds completed fast, so they can release security advisories with minimal delay. For this reason, Moritz from the security team expressed being unpleased with the speed of a Thecus and want something faster as buildd. I wonder if the security team is being entirely reasonable with their expectations for speed. The team typically wants to get a package built on all architectures before releasing an advisory, but since we don't have a rule that supported architectures all need to have hardware that's even within an order of magnitude of the speed of other architectures[2], isn't that a fundamentally unreasonable expectation for the security team to have? Perhaps it would be better for them to develop policies to deal with slower architectures. Some currently fast-enough architectures will likely fall into the too slow category in years to come no matter what happens with arm now. New, faster alpha chips are not being designed, for example. One such policy might be to have a set of architectures that advisories are never delayed for. I'm also unsure based on Moritz's mail exactly what kind of speed they're looking for from an arm security buildd. He mentioned something on the order of 14 hours to build xulrunner -- would an arm box that builds it in 9 hours[1] be a worthwhile improvement, or will that still leave the security team waiting until the next day for arm to catch up? Based on these xulrunner build times, we'd need an arm box that's 4x as fast as the current one to be "competitive": i3860:42 (amd64 is of course approx. the same) alpha 1:02 s3901:10 sparc 1:59 ia641:19 hppa2:39 powerpc 2:45 mips3:09 mipsel 3:19 arm 13:01 m68k46:22 (build failed incomplete) Can scratchbox and cross-compiling fulfill the need for speed and still produce good packages? I read a web article that was talking about scratchbox+qemu making the arm builds for debian be only 2-3 times as slow as x86 vs native compilation. Could it be used for other architectures? Even if qemu/scratchbox was only a little faster than native ARM, could the plethora of spare x86 boxes make up for that? Jason -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: armel boxes for Debian
On Wed, Feb 20, 2008 at 06:12:10PM -0500, Joey Hess wrote: > Riku Voipio wrote: > > The security buildd is a different story. Parallell buildd's compiling > > several packages at time don't help[1], they want single builds > > completed fast, so they can release security advisories with minimal > > delay. For this reason, Moritz from the security team expressed > > being unpleased with the speed of a Thecus and want something faster > > as buildd. > > I wonder if the security team is being entirely reasonable with their > expectations for speed. The team typically wants to get a package built > on all architectures before releasing an advisory, but since we don't > have a rule that supported architectures all need to have hardware > that's even within an order of magnitude of the speed of other > architectures[2], isn't that a fundamentally unreasonable expectation for > the security team to have? > > Perhaps it would be better for them to develop policies to deal with > slower architectures. Some currently fast-enough architectures will > likely fall into the too slow category in years to come no matter what > happens with arm now. New, faster alpha chips are not being designed, > for example. One such policy might be to have a set of architectures > that advisories are never delayed for. We could just declare arm a second-class architecture for security updates, i.e. DSAs being released once all archs are available except arm and arm updates being released once available. For small to medium packages most updates would still be released in sync, since we're not available to release updates 24/7. But we should stop withholding security updates because we wait on slow archs. Debian was the first to fix the vmsplice root exploit. If we had released all archs in sync we would have left most of our users unprotected for two more days. > I'm also unsure based on Moritz's mail exactly what kind of speed > they're looking for from an arm security buildd. He mentioned something > on the order of 14 hours to build xulrunner -- would an arm box that > builds it in 9 hours[1] be a worthwhile improvement, or will that still > leave the security team waiting until the next day for arm to catch up? 9 instead of 14 would still help. I also think a second security buildd would help: It wouldn't address the spikes of giga packages like xulrunner, but it would help for cases, where several updates are building in parallel. Cheers, Moritz -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: armel boxes for Debian
Riku Voipio wrote: > The security buildd is a different story. Parallell buildd's compiling > several packages at time don't help[1], they want single builds > completed fast, so they can release security advisories with minimal > delay. For this reason, Moritz from the security team expressed > being unpleased with the speed of a Thecus and want something faster > as buildd. I wonder if the security team is being entirely reasonable with their expectations for speed. The team typically wants to get a package built on all architectures before releasing an advisory, but since we don't have a rule that supported architectures all need to have hardware that's even within an order of magnitude of the speed of other architectures[2], isn't that a fundamentally unreasonable expectation for the security team to have? Perhaps it would be better for them to develop policies to deal with slower architectures. Some currently fast-enough architectures will likely fall into the too slow category in years to come no matter what happens with arm now. New, faster alpha chips are not being designed, for example. One such policy might be to have a set of architectures that advisories are never delayed for. I'm also unsure based on Moritz's mail exactly what kind of speed they're looking for from an arm security buildd. He mentioned something on the order of 14 hours to build xulrunner -- would an arm box that builds it in 9 hours[1] be a worthwhile improvement, or will that still leave the security team waiting until the next day for arm to catch up? Based on these xulrunner build times, we'd need an arm box that's 4x as fast as the current one to be "competitive": i3860:42 (amd64 is of course approx. the same) alpha 1:02 s3901:10 sparc 1:59 ia641:19 hppa2:39 powerpc 2:45 mips3:09 mipsel 3:19 arm 13:01 m68k46:22 (build failed incomplete) -- see shy jo [1] not a hypothetical number.. [2] the closest such rule being that an architecture needs to be able to keep the archive 95% up-to-date with only 3 buildds signature.asc Description: Digital signature
Re: armel boxes for Debian
On Tue, Feb 19, 2008 at 05:23:34PM +, Wookey wrote: > Givenwhat Riku said about buildd plans (3 more thecus, existing ones-> > porter machines, search for something even faster for security work), > it suggests that this Kuro box is probably only useful for developing > installer support? Usefull for working Installer and kernel for our endusers. Remember, our real goal is to provide a port for endusers, not just to fulfill cabal requirements ;) > Suggestions for the most useful thing to be done with it welcome. Bring it to to FOSDEM, I'm coming there too. While I'm interested in working on d-i/kernel for orions, it's even better if we can find some new blood to work on the port. -- "rm -rf" only sounds scary if you don't have backups signature.asc Description: Digital signature
Re: armel boxes for Debian
On 2008-02-19 16:55 +, Wookey wrote: > On 2008-02-19 16:05 +, Colin Tuckley wrote: > > A box for use as a porter machine would help with fixing problems. > > I have a kurobox pro here donated by marvell for Debian use. Givenwhat Riku said about buildd plans (3 more thecus, existing ones-> porter machines, search for something even faster for security work), it suggests that this Kuro box is probably only useful for developing installer support? Suggestions for the most useful thing to be done with it welcome. Wookey -- Principal hats: Balloonz - Toby Churchill - Aleph One - Debian http://wookware.org/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: armel boxes for Debian
* Wookey <[EMAIL PROTECTED]> [2008-02-19 16:55]: > Does anyone know if current installer images will work? I find some > rather old (2.4 kernel) stuff here: Not yet, but I'm currently working on Orion support for d-i. > If there is something I can expect to 'just work' then I'll have a Nope. -- Martin Michlmayr http://www.cyrius.com/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: armel boxes for Debian
On 2008-02-19 16:05 +, Colin Tuckley wrote: > Talking to tbm on irc this afternoon, it seems there is some confusion about > how many armel buildd boxes there are and how many are needed. > > Current situation: > > There are 3 buildd boxes (all of them Thecus N2100 boxes), two hosted by > Riku and the third by me. They are almost keeping up with the archive. > > Additionally we have another Thecus box hosted by Martin Guy which is being > used for armel work and is available for people to use. > > Wookey also has a balloon based box which I believe is also available for > use via the net. Correct. The external IP access/firewalling has even been sorted out so it will have a permanent IP, and be sited outside the company firewall for easier access soon. > The future: > > It looks like we need one more buildd of the same capacity as the current > Thecus boxes. > > It would probably be good to have a box for the security team too although > that could be one that was generally available perhaps. > > A box for use as a porter machine would help with fixing problems. I have a kurobox pro here donated by marvell for Debian use. It needs debianising and siting. I could site it but am terifically short of time for the debianising bit (which is why it hasn't happenened yet). it's 128MB, not 256MB which isn't ideal, but is otherwise reasonably quick. Does anyone know if current installer images will work? I find some rather old (2.4 kernel) stuff here: and some newer stuff but with kernel for the Kuro-HG box here: http://buffalo.nas-central.org/index.php/Debian_sylver If there is something I can expect to 'just work' then I'll have a go. Otherwise I'd prefer to give it to someone with more time/interest to fight with. Any takers? Handover at FOSDEM, perhaps? Wookey -- Principal hats: Balloonz - Toby Churchill - Aleph One - Debian http://wookware.org/ -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: armel boxes for Debian
Firstly, congrats for Zobel, we understand that you have more important stuff to do than read this mail now :) On Tue, Feb 19, 2008 at 04:05:17PM +, Colin Tuckley wrote: > There are 3 buildd boxes (all of them Thecus N2100 boxes), two hosted by > Riku and the third by me. They are almost keeping up with the archive. Well, you could have asked from me on IRC as well? :) The original plan was to Zobel to order 4 Thecuses (3 buildd's, 1 Porter machine for DD's to log into), and retire the current Buildd's to kernel/toolchain/whatever porting. However, since then 3 buildd's have proven not be enough. Many packages that are now buildable due to the porting work and gfortran transition are really slow to build, and it's only going to get the closer we get to 99%. I'm still not really worried, we can allways allocate the three old Thecus buildd's back as official debian buildd's. As long as they are all identical hardware and identically configured, the maintainence overhead should be quite tolerable. The security buildd is a different story. Parallell buildd's compiling several packages at time don't help[1], they want single builds completed fast, so they can release security advisories with minimal delay. For this reason, Moritz from the security team expressed being unpleased with the speed of a Thecus and want something faster as buildd. Cheers, Riku [1] in some point of future using DEB_BUILD_OPTIONS parallel=X and distcc could make that possible, but those options need much more testing first. -- "rm -rf" only sounds scary if you don't have backups signature.asc Description: Digital signature
armel boxes for Debian
Talking to tbm on irc this afternoon, it seems there is some confusion about how many armel buildd boxes there are and how many are needed. Current situation: There are 3 buildd boxes (all of them Thecus N2100 boxes), two hosted by Riku and the third by me. They are almost keeping up with the archive. Additionally we have another Thecus box hosted by Martin Guy which is being used for armel work and is available for people to use. Wookey also has a balloon based box which I believe is also available for use via the net. The future: It looks like we need one more buildd of the same capacity as the current Thecus boxes. It would probably be good to have a box for the security team too although that could be one that was generally available perhaps. A box for use as a porter machine would help with fixing problems. Comments? Colin -- Colin Tuckley | +44(0)1903 236872 | PGP/GnuPG Key Id Debian Developer | +44(0)7799 143369 | 0x1B3045CE What would life be like if there were no hypothetical questions? -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
