Re: [Mailman-Developers] Killing off Pipermail and the effects on scrubbing in Mailman 3
On 03/19/2012 09:14 PM, Mark Sapiro wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 3/19/2012 4:25 AM, Barry Warsaw wrote: There are lots of options here. As mentioned, virtualbox should be a good option, and you can probably use something like wubi to create an Ubuntu desktop running along side Windows. Thanks to all for the advice. I have some learning to do. I actually have VirtualBox installed and for the sprint I had a Vagrant VM set up running the basic http://files.vagrantup.com/lucid32.box. I had set up a virtualenv on this box with Sphinx installed for the Sphinx tutorial at PyCon, and this worked really well. When I got to the Mailman sprint, I tried to install MM 3 on this Vagrant VM, and I ran into some error with buildout trying to compile some package and failing with a missing Python.h file. I ran 'sudo apt-get install python-dev' on the VM which I thought would fix it, and the install seemed to run OK, but I still had no Python.h. At that point, I gave up and went to a virtualenv on my production server. At this point, I don't know whether the best approach is look for a more complete development box for Vagrant or figure out how to provision the basic Vagrant box with what I need or forget Vagrant and install directly in VirtualBox or go with a dual-boot. Anyway, it's going to be interesting to figure this out. Since I was curious to see I created a vagrant VM with lucid and I only needed to install the following: To install (Lucid32): - python - python-dev - bzr - python-setuptools - gcc To make everything work and be able to run buildout and bin/test in mailman bzr version. So now I was thinking to transform this manual steps in a Vagrantfile, and we could use this to ship for people that want to try, other things to do: Optional packages: - Gnome / XFCE / Other - vim To check/set: - launchpad login, possibly passing the authentication and the SSH key somehow to the vagrant file - dimension of the VM to make sure it's not too small PS. I get 6 failures running the tests though, as File /home/vagrant/mailman/src/mailman/model/docs/users.rst, line 172, in users.rst Failed example: user_2.preferred_address = anne Differences (ndiff with -expected +actual): Traceback (most recent call last): - ... - UnverifiedAddressError: Anne Person a...@example.com + File /usr/lib/python2.6/doctest.py, line 1248, in __run + compileflags, 1) in test.globs + File doctest users.rst[41], line 1, in module + user_2.preferred_address = anne + File /home/vagrant/mailman/src/mailman/model/user.py, line 112, in preferred_address + raise UnverifiedAddressError(address) + UnverifiedAddressError: unprintable UnverifiedAddressError object ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Killing off Pipermail and the effects on scrubbing in Mailman 3
On Mar 19, 2012, at 05:41 PM, Richard Wackerbarth wrote: I, too, am attempting to get MM3 running on my new laptop (Mac OSX 10.7). Because of the way Xcode 4.3 and Python are set up, compiling the .c extensions in storm is failing. I need to figure out where I can tell setup.py to add an additional include path when it calls the c compiler. Anyone know how? I've never done it, but I found this. Hopefully that helps. http://www.buildout.org/docs/tutorial.html#custom-egg-building -Barry signature.asc Description: PGP signature ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Killing off Pipermail and the effects on scrubbing in Mailman 3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Mar 17, 2012, at 04:43 PM, Mark Sapiro wrote: Next step for me may be to learn more about how the unittests fit within their framework so I can create some. It's *relatively* simple. Essentially all unittests live in a test_foo.py module inside a tests/ directory (which has an __init__.py so it's a package). I generally name unittest.TestCase subclasses as TestFoo and individual tests inside there as test_foo() methods. There are lots of examples spread throughout. It's only a little extra work if you have to create a new tests/ subdirectory or test_foo.py module, but again, it should be pretty straightforward. These all get auto-discovered by the zope.testing framework, so there's nothing else necessary to hook it all up. The one other tricky thing is the layers, which are a zope.testing mechanism to share more generic resources among a suite of tests. E.g. if you need the rest server to be started, you'd use the RESTLayer (see mailman.testing.layers for what's currently defined). To enable a layer, you just need to set the `layer` attribute of the TestFoo class. Almost all the tests will require at least the ConfigLayer since that brings up the Zope component architecture, connfiguration system, etc. zope.testing is a bit weird here because the normal rules of subclasses don't actually apply. E.g. the LMTPLayer derives from ConfigLayer which derives from MockAndMonkeyLayer. All the appropriate setUp()s get called by zope.testing so you don't need to up-call explicitly in the layer class. Yes, this kind of sucks since it's magical, and it's one reason why eventually I'd like to switch to something like testresources and a more sane testing framework, but that's a lot of effort for little immediate value. Note that doctests are all set up in test_documentation.py so they're not really all that special. Also, I need to figure out a better development platform for Windows boxes. I had a perfect opportunity to scrap Windows all together when I had to recover from a hard drive crash on my main development box last year, but the dice fell the wrong way on that one. There are lots of options here. As mentioned, virtualbox should be a good option, and you can probably use something like wubi to create an Ubuntu desktop running along side Windows. Of course, all the Linuxes should give you a way to dual-boot your Windows machine if you're so inclined. You may even be able to boot off a USB drive or stick, though that will be slower. Commercial options such as VMware are also a possibility, though of course you should try the free options first. :) Maybe I can organize a sprint at the next PyCon - Migrating to Linux and killing Windows one PC at a time. Definitely. :) - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBCAAGBQJPZxexAAoJEBJutWOnSwa/jlQQAKqzGaL5Vrnm5vS8+NSlN+LH gY/Gyh/aT8F4V6IYd3Hn47kPMtdMPv6m+FNhSTOGkwf1EU1UtPgNvkcYUNuVsHBe A9Bh04DDk0t5S1WOdcKBSdkwt65YR3yNkvsiCZQpOXD4xDF4EDlSRwNTjtGk6ze7 84xBUL5DXJiO1L5OxBsZusYlR4vpO80AwCszil3uon9oDQIxsHuXvPfVwjJM8CfN hFf3Rq7LePLbOO0uFNLxsDxMyRZx0wrN/vg+2H81bnLNdheaqlmRjoZn2TkFZmTb eYN+XeFW9l2jlGgYfQAm8R5e+mNhE+a3HG7LwO2IyYn8JzR+YTFQeiR91XdOFYrt OPfMrXbUaRAb6QEwwWJY2HmbqIiihgFOPtmSBSRcIoiU/cWjC6LbajZN8QV24xhq yTrbkBxZBTGPPQ7tgd05bb/PYlqdWCRoXeL8zlbcDK98MFzPgfBMfmHqWObPd+8E q49pDoeDUlhW+SmhfTOqM3zQvuolIu6WYDTcTeIWYLzMsjT3pDJ6R1NIw4jNGmtl TqLYL+WuUzBXyNfGD234C2VnJscVyJLFduew2pig6dFG0MX9gAN/zTI3i8wZDlUC xDKDagCZ7uSNzjVEDQLzoZmrs3EzuPvCY5h4kzlVObUiKBvg8ALPEV8MJS6M8QDj IeBbG8Iniz01AoBaixZ1 =rxit -END PGP SIGNATURE- ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Killing off Pipermail and the effects on scrubbing in Mailman 3
On 03/19/2012 11:25 AM, Barry Warsaw wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Mar 17, 2012, at 04:43 PM, Mark Sapiro wrote: Next step for me may be to learn more about how the unittests fit within their framework so I can create some. It's *relatively* simple. Essentially all unittests live in a test_foo.py module inside a tests/ directory (which has an __init__.py so it's a package). I generally name unittest.TestCase subclasses as TestFoo and individual tests inside there as test_foo() methods. There are lots of examples spread throughout. It's only a little extra work if you have to create a new tests/ subdirectory or test_foo.py module, but again, it should be pretty straightforward. These all get auto-discovered by the zope.testing framework, so there's nothing else necessary to hook it all up. The one other tricky thing is the layers, which are a zope.testing mechanism to share more generic resources among a suite of tests. E.g. if you need the rest server to be started, you'd use the RESTLayer (see mailman.testing.layers for what's currently defined). To enable a layer, you just need to set the `layer` attribute of the TestFoo class. Almost all the tests will require at least the ConfigLayer since that brings up the Zope component architecture, connfiguration system, etc. zope.testing is a bit weird here because the normal rules of subclasses don't actually apply. E.g. the LMTPLayer derives from ConfigLayer which derives from MockAndMonkeyLayer. All the appropriate setUp()s get called by zope.testing so you don't need to up-call explicitly in the layer class. Yes, this kind of sucks since it's magical, and it's one reason why eventually I'd like to switch to something like testresources and a more sane testing framework, but that's a lot of effort for little immediate value. Note that doctests are all set up in test_documentation.py so they're not really all that special. Just as a suggestion, it would be nice to have a few lines in the documentation about this, something like you just explained would be already quite good, since it's not so clear by just looking at the code.. Cheers, Andrea ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Killing off Pipermail and the effects on scrubbing in Mailman 3
On Mar 19, 2012, at 12:03 PM, Andrea Crotti wrote: Just as a suggestion, it would be nice to have a few lines in the documentation about this, something like you just explained would be already quite good, since it's not so clear by just looking at the code.. Care to adapt my post into a merge proposal? :) -Barry signature.asc Description: PGP signature ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Killing off Pipermail and the effects on scrubbing in Mailman 3
On 03/19/2012 12:27 PM, Barry Warsaw wrote: On Mar 19, 2012, at 12:03 PM, Andrea Crotti wrote: Just as a suggestion, it would be nice to have a few lines in the documentation about this, something like you just explained would be already quite good, since it's not so clear by just looking at the code.. Care to adapt my post into a merge proposal? :) -Barry Yes sure I will do that ASAP.. ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Killing off Pipermail and the effects on scrubbing in Mailman 3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 3/19/2012 4:25 AM, Barry Warsaw wrote: There are lots of options here. As mentioned, virtualbox should be a good option, and you can probably use something like wubi to create an Ubuntu desktop running along side Windows. Thanks to all for the advice. I have some learning to do. I actually have VirtualBox installed and for the sprint I had a Vagrant VM set up running the basic http://files.vagrantup.com/lucid32.box. I had set up a virtualenv on this box with Sphinx installed for the Sphinx tutorial at PyCon, and this worked really well. When I got to the Mailman sprint, I tried to install MM 3 on this Vagrant VM, and I ran into some error with buildout trying to compile some package and failing with a missing Python.h file. I ran 'sudo apt-get install python-dev' on the VM which I thought would fix it, and the install seemed to run OK, but I still had no Python.h. At that point, I gave up and went to a virtualenv on my production server. At this point, I don't know whether the best approach is look for a more complete development box for Vagrant or figure out how to provision the basic Vagrant box with what I need or forget Vagrant and install directly in VirtualBox or go with a dual-boot. Anyway, it's going to be interesting to figure this out. - -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) iD8DBQFPZ6GnVVuXXpU7hpMRAlwwAJ91EQ7k4TtOLBvyexmrmyspzJZ8RACg3iFQ fdgRt+ylQxrcuoL0d3W+VMY= =lP82 -END PGP SIGNATURE- ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Killing off Pipermail and the effects on scrubbing in Mailman 3
On 03/19/2012 09:14 PM, Mark Sapiro wrote: Thanks to all for the advice. I have some learning to do. I actually have VirtualBox installed and for the sprint I had a Vagrant VM set up running the basic http://files.vagrantup.com/lucid32.box. I had set up a virtualenv on this box with Sphinx installed for the Sphinx tutorial at PyCon, and this worked really well. When I got to the Mailman sprint, I tried to install MM 3 on this Vagrant VM, and I ran into some error with buildout trying to compile some package and failing with a missing Python.h file. I ran 'sudo apt-get install python-dev' on the VM which I thought would fix it, and the install seemed to run OK, but I still had no Python.h. At that point, I gave up and went to a virtualenv on my production server. At this point, I don't know whether the best approach is look for a more complete development box for Vagrant or figure out how to provision the basic Vagrant box with what I need or forget Vagrant and install directly in VirtualBox or go with a dual-boot. Anyway, it's going to be interesting to figure this out. Well but why using Vagrant? Vagrant is great to create VMS for automatic testing, but if you want to create your own VM with all your things that you need I don't see how it can help, just install a normal Ubuntu/whatever on Virtualbox and you're fine.. And another possible advice is to always copy the errors you get, also because most of the times you can get the answer you want simply by a quick google search ;) ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Killing off Pipermail and the effects on scrubbing in Mailman 3
On Monday 2012-03-19 14:40 (-0700), Andrea Crotti andrea.crott...@gmail.com wrote: Well but why using Vagrant? Vagrant is great to create VMS for automatic testing, but if you want to create your own VM with all your things that you need I don't see how it can help, just install a normal Ubuntu/whatever on Virtualbox and you're fine.. I'd recommend this too. (Vagrant for repeatable deployments, you only need one). If you like Ubuntu, Ubuntu Server (an LTS version) is well worth using for this sort of thing as it is already headless, no desktop (or though you can add one if you want). Chris ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Killing off Pipermail and the effects on scrubbing in Mailman 3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Mar 19, 2012, at 02:14 PM, Mark Sapiro wrote: I ran 'sudo apt-get install python-dev' on the VM which I thought would fix it, and the install seemed to run OK, but I still had no Python.h. Hmm, I just tried this in a Lucid chroot and it worked for me. :/ I do get test failures (could be 2.6 related) but everything built. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBCAAGBQJPZ6mFAAoJEBJutWOnSwa/5IIP/3bo0dOX/ESg4wDbtd//sjcr 9AHwT7X3oOUIB9H8IAqRkSFNTIZDoxTtLrv7BzIHLgfIAkS9KaKcRSK3IGlulPrG 6KtCf93nVbtd8KdRYDxT2TmWhoR377g9DDnGDztcf6N39HiSVrPNzxLx2oSMrfYh cvLhQnGnI7xOloVox6uEvEB7fU1DQaLJe6G4obZxzr0YZGViZGv1u4h1Qi0dgPgO 7Av1cne29t7+4iyNbgoo8nvCmSBcE4Lbmg3e+mHr25BMpKUKuM0sce323RN3Pp/+ 5GZOtFuiWKYB/GHmm0OpIRCUCMX1oPf2SyVw94+MYW32wspDbUpOBevgtGPbFmQx WtKm1l6ps2TGX+q5WatNVemtmoZWVLx7N6CfxdcyZ7kdIvnGkxVqylq7o5RUWUQM 83rqo4EEDPZjnF7Aos7RgdSUAbdh99Rus4/8VoCJgg4sTbMjFFcH/PRiYeiMR+GK GJtFiZilYjiXUnCZ6jye3s6mBsy/YHtUd3ytYrjUtritVgMcMuPF0EwRTwb04OIN TWmE5BNvlgF8mn0U1UA3+V5A42GqjnK7PajZ9VpTwyymU65QgK7PSwIYyXNzATQb wHHokKct8sE4eNWt9Bs4Ai6MlMJKLoAFaOJvJQtZ2r0OBpuBcNomHb0c2yddlGSr Yj04lLFc/Sp3FFko9EbU =Ib4D -END PGP SIGNATURE- ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Killing off Pipermail and the effects on scrubbing in Mailman 3
I, too, am attempting to get MM3 running on my new laptop (Mac OSX 10.7). Because of the way Xcode 4.3 and Python are set up, compiling the .c extensions in storm is failing. I need to figure out where I can tell setup.py to add an additional include path when it calls the c compiler. Anyone know how? Richard On Mar 19, 2012, at 4:47 PM, Barry Warsaw wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Mar 19, 2012, at 02:14 PM, Mark Sapiro wrote: I ran 'sudo apt-get install python-dev' on the VM which I thought would fix it, and the install seemed to run OK, but I still had no Python.h. Hmm, I just tried this in a Lucid chroot and it worked for me. :/ I do get test failures (could be 2.6 related) but everything built. - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBCAAGBQJPZ6mFAAoJEBJutWOnSwa/5IIP/3bo0dOX/ESg4wDbtd//sjcr 9AHwT7X3oOUIB9H8IAqRkSFNTIZDoxTtLrv7BzIHLgfIAkS9KaKcRSK3IGlulPrG 6KtCf93nVbtd8KdRYDxT2TmWhoR377g9DDnGDztcf6N39HiSVrPNzxLx2oSMrfYh cvLhQnGnI7xOloVox6uEvEB7fU1DQaLJe6G4obZxzr0YZGViZGv1u4h1Qi0dgPgO 7Av1cne29t7+4iyNbgoo8nvCmSBcE4Lbmg3e+mHr25BMpKUKuM0sce323RN3Pp/+ 5GZOtFuiWKYB/GHmm0OpIRCUCMX1oPf2SyVw94+MYW32wspDbUpOBevgtGPbFmQx WtKm1l6ps2TGX+q5WatNVemtmoZWVLx7N6CfxdcyZ7kdIvnGkxVqylq7o5RUWUQM 83rqo4EEDPZjnF7Aos7RgdSUAbdh99Rus4/8VoCJgg4sTbMjFFcH/PRiYeiMR+GK GJtFiZilYjiXUnCZ6jye3s6mBsy/YHtUd3ytYrjUtritVgMcMuPF0EwRTwb04OIN TWmE5BNvlgF8mn0U1UA3+V5A42GqjnK7PajZ9VpTwyymU65QgK7PSwIYyXNzATQb wHHokKct8sE4eNWt9Bs4Ai6MlMJKLoAFaOJvJQtZ2r0OBpuBcNomHb0c2yddlGSr Yj04lLFc/Sp3FFko9EbU =Ib4D -END PGP SIGNATURE- ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/richard%40nfsnet.org Security Policy: http://wiki.list.org/x/QIA9 ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Killing off Pipermail and the effects on scrubbing in Mailman 3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Mar 16, 2012, at 10:11 AM, Mark Sapiro wrote: There are two things going on. There is content filtering, i.e., removal from the message of parts with unwanted MIME types or filename extensions. These parts are simply removed by pipeline/mime_delete.py (which probably needs some changes ported from 2.1, aargh...). Yeah, that's embarrassing ;). I've started down the road of adding unittests for the code in that module. You'll see the start of that land momentarily. Then there is what pipeline/scrubber.py does with the remaining message which is remove those message parts which can't be rendered well in a flat, text/plain message and store them aside and replace them by links in the message. The part we can't do in MM 3 is calculate a URL to display/download them. Yep. The easiest thing to do, and what I will probably do in my 'death-to-pipermail' branch is to simply scrub out the unwanted parts *after* a copy of the message is sent to the archive queue, but *before* the message is sent to the digest, usenet, and outgoing queues. I'm not sure about the *before* with respect to usenet and digest and certainly outgoing. Currently in 2.1, we don't scrub (as opposed to content filter) non-digest deliveries unless scrub_nondigest is Yes. We maybe should just drop that option. We also don't scrub messages for the MIME digest. I also don't think we scrub messages destined for usenet. I think we let usenet worry about that in the same way we propose to let whatever archiver is configured worry about it. I don't see a need to handle these differently in MM 3. ISTM that essence of the scrubber is to turn any remaining text/html parts into plain text, by various means. I think the MM2 scrubber.py module is essentially hopeless, but the basic functionality is useful. I've decided to remove the scrubber in the Pipermail-eradication branch, which will also land momentarily. I think it would be useful though to rewrite the scrubber, boil it down to its essential functionality, and add that to the appropriate spot in the pipeline. How would you like to take a crack at that? For now, I'm going to try to implement sending an unscrubbed copy of the message to the archivers and just throwing up our hands for the copy of the message sent to the list members. The nice side-effect of this is that it makes the scrubber *way* simpler! Perhaps we could keep the scrubber as is except for modifying it to not store scrubbed parts and put some kind of apology in the message rather than the link to the no longer stored content. Then my lp:~msapiro/mailman/scrubber-fix branch would still be relevant ;) Yeah, sorry about that. ;) I think scrubber.py was just too nasty to salvage. Something much simpler would still be useful though. Cheers, - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBCAAGBQJPZMQ9AAoJEBJutWOnSwa/nFEP/iMDCM+ETv1KV36nP8r/cZfB C50m+K1MUm/MaZpkpQI8980J96QWC1RoWvQ7sQGg2difvDvNwI0JZP4gMBJkHVUu sO/hJZu0BDa28cC9Ww94fRX4ujelm/jesc8td0v02s54FSHUIOgxxDr+sfWNFPvI OpLDJZVtC6LJbDt1IqI2ozxbq/b3hhuaXDbmzIsWqotyZZ/+fQDjgM4L9SCEjhrT tDwQjFhsZmH3m58pFRkP/cOJCV2lKs0MnMGMhELHGkatMGKtVFAuP1e3r24N20yX EVDX/7Dg20BzacNYnAVGnO28sYqb4JltRAb14+IvIMcRzIO+WKKAyJioKX3cohcT 14fhb0agtDPlMMBJw8J5AD9VEimMcZaMmISLpRY6jqkaHRu/4RxZlG3RRWtcBwdS dN0WZnnNx6B+wV5VUJ7Q5WaDO1Xtp0jGHuT96vOQlHDm/+iwwmWWvGH3DQg1yVDN gT2/JyLeXpDprP+qXNPLyWlMlADQjUCq7uvD51J0gcCC6aLanPnM9CuCQXdJRlFl 7g+zI9a17qCdniQcbNUgq+87ektXLi7JCp6nA1yEm0Zaelp3wJC2cB7up9ZaVR7O SX8qpMFnfqFkvsQLC2pLH7plplHpboXWOjLALITFBzasth4hS98oHH+gOJktTKni Erk1f+FsVR9l0Geu2q++ =z6a7 -END PGP SIGNATURE- ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Killing off Pipermail and the effects on scrubbing in Mailman 3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 3/17/2012 10:05 AM, Barry Warsaw wrote: ISTM that essence of the scrubber is to turn any remaining text/html parts into plain text, by various means. I think the MM2 scrubber.py module is essentially hopeless, but the basic functionality is useful. I've decided to remove the scrubber in the Pipermail-eradication branch, which will also land momentarily. I think it would be useful though to rewrite the scrubber, boil it down to its essential functionality, and add that to the appropriate spot in the pipeline. How would you like to take a crack at that? Sure. Now that I actually have a bit of an idea of what's going on in the MM 3 core, I'm happy to give it a go. Next step for me may be to learn more about how the unittests fit within their framework so I can create some. Also, I need to figure out a better development platform for Windows boxes. I had a perfect opportunity to scrap Windows all together when I had to recover from a hard drive crash on my main development box last year, but the dice fell the wrong way on that one. Anyway, Cygwin is not going to cut it for MM 3. At the sprint, I tried installing MM 3 in a vagrant VM, but there was too much missing (e.g., no Python.h) and even 'apt-get install python-dev' didn't fix that. I ended up working remotely inside a virtualenv on my production server. That actually seems to work OK, but I'm afraid that as I get deeper into it, there will be things I need to do that I won't want to do on my production box. Anyway, if anyone has any suggestions for me besides the obvious bite the bullet now and scrap Windows - it will only be worse later, I'm interested. I suppose I could always dual-boot Windows and some Linux side by side. Maybe I can organize a sprint at the next PyCon - Migrating to Linux and killing Windows one PC at a time. - -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) iD8DBQFPZSGEVVuXXpU7hpMRAnRAAKDSSRUhDdQe8HoIBzOh3coe8elMIQCfU+dP fKbzWiMB+H1wm4Jou28BV7g= =Ehz5 -END PGP SIGNATURE- ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Killing off Pipermail and the effects on scrubbing in Mailman 3
On 03/17/2012 11:43 PM, Mark Sapiro wrote: Also, I need to figure out a better development platform for Windows boxes. I had a perfect opportunity to scrap Windows all together when I had to recover from a hard drive crash on my main development box last year, but the dice fell the wrong way on that one. Anyway, Cygwin is not going to cut it for MM 3. At the sprint, I tried installing MM 3 in a vagrant VM, but there was too much missing (e.g., no Python.h) and even 'apt-get install python-dev' didn't fix that. I ended up working remotely inside a virtualenv on my production server. That actually seems to work OK, but I'm afraid that as I get deeper into it, there will be things I need to do that I won't want to do on my production box. Anyway, if anyone has any suggestions for me besides the obvious bite the bullet now and scrap Windows - it will only be worse later, I'm interested. I suppose I could always dual-boot Windows and some Linux side by side. Maybe I can organize a sprint at the next PyCon - Migrating to Linux and killing Windows one PC at a time. I would definitively suggest to get rid of Windows, but if you don't want or can't do the big step there is no reason to do a dual-boot, a just create a Linux virtual machine and you're done :) Unless you don't have much RAM is the best solution, and there are even pre-made virtual machines here: http://virtualboximages.com/ so it doesn't really take long to set up something working :) ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Killing off Pipermail and the effects on scrubbing in Mailman 3
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 3/15/2012 10:41 PM, Barry Warsaw wrote: We can still scrub messages of unwanted content type, but we can't save those parts on the file system and calculate a URL into Pipermail to display them. There are two things going on. There is content filtering, i.e., removal from the message of parts with unwanted MIME types or filename extensions. These parts are simply removed by pipeline/mime_delete.py (which probably needs some changes ported from 2.1, aargh...). Then there is what pipeline/scrubber.py does with the remaining message which is remove those message parts which can't be rendered well in a flat, text/plain message and store them aside and replace them by links in the message. The part we can't do in MM 3 is calculate a URL to display/download them. I can think of a few ways to handle this. The easiest thing to do, and what I will probably do in my 'death-to-pipermail' branch is to simply scrub out the unwanted parts *after* a copy of the message is sent to the archive queue, but *before* the message is sent to the digest, usenet, and outgoing queues. I'm not sure about the *before* with respect to usenet and digest and certainly outgoing. Currently in 2.1, we don't scrub (as opposed to content filter) non-digest deliveries unless scrub_nondigest is Yes. We maybe should just drop that option. We also don't scrub messages for the MIME digest. I also don't think we scrub messages destined for usenet. I think we let usenet worry about that in the same way we propose to let whatever archiver is configured worry about it. I don't see a need to handle these differently in MM 3. [...] For now, I'm going to try to implement sending an unscrubbed copy of the message to the archivers and just throwing up our hands for the copy of the message sent to the list members. The nice side-effect of this is that it makes the scrubber *way* simpler! Perhaps we could keep the scrubber as is except for modifying it to not store scrubbed parts and put some kind of apology in the message rather than the link to the no longer stored content. Then my lp:~msapiro/mailman/scrubber-fix branch would still be relevant ;) - -- Mark Sapiro m...@msapiro.netThe highway is for gamblers, San Francisco Bay Area, Californiabetter use your sense - B. Dylan -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (MingW32) iD4DBQFPY3QuVVuXXpU7hpMRAkN2AKCVzhwvBUITlLVgDMwg+V+da0cyJACXed7Q jAvaD7jeN2/4armc/nIxBw== =MsB3 -END PGP SIGNATURE- ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
[Mailman-Developers] Killing off Pipermail and the effects on scrubbing in Mailman 3
There's an aspect of the scrubber that isn't going to work in a Mailman 3 world where we have multiple, possibly external, archivers and especially where we don't have such tight integration with Pipermail (or Pipermail at all wink). We can still scrub messages of unwanted content type, but we can't save those parts on the file system and calculate a URL into Pipermail to display them. I can think of a few ways to handle this. The easiest thing to do, and what I will probably do in my 'death-to-pipermail' branch is to simply scrub out the unwanted parts *after* a copy of the message is sent to the archive queue, but *before* the message is sent to the digest, usenet, and outgoing queues. This makes sense because with a model of external archiving, those archivers may make different decisions about what should be removed or displayed from the original message. We can still include a little blurb saying that a part was scrubbed out, and since the messages can have the pre-calculated url to the message in one or more archivers, the user is always free to just click on the url to see the full message, displayed with whatever policy the archiver is configured with. One possibility is to save the scrubbed part inside the core and provide a url to the REST API for accessing this attachment. This can't be inserted into the scrubbed message directly though because this would be a non-public url to the resource, and it would have to be proxied by the web ui. We need better configuration for integrating the web ui with the core any way (e.g. to calculate the url to the user's options page), so this could be part of that. The interactions are trickier though because you would then have to inform the web ui that there's a new attachment it should proxy. The other, more elaborate option is to define an IScrubber interface, or alternatively a primary IArchiver, that the message can pass through, which would give it an opportunity to provide urls for each of the parts that will be scrubbed out. This is trickier because there can really be only one such thing defined in the system. I think it would be confusing if you received a message that had something like this: text/html part scrubbed, view it at one of the following: http://example.com/attachments/foo.html http://example.org/some/extra/path/bar.html http://another.archive.example.net/whatever/baz.html Besides, this may be nearly impossible to do without in-band communication with that external archiver, which is exactly what the RFC 5064 + message-id-hash was supposed to avoid. I think we definitely don't want to have to force such in-band communications to occur in order to scrub messages of unwanted parts. For now, I'm going to try to implement sending an unscrubbed copy of the message to the archivers and just throwing up our hands for the copy of the message sent to the list members. The nice side-effect of this is that it makes the scrubber *way* simpler! Any other suggestions? Cheers, -Barry signature.asc Description: PGP signature ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
