Re: [Mailman-Developers] MM3 REST API testing [was: Python 3]
FWIW some notes on mitmproxy and ngrep - helpful tools when intercepting and recording HTTP traffic. mitmproxy/mitmdump: I know doesn’t serve the same purpose as VCR but its helpful to be able to capture and see what is going on between server and client when Mailman tests are running. https://mitmproxy.org/doc/mitmdump.html mitmproxy itself seems to have Python 3 as its next major initiative. I configured Mailman tests to run through a proxy (I sent an email to this list a few weeks back with instructions for how to do this). Then: mitmdump -p 7001 -P http://127.0.0.1:9001 -w outfile Captures traffic to outfile. It can be loaded back into mitmproxy and modified and replayed. **ngrep ngrep is useful when digging deeper. it will show you traffic in real time on a given port. for example sudo ngrep -W byline -d lo port 9001 will show what is happening on the Mailman REST API port when configured for port 9001 Dunno if this is useful for anyone but everything helps when digging into HTTP. as On 2 Jan 2015, at 3:40 pm, Barry Warsaw ba...@python.org wrote: On Dec 27, 2014, at 01:16 PM, Florian Fuchs wrote: So far we haven't found a perfect solution for testing packages that rely heavily on the MM3 REST API. As far as mailman.client, Postorius and HK are concerned, testing without *some* core integration doesn't make too much sense IMO, because any changes in the API would go unnoticed. I'm playing with a solution that involves the use of the 'vcr' package, which as its name implies (well, at least for us, um, mature programmers ;) is a facility to record and replay HTTP traffic. It's compatible with both Python 2 and 3 and is available on the Cheeseshop. So the basic idea is to set up an MM3 installation (`tox --notest -r` suffices), then start up its REST server with a fresh mailman.db, and run the test suite against it. This will capture the traffic in a yaml file, which we could ship with mailman.client. Then to run the test suite, you'd use vcr in replay mode against the captured yaml file. Any time the API changes, you'd have to re-record the traffic against a fresh database. I have a branch that's working in this direction and it's promising, but I've also had to rewrite the testing infrastructure for mailman.client, and make some other changes. It also points out what I think are some bugs in Mailman core, so I'm working those out on my py3 branch in parallel. It's not yet ready for review, but if you want to keep an eye on things, look here: lp:~barry/mailman.client/bilingual Feel free to provide feedback on the approach, or the changes so far, but I will submit a merge proposal when/if it's ready to go. Cheers, -Barry ___ Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40supercoders.com.au Security Policy: http://wiki.list.org/x/QIA9 ___ Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] MM3 REST API testing [was: Python 3]
Am 02.01.2015 um 05:40 schrieb Barry Warsaw: On Dec 27, 2014, at 01:16 PM, Florian Fuchs wrote: So far we haven't found a perfect solution for testing packages that rely heavily on the MM3 REST API. As far as mailman.client, Postorius and HK are concerned, testing without *some* core integration doesn't make too much sense IMO, because any changes in the API would go unnoticed. I'm playing with a solution that involves the use of the 'vcr' package, which as its name implies (well, at least for us, um, mature programmers ;) is a facility to record and replay HTTP traffic. It's compatible with both Python 2 and 3 and is available on the Cheeseshop. I took a quick look at vcr and it really looks promising. Although apparently it doesn't support VHS, which is very sad. But we can still use it for testing I guess... :-) So the basic idea is to set up an MM3 installation (`tox --notest -r` suffices), then start up its REST server with a fresh mailman.db, and run the test suite against it. This will capture the traffic in a yaml file, which we could ship with mailman.client. Then to run the test suite, you'd use vcr in replay mode against the captured yaml file. Any time the API changes, you'd have to re-record the traffic against a fresh database. That sounds fantastic! Do you think it might make sense to ship the traffic dump with mailman instead of mailman.client (and work the recording into the test suite somehow)? That way there would only be a single file to observe for changes in the REST API. Sure, a change in the yaml file doesn't necessarily mean a change in the API (might as well be just an additional test case or different test data). But still, if the yaml file stays the same between commits, that would be a nice guarantee that the API stayed stable. I have a branch that's working in this direction and it's promising, but I've also had to rewrite the testing infrastructure for mailman.client, and make some other changes. It also points out what I think are some bugs in Mailman core, so I'm working those out on my py3 branch in parallel. It's not yet ready for review, but if you want to keep an eye on things, look here: lp:~barry/mailman.client/bilingual Feel free to provide feedback on the approach, or the changes so far, but I will submit a merge proposal when/if it's ready to go. Thanks for starting the port, Barry. Looking forward to seeing that merge proposal (although I don't expect I'll have a too much to add... ;-). Florian ___ Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Python 3
Am 29.12.2014 um 00:26 schrieb Barry Warsaw: On Dec 28, 2014, at 01:51 PM, Aurelien Bompard wrote: As I mentioned, I think LMTP *could* work, but REST (inside HK) could work too. Aurelien, what do you think? I'd go with REST, it seems more flexible and we already have nice libraries for it. REST will be very nice, and I think with something like Falcon, pretty easy to implement. You probably don't need the object-based (i.e. restish-like) traversal stuff, but if so, I'd be happy to split that out into a separately library that HK could use (and keep it bilingual). Keep it simple, but think through the resource tree and leave open some opportunities for expanding the API later, both for use by MM3 and perhaps other scripting applications. Are you planning to implement that in a way similar to the IArchiver interface? Like, say, having a REST-based equivalent for each of the IArchiver's `list_url`, `permalink` and `archive_message` methods? I think it would be nice if it worked more or less the same as the an IArchiver class, not only on the implementation side but also as far as the configuration is concerned. Although allowing multiple, potentially slow-answering, REST-based archivers could be a trap performance-wise (if those requests are not made concurrently). Florian ___ Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] new wiki, get-togethers, and 3.0 blockers
Am 02.01.2015 um 06:14 schrieb Sumana Harihareswara: * I'd love to get to hack with some of you in person. Are any of you planning on going to the PyCon sprints in Montreal April 13-16? Yup. * I saw in the https://mail.python.org/pipermail/mailman-developers/2014-November/024056.html conversation that I should be looking at https://bugs.launchpad.net/postorius/+bugs?field.tag=mailman3-suite-blocker for Mailman 3 blockers. It looks like there are 13 blocker bugs left; is that still right? Some of them already have patches attached to them and just haven't been merged yet, and a few others are already in progress. So there are currently 5 bugs with no one assigned to it: https://bugs.launchpad.net/postorius/+bug/987100 https://bugs.launchpad.net/postorius/+bug/1062963 https://bugs.launchpad.net/postorius/+bug/1004049 https://bugs.launchpad.net/postorius/+bug/1006356 https://bugs.launchpad.net/postorius/+bug/1201150 Would be great if any of these would spark your interest! :-) Florian ___ Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Python 3
Am 29.12.2014 um 00:27 schrieb Barry Warsaw: On Dec 27, 2014, at 08:41 AM, Florian Fuchs wrote: How about a third option, a generic pub/sub or event archiver, implemented in py3? It would meet all the above criteria (archivers on other systems, no dependency on py3 -- or Python for that matter). We could start supporting one backend (zeromq for instance) and maybe add more later. This would be pretty cool. Would you be interested in writing an IArchiver implementation for that? Sure. I don't know how much interest there is out there in something like this. But I think I like the idea enough to just do it anyway. :-) Florian ___ Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] MM3 REST API testing [was: Python 3]
My understanding is that VCR will capture the HTTP test request/responses and comparison to some baseline will indicate that the API is behaving as expected. Forgive me if I’m misunderstanding something fundamental about the VCR based testing approach. Some questions to helop me understand: Are the HTTP responses going to be the same between runs and between systems? How deep are the comparisons of the HTTP responses? Are you thinking of doing byte level comparisions of the entire response data, including headers etc? Often the contents of the HTTP headers vary for a whole bunch of reasons. Also if there is a proxy in the middle that can easily rewrite the whole request and response. I’m guessing a byte level comparison of the responses would fail. If it was the content of the responses being compared then that’s likely to work but that’s not really what VCR is about, right? Am I understanding the VCR approach correctly? as On 2 Jan 2015, at 3:40 pm, Barry Warsaw ba...@python.org wrote: On Dec 27, 2014, at 01:16 PM, Florian Fuchs wrote: So far we haven't found a perfect solution for testing packages that rely heavily on the MM3 REST API. As far as mailman.client, Postorius and HK are concerned, testing without *some* core integration doesn't make too much sense IMO, because any changes in the API would go unnoticed. I'm playing with a solution that involves the use of the 'vcr' package, which as its name implies (well, at least for us, um, mature programmers ;) is a facility to record and replay HTTP traffic. It's compatible with both Python 2 and 3 and is available on the Cheeseshop. So the basic idea is to set up an MM3 installation (`tox --notest -r` suffices), then start up its REST server with a fresh mailman.db, and run the test suite against it. This will capture the traffic in a yaml file, which we could ship with mailman.client. Then to run the test suite, you'd use vcr in replay mode against the captured yaml file. Any time the API changes, you'd have to re-record the traffic against a fresh database. I have a branch that's working in this direction and it's promising, but I've also had to rewrite the testing infrastructure for mailman.client, and make some other changes. It also points out what I think are some bugs in Mailman core, so I'm working those out on my py3 branch in parallel. It's not yet ready for review, but if you want to keep an eye on things, look here: lp:~barry/mailman.client/bilingual Feel free to provide feedback on the approach, or the changes so far, but I will submit a merge proposal when/if it's ready to go. Cheers, -Barry ___ Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40supercoders.com.au Security Policy: http://wiki.list.org/x/QIA9 ___ Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] MM3 REST API testing [was: Python 3]
I feel like prefacing all my questions with an apology if it’s a dumb question. Anyway... What’s the advantage of using VCR over a set of tests that use Kenneth Reitz requests library to send HTTP queries to the API and have a look at the response to see if it contains the expected HTTP codes/data? thanks On 2 Jan 2015, at 3:40 pm, Barry Warsaw ba...@python.org wrote: On Dec 27, 2014, at 01:16 PM, Florian Fuchs wrote: So far we haven't found a perfect solution for testing packages that rely heavily on the MM3 REST API. As far as mailman.client, Postorius and HK are concerned, testing without *some* core integration doesn't make too much sense IMO, because any changes in the API would go unnoticed. I'm playing with a solution that involves the use of the 'vcr' package, which as its name implies (well, at least for us, um, mature programmers ;) is a facility to record and replay HTTP traffic. It's compatible with both Python 2 and 3 and is available on the Cheeseshop. So the basic idea is to set up an MM3 installation (`tox --notest -r` suffices), then start up its REST server with a fresh mailman.db, and run the test suite against it. This will capture the traffic in a yaml file, which we could ship with mailman.client. Then to run the test suite, you'd use vcr in replay mode against the captured yaml file. Any time the API changes, you'd have to re-record the traffic against a fresh database. I have a branch that's working in this direction and it's promising, but I've also had to rewrite the testing infrastructure for mailman.client, and make some other changes. It also points out what I think are some bugs in Mailman core, so I'm working those out on my py3 branch in parallel. It's not yet ready for review, but if you want to keep an eye on things, look here: lp:~barry/mailman.client/bilingual Feel free to provide feedback on the approach, or the changes so far, but I will submit a merge proposal when/if it's ready to go. Cheers, -Barry ___ Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40supercoders.com.au Security Policy: http://wiki.list.org/x/QIA9 ___ Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
