On Mon, 25 May 2015 21:33:21 -0000, PyTonic <pyto...@i2pmail.org> wrote: > On 05/25/2015 07:04 PM, PyTonic wrote: > > I will post another message containing a simple use case for the new > > interface which only streams, decodes and stores base64 encoded > > payloads on the fly and uses the old method for everything else. It > > additionally uses two more callables inside its Message subclass: > > get_payload_file() and is_streamed(). > > > > It also contains some comments about unresolved issues like how > > decoding errors should be properly dealt with. And who is responsible > > for catching exceptions raised by the new interfaces so they can't > > break the FeedParser itself.
It is great that you are interested in working on this. Providing a way to process large emails without the current crazy memory consumption is a goal of mine, and we'll happily work with you toward making that a reality. However, anything along these lines is going to be a new feature, and therefore can only target 3.6 at this point, so any patch proposals need to be against the default branch of the cpython repository. In Python3 we now have the policy framework. I'm pretty sure it makes sense to leverage that for the new internal API. I agree that feedparser itself will need some changes in order to make all this work correctly. (Also, feedparser has gotten a couple of non-trivial performance enhancements in python3, so some of the code is different.) Note that the generator needs similar changes, and that problem may be much harder to solve, since the current algorithm is recursive and holds *everything* in memory. In addition, it seems like this would be a natural (even necessary?) place to introduce the 'store bodies on disk' interface that we've wanted for quite a while. Can you take a look at the policy framework and reformulate your proposal in light of that? We can certainly work on this one piece at a time, we just want to keep all of the moving parts in mind while we do so... --David _______________________________________________ Email-SIG mailing list Email-SIG@python.org Your options: https://mail.python.org/mailman/options/email-sig/archive%40mail-archive.com
