Re: Streaming sitemaps

2013-07-29 Thread julianb
But then it still makes it better on the Python side. You still have psycopg memory problems but don't you have that now all the time anyways? On Friday, July 26, 2013 11:50:57 AM UTC+2, Aymeric Augustin wrote: > > On 26 juil. 2013, at 09:40, julianb > wrote: > > > I

Re: Streaming sitemaps

2013-07-26 Thread Aymeric Augustin
On 26 juil. 2013, at 09:40, julianb wrote: > I don't think the queryset needs to be loaded into memory. There is an > iterator() method available: I don't think .iterator() does what you expect. See http://thebuild.com/presentations/unbreaking-django.pdf, slide 62 and

Re: Streaming sitemaps

2013-07-26 Thread julianb
I don't think the queryset needs to be loaded into memory. There is an iterator() method available: https://docs.djangoproject.com/en/dev/ref/models/querysets/#iterator On Thursday, July 25, 2013 11:56:09 AM UTC+2, Aymeric Augustin wrote: > > Hi Julian, > > Thanks for the suggestion. This is

Re: Streaming sitemaps

2013-07-25 Thread Roger Barnes
> Ah, yes, I saw mindsocket working on this at PyConAU sprints, but wasn't > sure how far through he'd got. The issues I've had with generator > responses in the past is when there's an error in rendering, they tend to > just stop. > I'm mindsocket :) The patch is ready for review insofar as I

Re: Streaming sitemaps

2013-07-25 Thread Curtis Maloney
On 26 July 2013 14:12, Roger Barnes wrote: > If you do want to go down the streaming path despite the caching, static > and other advice, I can see a couple of options: > 1) Write your own sitemap view that doesn't use the sitemap.xml template > and instead streams back

Re: Streaming sitemaps

2013-07-25 Thread Roger Barnes
If you do want to go down the streaming path despite the caching, static and other advice, I can see a couple of options: 1) Write your own sitemap view that doesn't use the sitemap.xml template and instead streams back results using a StreamingHttpResponse and generator based get_url methods

Re: Streaming sitemaps

2013-07-25 Thread Curtis Maloney
We ended up splitting out sitemaps up... and caching them heavily. Pre-generating periodically would be equally effective. I can, however, see the merit in a generator style SiteMap engine with a StreamingHttpResponse. It would allow the client to consume the data as it's being consumed,

Re: Streaming sitemaps

2013-07-25 Thread Benjamin Wohlwend
Hi Julian, after having a site almost taken down by an over-eager spider downloading the sitemaps dozens of times in parallel, I learned the lesson that large sitemaps shouldn't be rendered on demand. Since that day a few years ago, I always generate the sitemap periodically and let apache/nginx

Re: Streaming sitemaps

2013-07-25 Thread Aymeric Augustin
Hi Julian, Thanks for the suggestion. This is an interesting idea, however, I'd like to see evidence that the performance improvement is worth the extra complexity before going forwards. > Since 1.5 we have streaming responses. What is the state of contrib.sitemaps > in this regard? I have

Streaming sitemaps

2013-07-25 Thread julianb
Since 1.5 we have streaming responses. What is the state of contrib.sitemaps in this regard? I have some very large sitemaps and experimented with making them faster a few years ago. If the do not yet stream, I think this would be a good idea to get memory usage down. Is there anything to keep