But then it still makes it better on the Python side. You still have
psycopg memory problems but don't you have that now all the time anyways?
On Friday, July 26, 2013 11:50:57 AM UTC+2, Aymeric Augustin wrote:
>
> On 26 juil. 2013, at 09:40, julianb >
> wrote:
>
> > I don't think the queryset
On 26 juil. 2013, at 09:40, julianb wrote:
> I don't think the queryset needs to be loaded into memory. There is an
> iterator() method available:
I don't think .iterator() does what you expect. See
http://thebuild.com/presentations/unbreaking-django.pdf, slide 62 and 63. If
you're careful, m
I don't think the queryset needs to be loaded into memory. There is an
iterator() method available:
https://docs.djangoproject.com/en/dev/ref/models/querysets/#iterator
On Thursday, July 25, 2013 11:56:09 AM UTC+2, Aymeric Augustin wrote:
>
> Hi Julian,
>
> Thanks for the suggestion. This is an
> Ah, yes, I saw mindsocket working on this at PyConAU sprints, but wasn't
> sure how far through he'd got. The issues I've had with generator
> responses in the past is when there's an error in rendering, they tend to
> just stop.
>
I'm mindsocket :) The patch is ready for review insofar as I
On 26 July 2013 14:12, Roger Barnes wrote:
> If you do want to go down the streaming path despite the caching, static
> and other advice, I can see a couple of options:
> 1) Write your own sitemap view that doesn't use the sitemap.xml template
> and instead streams back results using a StreamingH
If you do want to go down the streaming path despite the caching, static
and other advice, I can see a couple of options:
1) Write your own sitemap view that doesn't use the sitemap.xml template
and instead streams back results using a StreamingHttpResponse and
generator based get_url methods in
We ended up splitting out sitemaps up... and caching them heavily.
Pre-generating periodically would be equally effective.
I can, however, see the merit in a generator style SiteMap engine with a
StreamingHttpResponse. It would allow the client to consume the data as
it's being consumed, instead
Hi Julian,
after having a site almost taken down by an over-eager spider downloading
the sitemaps dozens of times in parallel, I learned the lesson that large
sitemaps shouldn't be rendered on demand. Since that day a few years ago, I
always generate the sitemap periodically and let apache/nginx s
Hi Julian,
Thanks for the suggestion. This is an interesting idea, however, I'd like to
see evidence that the performance improvement is worth the extra complexity
before going forwards.
> Since 1.5 we have streaming responses. What is the state of contrib.sitemaps
> in this regard? I have som
Since 1.5 we have streaming responses. What is the state of
contrib.sitemaps in this regard? I have some very large sitemaps and
experimented with making them faster a few years ago.
If the do not yet stream, I think this would be a good idea to get memory
usage down. Is there anything to keep
10 matches
Mail list logo
