Re: Streaming sitemaps

[julianb]

But then it still makes it better on the Python side. You still have 
psycopg memory problems but don't you have that now all the time anyways?

On Friday, July 26, 2013 11:50:57 AM UTC+2, Aymeric Augustin wrote:
>
> On 26 juil. 2013, at 09:40, julianb <julia...@gmail.com > 
> wrote: 
>
> > I don't think the queryset needs to be loaded into memory. There is an 
> iterator() method available: 
>
> I don't think .iterator() does what you expect. See 
> http://thebuild.com/presentations/unbreaking-django.pdf, slide 62 and 63. 
> If you're careful, model instances will be garbage-collected, but you still 
> have the entire psycopg result set in memory. 
>
> To improve performance in this case you're probably better off with 
> .values() or .values_list(), or more likely by splitting your sitemap. 
>
> -- 
> Aymeric. 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Streaming sitemaps

[julianb]

I don't think the queryset needs to be loaded into memory. There is an 
iterator() method available:

https://docs.djangoproject.com/en/dev/ref/models/querysets/#iterator

On Thursday, July 25, 2013 11:56:09 AM UTC+2, Aymeric Augustin wrote:
>
> Hi Julian, 
>
> Thanks for the suggestion. This is an interesting idea, however, I'd like 
> to see evidence that the performance improvement is worth the extra 
> complexity before going forwards. 
>
> > Since 1.5 we have streaming responses. What is the state of 
> contrib.sitemaps in this regard? I have some very large sitemaps and 
> experimented with making them faster a few years ago. 
>
> And what were the results of this experiment? 
>
> > If the do not yet stream, I think this would be a good idea to get 
> memory usage down. Is there anything to keep an eye on? Would it be 
> valuable to Django if this is looked into? 
>
> Large sitemaps are usually generated from a queryset, and the queryset 
> will be loaded in memory as soon as the first item is accessed. Streaming 
> the sitemap won't help at this level. 
>
> -- 
> Aymeric. 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Streaming sitemaps

[julianb]

Since 1.5 we have streaming responses. What is the state of 
contrib.sitemaps in this regard? I have some very large sitemaps and 
experimented with making them faster a few years ago.

If the do not yet stream, I think this would be a good idea to get memory 
usage down. Is there anything to keep an eye on? Would it be valuable to 
Django if this is looked into?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.

Re: URL dispatcher fallthrough?

[julianb]

A single pattern with a catchall view is certainly a way to go. However, I 
really does not help to see what the views do. And if you have different 
views with different arguments, you cannot do your regex explicitly.
You are right that it is non-obvious when the view itself has to invoke the 
fallthrough. So I'm thinking it would be better in the urlpatterns itself. 
Then it is explicitly clear, rather than having some catchall magic do-all 
views and patterns. Right?

On Monday, March 18, 2013 4:44:19 PM UTC+1, Andrew Ingram wrote:
>
> The approach we take at work is to have a view that wraps around the 
> product, category and occasion views and takes care of the fallthrough 
> logic. I'm not a fan of this approach, because it means that you can't just 
> look at the urlconf and see which pattern maps to which view function. On 
> the other hand, I don't think your solution would solve this either, you'd 
> end up having 3 patterns that could all match, making it non-obvious what 
> the behaviour is.
>
> Obviously the ideal solution is to scope URLs correctly to avoid all this 
> in the first place, but I've found that in the real world this is easier 
> said than done. So it would be nice to have an idiomatic solution.
>
> An alternative might be a kwarg on the url function, such as 
> 'fallthrough_on_404'. But it all feels a bit hackish to me :/
>
> Regards,
> Andy
>
>
> On 18 March 2013 15:23, julianb <julia...@gmail.com > wrote:
>
>> Hi,
>>
>> imagine the following use case:
>>
>> You build an online store where you have sorted products into several 
>> categories and maybe associated an occasion. Now you want to build URLs. So 
>> the URL schema that all of the store's owners agree on is:
>>
>> //
>> //
>> //
>>
>> Look simple.
>>
>> Because product slugs should generally be different to category slugs and 
>> occasions, you expect no clashes here. And you don't want to bloat the URLs 
>> by putting product/ and category/ in there. And you also do not want to 
>> resort making it look cryptic by having just /p// and so on.
>> Then it comes to Django. How do you do that?
>>
>> Well, at the moment, as far as I am aware, you can't. The first URL will 
>> match everything all the time, not giving the other views a chance to kick 
>> in.
>>
>> So I propose some kind of URL fallthrough. The view could do
>>
>> raise UrlNotMatched
>>
>> and the dispatcher goes to the next possible matching view.
>>
>> Would that be good? Any thoughts?
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to 
>> django-d...@googlegroups.com
>> .
>> Visit this group at 
>> http://groups.google.com/group/django-developers?hl=en.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>  
>>  
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

URL dispatcher fallthrough?

[julianb]

Hi,

imagine the following use case:

You build an online store where you have sorted products into several 
categories and maybe associated an occasion. Now you want to build URLs. So 
the URL schema that all of the store's owners agree on is:

//
//
//

Look simple.

Because product slugs should generally be different to category slugs and 
occasions, you expect no clashes here. And you don't want to bloat the URLs 
by putting product/ and category/ in there. And you also do not want to 
resort making it look cryptic by having just /p// and so on.
Then it comes to Django. How do you do that?

Well, at the moment, as far as I am aware, you can't. The first URL will 
match everything all the time, not giving the other views a chance to kick 
in.

So I propose some kind of URL fallthrough. The view could do

raise UrlNotMatched

and the dispatcher goes to the next possible matching view.

Would that be good? Any thoughts?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: #717 (If-Modified-Since checked for exact match) should be fixed

[julianb]

Uuh, so what's with this? Nothing happened with the ticket and no
further comments here either... :(

On Aug 9, 12:28 pm, julianb <julian@gmail.com> wrote:
> #717 was posted 5 years ago and closed as wontfix, supposedly cause
> Django was in it's early stages and it was not an important issue. I
> propose that it's getting reopened and fixed now.
>
> Like the ticket description says, in ConditionalGetMiddleWare the Last-
> Modified date is checked for exact match with If-Modified-Since and
> only then a 'Not Modified' response returned. It should however check
> if Last-Modified <= If-Modified-Since. This behavior is specified 
> inhttp://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html(14.25 c)
>
> Currently a problem is e.g. Google send an If-Modified-Since header
> that is not equal to the Last-Modified, often newer in my case. So in
> the end, the resource got modified far earlier than Google requests,
> Google has the latest version, but Django still sends the full content
> instead of 304.
>
> A patch is already attached and it's working on my production server.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: #717 (If-Modified-Since checked for exact match) should be fixed

[julianb]

I just discovered that the condition decorator does check for exact
match only, too. Thus, the ticket should be changed to also include
this issue. Is someone going to reopen the ticket?

On Aug 9, 11:46 pm, Łukasz Rekucki  wrote:
> On 9 August 2010 22:48, Paul McMillan  wrote:> I agree with 
> the person who closed the ticket again, since this should
> > have been discussed on the mailing list prior to re-opening it.
>
> That would be me ;)
>
>
>
> > That said, I'm strongly +1 on this issue. I've had to write
> > workarounds for exactly the described behavior on other systems, and
> > it hasn't been good. Django should follow the RFC and do the sane,
> > rational thing. There are a myriad of use cases these days for
> > retreiving an object without storing the precise previous retrieval
> > time.
>
> The "follow RFC" argument is not that strong. It actually advises
> clients to use the exact date string that was previously returned by
> server if possible. It also notes a few problems when using arbitrary
> dates. A valid use case is far more convincing. If Google's crawler is
> causing problems, it's probably worth fixing.
>
> As for the patch, it only parses one of three date formats required by
> the RFC[1]. So if you want to strictly follow the RFC, it must parse
> all three.
>
> Just my 2 cents.
>
> [1]:http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.3.1
>
> --
> Lucas Rekucki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: #717 (If-Modified-Since checked for exact match) should be fixed

[julianb]

On Aug 9, 11:46 pm, Łukasz Rekucki  wrote:
> On 9 August 2010 22:48, Paul McMillan  wrote:> I agree with 
> the person who closed the ticket again, since this should
> > have been discussed on the mailing list prior to re-opening it.
>
> That would be me ;)

Okay, I'm sorry. I somehow forgot the rules ;)

> As for the patch, it only parses one of three date formats required by
> the RFC[1]. So if you want to strictly follow the RFC, it must parse
> all three.

Are we doing such parsing anywhere already, so I could update the
patch?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

#717 (If-Modified-Since checked for exact match) should be fixed

[julianb]

#717 was posted 5 years ago and closed as wontfix, supposedly cause
Django was in it's early stages and it was not an important issue. I
propose that it's getting reopened and fixed now.

Like the ticket description says, in ConditionalGetMiddleWare the Last-
Modified date is checked for exact match with If-Modified-Since and
only then a 'Not Modified' response returned. It should however check
if Last-Modified <= If-Modified-Since. This behavior is specified in
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html (14.25 c)

Currently a problem is e.g. Google send an If-Modified-Since header
that is not equal to the Last-Modified, often newer in my case. So in
the end, the resource got modified far earlier than Google requests,
Google has the latest version, but Django still sends the full content
instead of 304.

A patch is already attached and it's working on my production server.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Decision for ticket #6362 - Remove blank spaces with strip when validating the data

[julianb]

On May 13, 2:56 am, Russell Keith-Magee 
wrote:
> We need to wait until we're not trying to get v1.1 out the door.

Since the ticket is one year old, that time had come and passed.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

URL mapping and percent-encoded reserved characters

[julianb]

Hi,
currently it seems that the URL dispatcher makes no difference between
e.g. "/" and "%2F", which is the same if you decode it but has special
meaning in URLs. The "/" separates path segments and you would have to
encode it to "%2F" if used in such a segment. (http://en.wikipedia.org/
wiki/Percent-encode#Percent-encoding_reserved_characters)
Is that intended? I mean, it messes up things when you have paths that
are segmented with "/" and the segments can have _everything_ in them
(encoded of course)...
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Search facility for new documentation

[julianb]

On Aug 31, 12:07 am, "Ramiro Morales" <[EMAIL PROTECTED]> wrote:
> From what i?ve heard I'd say facilities offered by Sphinx
> (seehttp://sphinx.pocoo.org/markup/para.html#dir-versionadded)
> are going to be used for this.

That would be good then. Currently it is written directly as text,
which seems a bit inconvenient.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Search facility for new documentation

[julianb]

On Aug 30, 9:31 pm, "Jacob Kaplan-Moss" <[EMAIL PROTECTED]>
wrote:
> The latest docs will always be at //dev/; versions frozen with
> each release will be at ///.

On this occasion, why not change "New in Django development version"
remarks to "New in this version"? That makes much more sense, since
the docs get frozen for each version, so you know now and later this
was new in dev/1.0/1.1...
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Call for testing: new docs

[julianb]

I have just a tiny suggestions... could you make the headlines a bit
bigger or with more space around them? When scrolling, there is no
real distinction between different parts. I always found that a bit
unusable in the Django docs. You just have a large stream of text and
your eyes are unable to scan the parts of a document.
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---