Re: Ideas for a new DEP on long-term Django API compatibility

[Patryk Zawadzki]

As someone who has spent over 12 years working with Django, I'd like to 
offer an opposite point of view:

I love where Django is headed. I love all of those breaking changes that 
have to happen so we're not perpetually stuck with decisions from 2005.

What I truly miss is strong static typing support that would tell that when 
my code is not going to work before I even begin to test it. With static 
types and tools like mypy and pyright upgrading dependencies is usually a 
matter of a single afternoon.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f9eaf95a-8611-4715-a4bf-6aed7a284b3f%40googlegroups.com.

Re: Django LTS support time

[Patryk Zawadzki]

W dniu sobota, 10 sierpnia 2019 18:19:07 UTC+2 użytkownik Uri napisał:
>
> Thanks for your feedback. Eventually I found out that the Django Crispy 
> Forms issue was a CSS bug in our CSS code. Anyway not related to Crispy 
> Forms, it may take a lot of time and effort to upgrade Django for us, and I 
> would prefer to keep using Django 1.11 as much as we can.
>

If upgrading looks like a lot of work today, it will be even more work in a 
year or two. Today a single dependency may be blocking you from the 
upgrade. Unless you have a contract with that dependency's maintainer, 
there's no guarantee of the situation ever improving. Soon you may not be 
able to add a dependency because it will require a newer version of Django. 
Anecdotal evidence: we have an internal policy of not supporting Django 
versions older than the current LTS unless we have someone paying for the 
effort.

As for the LTS policy: I don't think the current one cuts it. There are 
companies who are fine with upgrading once every two years but these 
companies could probably also do it every release or two. Then there are 
companies who want LTS support and these are often projects that need to 
run largely unmodified for five to ten years. I don't think the current LTS 
support helps them in any way. I think a better option (for both sides) 
would be if the foundation offered paid support for the releases nominated 
as LTS and if that support became more expensive every year (to compensate 
for having to maintain outdated software and to motivate the company to 
eventually allocate the budget to a proper upgrade). Paid support could 
also include features like early vulnerability disclosure and instructions 
for determining whether an issue was exploitable in a particular project.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c4fd117b-830e-4274-b563-902973316344%40googlegroups.com.

Re: Django Async DEP

[Patryk Zawadzki]

>
> We've always considered that implicit queries on attribute access were an 
> intractable problem. I said it on stage an DjangoCon US 2013. I'm now 
> wondering if I was wrong all along! In an async ORM context, every time we 
> traverse a relation, we could create a Future that would execute the query 
> and resolve to its result. This would require one await per SQL query so 
> we'd still get the benefit of making queries explicit, although to a lesser 
> extent than with an explicit select/prefetch_related.
>

In theory there's nothing stopping you from turning prefetch_related int on 
explicit fetch_related:

queryset = Foo.objects.filter(Foo.publish_date <= datetime.date.today())

queryset = fetch_related(queryset[:10], Foo.author)  # with a future

results = await queryset.all()

fetch_related(results, Foo.category)  # with a result set, returns identity

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6c2d073d-fa21-43f2-94aa-db12c0f817da%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Patryk Zawadzki]

>
> I don't think it makes a significant difference from a readability 
> perspective in this example. It does have some advantages, though: 
>
> - It could be easier to implement a sync version based on the async one, 
> or vice-versa, if each one has its own class. It will probably be more DRY. 
> - It could also be a bit more usable by developers, especially in IDEs 
>

Speaking about convenience, it would be great if we could make the sync 
versions raise exceptions when used with an event loop attached and the 
async ones fail when there isn't a loop running. I assume synchronous views 
would be executed within a worker thread with the event loop explicitly set 
to None. 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5b6222c3-9f76-4eb3-8619-e3f44a17d6a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Patryk Zawadzki]

W dniu czwartek, 9 maja 2019 22:47:48 UTC+2 użytkownik J. Pic napisał:
>
> I'm a bit confused here, what benefit are you getting from async emails if 
> you're already retrying emails in the background in production ?
>

Anything that uses I/O should be async to unblock the worker to process 
other things while for example waiting for an SMTP server to return. 
Network I/O like that often requires tens to hundreds of milliseconds, in 
this time you could process another request from another browser.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2ce12106-b423-4dc1-a95a-56f112e63a0f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Patryk Zawadzki]

>
> I am also not a fan of the approach, but I did err towards being explicit. 
> Jannis suggested what I think might be a nicer approach on Twitter, which 
> is to add an async "proxy object" to access methods with, e.g.:
>
> cache.get("foo")
> cache.async.get("foo")
>
> This is still explicit but looks less ugly, and gives us a way in future 
> to make some modules transition to async-by-default (by e.g. supplying 
> cache.sync.get). What do you think?
>

I think I'd personally prefer to have to do `from django.asyncore.cache 
import cache` and if the synchronous cache raised an exception when 
attempting to use it with an event loop active. Like Aymeric, I don't see 
myself mixing async and sync code within one module.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d2c879cb-fad7-4e56-a1d9-1a31d5134e96%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Patryk Zawadzki]

>
> That said, I also think it's important to allow the ORM to support both 
> modes in the long term. I truly believe the best way to be able to write 
> async code is to _have the choice to write it_, rather than being made to 
> all the time; if we make people use a separate, async ORM, then we force 
> them to write every view asynchronously, with all the extra danger and 
> thinking that requires. It's much better for Django to do the hard work, 
> and say "hey, if you want to write asynchronously or synchronously, that's 
> fine - it takes literally zero extra effort to go either way".
>

Slightly off-topic but once we have an async ORM, making it synchronous is 
not impossible (I believe either Channels or Daphne already have shims that 
use a worker thread to spin the event loop until a future is fulfilled).

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/402dbe77-44fc-485b-b058-e619a1576ddb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Patryk Zawadzki]

>
> I'm not sure but for me the "What is Django" section answers the question. 
> For me Django is full of philosophy that seeds a great ecosystem of apps of 
> all sorts with a growing user base nonetheless, and a bunch of brilliant 
> hackers to look up to and inspire for more. Of course if you're into fixing 
> a particular technical issue such as "I want to make an efficient GraphQL 
> server", then even Django works even if it's not necessarily the most fit 
> for X reason. But for "I want to get a web page going with some forms and a 
> database and see where that goes" then Django is definitely enjoyable, and 
> at this point in time where tons of more sophisticated frameworks are born 
> it's clear that Django is still in the game and will be for the next 10 
> years. So, why not also get more features out of it while we've not been 
> abandoned by all the talented contributors ? It's not because we like 
> Django that we cant haz nice things ;)
>

I'm arguing the opposite, that by limiting the scope of the MVP to 
asynchronous views (even if middleware stays synchronous for now) and using 
an existing ORM we can make Django a viable solution for many new 
applications. And some of them don't need the full stack.

I'd go as far as arguing that fewer and fewer applications benefit from 
Django providing all of the pieces (as many new applications only implement 
an API and render in the client) so treating them all as blockers here may 
not be beneficial.

Best regards

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/314d7b71-1bf1-441b-ae49-e4fbeb5e1891%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Patryk Zawadzki]

Hey Andrew,

Great work on the DEP, the task at hand is humongous.

Do you think it's worth it to try and make the ORM async? It contains tons 
of magic that is inherently incompatible with explicit I/O that is required 
for async/await to work, things like silently fetching relations on first 
access, silently fetching deferred fields, queryset laziness etc. Those 
need to be explicitly awaited now and having to await regular field access 
for the sake of making deferred fields work is (I believe) not worth it.

Conveniently, all of these magic properties are things that silently break 
performance of applications leading to the n+1 queries problem so I'm more 
than happy to get rid of them ;)

Seeing that we can't get rid of anything in the old synchronous ORM, one 
idea would be to just depend on a separate, asyncio-based ORM and configure 
it to use the same tables and relations that the old ORM uses. It could 
work like model forms or formset factories, give it a "classic" model 
class, a list of overrides (we can't hope to automatically match old field 
types to anything understood by another engine) and receive an 
asyncio-capable table. There's an obvious candidate that Tom Christie is 
working on.

We could even improve the querying syntax to use the SQL Alchemy's 
filtering operators with the hope of eventually reaching full static type 
safety[2] (one can only dream).

The new ORM would only be usable in new async views and other features 
could be implemented gradually. You don't need to async forms (but with Tom 
Christie's typesystem[3] you don't really need async forms) and async 
template loading to achieve great things.

You can already use things like requests-async[4] to access external 
resources, you can use libraries like Ariadne[5] to implement a GraphQL API 
with WebSocket-based subscriptions without having any of those (disclaimer: 
I work on Ariadne). It's okay if some things can't be ported initially or 
if certain features are only available in their sync flavor. The community 
will pave the way and provide many missing pieces before we get to 
implementing them. It's okay if new async counterparts have different APIs 
if these APIs are either better or are a closer match to how async/await 
works.

[1] https://github.com/encode/orm
[2] https://docs.sqlalchemy.org/en/13/core/tutorial.html#operators
[3] https://github.com/encode/typesystem
[4] https://github.com/encode/requests-async
[5] https://github.com/mirumee/ariadne


PS: If Django provided abstractions for common event loop operations such 
as "await this collections of futures" and "ensure this task is executed 
even if not explicitly awaited" then I think it may be possible to avoid 
explicitly depending on asyncio.

All the best

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/da0d4c65-562b-44cc-abe0-e5c31e28d91d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: BitBounce Spam Replies From the Mailing List

[Patryk Zawadzki]

Sorry for resurrecting but this is still very much a problem. Same person, 
same autoresponder.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/41722075-8e89-4777-8872-ee4660f14b26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal to re-open #27017 (updating only dirty fields in save())

[Patryk Zawadzki]

s worth measuring the latter.
>
> Compatibility
>
> It was observed that the two approaches considered on the ticket
> (overriding __setattr__ or defining custom property setters) would not
> work with obj.__dict__.update
> <https://code.djangoproject.com/ticket/4102#comment:12>, which is
> apparently an optimization you can find prescribed on some blogs out int he
> wild. This supports the premise that this behavior should be opt in, so
> devs who are using incompatible techniques can stay away (unless/until
> they've removed those optimizations from their code).
>
> I had already suggested putting this functionality behind a setting, so I
> think we're good here.
>
> Correctness
>
> There were some bugs related to
> <https://code.djangoproject.com/ticket/4102#comment:39> multi-table
> inheritance <https://code.djangoproject.com/ticket/4102#comment:67>. It
> appears these were both followed by patches with proposed fixes.
>
> Another bug was that fields updated in pre_save such as
> DateTimeField(auto_now=True) were not being updated
> <https://code.djangoproject.com/ticket/4102#comment:60>. There was a
> patch to fix this as well.
>
> The point was raised that this approach poses a problem for mutable field
> types <https://code.djangoproject.com/ticket/4102#comment:68> (e.g.,
> ArrayField). Presumably one approach for addressing that would be the one
> employed by django-save-the-change here
> <https://github.com/karanlyons/django-save-the-change/blob/master/save_the_change/decorators.py#L121>
> .
>
> One gets the sense from following the thread that, surely, there would be
> more bugs discovered were someone to resurrect this work. However, again, I
> don't see any showstoppers here. We're talking about an advanced framework
> with a large set of features, and so naturally it's going to be difficult
> to get this right while taking all supported use cases into account.
>
> Conclusion
>
> As I said up top, I think it's worth re-opening #27017, or creating a new
> ticket for the behavior it was requesting (updating only dirty fields
> automatically in save()) where future contributors might discuss possible
> approaches and, with any luck, submit pull requests. I believe it would be
> both possible and helpful, and I also *suspect* that the process would go
> a bit more smoothly today, as the project has a huge suite of tests to
> catch possible edge cases and provide greater confidence about the
> correctness of any patches submitted.
>
> P.S. I'm a little worried that someone will now direct me to *another*
> long thread I should read filled with even more reasons this idea caused
> problems in the past. If so, no worries! I'll do it. I've already come this
> far :)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/f13f860e-8dc9-4a5d-80c5-f1201e741a9d%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/f13f860e-8dc9-4a5d-80c5-f1201e741a9d%40googlegroups.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANw2pUGuMXs1RmqAHZjaoHuTpNKzVnWRi68bioZegJxkv_QL9Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving Backward Compatibility

[Patryk Zawadzki]

W dniu czwartek, 7 grudnia 2017 08:12:52 UTC+1 użytkownik Aymeric Augustin 
napisał:
>
> The duration of the deprecation period doesn't make a big difference 
> because few people update their code until they have no other choice. 
> Keeping deprecated code around longer means it's harder to track down the 
> release where the corresponding changes were introduced and figure out what 
> changes are needed. This effect already occurs in Django 2.0 where some 
> removals are related to 1.9 and others to 1.10.
>

We've adopted the policy of only supporting two versions of Django in our 
open-source stack: the latest LTS and the most recent version. The 
rationale is that making the window longer does not really benefit anyone 
in the long run.

We as authors of code have to maintain a ton of separate code paths guarded 
by conditional checks based on versions. Each class method needs to be 
annotated with when it's safe to remove it. Reasoning about the code is 
harder and adding new features is tricky when you discover that there is no 
way to support them cleanly on Django 1.8. None of our projects use 1.8 so 
by supporting it we'd be effectively doing extra work for no benefit.

Library users don't benefit in the long term as giving them a longer window 
gives them more time at the exchange of making the eventual upgrade a 
nightmare. They either have to upgrade from 1.8 straight to 2.1 (I can't 
honestly recommend this to anyone as a responsible way to maintain 
software) or upgrade to 1.11 and effectively reduce their situation to the 
current one (having to upgrade every time a new LTS is released instead of 
every two LTS cycles).

Just my two cents.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/4670df99-e7f8-4b09-a24f-909b089f9798%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Templates: __html__ method support

[Patryk Zawadzki]

W dniu wtorek, 26 września 2017 14:34:29 UTC+2 użytkownik Jonas H napisał:
>
> Proposal: Support the __html__ method as an alternative/addition to the 
> __str__ for turning objects into strings in the template layer.
>
> If this has been discussed before, please point me to it; I couldn't find 
> anything with the search function.
>
> Some custom classes may have, in addition to a __str__ representation, a 
> natural representation that is better suited for HTML output.
>

Another proposal:

Introduce a single-dispatch html function (
https://docs.python.org/3/library/functools.html#functools.singledispatch).

Have the default implementation call param.__html__() if it exists and fall 
back to param.__str__().

Document using html.register(...) as the preferred extension method.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/429010b7-7606-4f57-b4ca-c449005dfa32%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Having a MongoDB connector for Django

[Patryk Zawadzki]

W dniu poniedziałek, 11 września 2017 05:28:55 UTC+2 użytkownik Nes Dis 
napisał:
>
> Thank you all for your very insightful comments. I personally am a big 
> user/contributor to the framework myself and would like to see it thrive 
> and progress with respect to other competing frameworks.
>
> I am sure most are aware, of this argument about MongoDB increasing in 
> popularity. Several members are of the opinion that not supporting this 
> backend (which in my opinion is not too difficult) will not dent Django's 
> popularity. 
>

This is my personal opinion so take with a grain of salt but I believe 
MongoDB is popular among two very different groups of people:

1) those who think convenience of project setup (ie. I can bootstrap my 
project within 15 minutes) and believe that going schema-less is saving 
them time

2) those who deal with terabytes upon terabytes of unstructured data

During my career I've been part of both groups.

All projects that fell under group 1 had to eventually migrate to SQL. 
MongoDB is easy to setup and convenient to use for proof-of-concept toys 
but it was built with a very specific goal in mind: to handle data at 
SCALE. This means there are no transactions (you can't lock data when 
transactions can take hours to complete), there is no atomicity (all 
updates are distributed and row updates happen in parallel, you can easily 
end up with partial successes where some rows are updated and some raise 
errors, there's no ghost read protection as all updates are final), there 
is also no way for the database to reject invalid data (as there is no 
schema enforcement, the only rule is: junk in; junk out). Whatever 
resources you save by initially choosing MongoDB you pay for ten-fold each 
time you encounter a problem that is easily solvable in the ACID RDBMS 
world and that just does not exist in group 2 projects.

All projects that fell under group 2 had to build their logic with 
MongoDB's architecture in mind. To take advantage of the scalability you 
need to carefully craft all of your commands. You only select data that you 
know you'll absolutely need (which means that you do actually have multiple 
implicit schemas or "interfaces" certain rows conform to), you maintain a 
fleet of background jobs that recalculate certain denormalizations to keep 
them in approximate sync with data (as updates can and do happen in 
parallel and recalculating aggregate queries on terabytes of data can be 
prohibitively costly), and you do complex updates by sending a JavaScript 
function along with the query instead of fetching millions of rows and 
issuing millions of update commands. There would be no advantage for us to 
be able to use a ModelForm to manage schema-less documents.

Cheers

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8166d971-5e77-42f5-b5db-27d9df7b8778%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Automatic prefetching in querysets

[Patryk Zawadzki]

W dniu środa, 16 sierpnia 2017 14:48:54 UTC+2 użytkownik Luke Plant napisał:
>
> I completely agree that visibility of this problem is a major issue, and 
> would really welcome work on improving this, especially in DEBUG mode. One 
> option might be a method that replaces lazy loads with exceptions. This 
> would force you to add the required `prefetch_related` or `select_related` 
> calls. You could have:
>
> .lazy_load(None)   # exception on accessing any non-loaded FK objects
>
> .lazy_load('my_fk1', 'my_fk2')  # exceptions on accessing any unloaded FK 
> objects apart from the named ones
>
> .lazy_load('__any__')   # cancel the above, restore default behaviour
>
> This (or something like it) would be a different way to tackle the problem 
> - just throwing it out. You could have a Meta option to do 
> `.lazy_load(None)` by default for a Model.  I've no idea how practical it 
> would be to wire this all up so that is works correctly, and with nested 
> objects etc.
>

Would love if Django could be told to require explicit data fetches. This 
and replacing QuerySet.__iter__ with an explicit fetch() (and lazy_fetch()) 
would make it much easier to reason about large codebases.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1fa90611-66cc-4d06-992c-55164913679d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: About migrations

[Patryk Zawadzki]

W dniu piątek, 7 lipca 2017 23:54:07 UTC+2 użytkownik Andrew Godwin napisał:
>
> There is already a run-before constraint you can add to migrations for 
> exactly this purpose! It's called "run_before" and is in the same format as 
> the dependencies IIRC.
>

The problem with "run before X" is that there is no "X" at the point in 
time where you write that migration.

I think it would be more robust if Django tried to run migrations _from 
other apps_ that depend on the currently executing migration before 
proceeding to run the next migration from the same app.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8e89de6a-3a4b-4a53-8455-b1b62ca293fd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: About migrations

[Patryk Zawadzki]

W dniu wtorek, 4 lipca 2017 23:49:54 UTC+2 użytkownik Carl Meyer napisał:
>
> On 07/04/2017 07:04 AM, Patryk Zawadzki wrote: 
> > Have DB backends understand certain field types expressed as strings 
> > ("varchar", "text", "blob", "decimal" and so on). 
> > 
> > Possibly some backends could implement a wider set than the others 
> > ("json", "xml", "rasterimage" etc.). 
> > 
> > Have each Field class deconstruct to a field name and params, eg: 
> > "decimal", {"digits": 12, "decimals": 2}. 
> > 
> > Then a model becomes essentially a list of tuples: 
> > 
> > [ 
> > ("title", "varchar", {"length": 100}), 
> > ("price", "decimal", {"digits": 12, "decimals": 2}), 
> > ... 
> > ] 
> > 
> > This is not far from what "render model states" does currently except 
> > that it compares much richer model descriptions that leads to no-op 
> > migrations being generated each time you change a label or a 
> > user-visible part of choices. 
>
> Right, and one reason for generating those "no-op" migrations is that 
> they aren't actually no-ops, if you value being able to write data 
> migrations in Python using the ORM. They keep the historical Python 
> models accurate.
>

I would argue that this is a fairly optimistic view of the current state :)

They are technically "historically accurate" but the point in history they 
represent is not necessary the one you had in mind unless you only have a 
single application and linear migrations (ie. no merge migrations). Our 
current dependency system only allows you to express "no sooner than X" but 
the graph solver can execute an arbitrary number of later migrations 
between the one you depend on and the one you wrote.

Imagine you have app A and migration M1 adds field F. You then create a 
migration M2 in another application B that needs to access F so you have it 
depend on (A, M1). Two months later field F is removed or renamed in 
migration M3. Django has two ways to linearize the graph: (A, M1), (B, M2), 
(A, M3) or (A, M1), (A, M3), (B, M2). Both are valid but the latter will 
result in a crash when migrating from an empty DB state. In practice we 
often have to add arbitrary dependencies to later migrations to force a 
Python migration to execute in the correct order.

Also I'd argue that (from my personal experience which is obviously 
limited) having access to historical "choices", a form field label or the 
hint are not all that useful. In fact I'd be happy with a limited migration 
system that always returns bare database values without executing any of 
the field code. Writing portable migrations would be a bit more work but 
it's mostly a price apps would pay as projects themselves rarely need to be 
portable.

Anyway, I don't want anyone to think that I complain as I don't have the 
resources to write yet another migration tool and both South and Django 
migrations beat writing SQL by hand.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ea0381de-d710-4467-8e4f-9d371f3154e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: About migrations

[Patryk Zawadzki]

W dniu piątek, 23 czerwca 2017 21:28:07 UTC+2 użytkownik Andrew Godwin 
napisał:
>
>
>>
>> The advantages comes from db type independency, this is true, but in the 
>> other side you're including the app layer dependency. 
>>
>> Let's imagine that one of builtin field will change it's definition. 
>> Running migrations on two different Django versions will produce two 
>> different outputs.
>> My perspective is more database-like than app-like, so I'm expecting same 
>> db schema as a result (for both cases).
>>
>> So the first thing that comes into my mind sounds: a complete definiton 
>> should be baked in migration file. Then, when app layer changes (i.e. 
>> upgrading framework or changing custom field definition), the migration 
>> system should identify the change and produce new migration with baked in 
>> definition. If it is possible to develop, you'll achieve less dependencies. 
>> The definition (a meta-description of the field) will be baked in, instead 
>> of depending on the field itself. And you'll preserve database type 
>> independency.
>>
>
> How do you propose to identify "when the app layer changes"? This is a 
> harder problem to solve that it first appears; the only thing you can rely 
> on to compare to are the migration files themselves, so that necessarily 
> means you need some description of the app layer in there.
>

Have DB backends understand certain field types expressed as strings 
("varchar", "text", "blob", "decimal" and so on).

Possibly some backends could implement a wider set than the others ("json", 
"xml", "rasterimage" etc.).

Have each Field class deconstruct to a field name and params, eg: 
"decimal", {"digits": 12, "decimals": 2}.

Then a model becomes essentially a list of tuples:

[
("title", "varchar", {"length": 100}),
("price", "decimal", {"digits": 12, "decimals": 2}),
...
]

This is not far from what "render model states" does currently except that 
it compares much richer model descriptions that leads to no-op migrations 
being generated each time you change a label or a user-visible part of 
choices.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5efab6d9-5c76-4338-a3e3-6f55ec752c31%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Missing support for gettext fallback translations

[Patryk Zawadzki]

W dniu niedziela, 21 maja 2017 01:35:28 UTC+2 użytkownik Ramiro Morales 
napisał:
>
> I'm also surprised by your findings. I guess it's something we simply took 
> for granted. It's mentioned in the [1]docs and has been so for [2]years.
>

It's not the same case though. Docs say that if `de-at` is not available, 
Django will use `de` and that's the case. What does not work is the case 
where `de-at` exists but is a sparse catalog of only translation 
differences specific to `de-at`. In that situation translations missing 
from `de-at` should be filled in by falling back to the generic `de` 
catalog but Django doesn't support it.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/257e7cd2-8866-452c-b5b8-0e4e8d41306b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Making __repr__ safe by default

[Patryk Zawadzki]

Hey, it's been a number of years and I'd like to revisit this topic.

I still consider it an anti-pattern to have __repr__ make a call to 
self.__unicode__ or self.__str__ as these can (and very often do) refer to 
related database objects. The last thing you want to happen while debugging 
a crash is for the mere act of observation to result in more database 
queries being executed. __repr__ is assumed to be safe to call by most 
debugging tools and is called by all crash reporting tools I've worked 
with. Having __repr__ itself crash (for example when the database 
connection is invalid) while handling an exception in a production 
environment can lead to exceptions being silently ignored or to 
misleading/useless stack traces being reported.

PS: Backwards compatibility was brought up as a potential issue but 
__repr__ is not usually an API consumed by application code.

Cheers

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/097746a7-2eec-4390-9f29-200a5213b9dc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Some thoughts about improving migration squashing

[Patryk Zawadzki]

W dniu poniedziałek, 20 lutego 2017 11:49:38 UTC+1 użytkownik Markus 
Holtermann napisał:
>
> On Thu, Feb 16, 2017 at 03:25:16PM -0800, rap...@makeleaps.com 
>  wrote: 
> >RemoveField('A', 'foo') also references A and foo, but does it reference 
> B? 
> >if it does, then it' s hard to have optimizations that pass through this, 
> >because this field could be referencing any model (theoretically). 
>
> No, that field can not reference any model. It reference exactly one 
> model (that even holds for FK to abstract models as fields from abstract 
> models are inlined in the concrete models in migrations). However, 
> RemoveField doesn't have the information to "just" figure out the 
> referenced model. RemoveField would need to look into the from_state's 
> apps and actually even look into the actual field that's referred. 
> ForeignKeys have an implicit to_field attribute, so, having 
>
>   AddField('A', 'foo', ForeignKey('B', to_field='bar')) 
>
> a 
>
>   RemoveField('A', 'foo') 
>
> references exactly one field on one particular model. Not more and not 
> less. The issue here is that RemoveField needs to take that information 
> from the state and not from one of its attributes. 
>

Technically it references _some_ model named "B" that was created not 
sooner than the current migration's explicit dependencies. It may be the 
model you saw when you created that migration or it may be some other 
model. You can tell your migration that it has to be executed no sooner 
than after another migration is complete but there is no way to say "but 
before model B is modified any further".

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b9964408-8dff-49ec-aa08-89826d35503c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Model translation and meta options

[Patryk Zawadzki]

W dniu wtorek, 14 lutego 2017 16:19:48 UTC+1 użytkownik jp...@yourlabs.org 
napisał:
>
> However, there's a third alternative to changing Meta that may be worth 
> suggesting: adding a new Field attribute ie. translatable=True, which could 
> be default for CharField and TextField perhaps. It might even turn to be an 
> advantage that no existing translation app supports it that way.
>

I'd vote against adding non-database-related attributes to model fields. I 
actually hope one day Django will drop some of the existing ones that are 
only used by admin (like `editable`).

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f58a8e66-e792-4468-91e6-7ca6e1b641c4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Model translation and meta options

[Patryk Zawadzki]

I think the proposed fields are mostly useful for translation utilities 
that modify your schema on the fly. I consider these to be the least clean 
solutions as they mean none of the translated fields can be marked as 
required. For other solutions (depending on separate tables and 
registration) there's no reason for the field list to live inside the 
parent model.

For anyone curious we're implementing translations as a regular Django 
model with a ForeignKey to the main model. The parent has a descriptor 
attribute that implements the language selection logic, the rest is a 
matter of doing proper joins or prefetches. Example:

from django.utils.translation import get_language


class TranslationWrapper(object):
def __init__(self, instance, locale):
self.instance = instance
self.translation = next(t for t in instance.translations.all() if 
t.locale 
== locale, None)

def __getattr__(self, item):
if self.translation is not None and hasattr(self.translation, item):
return getattr(self.translation, item)
return getattr(self.instance, item)


class TranslationProxy(object):
def __get__(self, instance, owner):
locale = get_language()
return TranslationWrapper(instance, locale)


class Foo(models.Model):
# ...
translated = TranslationProxy()

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7f972ca9-ddbd-4f17-8f0d-f63509da3bec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Methodology for increasing the number of PBKDF2 iterations

[Patryk Zawadzki]

W dniu poniedziałek, 16 stycznia 2017 18:55:25 UTC+1 użytkownik Martin 
Koistinen napisał:
>
> Also, if a developer is experienced/motivated enough to *lower* the hash 
> iterations, s/he'll be more likely to also be experienced/motivated enough 
> to put other controls in place to compensate.
>

Or they just copy-pasted from an out-of-date response to a Stack Overflow 
question. I'm concerned that if someone is motivated enough to do it they 
may not get the memo to bump it by 40% every year even if they are 
experienced and competent.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/63200c19-59fd-4345-bc76-eab94b158374%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: user_login_failed and user_logged_in signals sent at different levels

[Patryk Zawadzki]

Are you sure you need to depend on users "logging in" through an API? Most 
authentication methods used in APIs are stateless and there is no explicit 
login and logout process. You either include the credentials (be it bearer 
token, auth header, custom data or whatever) or not. The credentials are 
checked using authenticate but there is no "logging in" happening: you 
don't create a session, set cookies etc.

W dniu piątek, 13 stycznia 2017 00:30:11 UTC+1 użytkownik Federico Bond 
napisał:
>
> You are right. I was confusing the login view with the login method. The 
> more concrete problem was that Django REST framework calls authenticate 
> directly and user_login_failed is sent but never user_logged_in. I realize 
> though that some of the authentication methods provided by an API don't 
> have clear login semantics so it makes sense to only send failure events.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0d3da30d-7954-46f3-bcbd-9876ca4a27ce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: changing the default MySQL isolation level to READ COMMITTED

[Patryk Zawadzki]

To add some context, REPEATABLE READ can break get_or_create among other 
things (not sure how many invalid tickets Django gets related to that).

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/eeac7c52-78c6-4b9c-bf0a-1a96f5d0b432%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: two-phase commit / distributed transaction

[Patryk Zawadzki]

W dniu piątek, 2 grudnia 2016 12:05:11 UTC+1 użytkownik Mateusz Mikołajczyk 
napisał:
>
> What would you say about checking which CRUD operations were executed 
> within atomic() call (in order to serialize them and save into a special 
> model for databases which don't support this functionality) ? Is it 
> realistic? 
>

It would likely break the promise that distributed two-step transactions 
give you: that once all statements are prepared the transaction is unlikely 
to fail during commit. In this case the commit would mean "start over and 
try to repeat my steps" at which point any of the recorded statements is 
likely to fail constraint checks. (Even more so if your code used 
get_or_create().)

Also how would relations work? You begin a transaction, create a Foo 
instance and the returned PK is 5. You assign it to child models. At this 
point the transaction is saved and rolled back. During replay the insert 
returns PK = 7, at this point there's no way to detect that some of the 
stored fives should now be treated as sevens while some should remain fives.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/098bbb3a-1ba0-42e5-8b75-441474981855%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Rethinking migrations

[Patryk Zawadzki]

I've just hit another problem related to custom fields.

Currently migrations contain information about "rich" fields. If you use a
custom field type, the migration code will currently import your field type
from its Python module. This is highly problematic in case either the code
moves or you later stop using that field type and want to remove the
dependency.

I am currently in the process of rewriting some of my existing migrations
by hand to replace all instances of a custom field type with the type it
actually uses for storage. This will eventually allow me to drop the
dependency but it's not very nice.

Another problem is that for many custom field tapes makemigrations detects
changes made to arguments that do no affect the database in any way (as
they are returned by deconstruction).

If we could ever break backwards compatibility, I'd suggest having field
deconstruction only return the column type (and necessary arguments) it
wants the schema editor to create. This would prevent the migrations from
having external dependencies (which is a major win in itself).

I'd also consider having apps.get_model() just use introspection to read
the schema and return transient models with default field types for each
underlying column type (so a custom JSONField would become a regular boring
TextField inside migration code). This would save us tons of "rendering
model states" time for the relatively small cost of having to cast certain
columns to your preferred Python types inside a couple of data migrations.

Cheers,

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANw2pUG4quLkCUskBkTSoTLVVb7QsZVZgva0y4xOHui75_P1zA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Rethinking migrations

[Patryk Zawadzki]

niedz., 6.11.2016, 00:58 użytkownik charettes <charett...@gmail.com>
napisał:

> I have to agree with Marteen.
>
> From my experience what really slow down the migrate and makemigrations
> command is the rendering of model states into concrete model classes. This
> is something I concluded from my work on adding the plan object to
> pre_migrate
> and post_migrate signals.
>

Yes, rendering model states is very slow but in our case ordering them is
also taking quite some time.

I assume that with a linear chain of migrations we'd only have to render
model states when detecting database changes (makemigrations) and when
executing RunPython code?

As soon as an operation accesses state.apps the rendering kicks in which
> triggers the dynamic creation of multiple model classes and the computation
> of reverse relationships. There are mechanisms in place to prevent the
> whole
> project model classes from being rendered again when a model state is
> altered but if the operation is performed on a model referenced by many
> others the relationship chain might force a large number of them to be
> rendered again causing massive slow downs.
>
> Markus Holtermann has been working on teaching the migration framework
> how to perform database operations without relying on state.apps which
> should
> solve the remaining performance issues of the migrate command. In the case
> of makemigrations the last remaining issue in the master branch should be
> solved
> by stopping to rely on state.apps in RenameModel.state_forwards[1].
>
> Patryk, many improvement landed in 1.9 and 1.10 to speed up the commands
> dealing with migrations. Are you still seeing the same slowdown on these
> versions?
>

I run a mix of 1.9 and 1.10 and am aware of the recent optimisations as I
helped with some of them during previous DUTHs.

Simon
>
> [1] https://github.com/django/django/pull/7468
>
>
> Le dimanche 6 novembre 2016 00:32:04 UTC+1, Marten Kenbeek a écrit :
>
> On Saturday, November 5, 2016 at 4:53:49 PM UTC+1, Patryk Zawadzki wrote:
>
> 1. Dependency resolution that turns the migration dependency graph into an
> ordered list happens every time you try to create or execute a migration.
> If you have several hundred migrations it becomes quite slow. I'm talking
> multiple minutes kind of slow. As you can imagine working with multiple
> branches or perfecting your migrations quickly becomes a tedious task.
>
>
> Did the dependency resolution actually come up in benchmarks/profiles as a
> bottleneck? When I optimized and benchmarked the dependency graph code, it
> had no trouble ordering ~1000 randomly generated migrations with lots of
> inter-app dependencies in less than a second. I'd be surprised if this had
> any significant impact on the overall performance of migrations.
>
> An easy way to test this is the `showmigrations` command, which will only
> generate the graph without any model state changes or model rendering
> taking place. It does some other things, but nothing that should take in
> the order of minutes.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/4a012e54-fae5-4bba-97a9-f323f38e53bc%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/4a012e54-fae5-4bba-97a9-f323f38e53bc%40googlegroups.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANw2pUGrzkDD5hjD-PXh%3DX5GXoqReiebaxuB8%3DggivB4%2B9DTaQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Rethinking migrations

[Patryk Zawadzki]

W dniu sobota, 5 listopada 2016 19:57:38 UTC+1 użytkownik Aymeric Augustin 
napisał:
>
> My solution is to throw away and remake all migrations on a regular basis. 
> Then I `TRUNCATE TABLE django_migrations` and `django-admin migrate 
> --fake`. Obviously this isn’t a great solution. 
>
> Since I work mostly on small projects with just a couple developers on 
> staff, doing this every few months suffices to keep the run time below one 
> minute (which is still quite annoying). 
>
> There’s a risk to lose important, manually generated migrations, typically 
> those that create indexes. I diff the schema before / after with apgdiff to 
> avoid such problems. 
>

That's the main problem we're facing. I'm currently leading a project that 
predates dinosaurs and when it was switched from South to Django, all the 
data migrations were just carried over from the old code. They are holy 
gifts from the elder gods and are rich with eldritch symbols. Nobody wants 
to have to copy and paste them every month or so when we decide to redo all 
of the migrations.

There's also the problem of having many long-running (weeks to months) 
feature branches that make it hard to find a point in time where all 
migrations can be safely discarded.

I can also imagine it's much harder to redo initial migrations in projects 
where two-way relations exist between certain applications.

Cheers,

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/88ed6c68-1732-49bf-8d6b-3c190729472a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Rethinking migrations

[Patryk Zawadzki]

W dniu sobota, 5 listopada 2016 18:40:24 UTC+1 użytkownik Shai Berger 
napisał:
>
> > 2. Dependency resolution is only stable as long as the migration set is 
> > frozen. Sometimes introducing a new migration is enough to break 
> existing 
> > migrations by causing them to execute in a slightly different order. We 
> > often have to backtrack and edit existing migrations and enforce a 
> strict 
> > resolution order by introducing arbitrary dependencies. 
> > 
>
> So, you say you really have implicit dependencies between migrations -- 
> dependencies in substance, which aren't recorded as dependencies. This 
> seems 
> to indicate that you have a lot of manually-written migrations (data 
> migrations?), since the automatically-written ones do include relevant 
> dependencies. This seems odd -- it sounds like you're doing something out 
> of 
> the ordinary. 
>
> This would also explain some of your bad experience with squashing -- 
> indeed, 
> if you have many data migrations, squashing can become much less 
> effective. 
>

Let's not come to conclusions prematurely. Django only supports predicate 
dependencies. You can say "not earlier than after these are applied" but 
that does not mean "immediately after they are applied". Sometimes Django 
tries to run the migration much later. If you have your models scattered 
across a large number of applications (we use apps to gateway entire 
classes of related features) sometimes the late migrations tries to 
reference a column in another model that was long since removed by a much 
later added migration in its respective app.

> 3. Removing an app from a project is a nightmare. You can't migrate to 
> zero 
> > state unless the app is still there. There is no way to add "revert all 
> > migrations for app X" to the migration graph, it's something you need to 
> > run manually. There is no clean way to remove an app that was ever 
> > references in a relation. We were forced to do all kinds of hacks to get 
> > around this. Sometimes it's necessary to create an empty eggshell app 
> with 
> > the same name and copy all migrations there then add necessary data 
> > migrations and finally migrations that remove all the models, indices, 
> > procedures etc. Sometimes people just leave a dead application in 
> > INSTALLED_APPS to not have to deal with this. 
>
> Clear out (maybe even remove) models.py and type "makemigrations", and you 
> get 
> a migration that deletes everything. The answer to getting rid of the 
> historical migrations is squashing, but of course you first need squashing 
> to 
> work properly. 
>

I cannot clear out anything from an app that came from PyPI. That's why I 
mentioned creating fake empty apps that are just containers for their 
migration history. Squashing does nothing to help with that if you have 
another application reference any of those models. Squashing only helps you 
have fewer migrations. If the migrations were always in the correct order, 
the migration engine could collapse them automatically at execution time.

> 4. Squashing migrations is wonky at best. If you create a model in one 
> > migration, alter one of its fields in another and then finally drop the 
> > model sometime later, the squashed migration will have Django try to 
> > execute the alter first and complain about the table not being there. 
> Also 
> > the only reason we need to squash migrations is to prevent problem 1 
> above 
> > from becoming exponentially worse. If migrations were only as slow as 
> the 
> > underlying SQL commands, we'd likely never squash them. 
> > 
>
> If that's so, it's a bug you should report; it's also an issue you can 
> work- 
> around by editing the migration to remove the redundant operation. There 
> are   
> issues with squashing, to be sure, but I don't think this is one of the 
> serious ones. 
>

It's a bug that I will report at some point but I mostly encounter it in 
environments where I can't afford the time needed to properly debug.

> 6. Conflict detection and resolution (migrate --merge) is a make-believe 
> > solution. It just trains people to execute the command without 
> > investigating whether their migration history still makes sense. 
>
> It could be smarter, assuming it understood the content of migrations. We 
> could probably improve it to a point where, for most cases, it would 
> either 
> know to merge automatically or know that there really is a conflict. This 
> would 
> probably not help you if you have a lot of RunPython's in your migrations. 
>

Depends on the project. I really don't care about the framework trying to 
reason about the results of a git merge. Django does not have enough 
understanding of the code and the version control history to do the job of 
the person responsible for the merge. Getting stuff right 60% of the time 
is not reliable enough to depend on it but is reliable enough to get some 
people lazy.

> Some of these I need to dig deeper into and probably file proper 

Re: Rethinking migrations

[Patryk Zawadzki]

W dniu sobota, 5 listopada 2016 17:30:15 UTC+1 użytkownik Andrew Godwin 
napisał:
>
> Hello! I have opinions about this :)
>  
>
>> Possible solution (or "how I'd build it today if there was no existing 
>> code in Django core"):
>>
>> a. Make migrations part of the project and not individual apps. This 
>> takes care of problem 3 above.
>>
>
> It also means it's impossible for apps to ship migrations and define how 
> to upgrade from version to version. I realise that (c) below is part of a 
> proposed solution to this, but how do you propose to match up what's 
> already been run in the database without having names match (and then you 
> just have app migrations by another name)?
>

I would actually insist on keeping the names intact. It means that adding 
an external dependency could inject a migration with a date from the 
previous year but I think that's not a problem as it's guaranteed not to 
conflict with any other migrations.


>> b. Prefix individual migration files with a UTC timestamp 
>> (20161105151023_add_foo) to provide a strict sorting order. This removes 
>> the depsolving requirement and takes care of 1 and 2. By eliminating those 
>> it makes 4 kind of obsolete as squashing migrations would become pointless.
>>
>
> Unfortunately this does not help all the time as computers' clocks aren't 
> necessarily right or in sync, so it would merely be an approximation and 
> you'd still get the occasional clash.
>

You only need your own migrations to be ordered so you can safely assume 
the previous one to be applied before the one you're writing at the moment. 
For two unrelated changes the order pretty much does not matter.
 

> c. Have reusable apps provide migration templates that Django then copies 
>> to my project when "makemigrations" is run.
>>
>
> Would these be lined up with their own timestamp in the single serial 
> migration timeline? Would you have to make sure any of these templates from 
> any app update was copied across and put in the order before you used the 
> new columns?
>

I'd say use proper timestamps. This way two apps can depend on each other 
and the migrations will still get run in proper order.
 

> d. Maintain a separate directory for each database connection.
>>
>
> This I think might be a good idea, though I'd like to see a more 
> generalised idea of "migration sets" and you then then say which alias uses 
> which set (so you can share sets among more than one connection)
>

Agreed.
 

> e. Execute all migrations in alphabetical order (which means by timestamp 
>> first). When an unapplied migration is followed by an applied one, ask 
>> whether to attempt to just apply it or if the user wants to first unapply 
>> migrations that came after it. To me this would work better than 6.
>>
>
> This is basically what South used to do, and it worked reasonably well in 
> either being successful or exploding enough that people noticed. Given that 
> you're proposing per-project migrations, however, people are going to run 
> into this almost constantly, as they will clash significantly more than 
> per-app ones.
>

South was not perfect but I'd say the current solution is not better, it's 
just different. Some of my projects use a lot of long-running feature 
branches so I have an application where every other migration is a merge 
migration with accepted default values. We do try to make migrations 
backwards-compatible where needed but I don't think it's a common scenario 
to add conflicting changes on two feature branches. Most of our conflicts 
can be described as department A added a field they needed while department 
B added a data migration to fix a denormalized field.
 

> Of course we do have migration support in core and it's not compatible 
>> with most of the above list. Any ideas? I think serializing the dependency 
>> solver state and reusing it between runs could be a pretty low hanging 
>> fruit (like "npm shrinkwrap" or yarn's lock file).
>>
>
> I think not only could the dependency solver state be serialised but that 
> it would be a replacement for the datetimes-on-filename proposal in that 
> you could easily pull out a previously-serialised order from disk and then 
> work out what the new ones do.
>
> I am generally not keen on the idea of per-project migrations, though - it 
> makes what's in the database a property of the project, not the app, and 
> that's not how Django has worked traditionally. I think an effort to get a 
> more reliable, exposed global ordering of those individual app migrations 
> would go a long way towards the end goal without having to have migration 
> templates, upgrade instructions, and way more collisions between branches.
>

I do believe that the database is my property and I'd much rather see the 
project code hold reign over its structure. Some problems simply cannot be 
solved by submitting an upstream patch (project-specific or 
backend-specific indexes come to mind).
 

> At the end of the day, though, 

Re: RFC: DEP 7 - dependency policy

[Patryk Zawadzki]

W dniu sobota, 5 listopada 2016 13:24:28 UTC+1 użytkownik Jacob Kaplan-Moss 
napisał:
>
> Hi all -
>
> DEP 7 proposes a new dependency policy. In a nutshell, the policy is: 
> Python packaging is good now. Django can have dependancies.
>
> For full details, please check out the DEP: 
> https://github.com/django/deps/blob/master/draft/0007-dependency-policy.rst
>
> I'd appreciate any comments and feedback y'all have before I submit this 
> to the technical board for review. In particular, there are a couple of 
> things I'd like feedback on:
>
> - Are my criteria for "maturity" appropriate? Will they cover use-cases we 
> want to cover?
>
 
What are your concerns here? Couldn't Django eventually either take over or 
vendor-in the project if it ever becomes unresponsive?

Cheers,

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/af985f1f-7f37-4527-9cf7-6603ed992627%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Rethinking migrations

[Patryk Zawadzki]

Greetings, Jazz Guitarists,

I've briefly talked about this with Markus and he mentioned that the 
subject was already brought up by Tyson Clugg but I think it deserves a 
proper discussion here.

I'm typing this from the comfort of Django: Under the Hood sprints so 
please excuse poor grammar and the somewhat chaotic explanations that 
follow. I'm very tired and English is not my mother tongue. This is not a 
DEP but merely a stream of consciousness I'd love to get some feedback on.

Here are some of the problems we face when dealing with migrations:

1. Dependency resolution that turns the migration dependency graph into an 
ordered list happens every time you try to create or execute a migration. 
If you have several hundred migrations it becomes quite slow. I'm talking 
multiple minutes kind of slow. As you can imagine working with multiple 
branches or perfecting your migrations quickly becomes a tedious task.

2. Dependency resolution is only stable as long as the migration set is 
frozen. Sometimes introducing a new migration is enough to break existing 
migrations by causing them to execute in a slightly different order. We 
often have to backtrack and edit existing migrations and enforce a strict 
resolution order by introducing arbitrary dependencies.

3. Removing an app from a project is a nightmare. You can't migrate to zero 
state unless the app is still there. There is no way to add "revert all 
migrations for app X" to the migration graph, it's something you need to 
run manually. There is no clean way to remove an app that was ever 
references in a relation. We were forced to do all kinds of hacks to get 
around this. Sometimes it's necessary to create an empty eggshell app with 
the same name and copy all migrations there then add necessary data 
migrations and finally migrations that remove all the models, indices, 
procedures etc. Sometimes people just leave a dead application in 
INSTALLED_APPS to not have to deal with this.

4. Squashing migrations is wonky at best. If you create a model in one 
migration, alter one of its fields in another and then finally drop the 
model sometime later, the squashed migration will have Django try to 
execute the alter first and complain about the table not being there. Also 
the only reason we need to squash migrations is to prevent problem 1 above 
from becoming exponentially worse. If migrations were only as slow as the 
underlying SQL commands, we'd likely never squash them.

5. There's no simple way to roll back all the migrations introduced after a 
particular point in time which is very useful when working with multiple 
feature branches. In my current project dropping the database means having 
to reimport over 200 MB of data snapshots. Switching branches requires me 
to look at branch diffs to determine which migrations to revert.

6. Conflict detection and resolution (migrate --merge) is a make-believe 
solution. It just trains people to execute the command without 
investigating whether their migration history still makes sense.


Some of these I need to dig deeper into and probably file proper tickets. 
For example I have an idea on how to fix 4 but it would make 1 even slower.

I took some time to get a good long look at what other ORMs are doing. The 
graph-based dependency solving approach is rather uncommon. Most systems 
treat migrations as part of the project rather than the packages it uses.


Possible solution (or "how I'd build it today if there was no existing code 
in Django core"):

a. Make migrations part of the project and not individual apps. This takes 
care of problem 3 above.

b. Prefix individual migration files with a UTC timestamp 
(20161105151023_add_foo) to provide a strict sorting order. This removes 
the depsolving requirement and takes care of 1 and 2. By eliminating those 
it makes 4 kind of obsolete as squashing migrations would become pointless.

c. Have reusable apps provide migration templates that Django then copies 
to my project when "makemigrations" is run.

d. Maintain a separate directory for each database connection.

e. Execute all migrations in alphabetical order (which means by timestamp 
first). When an unapplied migration is followed by an applied one, ask 
whether to attempt to just apply it or if the user wants to first unapply 
migrations that came after it. To me this would work better than 6.

f. Migrating to a timestamp solves 5.


Of course we do have migration support in core and it's not compatible with 
most of the above list. Any ideas? I think serializing the dependency 
solver state and reusing it between runs could be a pretty low hanging 
fruit (like "npm shrinkwrap" or yarn's lock file).

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, 

Re: Making QuerySets composable

[Patryk Zawadzki]

pt., 22.01.2016 o 17:44 użytkownik charettes  napisał:

> Hi Patryk,
>
>
> > Currently we have Prefetch objects that kind of solve this problem for
> M2M
> > relations but perhaps it would be nice to also be able to use QuerySets
> in
> > select_related() or even in filter(). I don't think Prefetch objects are
> best
> > suited for that and I personally find having to instantiate them
> explicitly
> > kind of ugly.
>
> From what I understand you're not a fan of Prefetch objects but I think
> most
> of your problems can be solved by using them and custom managers as the
> formers
> also support foreign keys[1].
>

Funny, I was aware of the featurebut always assumed prefetch_related would
always execute an extra query instead of becoming an inner join.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANw2pUHOmpeq%3Db4jbdQG%3D8qt_p_tTjDouaPMPe8AD3VnyMoYLw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Making QuerySets composable

[Patryk Zawadzki]

Hi,

Currently the way QuerySets work makes it hard to reuse code. Consider an
example where I have template tags to render an author and a book (that in
turn displays author). I don't want my template code to execute any
additional queries per row displayed so I am forced to prefetch everything
upfront. Now if I want to display a list of authors and a list of books my
code is vastly different:

```
authors = Author.objects.filter(visible=True)
authors_for_display = authors.annotate(num_books=Count('books'),
num_awards=Count('awards')).prefetch_related('foo')
```

vs.

```
books = Book.objects.filter(visible=True, author__visible=True)
books_for_display = books.annotate(author_num_books=Count('author__books'),
author_num_awards=Count('author__awards')).select_related('author').prefetch_related('author__foo')
```

Both cases need to do the exact same filtering, annotations and prefetches
for the Author object but notation is so different that no code reuse can
happen ("baz" vs. "bar__baz" vs. "foo__bar__baz"). The behaviour of
annotations is also different as now they end up on the Book object which
forces me to write some glue code. If my structure gains another level of
depth I will be forced to go to even greater extents.

Now consider that I have many other objects that reference the Author and
thus I have many places in code that implement the same logic but each time
with a slight syntactic twist. What if a front-end engineer asks me to
change the information fetched about the Author? Or perhaps I discover that
half of the query time is spent fetching a megabyte of Author's extended
biography in HTML that the front-end never uses outside of that Author's
page. I am forced to hunt down and update all of these.

Currently we have Prefetch objects that kind of solve this problem for M2M
relations but perhaps it would be nice to also be able to use QuerySets in
select_related() or even in filter(). I don't think Prefetch objects are
best suited for that and I personally find having to instantiate them
explicitly kind of ugly. To me having another SelectRelated object is
probably a no-go.

Ideas?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANw2pUHwXq_9sVCPaK5CFXLzowd8HmemxVBe4%2BU7NJZ6QAccOw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feature proposal: Target API version

[Patryk Zawadzki]

2015-11-12 17:40 GMT+01:00 Carl Meyer <c...@oddbird.net>:
> On 11/12/2015 05:48 AM, Patryk Zawadzki wrote:
>> * Setting the target version to a higher value could turn deprecation
>> warnings to exceptions for all features deprecated in the target
>> version and earlier (which could make future-proofing your code
>> easier).
> You're proposing this would only activate if your TARGET_DJANGO_VERSION
> is higher than your actual Django version? I'm not sure how this feature
> is even possible as you've described it, since if you have Django 1.8
> installed and you set TARGET_DJANGO_VERSION to (1, 9, 0), that doesn't
> magically give Django 1.8 knowledge of future deprecations! Perhaps I'm
> misunderstanding.

Consider Django version 1.9 that also happens to support API versions
1.7 and 1.8 and an application that originally targetted 1.7.

I could leave my target API at 1.7 and get the app to work. This
requires minimum resources. I would assume during normal operation
deprecations introduced in 1.8 and later would not show up since I'm
explicitly targetting 1.7.
I could also bump my target API to 1.8 and get some more deprecation
warnings for things deprecated in 1.8 (but not in 1.9). At this point
things deprecated since 1.7 should stop working so I can make sure
that I am actually targetting the "intended" APIs of 1.8.
I could finally go all the way up to 1.9 and possibly get different
behaviour when calling the same functions I did while targetting 1.8.
See below for discussion.

> Plus, it's not hard to use filterwarnings to turn all deprecation
> warnings into exceptions (I often do this on my projects, at least for
> the test suite). Perhaps this is a technique we should recommend and
> demonstrate somewhere in our upgrade docs?

Yes, this is how I imagine this to work. Specifying the target could
among other things configure which warnings are ignored and which are
explicitly turned into errors.

>> * Django could introduce A/B feature switches that introduce
>> backwards-incompatible behaviour if a high-enough target version is
>> specified without breaking existing projects.
> This is tempting at first blush, because it allows us to claim that
> you've opted in to whatever backwards-incompatible changes we feel like
> making by bumping your TARGET_DJANGO_VERSION. I think this is false,
> though. You don't know what all you're opting into by bumping your
> TARGET_DJANGO_VERSION unless you carefully read the release notes.
> (Unlike other opt-ins which typically involve changing the use of
> specific APIs in your code.) So how is it any different from us claiming
> that you've already "opted in" to backwards incompatible changes simply
> by upgrading Django? (In practice, we _do_ claim this already, but we
> try to minimize such changes.)
>
> I guess the idea is that you could first upgrade Django but leave your
> TARGET_DJANGO_VERSION alone, and nothing would break, and then you could
> at some point bump up your TARGET_DJANGO_VERSION when you're ready to
> deal with the breaking changes? In the end I'm not sure that's any
> better than today's story, which is just "upgrade Django when you're
> ready to deal with the breaking changes."

Yes, you could upgrade Django safely at least X times before having to
touch any code. You could also target a major version like 2.5 and not
experience potentially incompatible but more secure change to default
settings introduced in its point release, 2.5.7, while allowing new
projects to benefit from that change.

Cheers,

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANw2pUF3CsY0%2B_q_ABk2bfHeHzF3HVF9m39iYXapM0ox7Te%2BxQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Feature proposal: Target API version

[Patryk Zawadzki]

Hello everyone,

I have spent quite a while thinking about this and am still not sure
what the best approach would be or even if there are enough benefits
to justify the change.

If you've ever worked with compiled code, you're probably familiar
with ABI versioning in dynamic libraries. This is somewhat similar.


The idea is to allow a project to specify that is _assumes_ Django to
behave like a particular version or release. It could do so by
specifying a TARGET_DJANGO_VERSION (better name needed) setting that
is a valid semantic version tuple.

Example:

TARGET_DJANGO_VERSION = (1, 8, 6)

New projects would start with the variable set to whatever Django
version was used to create them. If the variable is not set, Django
could raise an ImproperlyConfiguredError telling the user what to put
in the settings file or it could simply issue a warning and assume the
current version.


Now what that gives us:

* Django version Y could outright tell you that the project that
targets version X is not compatible and point you to the *relevant*
upgrade instructions (since we know both the old and the new version).

* Setting the target version to a higher value could turn deprecation
warnings to exceptions for all features deprecated in the target
version and earlier (which could make future-proofing your code
easier).

* Django could introduce A/B feature switches that introduce
backwards-incompatible behaviour if a high-enough target version is
specified without breaking existing projects. It would also make it
theoretically possible for the latest stable version to support the
last LTS version from the same code tree (although it would probably
require an enormous legacy/compat module).


Is that something that was discussed in the past? Does it make any sense?

Cheers,

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANw2pUGty5ndZsREqjbqjT-RZYc3Vjo5%2Begc7Et3C2nbj5OzNw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: #25227 Add utility method `get_updated_model()` to `ModelForm`

[Patryk Zawadzki]

2015-08-07 16:51 GMT+02:00 Martin Owens <docto...@gmail.com>:
> Could we use something simple like 'update()' and 'commit()' to which save
> would call both?

Or `.apply()` and `.commit()` not to suggest that the form gets
updated in any way.

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANw2pUF-x9JdURrsHL-_nahsqj4bVyNMDw%3D_DYLJMR%2BTVmxQBg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Making management forms for formsets optional

[Patryk Zawadzki]

2015-06-08 20:22 GMT+02:00 Carl Meyer <c...@oddbird.net>:
> But I think such a project needs to be approached as a full rewrite,
> probably first as a third-party module and then later (if it works out
> well) considered for merge as a separate replacement module. It does not
> seem likely to me that tugging on individual threads like "remove
> management form" and "remove initial forms" will result in changes to
> the current formset code that are both backwards-compatible and an
> improvement in the design. I'm willing to be proven wrong, but so far
> the current pull request only strengthens that feeling.

This could very well be true. We could also try implementing a field
wrapping a subform and a field wrapping a multitude of fields. I am
not sure whether there would be an obvious way to render the resultin
form as HTML but it would mean that we'd end up only having a single
thing to deal with (forms) and would allow existing views to
transparently handle situations where more than one form is needed
(just create a form that consists of two subform fields).

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANw2pUEM9q%2BbWymWATKhoXSU9eSw3jagVY5bjVJhGGrqzv-zdw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Making management forms for formsets optional

[Patryk Zawadzki]

2015-06-07 10:18 GMT+02:00 Shai Berger <s...@platonix.com>:
> Semi-devil's-advocate suggestion: Replace the management form with a
> management field, a hidden field with
>
> name = "__manage__%s" % (formsetname)
>
> and
>
> value="x"
>
> Then instead of taking the number of forms from a field on a management form,
> we can just take the accumulated length of the management field (which would 
> be
> returned as an array). This also offers a better solution to the checkboxes-
> only-form problem, I think.

This is what Rails prefers but I think it is only added for checkbox fields.

> Do we use the management form for anything other than counting forms in the
> formset?

Yes, we currently have the concept of initial forms even for bound
formsets. This does not mirror form behaviour as bound forms don't
have the concept of initial data. I'd like to get rid of that as well
but Russell is not sure whether there are cases that require initial
forms to work correctly.

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANw2pUEorJLRRyJ1wQbVRcc3bQ6HnuGxe22fRE8LRRpjvrsd9A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Making management forms for formsets optional

[Patryk Zawadzki]

Playground: https://github.com/django/django/pull/4797

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANw2pUFOivXtNqSXgP1BbO7LKYbH_qfbdXcOQ75Q62G%2BPJ0%3DsQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Making management forms for formsets optional

[Patryk Zawadzki]

Hi folks,

I've talked to Marc about fixing the case where a formset will raise
an uncaught ValidationError when instantiated with a missing or broken
management form. This has caused us great pain when dealing with
vulnerability scanners that tend to POST random crap to each endpoint
they find as it means formsets—unlike forms—are not safe to
instantiate with untrusted data.

Now wrapping each and every formset instance in a try/except block and
having to manually handle the fallback case is not fun. But as Marc
pointed out, forgetting a management form is one of the most common
pitfalls people run into when dealing with formsets.

Therefore I propose making the management form optional. The only case
that requires an explicit TOTAL_FORMS variable are forms that consist
of nothing but checkboxes (as unchecked checkboxes are not included in
POST data). In other cases (including all ModelFormSets as they always
contain a hidden PK field) we can deduct the number of submitted forms
from the data itself.

Ideally I would get rid of the management form entirely and document
that a form that is nothing but checkboxes is an unsupported case and
that a workaround would be to include an extra hidden field. Honza may
or may not kill me for suggesting that.

For now, I would like to make the formset optional and document the
cases where you need to include it. If included, it would take
precedence to keep the existing deployments working.

Thoughts?

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANw2pUFwGQmOSXcCMwUB6Bb3%2BFtf627aweHUCcgC6_UyM%2B5cnQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Why apps have to have unique names? [related with subapps and name collision]

[Patryk Zawadzki]

Related 
thread: 
https://groups.google.com/d/topic/django-developers/9piN0wRFbLs/discussion

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Make sure QuerySet.get() does not fetch more rows than it absolutely needs

[Patryk Zawadzki]

Here's the ticket:

https://code.djangoproject.com/ticket/6785

tl;dr: Calling .get() on a badly filtered queryset can result in lots
of database rows being fetched and wrapped in Python objects for no
gain.

Currently .get() logic is as follows: clone the queryset, prepopulate
its full cache, check if more than one row was returned and if so,
raise MultipleObjectsReturned.

This works fine if the query returns two or five rows. It's not so
fine if the query results in thousands or millions of rows being
returned. While working on a large online store we had one of the
production servers die while trying to fetch twenty something million
rows because of a programming error that resulted in wrong Q object
being used as a filter.

The only reason all rows are fetched is to return an accurate counter
as part of the exception message. This counter is not accessible
programmatically and having the exact number there is of little value.
The ticket proposes to add a safeguard by limiting the database to at
most two results. The logic then is simple: if zero rows are returned,
raise ObjectDoesNotExist, if two, raise MultipleObjectsReturned. The
latter could possibly execute a second COUNT query if DEBUG is enabled
and the counter is deemed important enough to keep.

Of course you can work around the whole problem by not writing buggy
code but we all know how it works. In the mentioned case even unit
tests did not help as there were not enough records in the database to
trigger MultipleObjectsReturned.

Related pull request: https://github.com/django/django/pull/1139

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Making __repr__ safe by default

[Patryk Zawadzki]

2013/5/19 Karen Tracey <kmtra...@gmail.com>:
> I agree with Anssi, repr should stay as-is. I do a lot of shell/pdb work and

In interactive mode it's just as easy to call __unicode__ as it is to
call __repr__. On the other hand __repr__ is used by automated tools
such as crash reporters (raven).

> I can't recall ever encountering a problem with "unsafe" repr.

We had it explode a number of times when coupled with postgres. In
that case instead of the original exception, all you get is:

psycopg2.ProgrammingError: current transaction is aborted, commands
ignored until end of transaction block

> I think many
> people would find it annoying if suddenly repr would tell you no more than
> the pk of the object.

I'd argue that the PK is actually more important than whatever
__unicode__ returns. It's really annoying to find a stack trace that
only contains product's title when you have tens of millions of
products on production and title is not an indexed field. And by that
time it's already too late to override __repr__. On the other hand
it's fairly easy to extend __repr__ to display other information if
you find it more useful than the PK.

Backward compatibility should not be a concern here as no code should
depend on whatever __repr__ chooses to return.

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Not calling things twice in templates

[Patryk Zawadzki]

2013/6/3 Andre Terra <andrete...@gmail.com>:
> Well, Russ, you asked for suggestions, so here's a couple half-hearted
> attempts.
>
> Perhaps we could allow for if clauses in the with block or vice-versa? It
> could be argued that it would reduce readability and/or induce confusion
> with conditional expressions[0], and I would have to agree.
>
> {% with my_bonnet.bees as bees if my_bonnet.bees %}
> {# could be confused with conditional expressions #}
> 
> {% for bee in bees %}
> {{ bee }}
> {% endfor %}
> 
> {% else %}
> No bees!
> {% endwith %}

What about:

{% with nonempty my_bonnet.bees as bees %}

{% for bee in bees %}
    {{ bee }}
{% endfor %}

{% endwith %}


-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Making __repr__ safe by default

[Patryk Zawadzki]

18 maj 2013 18:48, "Anssi Kääriäinen"  napisał(a):
> There was very similar discussion recently (maybe in some ticket, I
> don't remember where) about QuerySet.__repr__(). IIRC the conclusion
> was that executing SELECT queries as part of __repr__() is OK.

I've been bitten by that one already. At least with postgresql if you
invalidate the transaction executing any query other than rollback will
raise an exception. It's not something you plan for ahead and it's a real
pain to find the real cause.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Making __repr__ safe by default

[Patryk Zawadzki]

As suggested by Marc Tamlyn I am posting this here for discussion:

https://code.djangoproject.com/ticket/20448

tl;dr:

Currently __repr__() calls __unicode__() which can be a very bad
thing. You really don't want things like an exception being raised or
debugger being used to cause additional side effects (like executing a
database query inside __unicode__).

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: #10863 - Email full stack traces (feedback please!)

[Patryk Zawadzki]

On Wed, Oct 5, 2011 at 12:40 PM, Cal Leeming [Simplicity Media Ltd]
<cal.leem...@simplicitymedialtd.co.uk> wrote:
> Could I bring the latest comment on the following ticket to the attention of
> you guys:
> https://code.djangoproject.com/ticket/10863#comment:17
> Any feedback would be much appreciated.

I'm not a core dev but since you're asking for feedback…

Instead of emailing people with HTML I'd like to see a _useful_ stack
trace in text format instead. I suggest you take a look at a debugging
tool I wrote some time ago, it manages to get useful info without
resorting to any rich formatting (in fact it only uses the logging
library):

https://github.com/patrys/great-justice

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

#16572 — looking for directions

[Patryk Zawadzki]

Hi,

As you can see in the ticket¹, I've identified the source of the
exception and provided a naïve patch. I am however not familiar with
internals of the SQL compiler enough to judge whether it's the correct
place to patch. Can someone with more knowledge hold my hand here?

I am interested in this bug as it couses us quite a headache in the
Satchless project. Being unable to select_related() we currently have
to choose between making an extra query (ouch) and ignoring the ORM
(eww).

Cheers,

¹ https://code.djangoproject.com/ticket/16572

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: NoSQL support

[Patryk Zawadzki]

On Tue, May 10, 2011 at 9:40 PM, legutierr <leguti...@gmail.com> wrote:
> Maybe it is inevitable that this kind of debate will crop up in any
> discussion of django-nonrel or NoSQL, but I very much hope that the
> philosophical debate does not detract from this fact: that django-
> nonrel has demonstrated in very real terms that the actual changes
> needed for Django's ORM to interface with a diverse set of non-
> relational systems, are, in the general scheme of things, relatively
> minor.  Because they are localized and relatively minor, if those
> changes do not have a negative impact on the usability and stability
> of the ORM, and if they do not introduce noticeable backwards
> incompatibility, that small set of changes should, in my opinion, be
> considered for acceptance into Django.

Please don't get me wrong. I have worked with RDBMS for more than a
decade but I alse use django-nonrel with MongoDB on a daily basis. I
also think that the approach django-mongokit takes is much more
natural for NoSQL data than just reusing the ORM. The ORM has no way
to express complex structures and if such support is added, you will
always have to choose which subset to use. For relational tables you'd
get foreign keys and for non-relational you'd get structure semantics.
Then we have the ModelForms that would need to start producing
sub-formsets for certain structures. In the end you end up with one
swiss army knife instead of a fork and a knife. While possible, it's
not very convenient to dine using a swiss army knife.

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: NoSQL support

[Patryk Zawadzki]

On Wed, Apr 27, 2011 at 10:12 AM, Waldemar Kornewald
<wkornew...@gmail.com> wrote:
> Hi,
> we (the Django-nonrel developers) would like to work on official NoSQL
> support for Django. We'd like to focus only on databases similar to
> App Engine, MongoDB, and Cassandra 0.7+ with secondary indexes. Our
> goal is not to support all native query features of key-value stores
> like Redis.

Did you guys consider providing a Document class that is entirely
separate from models.Model?

Technically speaking teaching the ORM non-relational tricks is of
course possible but in reality the philosophy is entirely different
and you need to plan for NoSQL from the very beginning. Traditional
models are flat and have a schema, NoSQL documents can have extra
fields and each of them can hold a fairly complicated structure,
possibly involving numerous other (python-enforced) schemas at
different points in the tree.

In the end you won't be able to move models or logic between
traditional RDBMS and NoSQL engines anyway. What we get instead is
either a whole bunch of NotImplementedErrors or a heap of hacks to
simulate traditional relations in a world that does not need them.

Of course as much of the ORM API as it makes sense should be supported
by the Document but I really feel these should be designed as separate
object types.

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: EmailField max_length of 75 characters should be 256 acccording to RFC 5321

[Patryk Zawadzki]

On Thu, Feb 18, 2010 at 1:15 AM, Artem Skvira <artem.skv...@gmail.com> wrote:
> Hi all,
>
> Following the django's tracker policy I'dlike to discuss this issue
> here instead of re-opening the ticket (http://code.djangoproject.com/
> ticket/12900#comment:1)
>
> My problem with current limitation of 75 characters is that it does
> not allow me to use auth.User.email for storage of login details.
> While there's certainly a workaround (using custom profile model) I am
> wondering what are the potential problems with the change to auth.User
> itself and why it might not be backwards compatible for existing
> django applications already using this field in User?

This problem is now more severe due to Facebook returning proxied
e-mail addresses in this format:

apps+111.22.abcdef0123456789abcdef0123456...@proxymail.facebook.com

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Proposal: make 1.4 more explicit

[Patryk Zawadzki]

On Fri, Apr 1, 2011 at 12:01 AM, Luke Plant <l.plant...@cantab.net> wrote:
> On 31/03/11 19:35, Patryk Zawadzki wrote:
>>  * for 1.4 provide a number of fallback settings that can be used to
>> enable legacy behavior in each of the first three points
> but this is where it falls down, because the devil is in the details
> with regards to backwards compatibility.
>
> Settings to control this is not really an option, because it means you
> have to get everything upgraded together, which is just infeasible with
> the current app ecosystem. For each bullet point, you'll need a more
> fine-grained approach that allows upgrading at least on the level of an
> individual app, and users of that app to be able to use old apps.

I meant a set of separate settings. Like a list of apps that should
still use legacy table names. As for template loading, the ticket I
mentioned in the original post has a backwards-compatible patch. A
future version of Django can just remove the fallback part of the
code. The hardest part is making the internals work with fully
qualified paths.

> We would also need tools to migrate existing tables. Don't forget the
> contenttypes app, which contains data that use the existing app names.
> Remember also that we don't have any tools in core for upgrading
> existing tables or data.

It would be much easier if Django contained a migration mechanism that
apps such as the content types framework could use. We also have to
support a way to convert existing fixtures.

> This sounds to me more like the kind of change we might do for Django
> 2.0, not 1.4...

I hope that it would be possible to roll these out gradually. newforms
didn't happen overnight either :)

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: ./manage.py loaddata calls .save() on all models... should it?

[Patryk Zawadzki]

On Wed, Mar 30, 2011 at 10:55 AM, George Karpenkov
<true.chesh...@gmail.com> wrote:
> Oh thanks Russel!
>
> Turns out django-tagging was creating those objects via the post-save
> signal hook.
>
> http://code.djangoproject.com/ticket/8399 <- here is the ticket which
> proposes an option to disable the signal handling during the loaddata
> operation.
>
> Malcolm says that some people might want to use signals while running
> loaddata - ie to create related objects.
>
> I don't really understand why anyone would want that -- the only use
> case I've ever seen for the loaddata operation was "dump the entire
> database -> load entire database", though I guess use cases differ.
>
> Any updated comments on disabling the signal handling for the loaddata
> operation?

Rather simple fix to the signal handler:

def on_something_saved(sender, instance, created, **kwargs):
if not kwargs.get('raw', False):
do_stuff()

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Proposal: make 1.4 more explicit

[Patryk Zawadzki]

Hi group,

Currently Django has its own concept of namespacing (apps) and mostly
ignores Python modules. I'd like to propose using Python paths
wherever we can instead of apps. This would solve a lot of problems
with conflicting submodule names that plague maintainers of reusable
apps and frameworks. Currently if you try to provide a base
frobnicator model and you choose to do so in
"fooframework.frobnicator.models", you pretty much forbid the users
from putting derived classes in a local module named
"frobnicator.models". Bad things happen when two apps start to provide
"frobnication_tags".

I propose the following:
 * make the internal model handling operate on full paths to the
classes, rather than just the app names and class names
 * make the DB table names use full python paths instead of just the
app name and the class name
 * make templatetag loading use full paths rather than just the suffix
 * allow fully qualified templatetag names ("{%
foo.templatetags.frobnication_tags.frobnicate baz %}") and possibly
loading aliases ("{% import foo.bar.tags as bar %}") to cover the case
when conflicts are hard to avoid (different app authors etc.)
 * for 1.4 provide a number of fallback settings that can be used to
enable legacy behavior in each of the first three points

PS: http://code.djangoproject.com/ticket/12772

PPS: Please don't shoot me. :>

-- 
Patryk Zawadzki
I solve problems.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: #6735 -- Class based generic views: call for comment

[Patryk Zawadzki]

On Sat, Oct 2, 2010 at 1:40 PM, Russell Keith-Magee
<russ...@keith-magee.com> wrote:
> On Sat, Oct 2, 2010 at 7:05 PM, Patryk Zawadzki <pat...@pld-linux.org> wrote:
>> Classes that represent real objects have state. Like cars have color,
>> make and registration number, your models have attributes that
>> differentiate them from other objects of same class.
>>
>> "Utility" classes (usually singletons) that represent a function
>> (service) should not have state.
> Ok, now show me keyword in Python that identifies a class as a
> "utility class". Or the page in the Python documentation where it
> talks about "utility classes" and how you identify them and use them
> properly.

There is no Python documentation that teaches you how to write good
and robust code. There are only docs that tell you what "print" or
"raise" does. I wouldn't pick software engineering during my studies
if I expectes language references to teach me all there was about
programming.

> You can't, because Python doesn't make that distinction. Python has
> classes. Class instances have state. Period. People certainly write
> classes that are stateless. But providing a base API based around
> classes, encouraging people to subclass and add their own behavior,
> and expecting *documentation* to prevent them from using self is
> complete folly IMHO.

I was going to respond to that. I wanted to mention things like
compiled languages and their expectance of you being able to write
good code. I respect you too much to drag you through that. After all
I am not forced to use whatever the standard becomes. If I am ever
forced to use this sort of class-based views, I'll file bugs, until
then I'll try to remain quiet. There are more important decisions to
be made.

Cheers,

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: #6735 -- Class based generic views: call for comment

[Patryk Zawadzki]

On Sat, Oct 2, 2010 at 1:05 PM, Patryk Zawadzki <pat...@pld-linux.org> wrote:
> But until Joey appears to give you money (...)

s/until/before/

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: #6735 -- Class based generic views: call for comment

[Patryk Zawadzki]

On Sat, Oct 2, 2010 at 12:49 PM, Russell Keith-Magee
<russ...@keith-magee.com> wrote:
> Python classes have state. People *will* assign variables to self,
> because that's something they have done with every other Python class
> in existence. And as soon as their code hits production, their code
> will break, in unpredictable ways. Claiming that users should
> RTFM/should know better isn't an acceptable excuse in this case, IMHO.
> When an API will break when you use it in the obvious way, that isn't
> something you can hide in documentation and say "we told you so".

Classes that represent real objects have state. Like cars have color,
make and registration number, your models have attributes that
differentiate them from other objects of same class.

"Utility" classes (usually singletons) that represent a function
(service) should not have state.

Imagine walking into a bank and talking to a clerk. She asks for your
name and purpose. You say you want to withrdraw some money. She then
walks up to a giant whiteboard, wipes it and writes your name and your
bank account number there. Then she yells "Joooy" and walks away.
But until Joey appears to give you money, you see another clerk walk
up to the whiteboard, wipe your data and replace it with John Doe. She
then proceeds to exclaim "Joooy" and walks away. That's what a
stateful view is :)

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: #6735 -- Class based generic views: call for comment

[Patryk Zawadzki]

On Fri, Oct 1, 2010 at 3:55 PM, Alex Gaynor <alex.gay...@gmail.com> wrote:
> On Fri, Oct 1, 2010 at 9:53 AM, Vinay Sajip <vinay_sa...@yahoo.co.uk> wrote:
>> On Oct 1, 11:16 am, Johannes Dollinger <emulb...@googlemail.com>
>> wrote:
>>> Could you (or anyone knowledgable) add a section, that explains why each 
>>> request should have its own view instance?
>>> The thread-safety argument alone does not suffice: if all _request_ state 
>>> would be stored on request instead of the view, you wouldn't need new 
>>> instances per request. You could also pass around state explicitly - which 
>>> admittedly gets messy quickly.
>>> So is this purely about preventing users from shooting themselves in the 
>>> foot? (hyperbole: Will there be protection from globals in django 1.4?)
>> I think Johannes has a point. The copy-on-call may seem useful at
>> first sight, but as it's a shallow copy, it may perhaps engender a
>> false sense of security. So
>>
>> view_instance.attr = x
>>
>> will not result in threads stepping on each other's toes, but
>>
>> view_instance.container_attr.attr = x
>>
>> surely will?
>>
>> It seems better to stress thread-safety dos and don'ts in the
>> documentation.
> Without wanting to beat a dead horse, I concur.  Fundamentally we need
> to have a little faith in your users, the admin has the exact same
> pattern of a single instance, and we haven't done anything special
> there.

I second that view. We should tell people how to write thread-safe
code, not try to work around broken solutions. Thread-safe views are
easy. Just don't write to self once you're out of __init__.
Workarounds are not trivial, can make your life hard in wonderful ways
(like "why is __new__ called twice?") and intentionally break best
practices. I'd really prefer to teach people not to stab themselves
instead of forcing everyone to wear a shiny set of armor.

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: proposal for lazy foreignkeys

[Patryk Zawadzki]

On Mon, Sep 27, 2010 at 5:46 PM, Luke Plant <l.plant...@cantab.net> wrote:
> On Mon, 2010-09-27 at 11:36 +0200, Patryk Zawadzki wrote:
>> With the risk of being ignored once again, I dare to link to a working
>> solution that does not need any changed to the framework itself (other
>> than perhaps including the factory class):
>>
>> http://gist.github.com/584106
> This looks rather good to me.  It may have been ignored before because
> it has no comments and some things are not immediately obvious.  For
> example, you are basically proposing that the concrete models are passed
> into view functions via URLconf, and from there are passed into any
> functions which need them, and so they would never actually need to be
> imported by the app that defines the abstract model.
>
> I for one would be much happier to not add any more machinery via Meta
> options. With some cleanup, and some documentation of this pattern, and
> possibly a better name, I think the AbstractMixin class you propose
> could be a good candidate for inclusion in core.
>
> Some notes:
> 1) it seems like line 15 in abstract.py should say 'abstract':'False',
> not 'True' - did I miss something?

It is there because I inherit from the .construct() result rather than
taking it directly as a solid class. This helps with debugging as all
the tracebacks show proper module for the model.

> 2) there would need to be some way of merging the concrete class's own
> Meta options with the abstract class's Meta options

True.

> 3) why do we need the _classcache?  Is the key used specific enough -
> what if two different apps both create 'MyCategory' based on
> 'CategoryFactory', using them in different situations?

I did it so there was no way to create two different mechanics for the
same model accidentally.

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: proposal for lazy foreignkeys

[Patryk Zawadzki]

On Sun, Sep 26, 2010 at 8:10 AM, Russell Keith-Magee
<russ...@keith-magee.com> wrote:
> My biggest technical concern is the same as Alex's -- that it doesn't
> address the 'FK to multiple models' problem. While I agree with your
> 'no silver bullet' response to Alex, I also don't want to end up with
> two (or more) completely different ways of solving the same problem.
> At the very least, I'd like to have some certainty that the solution
> for single concrete class problem will be conceptually similar to the
> multiple concrete class problem.

With the risk of being ignored once again, I dare to link to a working
solution that does not need any changed to the framework itself (other
than perhaps including the factory class):

http://gist.github.com/584106

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: parameterized apps (was: Re: Eric Florenzano's presentation slides)

[Patryk Zawadzki]

On Thu, Sep 9, 2010 at 10:59 PM, Russell Keith-Magee
<russ...@keith-magee.com> wrote:
> The LazyForeignKey pattern has been proposed by a number of parties;
> it's an interesting idea that is worth some serious consideration.
> There are some issues with implementation (e.g., exactly how to
> describe the relationship in a clean way that doesn't require us to
> import settings in order to define models), but these issues shouldn't
> be too hard to sort out. I'll see if I can get some discussion going
> at the sprints over the next couple of days.

See the blog link I posted, there I propose a factory pattern that
allows you to (1) extend the base factory class (2) use the factory to
create the actual Django model in your own app, passing any missing
bits the factory might be expecting.

It solves the "lazy binding" case and also lets you create more than
one model with the same structure and functionality attached.

As for views I recommend something along the lines of:

url(r'^bar/', include('fooapp.urls'), {'model': Bar}),
url(r'^baz/', include('fooapp.urls'), {'model': Baz}),

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: parameterized apps (was: Re: Eric Florenzano's presentation slides)

[Patryk Zawadzki]

On Thu, Sep 9, 2010 at 4:24 PM, Javier Guerra Giraldez
<jav...@guerrag.com> wrote:
> from Eric Florenzano's slide 41:
> (...)

You might want to take a look at my little sandbox here:

http://room-303.com/blog/2010/04/27/django-abstrakcji-ciag-dalszy/

(Sorry the post is in Polish but the only interesting part is the code)

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: CSRF Middleware/SSL/Firefox 3.6.8 bug

[Patryk Zawadzki]

> On Tue, Sep 7, 2010 at 10:04 PM, Luke Plant <l.plant...@cantab.net> wrote:
>> Your method is quite flawed:

Just to make sure we're on the same side -- I'm not trying to make
_CSRF_ more secure (as there's not much more we can do here).

I'm trying to:

* make it work without resorting to Referer checks
* make it not use cookies if it does not have to
* move it to the request class, making the csrf middleware just call
it and show the error if appropriate (this allows certain views to do
the check manually and for example ask the user to verify and
evetually resubmit data instead of showing an error -- I'm strongly
interested in that one)

As for the IP address, User-agent etc. -- feel free to replace both of
them by just md5(user-agent) -- it is guaranteed not to change between
form generation and submission. You can even default the TTL to None.
Just please make sure it does not rely on the Referer header :)

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: CSRF Middleware/SSL/Firefox 3.6.8 bug

[Patryk Zawadzki]

On Tue, Sep 7, 2010 at 10:04 PM, Luke Plant <l.plant...@cantab.net> wrote:
> On Mon, 2010-09-06 at 22:39 +0200, Patryk Zawadzki wrote:
>> Another approach would be not to use a cookie at all. For each {%
>> csrf_token %} use a slightly modified variant of the above
>> encode_cookie function with:
>>
>> values = {
>>     'host: request.META['HTTP_HOST'],
>>     'scheme': request.is_secure(),
>>     'user_ip': request.META['REMOTE_ADDR'],
>>     'user_agent': request.META['HTTP_USER_AGENT'],
>>     'ttl': time.time() + 30*60,
>> }
> Your method is quite flawed:
>
> 1) Use of IP address - a bad idea for the reasons I mentioned in my
> other message.

A lot of web frameworks I've worked with use the IP to lock the
session cookie to prevent easy cookie theft. None of the users
(including mobile ones on GPRS) complained about being constantly
logged out. I didn't use it here as a strong counter-measure, a mild
one at best.

> 2) The use of user agent does nothing to stop an attacker, even for an
> attacker who isn't a MitM:

It's not there as a counter-measure at all. Just to make the resulting
token vary a bit more.

> Consider an attacker who lures you to his site e.g.:
> http://evil.com/somepage.php
>
> somepage.php can read your user agent, and make a request to
> http://target.com/some-page-with-form/ with the same user agent to get a
> valid CSRF token.  (If he is behind the same firewall as you, he will
> automatically have the same public IP, so adding the IP address is not a
> perfect cure for this, even if we could do it, which we can't).

Again, it's not there to counter attacks. Think of it as the
equivalent of the "csrftoken" cookie which could be read in the exact
same way. I just wanted a couple of strings that are not likely to
change between form generation and submission.

> I don't think that adding a timeout will really ever help with CSRF.
> The nature of CSRF attacks means that the attacker is massively more
> likely to be able to attack 'now' (which is within a few seconds, the
> time for a few HTTP requests to complete) rather than 'later', so
> timeouts just don't help. In addition, they add nuisance for the user -
> it's quite possible for someone to leave their machine for 30 minutes
> and come back to it and want to carry on what they were doing.

The timeout serves it purpose beyond CSRF. I'd like the "sign_request"
method (and "check_request_signature" -- see the patch) to be able to
take custom TTL and eventually a custom dict to both encode at signing
time and check against at validation time. This would allow one to
easily implement other mechanics, such as secure API tokens (you don't
want to rely on Cookies there) or easy HTTP→HTTPS transitions (detect
that the token was signed and issued over unsecure connection and ask
the user to resubmit).

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: #13870: Correctly handling database isolation (in PostgreSQL)

[Patryk Zawadzki]

On Mon, Sep 6, 2010 at 7:12 PM, Patryk Zawadzki <pat...@pld-linux.org> wrote:
> It would be more useful if you could explicitly
> enter_isolation_block() and leave_isolation_block() as needed
> (currently there is no way to commit the isolating transaction other
> than doing a raw SQL query or accessing psycopg internals).

Another usecase I just came along:

Say I have to generate some unique string for the database. Something like that:

potential = base = slugify(obj.bar)
suffix = 0
while True:
if not Baz.objects.filter(foo=potential).exists():
obj.foo = potential
obj.save()
break
suffix += 1
potential = '%s-%s' % (base, suffix)

Except it's possible that another process or thread commits an
identical object right between the call to exists() and the save().
What I'd really want to do is something closer to this:

potential = base = slugify(obj.bar)
suffix = 0
found = False
while not found:
enter_isolation_block(ISOLATION_SERIALIZABLE)
if not Baz.objects.filter(foo=potential).exists():
obj.foo = potential
obj.save()
found = True
leave_isolation_block()
suffix += 1
potential = '%s-%s' % (base, suffix)

Of course with new python versions we can get the extra sugar coating:

with db.isolation(ISOLATION_SERIALIZABLE):
    # ...

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: #13870: Correctly handling database isolation (in PostgreSQL)

[Patryk Zawadzki]

On Tue, Sep 7, 2010 at 2:57 PM, David De La Harpe Golden
<david.delaharpe.gol...@ichec.ie> wrote:
> On 06/09/10 14:29, Patryk Zawadzki wrote:
>> The problem only exists when Django sets isolation level to 1, if you
>> use the deprecated "autocommit" setting, you will not be affected.
> Uh. Is this deprecated? At what layer? Since when?  I've been using it*
> happily for a while (modulo that one known "make sure to read once
> before writing" issue with django 1.1), was not aware of any deprecation
> of it?

Not deprecated in Django sense. I don't think it's going away any time
soon. Just that it's been discouraged for a while at the
psycopg/database level.

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: CSRF and Forms

[Patryk Zawadzki]

On Tue, Sep 7, 2010 at 8:38 AM, Russell Keith-Magee
<russ...@keith-magee.com> wrote:
> Firstly, I'm not wild about "secure=request.validated". This looks
> like a really simple way for people to say "secure=True" as a way of
> "fixing" CSRF support that they can't get to work. The choice of
> argument on the attribute isn't optional -- it *must* be
> request.validated. So really, it's the request that is the argument
> that needs to be passed in. The good news on this point is that a
> "request aware form" is something that has been floated in other
> discussions recently. I'll be sure to raise it at the DjangoCon
> sprints this week as a topic for discussion.

After giving it a second thought on my ride to work I think
request.is_valid() would be just as useful and would not require us
breaching the form/request layer separation.

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: CSRF and Forms

[Patryk Zawadzki]

On Tue, Sep 7, 2010 at 8:38 AM, Russell Keith-Magee
<russ...@keith-magee.com> wrote:
> Firstly, I'm not wild about "secure=request.validated". This looks
> like a really simple way for people to say "secure=True" as a way of
> "fixing" CSRF support that they can't get to work. The choice of
> argument on the attribute isn't optional -- it *must* be
> request.validated. So really, it's the request that is the argument
> that needs to be passed in. The good news on this point is that a
> "request aware form" is something that has been floated in other
> discussions recently. I'll be sure to raise it at the DjangoCon
> sprints this week as a topic for discussion.

Agree. In the other CSRF thread I provided a shorter, slightly safer,
cookie-less implementation that works withut Referer headers and
limits each form's lifespan to 30 minutes. I have since implemented a
test form that works like this:

form = Form(request.POST or None, request.FILES or None, request.META)

The form is able to both issue a signed CSRF token (you can just write
{{ form.as_p }} or {{ form.csrf_token }}) and validate it (checks if
POST contains a token, checks the signature and fails form validation
if needed). It could cooperate with the middleware to redirect to an
error if no token was present at all.

Having it this way is useful as it's possible for someone to keep a
form open longer than said 30 minutes and it's more practical to
simply ask for confirmation than to drop all the entered data and
redirect to an error page.

> Secondly, IMHO there is a lot of value in the fact that Django forces
> the raising of a 403, rather than a 200 with an error message on the
> page. This isn't a form input error. It's a catastrophic problem that
> should only be observed when the user is actually under attack, or if
> cookies aren't available. Displaying a CSRF failure in the same way as
> an error for not putting 12 digits in your credit card number strikes
> me as the wrong way to visualize the error case. It implies that the
> problem can be fixed by user interaction when it can't -- at least,
> not by fixing form inputs. In order to give instructions on why
> cookies are needed and how to enable them, we need a lot more real
> estate... like a CSRF view.

See above, I see the value of the error in case of a CSRF attack. In
such cases there will be no valid token present in the payload. My
implementation can however differentiate between a missing token, an
invalid one and one that is simply too old to accept it.

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: #13870: Correctly handling database isolation (in PostgreSQL)

[Patryk Zawadzki]

On Tue, Sep 7, 2010 at 8:29 AM, Thomas Guettler <h...@tbz-pariv.de> wrote:
> If you have a daemon that lives forever, I would do it like this (untested
> if this does not leave an idle transaction open):
> The place where the daemon hangs around if nothing can be done lives must
> not use the ORM. If there is something to be done, the daemon calls methods
> that use the commit_on_success decorators.

Unfortunately you don't always get to choose what to call with what
decorators. For example we have a DBus daemon that acts as an RPC
server. Creating a separate function for each select is not really
practical :)

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: CSRF Middleware/SSL/Firefox 3.6.8 bug

[Patryk Zawadzki]

On Mon, Sep 6, 2010 at 10:39 PM, Patryk Zawadzki <pat...@pld-linux.org> wrote:
> ...

First stab at a patch attached. Did not try to run it yet so it might
contain syntax errors.

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



django-signed-csrf.patch
Description: Binary data

Re: CSRF Middleware/SSL/Firefox 3.6.8 bug

[Patryk Zawadzki]

On Mon, Sep 6, 2010 at 9:11 PM, Patryk Zawadzki <pat...@pld-linux.org> wrote:
> Consider the following example. It was based on the code I wrote to
> handle the new Facebook API so it might be a bit of an overkill but it
> shows it's possible. A simpler solution would be to return (TTL + ':'
> + md5(token + user_ip + secret_key + TTL)).
>
> def encode_cookie(token):
>    values = {
>        'TTL': time.time() + 60*60,
>        'token': token,
>    }
>    payload = simplejson.dumps(values)
>    digest = hmac.new(settings.SECRET_KEY, payload, hashlib.sha256).digest()
>    encoded_payload = base64.b64encode(payload)
>    encoded_digest = base64.b64encode(digest)
>    return '%s.%s' % (encoded_digest, encoded_payload)
>
> def decode_cookie(value):
>    encoded_sig, encoded_payload = map(str, value.split('.', 1))
>    sig = base64.b64decode(encoded_sig)
>    payload = base64.b64decode(encoded_payload)
>    data = simplejson.loads(payload)
>    digest = hmac.new(settings.SECRET_KEY, payload, hashlib.sha256).digest()
>    if str(digest) != sig:
>        return None
>    if data.get('TTL', 0) < time.time():
>        return None
>    return data.get('token', None)

Another approach would be not to use a cookie at all. For each {%
csrf_token %} use a slightly modified variant of the above
encode_cookie function with:

values = {
'host: request.META['HTTP_HOST'],
'scheme': request.is_secure(),
'user_ip': request.META['REMOTE_ADDR'],
'user_agent': request.META['HTTP_USER_AGENT'],
'ttl': time.time() + 30*60,
}

Then when handling a POST request, decipher the token and compare each
META field with the ones from the request and validate ttl against
time.time(). I believe it's not less secure than the current
implementation and solves two problems:

1) each form served gets its own ttl, an attacker can't keep pinging
the server to keep the token alive
2) each token serves for a single use and will inevitably timeout in
30 minutes while still allowing you to open two forms in two browser
tabs and submit each of them separately

Attaching such an encoded string to the request should be harmless, it
won't be much longer than the cookie header, will not be sent with
each request (only sent when rendering a form, only received with form
submission).

If you like this approach, I suggest letting the form both generate
and validate the token if the middleware is not active. See my other
CSRF thread for the use case.

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

CSRF and Forms

[Patryk Zawadzki]

Hi,

Since CSRF is already being reafactored up-side down in the trunk, I
thought it might be a good idea to propose a slight modification.

Currently CSRF either falls through to the resolved view function or
calls settings.CSRF_FAILURE_VIEW. More than once I've found myself
wanting something in between. In such cases I'd like the logic flow to
be able to reach the view, just telling me that CSRF did not validate.

Let's say we add a new attribute to the request, call it "validated"
and make it default to True for GET and False for everything else. Now
split the CSRF middleware into two separate pieces of code. One
middleware that does the validation and sets request.validated to True
on success. One middleware that checks for (request.method == 'POST'
and not request.validated) and in such cases returns
settings.CSRF_FAILURE_VIEW.

"How is that useful?" I hear you ask.

class SecureForm(forms.Form):
def __init__(self, *args, **kwargs):
self.secure = kwargs.pop('secure', False)
return super(SecureForm, self).__init__(*args, **kwargs)
def _clean_form(self, *args, **kwargs):
if not self.secure:
self._errors[NON_FIELD_ERRORS] = self.error_class([
'We could not confirm that the request originated from
your machine. Please resubmit to continue.'
])
else:
super(SecureForm, self)._clean_form(self, *args, **kwargs)

def MyForm(SecureForm):
foo = forms.CharField()

def my_view(request):
myform = MyForm(request.POST or None, request.FILES or None,
secure=request.validated)
if myform.is_valid():
# ...
pass
return direct_to_template(request, 'my.html', {'form': myform})

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: CSRF Middleware/SSL/Firefox 3.6.8 bug

[Patryk Zawadzki]

On Mon, Sep 6, 2010 at 8:25 PM, Luke Plant <l.plant...@cantab.net> wrote:
> Almost everything in this discussion and all your misconceptions are
> covered here:
>
> http://code.djangoproject.com/wiki/CsrfProtection

I've read it before joining the discussion.

> For HTTP (not HTTPS), MitM attacks are out of scope for our CSRF
> protection, because a MitM sees everything in the clear and will easily
> be able to defeat any defences we put up.  Your suggested alterations do
> nothing to stop that.

I wasn't targetting HTTP at all. But I can't agree that there's
nothing we can do.

We can encode TTL into the CSRF cookie¹ to make sure being able to
eavesdrop at one point in time does not grant you permanent access to
the server.

> In the context of HTTPS, the Referer header check does indeed add a
> necessary and effective protection. The nature of a CSRF attack is that
> the attacker has to gets the *user's* browser to make the request, not
> the attacker's.

Plain CSRF is not interesting to me, requiring any non-trivial token
is enough to prevent blind CSRF attacks.

I am more interested in preventing an attack that consists of
eavesdropping a cookie sent over HTTP and using it to forge a POST
using a secure connection.

> The curl command line says nothing at all - I'm well
> aware that the attacker can do that from their own machine if they want
> to, but it won't do them any good! I could not understand how you
> thought that showing a curl command line is at all relevant.

See above, I'm concerned about the case when the attacker is able to
perform a replay attack. If you validate CSRF against the client's IP
(and possibly against a TTL field¹), you effectively prevent that type
of attack. If the CSRF tokens were stored in the database (much like
sessions are), you'd gain additional security of all tokens expiring
after one use (or TTL reached).

> A MitM cannot tamper with requests that are sent over SSL.  The only
> exception to this is that if HTTP -> HTTPS POST requests are allowed,
> the MitM can indeed generate any request they like and get the user's
> browser to send it, with potentially damaging consequences.

The MitM can also trick the user into POSTing same data over HTTP and
forge a HTTPS request of his own. That's why I propose using a
separate, secure CSRF cookie (using a different name) for HTTPS
requests. Even if you trick the user into sending data over HTTP, you
will not gain access to the secure token. Any views that require HTTPS
will not be exploited no matter what the attacker sends to the
browser.

> To stop
> this we, we simply refuse HTTP -> HTTPS POST requests. In the context of
> an HTTPS request from a browser, we can indeed trust the Referer header
> since no-one can tamper with it in transit, and the MitM will not be
> able to convincingly send the user's browser a page that looks like it
> comes from https://example.com.

I believe you can use mere cookies to prevent this (see above) and
keep the site working for people who don't and can't send Referer
headers.

> Please distinguish between the HTTP/HTTPS context if you've still got
> questions.

As I said, I am only concerned about using HTTP as an attack vector to
access HTTPS.


¹ Consider the following example. It was based on the code I wrote to
handle the new Facebook API so it might be a bit of an overkill but it
shows it's possible. A simpler solution would be to return (TTL + ':'
+ md5(token + user_ip + secret_key + TTL)).

def encode_cookie(token):
values = {
'TTL': time.time() + 60*60,
'token': token,
}
payload = simplejson.dumps(values)
digest = hmac.new(settings.SECRET_KEY, payload, hashlib.sha256).digest()
encoded_payload = base64.b64encode(payload)
encoded_digest = base64.b64encode(digest)
return '%s.%s' % (encoded_digest, encoded_payload)

def decode_cookie(value):
encoded_sig, encoded_payload = map(str, value.split('.', 1))
sig = base64.b64decode(encoded_sig)
payload = base64.b64decode(encoded_payload)
data = simplejson.loads(payload)
digest = hmac.new(settings.SECRET_KEY, payload, hashlib.sha256).digest()
if str(digest) != sig:
return None
if data.get('TTL', 0) < time.time():
return None
return data.get('token', None)

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: #13870: Correctly handling database isolation (in PostgreSQL)

[Patryk Zawadzki]

On Mon, Sep 6, 2010 at 4:47 PM, Robert Gravsjö <rob...@blogg.se> wrote:
> Patryk Zawadzki skrev 2010-09-06 15.29:
>> The isolating transaction keeps going on until you either (1) commit,
>> (2) rollback or (3) disconnect. Django only commits/rollbacks the
>> transactions it explicitly starts, it does not care about the
>> implicitly started isolating transaction. That's what results in
>> "  in transaction" and I can reproduce it with a two-line view
>> that does a simple SELECT with no transaction middleware involved.
> Can you please show me the code you're running to reproduce this?

Right, I misremembered the original problem. I've now found the
testing environment.

The problem is not with regular views but with Celery tasks,
long-running management commands such as daemons and any other place
where you access the ORM from outside of the usual
request→dispatcher→view→response flow. These cases all end up with an
isolating transaction spanning their whole life span. In case of
daemons it results in permanently blocking the database structure (for
example causing "VACUUM FULL" to hang).

It would be more useful if you could explicitly
enter_isolation_block() and leave_isolation_block() as needed
(currently there is no way to commit the isolating transaction other
than doing a raw SQL query or accessing psycopg internals).

Sorry about the confusion, it's not really related to views.

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: CSRF Middleware/SSL/Firefox 3.6.8 bug

[Patryk Zawadzki]

On Mon, Sep 6, 2010 at 6:34 PM, Luke Plant <l.plant...@cantab.net> wrote:
> OK, that does it.  I call 'troll'.  If you're not trolling, my
> apologies, but I have run out of energy trying to explain this to you.

I'm still waiting for you to explain anything.

You said you were afraid of a third-party injecting cookies over HTTP
that would be then sent over HTTPS.

I replied by telling you that currently it does not have to do any
injecting as it can just forge both the cookie and the token when
doing a malicious POST (attaching any other cookies and post data
needed to DoEvil™) -- current CSRF implementation does not stop such
attacks at all.

Moreover I proposed a more secure cookie header that is both immune to
stealing (as it's calculated and checked using your IP address) and
forging (as you no longer ask the client to send you the same string
twice, both post payload and headers being in plaintext).

The curl example was there to illustrate the problem of the current
implementation: a malicious client or proxy is free to replace both
the cookie and post payload and CSRF will validate. If I can capture a
HTTP request (the MitM case) and read the headers, it's just a matter
of taking the cookies, replacing the CSRF token cookie with "foo" and
I am free to make any request on the site I want for as long as I
want. Requiring me to send a "Referer" header is a mild inconvenience
at most (just send "https://domain.com/;).

Your only reply is calling me a troll.

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: CSRF Middleware/SSL/Firefox 3.6.8 bug

[Patryk Zawadzki]

On Mon, Sep 6, 2010 at 5:56 PM, Patryk Zawadzki <pat...@pld-linux.org> wrote:
> In fact current implementation is weaker than the old one. You are now
> depending on the client delivering the challenge and the response,
> both being sent over HTTP. An attacker no longer has to steal a valid
> session cookie, it's enough to pick any string ("foo" will suffice)
> and send it both in cookie headers and in POST payload.

Mandatory demo attack:

$ curl --cookie \
csrftoken=foo \
--form csrfmiddlewaretoken=foo \
--referer https://example.com/innocent
https://example.com/do/something/evil

:)

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: CSRF Middleware/SSL/Firefox 3.6.8 bug

[Patryk Zawadzki]

On Mon, Sep 6, 2010 at 5:54 PM, Luke Plant <l.plant...@cantab.net> wrote:
> On Mon, 2010-09-06 at 16:42 +0200, Patryk Zawadzki wrote:
>> If you use a separate *secure* cookie for CSRF over SSL then it's not
>> possible to submit data from an unsafe page to a safe processor.
>> Secure cookies are only sent to SSL targets so it's not possible to
>> intercept the cookie by any means.
> No - the 'secure' flag for cookies only stops the cookie being sent over
> a non-secure connection.  It *cannot* be used to stop them being *set*
> over non-secure connections, which is what we need. And cookies without
> the secure flag will still be sent over a secure connection.

If you use separate cookie names for SSL and non-SSL requests, the SSL
one will never be set over unsecure connections by Django itself.

If you try to prevent the cookies from being set by a malicious proxy,
the current security scheme is so weak, you cannot guard against it
anyway. The attacked does not need to set a cookie or even steal it,
it's enough for him to make a request forging both POST and the Cookie
header - see my other reply in the thread.

> And, once again, this is not about the cookie being intercepted, it is
> about it being *set* over an HTTP connection and then being *sent* over
> HTTPS.

That part is taken care of above.

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: CSRF Middleware/SSL/Firefox 3.6.8 bug

[Patryk Zawadzki]

On Mon, Sep 6, 2010 at 4:42 PM, Patryk Zawadzki <pat...@pld-linux.org> wrote:
> On Mon, Sep 6, 2010 at 3:55 PM, Luke Plant <l.plant...@cantab.net> wrote:
>> On Sun, 2010-09-05 at 19:49 +0200, Patryk Zawadzki wrote:
>>> As for the vulnerability -- it's only there if you implement it
>>> yourself. If you send the initial login form over SSL (we do it this
>>> way for various reasons), the cookies are never prone to be
>>> intercepted.
>> No, the vulnerability we are talking about is there even if *you* don't
>> send the forms over HTTP, because the MitM can insert the forms
>> themselves (and send a matching CSRF cookie) into any page served over
>> HTTP, and have those forms target HTTPS connections, and submit them
>> automatically by javascript.  It's not about cookies being intercepted,
>> it's about them being set by a MitM over HTTP.
> Of course if this is a true MitM attack, the attacker could just take
> your cookie and forge a request of his own, including a perfectly fine
> REFERER header.

Actually after reading the code for the current implementation¹ it
seems it does nothing to prevent MitM attacks. It only prevents basic
CSRF attacks where the attacker is not able to eavesdrop the
request/response cycle. The site still needs to rely on mechanisms
such as django.contrib.session to prevent MitM attacks.

In fact current implementation is weaker than the old one. You are now
depending on the client delivering the challenge and the response,
both being sent over HTTP. An attacker no longer has to steal a valid
session cookie, it's enough to pick any string ("foo" will suffice)
and send it both in cookie headers and in POST payload.

I'd suggest setting the cookie to something like:

md5(META["CSRF_TOKEN"] + META["REMOTE_ADDR"] + settings.SECRET_KEY)

¹ http://code.djangoproject.com/browser/django/trunk/django/middleware/csrf.py

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: CSRF Middleware/SSL/Firefox 3.6.8 bug

[Patryk Zawadzki]

On Mon, Sep 6, 2010 at 3:55 PM, Luke Plant <l.plant...@cantab.net> wrote:
> On Sun, 2010-09-05 at 19:49 +0200, Patryk Zawadzki wrote:
>> As for the vulnerability -- it's only there if you implement it
>> yourself. If you send the initial login form over SSL (we do it this
>> way for various reasons), the cookies are never prone to be
>> intercepted.
> No, the vulnerability we are talking about is there even if *you* don't
> send the forms over HTTP, because the MitM can insert the forms
> themselves (and send a matching CSRF cookie) into any page served over
> HTTP, and have those forms target HTTPS connections, and submit them
> automatically by javascript.  It's not about cookies being intercepted,
> it's about them being set by a MitM over HTTP.

If you use a separate *secure* cookie for CSRF over SSL then it's not
possible to submit data from an unsafe page to a safe processor.
Secure cookies are only sent to SSL targets so it's not possible to
intercept the cookie by any means. I suggest you enforce that by
default (if CSRF_ALTERNATIVE is False).

Now if CSRF_ALTERNATIVE is True, feel free to use the same cookie for
HTTP and HTTPS and rely on the REFERER header. This will of course
break for a small minority of internet users, probably tech-savvy ones
who are respected enough to convince other people not to use my
application.

Of course if this is a true MitM attack, the attacker could just take
your cookie and forge a request of his own, including a perfectly fine
REFERER header.

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: #13870: Correctly handling database isolation (in PostgreSQL)

[Patryk Zawadzki]

On Mon, Sep 6, 2010 at 2:00 PM, Robert Gravsjö <rob...@blogg.se> wrote:
> I'm not sure what you think you are doing but if you end up with "
>  in transaction" that means you have not commited your transactions.

See below.

> For instance, open two connections with psql and run BEGIN in one and
> then take a look at pg_stat_activity you will have that connection
> marked as idle in transaction.
>
> I believe you are confused about isolation levels. They control the
> visibility of transactions.

Maybe I am indeed confused but quoting the psycopg documentation¹:

"psycopg2.extensions.ISOLATION_LEVEL_READ_COMMITTED:

This is the default value. A new transaction is started at the first
execute() command on a cursor and at each new execute() after a
commit() or a rollback(). The transaction runs in the PostgreSQL READ
COMMITTED isolation level."

The isolating transaction keeps going on until you either (1) commit,
(2) rollback or (3) disconnect. Django only commits/rollbacks the
transactions it explicitly starts, it does not care about the
implicitly started isolating transaction. That's what results in
" in transaction" and I can reproduce it with a two-line view
that does a simple SELECT with no transaction middleware involved.

The problem only exists when Django sets isolation level to 1, if you
use the deprecated "autocommit" setting, you will not be affected.

¹ http://initd.org/psycopg/docs/extensions.html#isolation-level-constants

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: #13870: Correctly handling database isolation (in PostgreSQL)

[Patryk Zawadzki]

On Mon, Sep 6, 2010 at 1:35 PM, burc...@gmail.com <burc...@gmail.com> wrote:
> Hi Patryk,
>
> This was done to make Django faster, so it doesn't create connection
> to database every time new SQL is executed.

What was done? Commiting the isolating transaction of a connection
does not terminate it.

> Do you mean you wanted to set up timeouts for idle database connections?
> I guess, nobody asked this before.

No, I think I stated quite clearly what my proposal was.

http://initd.org/psycopg/docs/extensions.html#isolation-level-constants

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: #13870: Correctly handling database isolation (in PostgreSQL)

[Patryk Zawadzki]

On Mon, Sep 6, 2010 at 11:20 AM, Kirit Sælensminde (kayess)
<kirit.saelensmi...@gmail.com> wrote:
> Strange. We use Postgres and don't see any problem with this. We do
> encounter complications occasionally with the lack of composition of
> Django transaction handling, but other than that find the transaction
> handling adequate.
>
> Are you actually using transactions in your code? You need to be. The
> transaction middleware does a good enough job for most things, but for
> external processing you will need to carefully design your transaction
> handling. With the transaction middleware we see fully isolated
> updates until the transaction is committed using psycopg2.

I'm not sure you understand the problem at all. The problem is not
lack of the isolation. The problem is permanent isolation. The
isolating transaction is never terminated, thus remaining alive for
indefinite amounts of time.

You can check this by creating a fresh project using psycopg2 and
creating a model. Then write a view that queries the database and
invoke it. Now, leaving the server running, open up your database
shell. The pg_stat_activity table will report " in transaction"
for hours.

BTW: This has nothing to do with Django transactions or TransactionMiddleware.

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: CSRF Middleware/SSL/Firefox 3.6.8 bug

[Patryk Zawadzki]

On Sat, Sep 4, 2010 at 1:51 AM, Luke Plant <l.plant...@cantab.net> wrote:
> Barth, Jackson and Mitchell [1] collected some data that said that for
> same-domain HTTPS POST requests, the header is missing in only 0.05% to
> 0.22% of cases.  They've also got strong evidence that the header is
> suppressed in the network, not by the browser.

17.6% of statistics are wrong unless you can prove it otherwise ;)

It does not matter if only 5 of 10k surveyed computers didn't send the
header. There are whole class C sub-networks that don't by policy.

As for the vulnerability -- it's only there if you implement it
yourself. If you send the initial login form over SSL (we do it this
way for various reasons), the cookies are never prone to be
intercepted. I have a strong feeling that the framework should not
hinder interoperation to try and save the developer from his stupidity
unless explicitly asked to do so.

If you really want to use POST in HTTP → HTTPS transitions, introduce
settings.CSRF_WHATEVER, document it thoroughly and make it default to
False.

Now everyone is happy :)

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

#13870: Correctly handling database isolation (in PostgreSQL)

[Patryk Zawadzki]

Some time ago I've reported ticket #13870¹. Here's a brief explanation
of the problem:

PostgreSQL has a concept called "isolation levels". These are various
types of meta-transactions.

1. There's level 0 which means "off" and results in ghost reads
(SELECT returning data not yet commited).
2. The most useful level is 1 which prevents ghost-reading uncommited
data yet does not guarantee atomicity (it's possible for two identical
SELECTs to return different results).
3. There's also a third "serializable" level that guarantees full
atomicity but it's not used by Django.

In psycopg(2) the default isolation level is 0 or "no isolation at
all". If enabled, a meta-transaction is designed to work as follows:

1. The first query you execute results in an implicit "BEGIN" being
called internally by PostgreSQL
2. Any consecutive queries are executed in the same transaction
3. You terminate the meta-transaction by calling connection.commit()
or connection.rollback()
4. The first query you execute results in an implicit "BEGIN" being
called internally by PostgreSQL
5. Any consecutive queries are executed in the same transaction
6. You terminate the meta-transaction by calling connection.commit()
or connection.rollback()

...and so on.

The problem is that this is not true for Django. The backend
initializes the meta-transaction at connection time and never bothers
to terminate it. As the transaction is merely a ghost-read-preventing
one, it does not stop data from being saved in the database, but it
does result in a parmanently open transaction. This both wastes
resources on the server and prevents any other client from touching
the database structure (any ALTER or VACUUM command will hang waiting
for the remaining transactions to finish).

The ticket contains a naïve workaround but as described there, I feel
a proper solution would look like this:

1. Introduce IsolationMiddleware that does something like this (pseudocode):

class IsolationMiddleware(object):
def process_request(self, request):
for db in database_connections:
db.enter_isolation_block()
def process_response(self, request, response):
for db in database_connections:
db.leave_isolation_block()

2. Make the middleware default and describe it in the migration guide.

3. Introduce no-op enter_isolation_block() and leave_isolation_block()
that just "pass"

4. Override both methods in the pgsql backends (pseudocode):

def enter_isolation_block(self):
if self.isolation_level:
self.connection.set_isolation_level(self.isolation_level)

def leave_isolation_block(self):
if self.isolation_level and self.connection.get_transaction_status() == \
psycopg2.extensions.TRANSACTION_STATUS_INTRANS:
if self.commited:
self.connection.commit()
else:
self.connection.rollback()

5. Remove code that sets isolation level in the connection initialization code.

¹ http://code.djangoproject.com/ticket/13870

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: contrib.admin:list_editable - ForeignKey Performance is O(m*n)?

[Patryk Zawadzki]

On Tue, Jun 29, 2010 at 6:04 PM, chadc <chadco...@hotmail.com> wrote:
> Hi there,
>
> I have been experiencing poor performance with django-admin when
> list_editable includes ForeignKey fields. In particular, rendering the
> changelist requires O(m*n) database queries where m is the number of
> ForeignKey fields included in list_editable and n is the number of
> rows in the changelist. I have searched extensively for possible
> causes and, after finding nothing and receiving no response on either
> django-users ('list_editable duplicate queries') or IRC, I am starting
> to think that it is a legitimate bug.

While this is worth to cache, you need to index the cache using a
serialized query, not the model.

It is possible to have a different set of choices for each model
instance (for example applying extra filter using another field's
value).

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Suggestion of exception Http301 add as new feature

[Patryk Zawadzki]

On Mon, Jun 28, 2010 at 12:07 PM, kernel1983 <kernel1...@gmail.com> wrote:
> There is exception Http404 in the system.
>
> During django programming, we often need to do some data valid:
>
> def A(request):
>    valid()
>    return render_to_response(...)
>
> def B(request):
>    valid()
>    return render_to_response(...)
> ...
>
> def valid():
>    throw Http301(url)

from django.shortcuts import redirect

def valid(method):
def decorated(request, *args, **kwargs):
if not 'Tim' in request.user.called_by_some_people():
return redirect('some-view', ...)
return method(request, *args, **kwargs)
return decorated

@valid
def A(request):
    return render_to_response(...)

@valid
def B(request):
    return render_to_response(...)

print 'There you go.'

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Class based generic views in 1.3?

[Patryk Zawadzki]

On Fri, Jun 18, 2010 at 9:22 AM, Waldemar Kornewald
<wkornew...@gmail.com> wrote:
> On Thu, Jun 17, 2010 at 10:45 PM, Jacob Kaplan-Moss <ja...@jacobian.org> 
> wrote:
>> On Thu, Jun 17, 2010 at 3:41 PM, Alex Gaynor <alex.gay...@gmail.com> wrote:
>>> So here's an idea, of dubious quality ;)
>> Okay, folks: at this point the discussion has pretty much descended
>> into bikeshedding territory. I'm going to read through everything
>> posted so far and try to post a summary and round-up to help us get
>> refocused; gimme a few hours to pull that together and then let's try
>> to reach towards a consensus.
> I've started putting everything into a summary spreadsheet. Feel free
> to change/extend it:
> https://spreadsheets.google.com/ccc?key=0AnLqunL-SCJJdGhxSVZaQkNCcTlzM2d4OEc5dFRPUUE=en

Some of the approaches also allow for easy grouping of similar views.
For example I'm currently working on an ArchiveView with methods:

* index(self, request)
* details(self, request, year, month, day, id)
* day(self, request, year, month, day)
* week(...)
* month(...)
* year(...)

So you only instantiate one object (telling it which templates to use,
what month format you prefer etc.) and bind its methods to various
URLs.

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Class based generic views in 1.3?

[Patryk Zawadzki]

On Thu, Jun 17, 2010 at 8:00 PM, Waldemar Kornewald
<wkornew...@gmail.com> wrote:
> Imagine what the code would look like with parameter passing:
>
> class View(object):
>    def __call__(self, request, **kwargs):
>        qs = self.get_queryset(request, **kwargs)
>        .
>
>    def get_queryset(self, request, **kwargs):
>        ...

Why would you hardcore the queryset inside the view? Pass the
pre-filtered queryset to the constructor and use get_objects_or_404 to
filter it basing on what is passed to __call__.

Actually as I see it __call__ is not the best idea:

class GenericView(object):
def __init__(self, queryset):
self.qs = queryset

def details(self, request, id):
obj = get_object_or_404(self.qs, id=id)
return direct_to_template(request, 'details.html', {'object': obj})

def list(self, request):
objs = get_objects_or_404(self.qs)
return direct_to_template(request, 'list.html', {'objects': objs})

# ... and so on

> Now someone writes a reusable view for S3 file management which only
> has one parameter (bucket_name) and publishes the code as an
> open-source project:
>
> class S3View(View):
>    def get_queryset(request, bucket_name, **kwargs):
>        self.get_bucket(bucket_name)
>        ...
>    ...

Are you sure you need to pass the bucket name each time you _call_ the
view? I'd pass it at construction time instead.

> Then you want to use his code in your project:
>
> class MyS3View(S3View):
>    def get_bucket(self, bucket_name):
>       # oh no! I need the request and the other configuration
> paramters here! :(

Request sure, but what other configuration parameters? Default
implementation should under no circumstances pass and silently swallow
**kwargs.

> Now you have a problem because the other developer was too lazy to
> pass request and **kwargs to that little "unimportant" get_bucket
> method. This happens in practice and our job is to come up with a
> design that makes this impossible or very unlikely. If the
> one-instance-per-request solution solves this and saves you from
> typing **kwargs all over the place then it absolutely is better, hands
> down.

See above.

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Class based generic views in 1.3?

[Patryk Zawadzki]

On Thu, Jun 17, 2010 at 6:49 PM, Waldemar Kornewald
<wkornew...@gmail.com> wrote:
> The one-instance approach is no more thread-safe than having a global
> variable. In your example nothing bad will happen, but once we get to
> some real-world reusable views you'll quickly see a need for saving
> thread-local state somewhere. We had that problem with Django's Feed
> class, for example, where we needed access to the request in one of
> the methods and the only workaround was to instantiate a new Feed for
> every request and manually inject a _request attribute. The point is
> that in a class-based view you normally have lots of methods for
> customizing the view's behavior, so either you pass the whole
> thread-local state as arguments to every single method (which is
> *very* ugly and tedious) or you save that state in self. Also, the
> problem with passing lots of arguments around is that it can make
> adding more variables to the current state and thus customizability
> difficult. There's no way around allowing thread-local state and the
> one instance per request approach solves this elegantly.

So the only limitation is that you don't want to pass parameters to methods?

Good thing self is passed implicitly or classes would be
totally useless! ;)

More seriously:

Explicit is better than implicit.
Simple is better than complex.
[...]
If the implementation is hard to explain, it's a bad idea.

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Class based generic views in 1.3?

[Patryk Zawadzki]

2010/6/17 Ian Lewis <ianmle...@gmail.com>:
> The example he provided isn't terribly good but you don't need an view
> instance per request (the example will work unless you start adding
> stuff to self or overwriting self.qs etc). Only shared state is stored
> at the class level and view customization is done at the class
> instance level as well. That state shouldn't change.
>
> Attaching all kinds of stuff to self tends to depend on the order of
> the methods called and allows for some ugly coding styles. In that
> case the slightly strange idea of merging the request and the view by
> extending HttpRequest starts to make sense since otherwise you would
> have some state on the request object (like the user etc. as it is
> now) and some state on the view object.
>
> That said, both extending HttpRequest and using __new__ seem like a
> hacks to me. Personally the idea of a view being a callable class
> instance or method is simple and flexable enough to handle any use
> cases we are talking about. You just have to think of self as the
> *view* rather than a thread safe request because that's what it is.
>
> Having a view object per request makes no sense. You should have a
> request object per request.  It is largely a result of wanting the
> convenience of being able to write sloppy code that writes all kinds
> of state to self rather than a technical necessity.

You, sir, deserve a beer!

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Class based generic views in 1.3?

[Patryk Zawadzki]

On Thu, Jun 17, 2010 at 2:08 PM, Waldemar Kornewald
<wkornew...@gmail.com> wrote:
> On Thu, Jun 17, 2010 at 1:42 PM, Patryk Zawadzki <pat...@pld-linux.org> wrote:
>> Here's an example of a thread-safe view that works happily with just
>> one instance:
>>
>>  8< 
>>
>> class MyView(object):
>>    def __init__(self, queryset, template='bar.html'):
>>        self.qs = queryset
>>        self.template = template
>>
>>    def __call__(self, request, id):
>>        instance = get_object_or_404(self.qs, id=id)
>>        return direct_to_template(request, self.template, {'object': 
>> instance})
>>
>> # ...
>>
>>    url(r'^foo/(?P\d+)/', MyView(queryset=Foo.objects.all(),
>> template='foo.html')),
> This is not at all thread-safe because all requests share the same
> view instance and thus the same state. Shared state is not thread-safe
> (unless you introduce some locking mechanism which is not an option
> here). Thread-safety can only be guaranteed if every thread has its
> own state. In other words, every request needs his own View instance.

You don't seem to get how threading works.

There is no state in this object past the assignments in __init__.
These are thread-safe as there is only one instance of the object.

Variables in local scope are always thread-safe as reetrancy grants
you a new local scope every time it enters a function.

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Class based generic views in 1.3?

[Patryk Zawadzki]

On Thu, Jun 17, 2010 at 1:20 PM, Waldemar Kornewald
<wkornew...@gmail.com> wrote:
> Please take a deeper look at his code. He doesn't use __init__. He
> uses __new__, so each request gets his own View instance.
>
> Instead of
>
> aa = AwesomeAdd()
> foo = aa(3, 5)
>
> the __new__-based approach allows to do this:
>
> foo = AwesomeAdd(3, 5)
>
> IOW, the "constructor" directly returns an HttpResponse (foo) instead
> of an AwesomeAdd instance.

There is no difference between using __init__ and __new__ as long as
you treat the instance as a junkyard for temporary variables. If done
properly, class-based views do _not_ need a separate instance for each
request.

Here's an example of a thread-safe view that works happily with just
one instance:

 8< 

class MyView(object):
def __init__(self, queryset, template='bar.html'):
self.qs = queryset
self.template = template

def __call__(self, request, id):
instance = get_object_or_404(self.qs, id=id)
return direct_to_template(request, self.template, {'object': instance})

# ...

url(r'^foo/(?P\d+)/', MyView(queryset=Foo.objects.all(),
template='foo.html')),

 8< 

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Class based generic views in 1.3?

[Patryk Zawadzki]

On Thu, Jun 17, 2010 at 1:44 AM, Nick Fitzgerald <fitz...@gmail.com> wrote:
> I have forked bfirsh's code as well on
> github: http://github.com/fitzgen/django-class-based-views
> I have changed the code to use __new__ rather than copying self. I have also
> merged in daonb's commit to make request implicit to all methods via
> self.request.
> All feedback is very welcome! :)

Is there any reason why all of you think not passing request around is a gain?

Would you recommend anyone to do:

 8< 

request = None

def my_view():
global request
return HttpResponse('Hi, %s' % (request.user.get_full_name(), ))

def my_other_view():
globl request
return HttpResponse('Hello again, %s' % (request.user.get_full_name(), ))

# ...

 8< 

As I seriously see no difference between this and pushing stuff to self.request.

Request objects are esoteric. They are only meaningful during that
short period of time you spend in the __call__ method. Once you return
a HttpResponse, request is ready to retire. Storing them at the level
objects grants them potentially eternal life for no reason.

Consider this:

 8< 

class AwesomeAdd(object):
def __call__(self, number1, nubmer2):
self.number1 = number1
self.number2 = number2
return self.do_the_math()

def do_the_math(self):
# OMG BBQ, we cleverly avoided passing params around!!1!one
return self.number1 + self.number2

aa = AwesomeAdd()

foo = aa(3, 5)

 8< 

Doesn't that scream "WTF"? :)

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Class based generic views in 1.3?

[Patryk Zawadzki]

On Wed, Jun 16, 2010 at 9:24 AM, daonb <bennyd...@gmail.com> wrote:
> Actually, the current code is not that sure on the second point,
> here's a line from base.py:
>
> view.request = request # FIXME: Maybe remove? Should this be
> encouraged?
>
> As I see it, if we're cloning the view, it should be encouraged. I
> forked Ben's code and refactored it so that instead of having the
> methods pass 'request' around, use self.request. I believe it made the
> generic views more readable and it will help users' make their views
> cleaner. My fork is at http://github.com/daonb/django-class-based-views

I consider using self to store temporary variables a bad design
practice. You wouldn't do that in C++ or Java. Having said that I
might be biased.

If you really want to do such hacks, use threading.local instead of
writing to self directly:

class View(object):
storage = threading.local()
def __call__(self, request, id):
self.storage.request = request
# ...

Cloning objects on each call to a method is a really really really
really bad idea and puts unnecessary stress on the garbage collector
(actually if any of the methods are careless enough to pass self to an
external function, it might result in a steady leak of non-freeable
memory).

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Class based generic views in 1.3?

[Patryk Zawadzki]

If you don't mind, I'll talk a bit more to myself.

On Tue, Jun 15, 2010 at 3:38 PM, Patryk Zawadzki <pat...@pld-linux.org> wrote:
> I really don't think we should add anti-moron filters at the framework
> level past documenting "taking care to ensire immutability of view
> instances".

This came out a bit harsh. What I mean is: teach users how to write
thread-safe views (classes or not) and ignore those who refuse to do
it. They either know what they're doing or there is really no point in
forcing the help upon them.

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Class based generic views in 1.3?

[Patryk Zawadzki]

After giving it more thought:

The supposed View class is a pure utility class that is close to a
module. It does not represent a state in an application, it takes
input and returns output, none of it should outlive a "__call__" call.
Therefore I see no reason for a proper implementation to not be
thread-safe. Just treat "self" as immutable past "__init__". In other
words, don't treat "self" as a junkyard for temporary variables.

This is WRONG:

class View(object):
def greet(self):
return HttpResponse('Hi %s' % self.request.POST.get('hello', ''))
def __call__(self, request):
self.request = request
return greet(self)

This is CORRECT and not any harder to design/implement:

class View(object):
def greet(self, request):
return HttpResponse('Hi %s' % request.POST.get('hello', ''))
def __call__(self, request):
return greet(self, request)

I really don't think we should add anti-moron filters at the framework
level past documenting "taking care to ensire immutability of view
instances".

A true moron will always outsmart us with creativity. It's perfectly
possible to write thread-unsafe code without using classes:

req = None

def greet(self, request):
global req
req = request
# ...
return HttpResponse('Hi %s' % req.POST.get('hello', ''))

or even:

def greet(self, request):
greet.foo = request
# ...
return HttpResponse('Hi %s' % greet.foo.POST.get('hello', ''))

:)

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Class based generic views in 1.3?

[Patryk Zawadzki]

On Fri, Jun 11, 2010 at 7:34 PM, Patryk Zawadzki <pat...@pld-linux.org> wrote:
> Maybe something simpler?

Actually you might want to have a couple of instances with different
__init__ params:

 8< 

from threading import local, Thread

class View(object):
_child = local()

def __new__(cls, *args, **kwargs):
if not hasattr(cls._child, 'dict'):
cls._child.dict = {}
param_hash = (args, tuple(kwargs.items()))
existing = cls._child.dict.get(param_hash, None)
if not existing:
print 'first time in this thread with args', param_hash
cls._child.dict[param_hash] = super(View, cls).__new__(cls)
return cls._child.dict[param_hash]

def __call__(self, foo):
print 'call', id(self), foo
return 'bar'

-- 
Patryk Zawadzki

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.