Re: Potential way forward for DATABASE_URL

[Raphael G]

To be clear, "pressure to get the API right on the first shot" is a 
statement of fact about adding APIs to a heavily used project in general, 
not a comment on anything said in this mailing thread. Just that in this 
case there's a lot of ways to get the ergonomics wrong.

On Wednesday, December 21, 2022 at 12:06:44 AM UTC+9 Raphael G wrote:

> OK after looking at this some more and trying to write up a review aid, 
> I'm giving up on this branch and trying to integrate DATABASE_URL support 
> into Django proper.
>
> A couple reasons:
>
> - I misread the original mailing list thread which made me think there was 
> a consensus on this branch, and there wasn't.
> - There are a lot of tiny backend-specific things going on that are in 
> because... well because django-database-url has that backend-specific 
> behavior
> - Lots of people want the cache backend as a part of this. I'd be happy to 
> have it. But basically every cache backend has weirdness (what's a service 
> URL for the dummy backend? do we really need one for the file backend?), so 
> ... I don't even know what makes sense there, honestly.
>
> So there's this dual thing of not wanting to "just" vendor 
> django-database-url, but really the original branch that I tried to revive 
> was either "just vendor that library" (battle tested) or "explore doing 
> this for caches as well" (not battle tested).  On top of all of this really 
> apart from postgres I'm having a hard time finding docs for URLs that are 
> "industry standard" for much of anything. 
>
> So I'm trying to write up examples or justification for code I'm barely 
> convinced of. 
>
> This might be a thing where if there was a workshop day at a conference 
> then a group of people with diverse experiences on various systems could 
> land on a convincing thing and build consensus, along with a wonderful 
> patch. But staring at this just feels like a tarpit for me. Especially 
> given the sort of pressure to get the API right on the first shot. 
>
> Anyways if I were the only person working on Django I would sidestep 
> all of this by throwing dj_database_url into django.contrib and relying on 
> the years of usage by everyone as "proof" that the thing works. And I think 
> the branch I tried reviving is "correct", I just don't have the background 
> in most of these backends to know if it's right.
>  
>
>
>
>
> On Tuesday, November 29, 2022 at 5:41:41 PM UTC+9 Raphael G wrote:
>
>> Alright, I'm writing up a review aid that tries to re-explain the actual 
>> changes in the PR I opened before. This document should go over all of the 
>> actual API changes that are exposed to users as well. I believe the API 
>> _is_ correct, and that future settings improvements could rely on these to 
>> implement their features (so in the larger discussion, this is offering a 
>> low-level API while an overarching high-level settings API is still being 
>> worked on). At the end of the day there are only so many ways to structure 
>> a dictionary containing a hostname, port, username, and password!
>>
>> But of course the specifics are important here, so I will finish up a 
>> review aid and paste that in the PR (along with actually getting that 
>> branch passing) discussion and crosslink it here.
>> On Tuesday, November 29, 2022 at 4:45:12 PM UTC+9 carlton...@gmail.com 
>> wrote:
>>
>>> Hey Raphael. 
>>>
>>> My only query is as we sure the API is correct going forward? 
>>> The answer could be yes there, but I didn't (as yet get to) review the 
>>> history in depth. 
>>>
>>> We **can** deprecate things, but we get an awful lot of complaints and 
>>> pushback, even changes that are clearly for the good. 
>>> I'd rather measure twice and cut once is all. 
>>> The whole point of the "Do it in a third-party app" approach is that we 
>>> get to make sure the APIs are right, without adding churn to Django, 
>>> and without being tied to the long-release cycle fixing the unforeseen 
>>> issues that arise. 
>>>
>>> Kind Regards,
>>>
>>> Carlton
>>>
>>> On Tue, 29 Nov 2022 at 06:45, Raphael G  wrote:
>>>
>>>> (I'm very sorry about the threading going on here, I originally replied 
>>>> to the very old mailing thread and then realized it had not generated 
>>>> consensus, so am going to try and make this thread a more focused 
>>>> discussion regarding concensus)
>>>>
>>>> In the other thread people are discussing more generalized setting 
>>&g

Re: Potential way forward for DATABASE_URL

[Raphael G]

OK after looking at this some more and trying to write up a review aid, I'm 
giving up on this branch and trying to integrate DATABASE_URL support into 
Django proper.

A couple reasons:

- I misread the original mailing list thread which made me think there was 
a consensus on this branch, and there wasn't.
- There are a lot of tiny backend-specific things going on that are in 
because... well because django-database-url has that backend-specific 
behavior
- Lots of people want the cache backend as a part of this. I'd be happy to 
have it. But basically every cache backend has weirdness (what's a service 
URL for the dummy backend? do we really need one for the file backend?), so 
... I don't even know what makes sense there, honestly.

So there's this dual thing of not wanting to "just" vendor 
django-database-url, but really the original branch that I tried to revive 
was either "just vendor that library" (battle tested) or "explore doing 
this for caches as well" (not battle tested).  On top of all of this really 
apart from postgres I'm having a hard time finding docs for URLs that are 
"industry standard" for much of anything. 

So I'm trying to write up examples or justification for code I'm barely 
convinced of. 

This might be a thing where if there was a workshop day at a conference 
then a group of people with diverse experiences on various systems could 
land on a convincing thing and build consensus, along with a wonderful 
patch. But staring at this just feels like a tarpit for me. Especially 
given the sort of pressure to get the API right on the first shot. 

Anyways if I were the only person working on Django I would sidestep 
all of this by throwing dj_database_url into django.contrib and relying on 
the years of usage by everyone as "proof" that the thing works. And I think 
the branch I tried reviving is "correct", I just don't have the background 
in most of these backends to know if it's right.
 




On Tuesday, November 29, 2022 at 5:41:41 PM UTC+9 Raphael G wrote:

> Alright, I'm writing up a review aid that tries to re-explain the actual 
> changes in the PR I opened before. This document should go over all of the 
> actual API changes that are exposed to users as well. I believe the API 
> _is_ correct, and that future settings improvements could rely on these to 
> implement their features (so in the larger discussion, this is offering a 
> low-level API while an overarching high-level settings API is still being 
> worked on). At the end of the day there are only so many ways to structure 
> a dictionary containing a hostname, port, username, and password!
>
> But of course the specifics are important here, so I will finish up a 
> review aid and paste that in the PR (along with actually getting that 
> branch passing) discussion and crosslink it here.
> On Tuesday, November 29, 2022 at 4:45:12 PM UTC+9 carlton...@gmail.com 
> wrote:
>
>> Hey Raphael. 
>>
>> My only query is as we sure the API is correct going forward? 
>> The answer could be yes there, but I didn't (as yet get to) review the 
>> history in depth. 
>>
>> We **can** deprecate things, but we get an awful lot of complaints and 
>> pushback, even changes that are clearly for the good. 
>> I'd rather measure twice and cut once is all. 
>> The whole point of the "Do it in a third-party app" approach is that we 
>> get to make sure the APIs are right, without adding churn to Django, 
>> and without being tied to the long-release cycle fixing the unforeseen 
>> issues that arise. 
>>
>> Kind Regards,
>>
>> Carlton
>>
>> On Tue, 29 Nov 2022 at 06:45, Raphael G  wrote:
>>
>>> (I'm very sorry about the threading going on here, I originally replied 
>>> to the very old mailing thread and then realized it had not generated 
>>> consensus, so am going to try and make this thread a more focused 
>>> discussion regarding concensus)
>>>
>>> In the other thread people are discussing more generalized setting 
>>> helpers. I am trying to avoid this, most because I think this work doesn't 
>>> exclude that work. But also I don't want to introduce more magic 
>>> personally, nor do the work involved in the settings magic personally. 
>>> os.environ is straightforward IMO.
>>>
>>> Carlton posted the following comment:
>>>
>>> > Given that it's a single import I might still lean towards seeing it 
>>> as an external package, at least for a cycle, so unknowns that come up can 
>>> be resolved, and folks on an older LTS can opt-in early, etc. 
>>> (But that's not a point of religion.) 
>>>
>>> I am OK with putting in work to have

Re: Potential way forward for DATABASE_URL

[Raphael G]

Alright, I'm writing up a review aid that tries to re-explain the actual 
changes in the PR I opened before. This document should go over all of the 
actual API changes that are exposed to users as well. I believe the API 
_is_ correct, and that future settings improvements could rely on these to 
implement their features (so in the larger discussion, this is offering a 
low-level API while an overarching high-level settings API is still being 
worked on). At the end of the day there are only so many ways to structure 
a dictionary containing a hostname, port, username, and password!

But of course the specifics are important here, so I will finish up a 
review aid and paste that in the PR (along with actually getting that 
branch passing) discussion and crosslink it here.
On Tuesday, November 29, 2022 at 4:45:12 PM UTC+9 carlton...@gmail.com 
wrote:

> Hey Raphael. 
>
> My only query is as we sure the API is correct going forward? 
> The answer could be yes there, but I didn't (as yet get to) review the 
> history in depth. 
>
> We **can** deprecate things, but we get an awful lot of complaints and 
> pushback, even changes that are clearly for the good. 
> I'd rather measure twice and cut once is all. 
> The whole point of the "Do it in a third-party app" approach is that we 
> get to make sure the APIs are right, without adding churn to Django, 
> and without being tied to the long-release cycle fixing the unforeseen 
> issues that arise. 
>
> Kind Regards,
>
> Carlton
>
> On Tue, 29 Nov 2022 at 06:45, Raphael G  wrote:
>
>> (I'm very sorry about the threading going on here, I originally replied 
>> to the very old mailing thread and then realized it had not generated 
>> consensus, so am going to try and make this thread a more focused 
>> discussion regarding concensus)
>>
>> In the other thread people are discussing more generalized setting 
>> helpers. I am trying to avoid this, most because I think this work doesn't 
>> exclude that work. But also I don't want to introduce more magic 
>> personally, nor do the work involved in the settings magic personally. 
>> os.environ is straightforward IMO.
>>
>> Carlton posted the following comment:
>>
>> > Given that it's a single import I might still lean towards seeing it as 
>> an external package, at least for a cycle, so unknowns that come up can be 
>> resolved, and folks on an older LTS can opt-in early, etc. 
>> (But that's not a point of religion.) 
>>
>> I am OK with putting in work to have it as a separate package for a 
>> cycle. The glib comment would be that dj-database-url was that package for 
>> many cycles, but this is not very true in practice. This is introducing 
>> extra things not originally present for cache configuration, and has this 
>> concept of the database backends holding the parsing logic. And of course 
>> there's an extremely valid underlying point here: the API really needs to 
>> be "right". Personally I believe that Django's very good deprecation 
>> strategy means that big mess-ups are fortunately fixable, but it's work for 
>> everyone (and likely would involve either some weird hack in the 
>> intermediate steps). 
>>
>> I would like to offer an alternative narrative to the background here, 
>> that I think more strongly justifies introducing this into Django proper. 
>> It is not the real narrative, but it is a narrative. 
>>
>> URL-based configuration conventions exist for database backends and cache 
>> backends in various libraries. This lets us pass in credentials as one 
>> string rather than a bunch of components to be assembled. But each backend 
>> will handle things like configuration options within those URLs 
>> differently. Overall URL parsing logic is all very similar, with important 
>> differences coming from how the database name might get passed in, how 
>> certain connection options get passed in, etc.
>>
>> So it would be helpful to provide both a method on DatabaseWrapper that 
>> does basic URL parsing (to pull out the host/username/password), and for 
>> Django's supported DB backends to override this URL parsing method based on 
>> whatever convention is being applied by other libraries (or from 
>> backend-specific tooling). Same thing for caches. 
>>
>> Because this is ultimately a bit backend-specific, having this logic 
>> close to the actual backend connection logic (so on these classes 
>> themselves) is the most natural, more so than having separate dictionaries 
>> with mappings to backends. New configuration option? Would be good to make 
>> sure it's handled in the URL parser as well, som

Re: Potential way forward for DATABASE_URL

[Raphael G]

(I'm very sorry about the threading going on here, I originally replied to 
the very old mailing thread and then realized it had not generated 
consensus, so am going to try and make this thread a more focused 
discussion regarding concensus)

In the other thread people are discussing more generalized setting helpers. 
I am trying to avoid this, most because I think this work doesn't exclude 
that work. But also I don't want to introduce more magic personally, nor do 
the work involved in the settings magic personally. os.environ is 
straightforward IMO.

Carlton posted the following comment:

> Given that it's a single import I might still lean towards seeing it as 
an external package, at least for a cycle, so unknowns that come up can be 
resolved, and folks on an older LTS can opt-in early, etc. 
(But that's not a point of religion.) 

I am OK with putting in work to have it as a separate package for a cycle. 
The glib comment would be that dj-database-url was that package for many 
cycles, but this is not very true in practice. This is introducing extra 
things not originally present for cache configuration, and has this concept 
of the database backends holding the parsing logic. And of course there's 
an extremely valid underlying point here: the API really needs to be 
"right". Personally I believe that Django's very good deprecation strategy 
means that big mess-ups are fortunately fixable, but it's work for everyone 
(and likely would involve either some weird hack in the intermediate 
steps). 

I would like to offer an alternative narrative to the background here, that 
I think more strongly justifies introducing this into Django proper. It is 
not the real narrative, but it is a narrative. 

URL-based configuration conventions exist for database backends and cache 
backends in various libraries. This lets us pass in credentials as one 
string rather than a bunch of components to be assembled. But each backend 
will handle things like configuration options within those URLs 
differently. Overall URL parsing logic is all very similar, with important 
differences coming from how the database name might get passed in, how 
certain connection options get passed in, etc.

So it would be helpful to provide both a method on DatabaseWrapper that 
does basic URL parsing (to pull out the host/username/password), and for 
Django's supported DB backends to override this URL parsing method based on 
whatever convention is being applied by other libraries (or from 
backend-specific tooling). Same thing for caches. 

Because this is ultimately a bit backend-specific, having this logic close 
to the actual backend connection logic (so on these classes themselves) is 
the most natural, more so than having separate dictionaries with mappings 
to backends. New configuration option? Would be good to make sure it's 
handled in the URL parser as well, somehow.

Rambling a bit, but really am open to any(?) way forward that leads to "I 
will not need to install an extra package to handle this, nor am I 
personally parsing the URL with urllib.parse", and am ready to do the 
legwork. 

Speaking to that, if we have consensus on the principle, what would be the 
right step forward? An actual DEP?

Raphael

On Tuesday, November 29, 2022 at 1:47:24 PM UTC+9 Adam Johnson wrote:

> I’m happy with this approach, it’s a little step forwards towards 
> maintainable settings files.
>
> On Sun, 27 Nov 2022 at 20:37, 'Tobias McNulty' via Django developers 
> (Contributions to Django itself)  wrote:
>
>> Hi Raphael,
>>
>> Thanks for taking this on.
>>
>> Starting with a limited scope seems like a good idea to me.
>>
>> A couple other things I like about this approach:
>>
>> - It tackles cache URLs at the same time (it makes sense for them to 
>> mirror one another, IMO).
>> - No implicit usage of DATABASE_URL, but as you said it still supplies an 
>> easily searchable answer for "Django DATABASE_URL."
>>
>> Cheers,
>> Tobias
>>
>>
>> On Sun, Nov 27, 2022, 2:40 PM Raphael G  wrote:
>>
>>> Some base industry background. It's a pretty common convention to share 
>>> credentials in environment variables. For many PaaS, it's common to use 
>>> connection URLs to do so. So DATABASE_URL will have a URL like 
>>> postgres://my_user:mypassword@somedomain/database stuffed into a single 
>>> environment variable.
>>>
>>> Django expects a configuration dictionary for its drivers. So what do 
>>> people do? People install django-database-url, and pass in the string 
>>> into that library (or rather, the library will read a blessed environment 
>>> variable). Absent that they'll need to manually parse out the information 
>>> and build the configuration dictionary. So if you just hav

Potential way forward for DATABASE_URL

[Raphael G]

Some base industry background. It's a pretty common convention to share 
credentials in environment variables. For many PaaS, it's common to use 
connection URLs to do so. So DATABASE_URL will have a URL like 
postgres://my_user:mypassword@somedomain/database stuffed into a single 
environment variable.

Django expects a configuration dictionary for its drivers. So what do 
people do? People install django-database-url, and pass in the string into 
that library (or rather, the library will read a blessed environment 
variable). Absent that they'll need to manually parse out the information 
and build the configuration dictionary. So if you just have Django time to 
futz around with urlparse

There have been some discussions about how to make this better, including:

   - adopting dj-database-url as an "official django project" via DEP 7 
   
   - instead adopt a behavior where Django will automatically handle a 
   string as an alternative to a configuration dictionary 
    (this is discussed in 
   the first link as PR 10786)
   - Integrate dj-database-url but expand on it a bit to also help with 
   cache configuration, and have backends handle URL parsing 
   

These haven't seen much movement in the past couple of years. A comment in 
one of these e-mail threads:

> I suspect this is a "good enough is good enough" situation. Something 
like what Raffaele is talking about, or dsnparse, or whatever would 
probably be ideal. And for something to be merged into core, I think it'd 
need to be a more full solution than just dj-database-url.

dj-database-url takes something from an environment variable and provides a 
configuration dictionary. There's this feeling that having Django directly 
accept a string would feel more natural and correct. There are also other 
libraries like dsnparse, and people proposing things like adding a DSN name 
into settings.

I think of all the options, the third option (the proposal by Tom Forbes) 
is a very good option. What it looks like in practice is the addition of 
the following:

   - the ability for database backends to register protocol names for URLs, 
   so that postgres://localhost:5432 will properly map to the 
   django.db.backends.postgresql backend, but people can show up with their 
   own mappings.
   - A configure_db(url) function, that will return a configuration 
   dictionary meant for DATABASES
   - A similar configure_cache(url) function that will give cache 
   configuration dictionaries meant for CACHES

A thing that is notably absent here is any blessing of DATABASE_URL. You 
have to do

DATABASES = {
'default': configure_db(os.environ['DATABASE_URL'])
}

yourself. It's not "ideal" in that you don't magically get behavior from 
your URLs, but that also means you're just doing something in a 
straightforward way that should be easy to debug with some print statements 
when needed. It feels way less likely for this to be a major design miss.

The motivating examples for the two above being supported is that Heroku 
will provide DATABASE_URL and REDIS_URL.

The nice thing about this solution is that it doesn't block future design 
space. We get a configuration dictionary that matches the existing 
functionality, because the added API is simple it's easy for people to 
inspect the results, and of course it doesn't preclude people from keeping 
on with their existing solutions. There isn't even an assumed usage of 
DATABASE_URL like with dj-database-url! Mostly magic free.

I tried to rebase the PR 
 including 
the above functionality from a couple years ago, and added some basic 
documentation. This doesn't try and convince users to use this, but I 
believe the usage would be sufficient for simple cases.

So my ask here: how do people feel about moving forward with this limited 
scope? Previous discussions talked about wanting a larger scope for it to 
get merged into core. I believe that instead targetting a smaller scope 
means we can at least provide a workable answer to the DATABASE_URL question 
in the near term. And when consensus coalesces around a good overall answer 
to settings, the actual URL parsing logic will already be present and even 
more battle tested.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d990fef4-586b-447f-afd1-b53f23237a3an%40googlegroups.com.

Re: Integrate dj-database-url into Django

[Raphael G]

Alright, I tried to revive Tom Forbe's work on this in  
https://github.com/django/django/pull/16331

My honest feeling here is that if Django existed just for me, I would 
really just want to get this pulled in as an option, and trying to increase 
the scope beyond "given a URL, get a dictionary of settings" is a tarpit 
that is not worth venturing into in a first merge. Though it's definitely a 
tarpit worth venturing into as a future step! 

So my strategy here was to try and get this as close to mergeable as 
possible, with as few points of discussion needed as possible. I tried to 
document what Tom did, more or less disregarded the idea of providing DSN 
support in settings, and for memcache avoided blessing an "official" 
backend. The main thing should be that if people google Django DATABASE_URL 
there should be a good answer in the docs, IMO.

As someone who's dealt with custom backends in various parts of Django, and 
with libs, I think Tom's simple registry pattern for custom protocols is 
more than sufficient for library writers. 

Documentation is minimal, but exists. While I basically expect people 
writing custom DB backends or cache backends to directly look at the 
source, at the very least we are pointing to the existence of functions for 
registration in the docs in this PR.

I'm sure we all have these simple projects that really don't need to do 
much, we all have been pulling in dj_database_url on the side to get this 
and feeling a bit off about it, so let's try and get that use case handled 
in a way that doesn't prevent further niceness in the future! 

Raphael
On Monday, December 24, 2018 at 7:01:23 AM UTC+9 Raffaele Salmaso wrote:

> Hi all,
> I'm working on https://github.com/django/django/pull/10786 (which is a 
> port of https://pypi.org/project/django-service-urls/ , which is a 
> 'fork/rewrite' of Tom PR).
> I need to (re)read all these emails to find ideas to improve the 
> PR/package.
>
> On Sat, Jul 28, 2018 at 9:44 PM Tom Forbes  wrote:
>
>> So in the PR I proposed I only bits I took verbatim from dj-database-url 
>> are the tests. The rest is re-implemented. I think it's a pretty good POC 
>> but I haven't touched it in a while.
>>
>> In any case we have to implement our own parsing for backends that do not 
>> support passing in a URL to the connection library. 
>>
>> Making postgres skip our parsing step and passing it in directly is an 
>> implementation detail and there are much more important questions around 
>> the API design to answer before this has any chance of being included.
>>
>> On Sat, 28 Jul 2018, 12:57 Maciej Urbański,  wrote:
>>
>>> I would agree that DSN support seems like a nicer alternative to just 
>>> copying dj-database-url, because it not only focuses on 12factor 
>>> configuration in enviroment variables, but also enables some additional 
>>> flexibility for the database connection option passing.
>>>
>>> As for 12factor, I think https://gist.github.com/telent/9742059 is a 
>>> good read as to why exposing as enviroment variables maybe not the best 
>>> motivation. Having to accommodate settings, like database connection 
>>> information, just so they can be fitted into single string put conveyable 
>>> by enviroment variable is a case in point. We likely can do the same for 
>>> Cache addresses as mentioned previously, although we may end up inventing 
>>> new URI schemes do to that.., but django overall has multitude of other 
>>> options that are not as easy to stringify.
>>>
>>> On Friday, 27 July 2018 19:14:12 UTC+2, gw...@fusionbox.com wrote:
>>>>
>>>> I'd like to approach this as 'support database urls in django', rather 
>>>> than 'copy/paste dj-database-url into django'. For postgres (I'm not sure 
>>>> about other backends), a database url can be passed directly to psycopg2. 
>>>> The postgres connection string format actually supports more features than 
>>>> is possible with django's HOST/USER/PORT... style settings. For example, 
>>>> you can pass multiple hosts and psycopg2 will attempt to connect to one of 
>>>> them: https://paquier.xyz/postgresql-2/postgres-10-multi-host-connstr/. 
>>>> Any attempt to parse the url in django will result in a loss of those 
>>>> features.
>>>>
>>>> The only problem I see is that we have to parse the database backend 
>>>> out of the url, and you can't pass a url like 'postgis://' to 
>>>> psyscopg2. I'd like to be able to do something like:
>>>>
>>>> DATABASES = {
>>>> 'default': {
>>>> 'DSN': 'po

Re: Proposal for a transaction.on_before_commit

[Raphael Michel]

Hi,

Am Sun, 10 Oct 2021 18:38:55 +0300
schrieb Shai Berger :
> Why is a before-commit signal preferable to a vanilla Python
> context-manager around the code? Or, if it is in the context of
> requests, a middleware?

basically mostly because you can forget to put the context manager
around it and because it might become very verbose adding context
managers everywhere.

Best
Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20211012112913.26138880%40kvothe.

Re: Proposal for a transaction.on_before_commit

[Raphael Michel]

Hi Aymeric,

thank you for the insightful reply. Indeed I have overlooked the issue
with savepoints which makes it much more fragile. In our case, "not
using savepoint rollbacks any more" would be a trade-off that we'd
happily make (there are enough other problems with savepoints to begin
with), but I understand that this kinda disqualifies it for a
first-class feature in Django :(

A middleware is not an option for us, as we intentionally don't use
ATOMIC_REQUESTS and not all code is triggered through requests (celery
tasks, management commands, …) and we'd like to avoid to deal with all
of these separately.

Best
Raphael

Am Sun, 10 Oct 2021 21:38:13 +0200
schrieb Aymeric Augustin :

> Hello Raphael,
> 
> Oh - a use case for django-transaction-signals
> <https://github.com/aaugustin/django-transaction-signals> ;-) I'm
> bringing up this elaborate joke because you're essentially asking for
> a "pre-commit" signal here and the README contains a good list of
> things that can go wrong with transaction signals. (Please ignore how
> this package demonstrates a way to do it as third-party code *cough*
> *cough* *cough*)
> 
> > I figured a perfect way to do this would be using `save()` or
> > `post_save` to add the changed model instance to some kind of
> > thread-local list, and then using `transaction.on_commit` to
> > "schedule" the aggregation and create the log entries when all
> > changes have been made. However, this obviously is not a good
> > enough because `on_commit` runs *after* the `COMMIT` statement and
> > thus we're not guaranteed that all log entries are persisted to the
> > database.  
> 
> 
> In my opinion "saving the log entries may fail after a successful
> transaction" isn't the main design problem here. The bigger problem
> is "log may contain entries for writes that don't actually happen,
> because some savepoints were rolled back, typically due to atomic
> blocks exiting with an exception". And then you get dragged into the
> whole complexity that the README of django-transaction-signals
> outlines and that we're trying to avoid in Django.
> 
> (If you don't have any savepoint rollbacks, then your code sounds
> sufficiently constrained to implement logging of changes explicitly
> at the application layer rather than at the framework layer.)
> 
> If you run with ATOMIC_REQUESTS, I would suggest to replace it by a
> custom middleware that wraps get_response(request) in an atomic
> block. Then you know that this is the outermost traction and you can
> do whatever needed before exiting the atomic block. You also need the
> same in all management commands, which could be a problem if you
> depend on third-party management commands.
> 
> Failing that, in order to log changes with transactional correctness
> enforced by the ACID guarantees of PostgreSQL, I'd recommend doing it
> at that database level with triggers — which always execute in the
> same transaction. I realize it may be difficult to perform the kind
> of aggregation you have in mind.
> 
> As a last resort, I'd try a custom database backend to track
> accurately transactions and savepoints and maintain an up-to-date
> record of changes that will happen when the transaction is committed.
> 
> Hope this helps!
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20211012112820.57a47da3%40kvothe.

Proposal for a transaction.on_before_commit

[Raphael Michel]

Hi everyone,

I've spent some days thinking about a use case we have internally where
we want to create some database records automatically based on some
other records, think e.g. of an audit log where whenever an instance of
model X is changed, we want to create an instance of model Y as a log
record. Of course, there are *lots* of ways and libraries that allow you
to do this with Django, mostly somehow circling around overriding
`save()` or using the `post_save` signal.

However, we want to do this in an *aggregated* way, i.e. even if you
add hundreds of model instances in the same transaction, we only want
*one* new line in our audit log to be created. This is impossible with
`save()` or `post_save` alone since we can't know which change is the
"last" that should trigger the audit log to be created.

I figured a perfect way to do this would be using `save()` or
`post_save` to add the changed model instance to some kind of
thread-local list, and then using `transaction.on_commit` to "schedule"
the aggregation and create the log entries when all changes have been
made. However, this obviously is not a good enough because `on_commit`
runs *after* the `COMMIT` statement and thus we're not guaranteed that
all log entries are persisted to the database.

So I was wondering if there was potential for a
`transaction.on_before_commit` that works just like
`transaction.on_commit` except that it is executed before the `COMMIT`
statement. I imagine there are lots of other possible uses as well and
without looking into it much deeper, it seems easy enough to implement.

Does anyone have opinions on whether or not this would be a good
feature to add to Django? Unfortunately it doesn't seem possible to do
this as third-party code (except if we were shipping entire database
backends) so it would need to be an acceptable change to Django to be a
viable option.

Thanks
Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20211010155522.6489b86d%40amara.localdomain.


pgpl51uPmIu7I.pgp
Description: Digitale Signatur von OpenPGP

Re: Faster Migrations! But at what cost?

[Raphael Gaschignard]

Hi Markus, Simon,

 Both of you, thank you for the detailed replies and status report on 
this sort of stuff.



Did you look into squashing these 500 migrations by any chance?


Yeah so I'll go in and squash things, though (partly because effective 
squashing requires moving models around, though we're still at Django 
1.11 so it might become easier, partly because of cross-app references) 
it's a decent amount of work.


I do like the idea of reusing a "effectively production DB" for things, 
not least because it's actually acurate.


OK so the Operation API is effectively documented. I am tempted to try 
modifying the `database_forwards` and `database_backwards` to not 
require `from_state` (probably through adding a hook that gets called 
_before_ `state_forwards` to allow capturing the small details of the 
old state). It might actually be possible to make this backwards 
compatible by porting Django migrations to use an `apps`-free workflow, 
but falling back to the old "re-render the universe" mechanism for 
operations that don't apply it.


This is very "that xkcd comic about time spent versus time saved"

> will cause trouble with RunSQL and other operations that use related 
field or model attributes


So one thing I felt like was an invariant in this code was that field 
sharing was expected? From the docstring of ModelState:


    Note that while you are allowed to mutate .fields, you are not allowed
    to mutate the Field instances inside there themselves - you must 
instead

    assign new ones, as these are not detached during a clone.

Also maybe you meant to refer to RunPython instead of RunSQL. But I get 
your point in general here. Related models can be a problem


One throwaway idea would be to not allow related model/related field 
access in these models? There's already a lot of documentation related 
to not allowing general model methods (effectively establishing that 
"migration models are _not_ normal models"), so there's a bit of 
precedent. But beyond the general backwards incompatability, it might 
not actually even be obvious how one would implement this. And you kinda 
need this info for foreign keys and the like anyways.



Working directly off of `ModelState` is interesting, and I think there 
might be a backwards-compatible way forward there, where we still allow 
for rendering on certain operations but hold off on it on the basic 
ones. Even in our large project, most of our migration operatiosn are 
dirt-simple, so if the core django migrations could work off of 
`ModelState` then we could get a fast path through there.


Thanks for your input, both of you. I have a couple ideas now that I'm 
pretty tempted to try out, mainly around the "fast path and slow path" 
strategies that should offer backwards compatibility.


 Raphael

Markus Holtermann wrote on 2019/05/21 2:26:

Thanks Raphael for bringing this topic up and Simon for your input already.

I just left a note on your PR: 
https://github.com/django/django/pull/11388#issuecomment-494076750 . I'll quote 
it here for ease of readability:

As far as I can see right now, a similar caching happened as a first approach 
to the Django 1.8 release but cause significant problems, specifically with 
regards to relational fields. Relational fields (ForeignKey, OneToOneField, 
ManyToManyField) keep an instance reference to the related model in 
`.related_model` or the related fields in `.related_fields`. The problem now 
is, if you reuse a field (and you do because you're only calling `list()` on 
`self.fields` to copy but not deepcopy the list), you're screwing up references 
between the models that _will_ cause trouble with `RunSQL` and other operations 
that use related field or model attributes.

https://github.com/django/django/blob/1d0bab0bfd77edcf1228d45bf654457a8ff1890d/django/db/models/fields/__init__.py#L495-L499

 From my work on migrations, I see essentially 2 approaches that are viable and 
would lead to significant performance improvements:

## 1. Make the schema editor work with model states

There's a _very old_ branch on my fork that tries to implement this approach: 
https://github.com/MarkusH/django/commits/schemaeditor-modelstate

I lost interest and eventually also didn't have the time to pursue this 
approach further. I think that's the better of the two approaches, as it gets 
rid of the piece of code that actually _causes_ the slow behavior: turning a 
model state into a model class to be used in the schema editor.

However, making the schema editor backward compatible has been proven difficult 
and to be a huge pain (just check out the commits :wink: )

## 2. Don't resolve models/fields to instances but rely on string references.

This approach is IMO merely a workaround as it would allow us to cache the fields and models as you're 
doing in your PR. `model._meta.get_field("author").related_model` would not return `` but `"my

Faster Migrations! But at what cost?

[Raphael Gaschignard]

Hi Developers,

  We have a decently-sized "large project", around 240 models across 90 
apps, with roughly 500 migrations to work off of. We do periodically squash 
migrations to keep the migration count under control, but because of all 
this migrations in our testing server take 3-5 minutes to run to 
completion. 

I am not sure about what the size of a typical Django project is (or 
rather, a typical "large project") so it's hard for me to quantify how big 
of an issue this is.

Looking through the migration code and some profiling I found a place where 
caching was possible (on the ModelState -> Model rendering, based on some 
of the invariants stated in ModelState code), which would cut *our* full 
migration from 230 seconds to 50 seconds (on my machine at least). On the 
specific caching I did, I was hitting a 90% cache hit rate on our full 
migration run.

Caching is always a bit scary, though, and there are a *lot* of places in 
the apps registry code/model registration code in particular where caches 
are constantly being wiped. So this stuff scares me quite a bit. In my 
personal ideal, I would love to be able to check in my caching thing but 
have it be behind some MIGRATIONS_FASTER_BUT_MAYBE_UNSAFE flag. I am not 
recommending this for Django because it's not how the project tends to do 
things, this is just my personal feeling. After all, you're rarely running 
all  your migrations in production, so this is a testing problem more than 
anything.

I do think there would be an alternative way to move forward though. 
Currently the migrations Operation class relies on having the from_state 
and to_state for DB operations in particular. But I think that we could 
change up this API based on how these properties are used in 
Django-provided Operation classes to avoid having to copy the state to 
provide from_state and to_state. I haven't gone through with this 
investigation too much yet but I think this would improve things a bit.

So this is a multi-pronged question:

- Have there ever been any surveys about how the size of Django projects? I 
don't know the value of investigating this further except for our own usage.

- Does the caching of ModelState.render as done in this PR 
<https://github.com/django/django/pull/11388> (still need to work through a 
couple failing tests) sound reasonable? Or is this veering too far in the 
performance/safety guarantee tradeoff?
- Is the migration operation infrastructure considered a public API? As in, 
would changing the Operation model API (potentially breaking subclasses) be 
considered a major undertaking? Or would it be an acceptable cost to pay 
for some performance improvements?

I am still trying to wrap my head around some of this problem space, so any 
insight will be very appreciated

Thanks, 
   Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/fd945497-ef84-4135-b92a-5473ca098809%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: GSoC Proposal: Add Cross-DB JSONField, ArrayField, and HStoreField

[Raphael Michel]

Hi Sage, Hi everyone,

I lacked the time to read this mailing list in the past months, and
someone at DjangoCon Europe just pointed me to this thread, so just for
reference, I want to add some prior work that I did to the list:

At DjangoCon US and thereafter, I developed a third-party
implementation that does a little more than the ones that are more
well-known and mentioned in the proposal:

https://github.com/raphaelm/django-jsonfallback

django-jsonfallback supports native JSON fields on PostgreSQL, MariaDB
*and* MySQL (which are quite different in that regard). On all other
databases, it falls back to the behaviour of just storing strings in
text fields. Therefore, it's still different from what is being
proposed here, and also the implementation is quite unclean in the
sense that it needs to check for database backends on the level of
database fields. When implemented in core, this could be done much more
nicely.

Still, it might be an interesting thing to look at when doing this, if
only for the test cases and the weird MySQL/MariaDB things. I'd love to
see this in core and are happy to help wherever I can with my expertise.

Best
Raphael

Am Tue, 2 Apr 2019 04:41:36 -0700 (PDT)
schrieb "Sage M.A." :

> Hello, everyone! My name is Sage. I'm a 19-year-old computer science 
> student from Indonesia. I'm planning to join the Google Summer of
> Code (GSoC) this year, and I want to contribute to Django. I have
> written a draft for my proposal in this gist 
> <https://gist.github.com/laymonage/b53a1acbbab36b6cd526b48fd2a5>.
> I have submitted two small patches for Django, and I hope to
> contribute more in the future. Feedbacks are much appreciated, thanks!
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20190416114408.35c8afa4%40kvothe.
For more options, visit https://groups.google.com/d/optout.


pgp5YMkBtqawZ.pgp
Description: Digitale Signatur von OpenPGP

Re: Request to reconsider ticket #27910: using an Enum class in model Field choices

[raphael]

I saw this thread so wanted to share a bit of a gotcha we had with enums 
internally for anyone trying to handle this stuff

Internally, we have a wrapper around CharField to work with enums (we go 
further than just choices and actually have the values be enums), but 
there's unfortunately still a good amount of issues in certain use cases 
that make it not a great fit to be put into general usage for Django. I do 
think it's possible to get things right, but you have to establish certain 
rules because the way enums work in Django can sometimes be surprising.

We notably had an issue with translations, and I believe the problem would 
occur with other context-sensitive values. Because database operations will 
often copy values before using them to save to the database, you're 
beholden to (what I believe to be) slightly busted semantics on `copy.copy` 
for enums. Probably defining a django-y enum subclass that requires 
identity/lookup helpers would make this more usable for the general public 
(much like what other people have said). 

In [10]: from enum import Enum
In [11]: from django.utils.translation import ugettext_lazy, override
In [12]: class C(Enum):
...: a = ugettext_lazy("Some Word")

In [13]: with override('en'):
...: elt = C.a
...: with override('ja'):
...: copy.copy(elt)
...:
---
ValueErrorTraceback (most recent call last)
 in 
  2 elt = C.a
  3 with override('ja'):
> 4 copy.copy(elt)
  5

/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py
 
in copy(x)
104 if isinstance(rv, str):
105 return x
--> 106 return _reconstruct(x, None, *rv)
107
108

/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/copy.py
 
in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
272 if deep and args:
273 args = (deepcopy(arg, memo) for arg in args)
--> 274 y = func(*args)
275 if deep:
276 memo[id(x)] = y

/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py
 
in __call__(cls, value, names, module, qualname, type, start)
289 """
290 if names is None:  # simple value lookup
--> 291 return cls.__new__(cls, value)
292 # otherwise, functional API: we're creating a new Enum type
293 return cls._create_(value, names, module=module, 
qualname=qualname, type=type, start=start)

/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py
 
in __new__(cls, value)
531 return member
532 # still not found -- try _missing_ hook
--> 533 return cls._missing_(value)
534
535 def _generate_next_value_(name, start, count, last_values):

/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/enum.py
 
in _missing_(cls, value)
544 @classmethod
545 def _missing_(cls, value):
--> 546 raise ValueError("%r is not a valid %s" % (value, 
cls.__name__))
547
548 def __repr__(self):

ValueError: 'Japanese Version of Some Word' is not a valid C

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ad1b1890-3ce3-46fb-9801-6408de4ed256%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding a database-agnostic JSONField into Django

[Raphael Michel]

Hi,

On Saturday, October 20, 2018 at 10:48:04 PM UTC+2, Adam Johnson wrote:
>
> The main thing is that MySQL added a binary JSON type, whilst MariaDB just 
> made the JSON type an alias for TEXT (plus the corollaries that implies): 
> https://mariadb.com/kb/en/library/json-data-type/
>

Yes, this is the main difference, as MySQL requires CAST(x AS JSON) calls 
in many places while MariaDB does not support CAST(x AS JSON). I also noted 
a number of oddities with regards to case-(in)sensitivity when comparing 
values extracted from JSON, but I didn't research this deeper.

Best
Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3d2e6bf8-aa2a-4ece-80bc-5867f8105e04%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding a database-agnostic JSONField into Django

[Raphael Michel]

Hi everyone,

I used the sprints at DjangoCon US to work on this issue in form of a 
third-party package. Mainly, I created a subclass of 
django.contrib.postgres.fields.JSONField that

   - includes code from django-mysql to work on MySQL 5.7+ as well
   - does some nasty hacks to even work on MariaDB 10.2.7+
   - gracefully falls back to storing JSON strings in a TextField on all 
   other databases

Unfortunately, the differences between the JSON implementations (especially 
between MariaDB and MySQL) seem to be quite large. Since my primary 
objective was to prove that it is possible to create a multi-database 
field, I resolved this mostly with some back-and-forth data conversion 
hacks, instead of searching for a clean solution. Not all features of the 
PostgreSQL version are possible on MySQL/MariaDB, but the most useful ones 
all are.

This would probably need a lot of cleaning up before considering to move it 
to Django core, and you'd likely also want to explore integrating 
Oracle/SQLite for that. I will probably not have time to do that, but if 
someone else wants to, this might provide a starting point.

Here's the repository: https://github.com/raphaelm/django-jsonfallback

Best
Raphael

P.S. Thanks to all organizers and attendees for an amazing conference <3

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/330b1ade-ab27-4135-9952-d6a188789c31%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding a bulk_save method to models

[Raphael Michel]

Hi,

I'd be very careful about calling it bulk_save(), since calling
it something with save() very strongly suggests that it calls pre_save
or post_save signals.

Best
Raphael


Am Fri, 14 Sep 2018 07:56:38 -0700 (PDT)
schrieb Tim Graham :

>  
> 
> I wanted to ask about naming of the new method. Currently the
> proposed name is "QuerySet.bulk_save()" but I think it's a bit
> confusing since it uses QuerySet.update(), not Model.save(). It works
> similarly to QuerySet.bulk_update() from
> https://github.com/aykut/django-bulk-update but the arguments are a
> bit different.
> 
> 
> Josh's comment on the PR: "Since this only works for instances with
> an pk, do you think that bulk_update would be a better name? The
> regular save() method can either create or update depending on pk
> status which may confuse users here."
> 
> And Tom's reply: "I considered this, but queryset.update() is the
> best 'bulk update' method. I didn't want to confuse the two, this is
> more about saving multiple model fields with multiple differing
> values, gene bulk_save. Open to changing it though."
> 
> 
> On Tuesday, January 23, 2018 at 7:38:18 AM UTC-5, Neal Todd wrote:
> >
> > Hi Tom,
> >
> > That's great, should be a helpful addition to core. Will follow the
> > ticket and PR.
> >
> > Neal
> >
> > (Apologies - I hadn't spotted that you'd already referenced 
> > django-bulk-update in your ticket when I left my drive-by comment!)
> >
> > On Monday, January 22, 2018 at 7:41:11 PM UTC, Tom Forbes wrote:  
> >>
> >> Hey Neal,
> >>
> >> Thank you very much for pointing that out, I actually found out
> >> about this package as I was researching the ticket - I wish I had
> >> known about this a couple of years ago as it would have saved me a
> >> fair bit of CPU and brain time!
> >>
> >> I think that module is a good starting point and proves that it’s 
> >> possible, however I think the implementation can be improved upon
> >> if we bring it inside core. I worked on a small PR to add this 
> >> <https://github.com/django/django/pull/9606/files#diff-5b0dda5eb9a242c15879dc9cd2121379R473>
> >>  
> >> and the implementation was refreshingly simple. It still needs
> >> docs, a couple more tests and to fix a strange error with sqlite
> >> on Windows, but overall it seems like a lot of gain for a small
> >> amount of code.
> >>
> >> Tom 
> >>
> >>
> >> On 22 January 2018 at 15:10:53, Neal Todd (ne...@torchbox.com)
> >> wrote:
> >>
> >> Hi Tom,
> >>
> >> A built-in bulk save that's more flexible than update would
> >> certainly be nice. Just in case you haven't come across it though,
> >> there is a package called django-bulk-update:
> >>
> >> https://github.com/aykut/django-bulk-update
> >>
> >> I've found it very useful on a number of occassions where update
> >> isn't quite enough but the loop-edit-save pattern is too slow to
> >> be convenient.
> >>
> >> Probably some useful things in there when considering the API and 
> >> approach.
> >>
> >> Cheers, Neal 
> >>
> >> On Friday, January 19, 2018 at 5:49:48 PM UTC, Tom Forbes wrote:   
> >>>
> >>> Hello all,
> >>>
> >>> I’d love for some feedback on an idea I’ve been mulling around
> >>> lately, namely adding a bulk_save method to Dango.
> >>>
> >>> A somewhat common pattern for some applications is to loop over a
> >>> list of models, set an attribute and call save on them. This
> >>> unfortunately can issue a lot of database queries which can be a
> >>> significant slowdown. You can work around this by using
> >>> ‘.update()’ in some cases, but not all.
> >>>
> >>> It seems it would be possible to use a CASE statement in SQL to
> >>> handle bulk-updating many rows with differing values. For example:
> >>>
> >>> SomeModel.object.filter(id__in=[1,2]).update(
> >>> some_field=Case(
> >>> When(id=1, then=Value('Field value for ID=1')),
> >>> When(id=2, then=Value('Field value for ID=2'))
> >>> )
> >>> )
> >>>
> >>> I’ve made a ticket for this here: 
> >>> https://code.djangoproject.com/ticket/29037
> >>>
> >>> I managed to get a 70x performance increase using this technique
> >>> on a fairly large

Re: Dynamic app loading

[Raphael Michel]

Hi,

Am Tue, 31 Jul 2018 23:18:32 +0200
schrieb Christian González :
> Whow, I'm quite impressed. Never stumbled upon that. I'll recommend
> that for some collegues (to use it...)

Thanks =)

> But about the internals: Nice, this was the first way I implemented
> that as well, and maybe I will come back to that again. You have to
> restart the server as well if you want to enable a plugin, right?

If by enable you mean installing (in our terminology), yes. Normally,
you need to do something like

pip install pretix-myfancyplugin
python -m pretix migrate
python -m pretix rebuild
systemctl restart pretix

rebuild is a custom management command that compiles translation files,
SASS files and compressed JS files.

> From a security POV, I thought: who hinders anyone to write malicious
> code and provide an entry point for your system? maybe in a library
> you install... (but this is merely a philosophic question... don't
> bother.)

Nobody. But if you pip-install a library with malicious intents, you
have lost anyways, since its setup.py will likely be executed during
install (or at least some of its code will be executed once you import
it) and it can do whatever it wants anyway.

> But it's now replaced by entry points discovery. I had this too, (and
> will maybe come back to it). Setuptools' entry points discovery is a
> bit slow, but for the server start that doesn't matter, it's just
> once. (except development, you'll have to restart your server often)
> Your entrypoint is e.g. "pretix_pages=pretix_pages:PretixPluginMeta"
> - I don't understand that - this is an inner class of PluginApp. How
> can you  access it directly via the entry point? But generally I
> understand the workflow.

The part after the : in the entrypoint is probably not used at all at
the moment. With the entry point, I get the Django app. Then, with the
Django app, I get the inner class from the app registry:
https://github.com/pretix/pretix/blob/0b167aaa2cf607adddf4fe96c3da55b160b95602/src/pretix/base/plugins.py#L24

> * How do you use signals? e.g.
> 
>     @receiver(footer_link, dispatch_uid="pages_footer_links")
> What does this mean? How do you pass this to your frontend? Could you
> describe this in just a few words?

These are basically standard Django signals, with only slight
modifications to be able to turn them off per-client:
https://docs.djangoproject.com/en/2.0/topics/signals/

I then use a few custom helpers like a custom template tag that allows
to call a signal directly from a template:

{% eventsignal event "path.to.signal" argument="value" ... %}

This is just to avoid repetitive code, however.

> I'd like to add channels to the mix as well to enable a Vue frontend.
> Instead of copying your code and forking the parts that I need, It
> would be better OpenSource practice to maybe build a "plugin system"
> that is a library and can be used of both - this is not very
> difficult IMHO, b'cause most of it is already there.
> 
> Just tell me what you think about it - I can live with both - I just
> think that stability is better if more projects rely on one library
> and find security flaws and bugs alternatively ;-)

It would be great to have a library based on the ideas in pretix that
others can depend on. I'm not sure if pretix itself could use it, since
some of the things we do in the plugin API are very domain-specific,
like allowing plugin URLs to specify if they should be available only
when a shop is enabled or always.

A generic library would need to behave a bit different in design, and
changing that now will be hard for pretix, since there are already >50
plugins out there.

That said, I'd love to see it and maybe even collaborate on it!

> I have written a IMHO better plugin handler:
> https://gitlab.com/nerdocs/medux/MedUX/blob/develop/medux/extensionsystem/__init__.py
> …
> This way all plugins can just "implement" predefined Interfaces,
> supereasy.

I like it! It is more framework-agnostic than our approach and
therefore probably more flexible. Our approach is a bit more tied to
Django's internals and therefore makes some things easier, but some less
clean.

Best
Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20180801093159.49fd646d%40tavi.localdomain.
For more options, visit https://groups.google.com/d/optout.


pgpUvNQYOfvvI.pgp
Description: Digitale Signatur von OpenPGP

Re: Dynamic app loading

[Raphael Michel]

Hi Christian,

we are doing such a thing for quite a while in our open source project
pretix[1]. I'm not sure if its something that Django needs to do
better, since requirements for this tend to deviate a lot and there's
already a solid basis in the Python packaging toolchain to start from:
To discover apps/plugins, we rely on setuptools' entry point
feature[2], which allows us to easily load all compatible apps
installed in the local Python requirement.

We then disable/enable plugins on a per-client level by using a custom
subclass from Signal[4] and automatically wrapping all views with a
decorator that makes them conditional[6] through some URLConf inclusion
tricks[5].

>From a plugin author perspective, this makes for a pretty clean view[7]
and we are pretty satisfied with the approach. You can also watch me
explaining it here: https://www.youtube.com/watch?v=5NxRdzLTFik

Cheers
Raphael

[1] https://github.com/pretix/pretix
[2] https://packaging.python.org/specifications/entry-points/
[3]
https://github.com/pretix/pretix/blob/master/src/pretix/settings.py#L264
[4]
https://github.com/pretix/pretix/blob/master/src/pretix/base/signals.py#L21
[5]
https://github.com/pretix/pretix/blob/master/src/pretix/multidomain/maindomain_urlconf.py#L23
[6]
https://github.com/pretix/pretix/blob/master/src/pretix/multidomain/plugin_handler.py
[7] https://docs.pretix.eu/en/latest/development/api/index.html

Am Sun, 29 Jul 2018 13:26:04 +0200
schrieb Christian González :

> Hello DjangoDevs,
> 
> I'm new here to this group, and to be honest, just a "fake" developer.
> Doing that is not my main job. So please be patient with my maybe
> BadIdea(tm).
> 
> Another warning: much text. Hint: there is a TL;DR at the end.
> 
> I stumbled upon a "missing feature" in Django, which I call "dynamic"
> app loading, a while ago, since I try to create a Django based
> application which can dynamically add plugins to itself.
> 
> I first tried to google the internet, and found many Stackexchange Q
> where this topic is handled, but either in an insufficient way, not
> applicable to Django 2.0, or else.
> 
> Best ones:
> https://stackoverflow.com/questions/24027901/dynamically-loading-django-apps-at-runtime
> https://stackoverflow.com/questions/7933596/django-dynamic-model-fields
> 
> And my own question with no answer so far:
> https://stackoverflow.com/questions/51234829/dynamic-django-apps
> 
> So I began implementing my own way of handling this.
> 
> Let me first tell a "user story", so you can imagine what I mean.
> 
> My application should more or less be a framework that provides a
> loosely-coupled bunch of modules working together, with a dependency
> tree and versioning. There is a "core module", and others that depend
> on it (e.g. "notifications" etc.). Third party apps should be
> possible, and something like an "app store" should be created to
> dynamically download apps from within the program, and add that
> functionality to the main application.
> 
> So, my first approach was creating zip files with a predefined
> structure (models.py, schema.py, views.py, client stuff etc.), and
> tried to load this code during runtime. I soon realized that I had to
> re-implement most of the stuff Django does anyway, and doing
> migrations isn't an easy task when done barefoot.
> 
> I then changed my mind, and found the best way of having "dynamic
> plugins" is using "Django apps" as plugins.
> But: Django apps are not pluggable. They have to be inserted
> hardcodedly into INSTALLED_APPS to have a predictable order of
> loading. Yes, I've read
> https://groups.google.com/forum/#!searchin/django-developers/app-loading%7Csort:date/django-developers/_iggZzrYtJQ/FWFPgCflSnkJ
> - and I "kind of" understand the Django setup() process (see later).
> 
> I started to fiddle with INSTALLED_APPS, as recommended in
> Stackexchange etc., and dynamically searched a "plugins" directory to
> add some plugins into the list of other apps, just by extending
> INSTALLED_APPS. Django sees no difference, has no cache problems and
> happily loads all my plugins.
> 
> BUT: this is no way dynamic. First thing I recognized is: You can't
> simply call DB requests anywhere near the settings.py loading time.
> Because there is no DB at that moment, let alone models. I then
> stuffed the code into AppConfig.ready() of the core app, and was a
> step further, even if it's not recommended to call models there:
> I need to use models there: I want to check if a plugin app on disk is
> "deactivated", and NOT load it in that case. Aaaargh. Back to the
> start.
> 
> * In settings.py, you can tell Django to dynamically load plugins,
> using disk IO code th

Re: Why doesn't the framework provide support for translated database content?

[Raphael Michel]

Hi Bernhard,

fyi: a similar topic was recently discussed here[0].

My point of view:

I recently counted 19 packages for this purpose, six of them being
actively maintained. You already mentioned that there are two ways that
you can design the database layout (I argue there are three, but thats
nitpicky), but there are also multiple ways to design the Python-side
API.

And here's the point: For each and every of the available options,
there are really big disadvantages.

The JOIN approach is the cleanest database-wise but has its own
problems, for example it gets really messy when you want to deal with
multiple languages at the same time. Also, no implementation managed to
implement that without massively limiting the overall power of the ORM.

The "many columns" approach is bad, because it requires migrations
every time you want to change the supported languages and because it
might lead to very large data sets returned by the database -- but its
easier to work with multiple languages at the same time.

There are also two implementations (django-nece and django-i18nfield)
using a JSON-based approach which mixes some of the problems and
features of the other two approaches, but you give up some lookup or
filterting functionality.

There is absolutely no one-size-fits-all solution here in sight, not
even a works-for-90%-of-people soltion. That makes it really hard to
argue for one of them to be included in Django.

On DjangoCon Europe this year, I gave a talk comparing the six
maintained packages [1]. In an email earlier on this list [2] you can
find a very short summary of that. Unfortunately, the videos of
this conferences are not yet online. You can find my slides at [3],
although the Slides mostly contain the mere facts, and not the opinions
expressed in the talk ;)

Cheers
Raphael

[0]
https://groups.google.com/d/msg/django-developers/6a5Bwf5KMNA/HBj-S8ypBgAJ
[1]
https://2017.djangocon.eu/schedule/data-internationalization-in-django/
[2]
https://groups.google.com/d/msg/django-developers/6a5Bwf5KMNA/4q7HNXysBgAJ
[3] https://speakerdeck.com/raphaelm/data-internationalization-in-django

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20170613191025.14fade39%40kvothe.
For more options, visit https://groups.google.com/d/optout.


pgpVy13lQyhfG.pgp
Description: Digitale Signatur von OpenPGP

Re: Some thoughts about improving migration squashing

[Gaschignard, Raphael]

To clarify on my previous post, if we're in the first case, then the +M and
-F operations can be optimized in one path to remove the dependency, and
then the +M' and -M' operation can be optimized through.

In the second case (with an AddField operation), the +F and -F operations
will cancel each other out (since they will be "in between" +M' and -M')
and then the optimization can happen.

A bit more holistically, since M's field depends on M' in the code (which
is where these operations come from, after all), in order to remove M', you
would first need to remove M's field, so you're almost guaranteed that a
RemoveField will be "in between" any of its dependency's creations/removals.

If we have the field information in the RemoveField, we could check it. But
if we do not make an assumption on (no field information) RemoveFields, it
blocks a lot of possible optimizations. We could just do that (start
generating RemoveField with field info), though it would not allow for
older migrations to get optimized. I'd be good with doing either/both.

Raphael

On Fri, Feb 17, 2017 at 3:09 PM, Gaschignard, Raphael <raph...@makeleaps.com
> wrote:

> Hi Simon,
>
>I think it's a bit more general than that. Why does the `RemoveField`
> exist? Because somewhere, an `AddField`-esque operation exists before it,
> right?
>
>   Let's say we have m, m' as models.
>
>   Let -F be a RemoveField(m, 'foo', ForeignKey(m') operation. We also have
> two operations +M', a CreateModel(m'), and -M', a RemoveModel(m')
> operation. Because we have a removeField operation, we also have +M, a
> "CreateModel(m), somewhere
>
>  Let's assume that we have an operations list like:   [ ...(1)..., +M',
> ...(2)..., -F, ...(3)..., -M']. How do we know that we can reduce +M' and
> -M' together "through" -F?
>
> Because we have -F, we have +M somewhere. because m has a field for m',
> either:
>1 - The field is within the initial +M operation. Because the
> definition depends on M', it must be in group (2).
>2 - The field is not within the initial +M operation. Because the
> definition of the field depends on M', we need an AddField operation (or
> moral equivalent) +F in group (2)
>
> So, if we have -F in between +M' and -M', we will also have either a +M
> which depends on M' or a +F which depends on M' within (2).
>
> So if the -F is between the two, there will also be another operation that
> will be present between the two that expresses the same dependencies. So
> the -F operation itself can avoid expressing its "RemoveField" indirect
> dependency to m' because another operation will do it for them.
>
> 
>
> It's hard to generalize this across everything because there are, after
> all, arbitrary migrations. But considering that RemoveField will only be
> reduced with a "moral equivalent" to AddField, I think we can expand this
> reasoning across everything to say that RemoveField will be sufficiently
> protected by the AddField operation's location (which will have to be after
> +M').
>
>
> Raphael
>
>
> On Fri, Feb 17, 2017 at 12:28 PM, charettes <charett...@gmail.com> wrote:
>
>> > RemoveField('A', 'foo') also references A and foo, but does it
>> reference B? if it does, then it' s hard to have optimizations that pass
>> through this, because this field could be referencing any model
>> (theoretically).
>>
>> I think we all agree on that.
>>
>> > But if we assert that RemoveField doesn't refer to any models
>> referenced to by its field, then our optimizer can take a couple more
>> liberties.
>>
>> Do you have suggestion on how we can assert that it's the case? The only
>> way I could come up with was to make sure RemoveField has a reference to
>> the field it's removing. e.g. It would be generated in the form
>> `RemoveField('A', 'foo', ForeignKey('B'))`.
>>
>> Simon
>>
>>
>> Le jeudi 16 février 2017 18:25:16 UTC-5, rap...@makeleaps.com a écrit :
>>>
>>> When you have AddField('A', 'foo', ForeignKey('B')), this operation
>>> references A and foo, but also references B.
>>>
>>> RemoveField('A', 'foo') also references A and foo, but does it reference
>>> B? if it does, then it' s hard to have optimizations that pass through
>>> this, because this field could be referencing any model (theoretically).
>>>
>>> But if we assert that RemoveField doesn't refer to any models referenced
>>> to by its field, then our optimizer can take a couple more liberties.
>>>
>>> Raphael
>>>
>>> On Friday, February 17, 2017 at 2:15:47 AM UTC+9, Markus Holtermann
>>> wrote:
>>

Re: Some thoughts about improving migration squashing

[Gaschignard, Raphael]

Hi Simon,

   I think it's a bit more general than that. Why does the `RemoveField`
exist? Because somewhere, an `AddField`-esque operation exists before it,
right?

  Let's say we have m, m' as models.

  Let -F be a RemoveField(m, 'foo', ForeignKey(m') operation. We also have
two operations +M', a CreateModel(m'), and -M', a RemoveModel(m')
operation. Because we have a removeField operation, we also have +M, a
"CreateModel(m), somewhere

 Let's assume that we have an operations list like:   [ ...(1)..., +M',
...(2)..., -F, ...(3)..., -M']. How do we know that we can reduce +M' and
-M' together "through" -F?

Because we have -F, we have +M somewhere. because m has a field for m',
either:
   1 - The field is within the initial +M operation. Because the definition
depends on M', it must be in group (2).
   2 - The field is not within the initial +M operation. Because the
definition of the field depends on M', we need an AddField operation (or
moral equivalent) +F in group (2)

So, if we have -F in between +M' and -M', we will also have either a +M
which depends on M' or a +F which depends on M' within (2).

So if the -F is between the two, there will also be another operation that
will be present between the two that expresses the same dependencies. So
the -F operation itself can avoid expressing its "RemoveField" indirect
dependency to m' because another operation will do it for them.



It's hard to generalize this across everything because there are, after
all, arbitrary migrations. But considering that RemoveField will only be
reduced with a "moral equivalent" to AddField, I think we can expand this
reasoning across everything to say that RemoveField will be sufficiently
protected by the AddField operation's location (which will have to be after
+M').


Raphael


On Fri, Feb 17, 2017 at 12:28 PM, charettes <charett...@gmail.com> wrote:

> > RemoveField('A', 'foo') also references A and foo, but does it reference
> B? if it does, then it' s hard to have optimizations that pass through
> this, because this field could be referencing any model (theoretically).
>
> I think we all agree on that.
>
> > But if we assert that RemoveField doesn't refer to any models referenced
> to by its field, then our optimizer can take a couple more liberties.
>
> Do you have suggestion on how we can assert that it's the case? The only
> way I could come up with was to make sure RemoveField has a reference to
> the field it's removing. e.g. It would be generated in the form
> `RemoveField('A', 'foo', ForeignKey('B'))`.
>
> Simon
>
>
> Le jeudi 16 février 2017 18:25:16 UTC-5, rap...@makeleaps.com a écrit :
>>
>> When you have AddField('A', 'foo', ForeignKey('B')), this operation
>> references A and foo, but also references B.
>>
>> RemoveField('A', 'foo') also references A and foo, but does it reference
>> B? if it does, then it' s hard to have optimizations that pass through
>> this, because this field could be referencing any model (theoretically).
>>
>> But if we assert that RemoveField doesn't refer to any models referenced
>> to by its field, then our optimizer can take a couple more liberties.
>>
>> Raphael
>>
>> On Friday, February 17, 2017 at 2:15:47 AM UTC+9, Markus Holtermann wrote:
>>>
>>> I'm not sure if it's related or not wo what you're investigating,
>>> RemoveField cannot "just" optimized through, as you might have another
>>> AddField operation afterwards adding another field with the same name.
>>>
>>> /Markus
>>>
>>> On Thu, Feb 16, 2017 at 08:19:01AM -0800, rap...@makeleaps.com wrote:
>>> >Hey Simon,
>>> >
>>> > I looked through your PR and added a couple comments. The main thing
>>> is I
>>> >think we can actually ignore the field context on "RemoveField", if
>>> only
>>> >because the executor doesn't need it. Even though the field might be
>>> >pointing to a related model, that doesn't prevent being optimized
>>> through.
>>> >
>>> > This is hard to explain, but intuitively, each "RemoveField" is paired
>>> >with an "AddField" or "CreateModel" that *does *depend on the related
>>> >model. So if we have a potentially dangerous optimization, those
>>> initial
>>> >operations will "protect" the causal order, not "RemoveField".
>>> >
>>> > Raphael
>>> >
>>> >--
>>> >You received this message because you are subscribed to the Google
>>> Groups "Django developers  (Contributions to Django itself)" group.
>>> >To unsubscribe

Re: Some thoughts about improving migration squashing

[raphael]

When you have AddField('A', 'foo', ForeignKey('B')), this operation 
references A and foo, but also references B. 

RemoveField('A', 'foo') also references A and foo, but does it reference B? 
if it does, then it' s hard to have optimizations that pass through this, 
because this field could be referencing any model (theoretically).

But if we assert that RemoveField doesn't refer to any models referenced to 
by its field, then our optimizer can take a couple more liberties.

Raphael

On Friday, February 17, 2017 at 2:15:47 AM UTC+9, Markus Holtermann wrote:
>
> I'm not sure if it's related or not wo what you're investigating, 
> RemoveField cannot "just" optimized through, as you might have another 
> AddField operation afterwards adding another field with the same name. 
>
> /Markus 
>
> On Thu, Feb 16, 2017 at 08:19:01AM -0800, rap...@makeleaps.com 
>  wrote: 
> >Hey Simon, 
> > 
> > I looked through your PR and added a couple comments. The main thing is 
> I 
> >think we can actually ignore the field context on "RemoveField", if only 
> >because the executor doesn't need it. Even though the field might be 
> >pointing to a related model, that doesn't prevent being optimized 
> through. 
> > 
> > This is hard to explain, but intuitively, each "RemoveField" is paired 
> >with an "AddField" or "CreateModel" that *does *depend on the related 
> >model. So if we have a potentially dangerous optimization, those initial 
> >operations will "protect" the causal order, not "RemoveField". 
> > 
> > Raphael 
> > 
> >-- 
> >You received this message because you are subscribed to the Google Groups 
> "Django developers  (Contributions to Django itself)" group. 
> >To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-develop...@googlegroups.com . 
> >To post to this group, send email to django-d...@googlegroups.com 
> . 
> >Visit this group at https://groups.google.com/group/django-developers. 
> >To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/9dfdcec6-b98c-44f2-86af-99aaa8857cc9%40googlegroups.com.
>  
>
> >For more options, visit https://groups.google.com/d/optout. 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7515c909-a015-451d-bdaa-f040e6322166%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Some thoughts about improving migration squashing

[raphael]

Hey Simon, 

 I looked through your PR and added a couple comments. The main thing is I 
think we can actually ignore the field context on "RemoveField", if only 
because the executor doesn't need it. Even though the field might be 
pointing to a related model, that doesn't prevent being optimized through. 

 This is hard to explain, but intuitively, each "RemoveField" is paired 
with an "AddField" or "CreateModel" that *does *depend on the related 
model. So if we have a potentially dangerous optimization, those initial 
operations will "protect" the causal order, not "RemoveField".

 Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9dfdcec6-b98c-44f2-86af-99aaa8857cc9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Some thoughts about improving migration squashing

[raphael]

Markus: Now that I've written a command to optimize a single migration 
file, I think that it's sufficient for the "squash, edit, optimize" 
workflow that I was doing before. It's more about offering people to get 
their squashing done well until our optimizer becomes omniscient. 

Florian: Having the command run on all migrations should be 
straightforward, I'll look into that!

Tim: It's not untested! I tested it locally by running it ;) I'll write up 
a couple test cases and some documentation.


I've posted one ticket  for 
the management command to pass a selected migration through the optimizer, 
and another ticket  for 
improvements to the optimizer itself.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/686a3cd9-c15c-4619-a2f7-5aa9f9978910%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Some thoughts about improving migration squashing

[raphael]

I ended up having some time today, so wrote up a management command for the 
first suggestion!

I called it "optizemigration"


>>> ./manage.py optimizemigration appname 0001_squashed
  # snipped django startup noise
  Optimized from 9 operations to 4 operations

Optimized migration /Users/rtpg/proj/projname/projname/appname/migrations/
0001_squashed_20170215.py


This reads in the migration file, runs the migration optimizer, and then 
outputs to the same file. Writing it has paid off almost immediately for me.

Those who are interested can take a look here 

.

How much testing/coverage requirements are there for management commands 
like these?


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b4924688-1001-4a2b-a8de-20e6a10ff83f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Some thoughts about improving migration squashing

[raphael]

Hello everyone,

   I spent a couple hours last night trying to improve the migration 
squasher optimizer (migrations were taking almost 15 minutes in CI). I came 
up with a couple ideas for anyone interested in improvements:
 
 1- Having an interactive mode for squashing would be interested. 
Currently, when squashing migrations, I do the following: 

   - Generate an initial squash
   - Edit it (namely, move around operations to get more optimizations to 
   work)
   - remove the "replaces" tag, then rerun migration squashing to 
   "re-optimize"
   - repeat until I get something I like, then add the original "replaces" 
   tag

   It would be cool if instead, the process were (with a flag):

   - Generate an initial squash, but have the process wait for confirmation 
   to "commit" this squash as final (though writing out the file)
   - Edit the file, and tell the process to try re-optimizing with the same 
   file (getting around the "no-squash of squashes" rule)
   - Potentially, allow us to also step back

 For example, the "squashmigrations" command output could look like:

generated 0001_squashed_mig.py
> optmize migration[yN]? 
> 
> regenerated 0001_squashed_mig.py
> ( 20 operations -> 10 operations)
> optimize migration[Ynr]? 
> 
> regenerated 0001_squashed_mig.py
> ( No change in operation count)
> optimize migration[Ynr]? 
> 
> rolled back to previous version
> optimize migration[Ynr]? 
> 
> Saved migration
>


A simpler version of this command would simply be to add an 
"optimizemigration" command that just reads in a single migration and 
optimizes the operations, without touching any of the squashiness. 


 2- The reducer might be a bit too pessimistic

 Currently, the optimizer lets "reduce" operations (that take 2 operations 
and return 0,1, or 2 operations, or None if nothing can be change) do 
whatever they want. Because of that, if you have [A, B, C ,D] and B depends 
on A, you can't reduce A and C because the reduction might remove A.

 In reality there are two kinds of reduction operations that we could be 
taking into account:

   - reducing "left". if you have [A, B, C, D] (for example, A is a 
CreateModel , C is an AddField for the same model),  and you can reduce A 
and C into just A' (A with C), giving [A', B, D], then it doesn't matter 
that B depends on A. 

The thing that matters is if C depends on B (for example, C adds a foreign 
key to a model created in B). This is actually already encoded in the 
CreateMode + AddField reduction, but is perhaps a more general case.

In a sense, reducing A and C "to the left" means that we're bringing A and 
C closer together only by moving C. This is a major part of the potential 
reductions that the current optimizer is missing.

   - reducing right. If you have [A, B, C, D] (for example, A is a 
CreateModel, C is a RemoveModel for the same model), and you can reduce A 
and C into just C' (C with A), giving [B, C', D], then it does matter that 
B depends on A. C can't depend on B (assuming causality holds in our 
universe)

This is the current mechanism, essentially. If B depends on A, then you 
can't move A past B. 

 Removing both operations is a special case of reducing right (You can make 
C' into a no-op).

I had monkeypatched a special case of  reducing left (taking CreateModel, 
AddField of different models and swapping them . For example 
[CreateModel(A), CreateModel(B), AddField(A.foo)] -> [CreateModel(A), 
AddField(A.foo), CreateModel(B)]) and got decent results, but I think 
making the optimization code express these two concepts separately would 
catch even more of the optimizations I saw that the optimizer didn't.


I hope some of this is useful 
 
  Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6042857b-fd48-467a-badc-1cf5305ecac4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Model translation and meta options

[Raphael Michel]

Hi Claude,

I spent some time looking at the implementations out there and in one
of my projects I'm running a custom one, that uses yet another approach
(that I plan to release as a library some day).

It is not only that we are not yet ready to bless one of them, I have a
feeling that we never will, as they are incredibly different from each
other. 

django-parler, django-hvad and django-model-i18n use seperate tables to
store the data, which is cleaner from a database point of view but
requires more JOINs. The former two use custom model base classes, the
latter uses a registration-based approach.

django-model-translation uses seperate columns for the different
languages, thus working with a "static" set of languages and
requiring a migration if you add translation capabilities or
change the set of languages. On the other hand, it saves you a lot of
JOIN queries. It uses a registration-based approach to register models
with the system.

My project uses only a custom field type that stores the translated
contents in a JSONField-type thing. This has some downsides with
regards to indexing and searching (at least on non-Postgres) and is
less clean from the DB side but requires neither migrations for new
langages nor any additional JOINs. (And I believe it is very clean on
the Django-side as it just requires you to replace a CharField with an
I18nCharField, no custom base classes, no registration, etc.).

Each of those approaches has severe advantages and disadvantages and
more importantly, they are fundamentally different. I don't believe a
new meta option would help you in any way, as for some of them you'd
probably *still* need a custom model base class.

It also would not allow you to easily switch between them or have a
reusable app to be compatible with multiple of them because it would
still be very undefined how you get to the translated versions in the
different systems, how you store the data into the field, etc.

Cheers
Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20170214145327.6c16263d%40arlen.
For more options, visit https://groups.google.com/d/optout.


pgpnFbYwH1Kt8.pgp
Description: Digitale Signatur von OpenPGP

Re: Guidelines to name python modules of Django applications?

[Raphael Hertzog]

Le jeudi 08 décembre 2016, Raphaël Barrois a écrit :
> If I understand correctly the input, would the following proposal be an 
> acceptable middle ground?

It would certainly be an improvement from my point of view, yes.

Cheers,
-- 
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20161217120749.yn74i73y2d7vyyfr%40home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.

Re: Guidelines to name python modules of Django applications?

[Raphael Hertzog]

Le mercredi 07 décembre 2016, Aymeric Augustin a écrit :
> In my experience:
> 
> - many Django-related packages use a django- prefix in the name of the package
>   on PyPI

And the mismatch between package name and module name means that we have
many packages which are not co-installable because they use the same
python module name.

At least using a matching package/module name means that those conflicts
are detected when you try to submit your package.

Here's a concrete example found in a few seconds:
https://pypi.python.org/pypi/cep/0.1.1
https://pypi.python.org/pypi/django-cep/1.1.2

> - very few use a django_ prefix in the name of the Python module, because this
>   module can only be used in the context of Django and the longer name makes
>   imports less readable

It's hard to know the reason. Maybe they do because they follow the
documentation to the letter?

I agree with your assessment of the current situation. A majority of
packages do not follow this recommendation, but a non-trivial part
of the packages do follow it correctly. I have seen many examples in
Debian and it's easy to find many more on PyPi.

> Considering the backwards-compatibility issues — if you change a package name
> all imports needs updating — I don’t think it’s realistic to go against the
> established practice of the ecosystem at this point, and for this reason I'm
> not in favor of making that recommandation.

I'm not asking to rename existing modules, I'm just asking to update
the recommendation in the documentation so that new Django applications do
follow the best practice that avoid namespace conflicts and namespace
pollution.

While the mismatch affects the majority of Django packages, and it's
an established practice for some Django developers, I don't see how
making this recommendation goes against anything. It's not like
we will break anything by making this recommendation.

> To sum up, I’m afraid that your suggestion priorizes the needs of packagers
> over those of developers and, unfortunately for you, developers are the
> heavier users there.

I don't think that developers have much to loose here and as you know,
developers and packagers work hand in hand pretty well.

Cheers,
-- 
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20161207115211.vzjefmdxpn4db2k4%40home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.

Guidelines to name python modules of Django applications?

[Raphael Hertzog]

Hello,

in Debian we recently had a discussion about the package name we use for
Django applications/extensions, it's usually python-django-. But the
generic naming policy we have for all Python modules implies that this
package would provide a "django_foo" Python module and this is not always
the case.

That said we do believe that it is a sort of best-practice and that
as such it should be documented somewhere in the Django documentation.

See the thread on the Debian side here:
https://lists.debian.org/debian-python/2016/11/threads.html#00082

In general, the python module name should match the name on pypi
(except for conversion between "-" and "_") and it should be prefixed with
"django_" to make it clear that this module is for use within the Django
framework.

Would you agree to add this recommendation somewhere in the official
Django doc? 

Looking at https://docs.djangoproject.com/en/1.10/intro/reusable-apps/
it already recommends a "django-" prefix, so the only missing
recommendation is really that the "polls" module should be renamed into
"django_polls" when included in a PyPi package to avoid namespace
pollution and have a direct match between the package name and the module
name.

Cheers,
-- 
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20161207103930.4idy6pqh3i2iw2xa%40home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.

Re: Should ugettext_lazy return instanceof unicode? Or are reusable apps responsible for calling force_text a lot?

[Raphael Michel]

Hello,

Am Fri, 21 Oct 2016 13:49:16 -0700 (PDT)
schrieb Mike Edmunds :
>1. Should the result of ugettext_lazy somehow inherit from
> unicode?

I believe this would break giant measures of code out there that use
"not isinstace(lazystr, unicode)" exectly to detect that it is a lazy
string and not a regular one.

Cheers
rami

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20161023183840.36da34cb%40kvothe.
For more options, visit https://groups.google.com/d/optout.


pgpvJXHU_ALwE.pgp
Description: Digitale Signatur von OpenPGP

Re: DEP pre-proposal for a simpler URLs syntax.

[Raphael Michel]

Hi,

Am Tue, 4 Oct 2016 00:11:28 +0200
schrieb Markus Holtermann <i...@markusholtermann.eu>:
> Thanks for the draft, Tom. I'm a bit concerned that the different
> `url*()` functions you can import will become confusing. Can we have
> `regex_url()` (with chim for `url()`) -- as proposed -- and
> `simple_url()` instead?

yes, we should at all cost avoid two functions named url() that behave
differently. The proposed import paths look to similar to be easily
differentiated on first sight and I normally just type `url` and hit
Alt+Enter to have my IDE do an automatic lookup. I believe this would
be a big burden for people used to the old syntax AND newcomers
(because of the autoimport that is not helping).

Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20161004100240.3bfb0bb9%40arlen.
For more options, visit https://groups.google.com/d/optout.


pgpyxuu_DX6sg.pgp
Description: Digitale Signatur von OpenPGP

Re: Suggestion: Context manager for translation.activate

[Raphael Michel]

Could you just ignore that I've ever sent this mail? Apparently, I
wasn't able to finde translation.override in the docs.

Sorry for the noise.
Raphael

Am Tue, 20 Sep 2016 14:11:11 +0200
schrieb Raphael Michel <m...@raphaelmichel.de>:

> Hi,
> 
> in my application, I regularly need to switch the active language for
> a short period of time. A popular example would be that a
> German-speaking user does something and I need to send out a
> notification to an English-speaking user.
> 
> Cluttering the code with translation.activate() statements is
> certainly not making the core more readable, which is why I'm now
> using a context manager:
> 
> with language(other_user.locale):
> send_mail()
> 
> language() is currently implemented in my project like this:
> 
> @contextmanager
> def language(lng):
> _lng = translation.get_language()
> translation.activate(lng or settings.LANGUAGE_CODE)
> try:
> yield
> finally:
> translation.activate(_lng)
> 
> This turned out to be very convenient and readable and I'd wanted to
> ask for your opinion if 
> 
> (a) something like this should be added to django
> (b) the activate() method itself should be changed to optionally work
> as a context manager
> 
> Cheers
> Raphael
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20160920141233.7d7036da%40kvothe.
For more options, visit https://groups.google.com/d/optout.


pgpMEgAif2NZY.pgp
Description: Digitale Signatur von OpenPGP

Suggestion: Context manager for translation.activate

[Raphael Michel]

Hi,

in my application, I regularly need to switch the active language for a
short period of time. A popular example would be that a German-speaking
user does something and I need to send out a notification to an
English-speaking user.

Cluttering the code with translation.activate() statements is certainly
not making the core more readable, which is why I'm now using a context
manager:

with language(other_user.locale):
send_mail()

language() is currently implemented in my project like this:

@contextmanager
def language(lng):
_lng = translation.get_language()
translation.activate(lng or settings.LANGUAGE_CODE)
try:
yield
finally:
translation.activate(_lng)

This turned out to be very convenient and readable and I'd wanted to ask
for your opinion if 

(a) something like this should be added to django
(b) the activate() method itself should be changed to optionally work as
a context manager

Cheers
Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2016092014.62ef589e%40kvothe.
For more options, visit https://groups.google.com/d/optout.


pgpxloUreQIqG.pgp
Description: Digitale Signatur von OpenPGP

Re: change commit message format to present tense?

[Raphael Hertzog]

Hi,

Le lundi 27 juin 2016, Reinout van Rees a écrit :
> > (You must) "add password validation to prevent the usage of..."!
> 
> "It might be better to think of it as...": it is exactly this extra
> thinkwork that everyone reading the messages has to do. We write it once and
> read it many times: what should we optimize for?
> In our source code, the answer is clearly that you should optimize for
> readability.
> Why is it suddenly different for commit messages?

The first persons reading the commit messages are the persons
reviewing the pull requests and for them, it is quite natural to see
the commit message as "(If merged, this will) fix foo".

Even for someone reading the commits after they have been merged,
when they are clearly history, it's not particularly complicated
to read them as "(This commit did) fix foo".

While this looks like some extra thinkwork, I don't think it's
particularly problematic, and it's defininetely not on the scale
of unreadable code. Whatever the tense used, if the commit message
has the required information, the reader will figure it out.

I have no special preference, but this thread pointed me towards some
widely agreed convention and I'm thankful for that since I tend to always
hesitate when writing a commit message. Now I will stick to the git
convention.

Cheers,
-- 
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20160627102828.GA22531%40home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding a database-agnostic JSONField into Django

[Raphael Hertzog]

Hi,

Le vendredi 24 juin 2016, Shai Berger a écrit :
> > Why would you want to discourage people? In the recent thread "Extending
> > JSONField serialization", you said yourself that the use-case for a
> > JSONField is clearly to store and retrieve data and not to make queries on
> > it with smart lookups.
> 
> So, what you're suggesting is actually a thin wrapper around TextField, which 
> deals with the JSON encoding?

That's the simplest answer that fullfills this requirement but I'm not
suggesting that.

I believe that a "core" JSONField should supersede all the existing
implementations:
- the external implementations which are close to your description
  (but got multiple related features over the years, not all of which are
  worth keeping)
- the django.contrib.postgres one
- the django-mysql one (that I wasn't aware of until Tim pointed out
  the older thread on this topic)

So, in my view, it builds on the simplest implementation while relying
on the native JSON database type when they are available and it should
be easy to extend for the non-portable JSON-related features that each
database might want to make available (special lookups, etc.).

Even with a "thin wrapper", as soon as you rely on the native database types,
you quickly get into non-trivial integrations problems:
- with migrations from TextField to JSONField and vice-versa
  ex: 
https://bitbucket.org/schinckel/django-jsonfield/issues/54/migrating-from-textfield-to-jsonfield-with
- with configuration requirements of the database connector that might have an 
impact
  on other parts of Django
  example where pyscopg does under the hood JSON conversion for us:
  
https://bitbucket.org/schinckel/django-jsonfield/pull-requests/32/make-from_db_value-compatible-with

On top of the model field, there would also be a FormField validating the
JSON input and a widget to present the JSON in a human-readable (indented)
form.

Among the other features that would be interesting to have is the
possibility to swap in a more advanced serializer (which would be able 
to deal with more types of objects than the default one).

> I'm not sure the scope justifies a DEP, but some discussion of the API should 
> happen. Personally, I'm not yet convinced that -- assuming indeed we are 
> talking about a thinly-wrapped TextField -- such a field needs to have a 
> canonical implementation (one declared so by the Django project).

Did my answer clear your doubts?

Cheers,
-- 
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20160624114954.GB21727%40home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding a database-agnostic JSONField into Django

[Raphael Hertzog]

Le vendredi 24 juin 2016, Aymeric Augustin a écrit :
> On 24 Jun 2016, at 09:01, Raphael Hertzog <raph...@ouaza.com> wrote:
> > Why would you want to discourage people?
> Well, this seemed to be the latest decision, but I don’t care much myself.

AFAIK only Marc Tamlyn insisted on such a requirement to have native JSON
support on all main databases.

While this is nice and should be encouraged, it's also the kind of
requirement which discourages people from working on it.

We have working codebase(s) for a generic JSONField. I would like to pick
one, clean it up, strip it down and make it easy to extend so that native
JSON support for more database can be easily added while still keeping
the same user facing API.

The non-portable parts (lookups mainly?) should ideally be enabled via
hooks that applications like django.contrib.postgres could rely on.

> Given the relatively isolated scope of this feature, I think that reaching
> consensus in this thread would suffice. (I don’t think we’re quite there yet.)

Yes, few core team members shared their opinions so far, but we clearly have
user interest for this feature.

> A DEP could help, though, so if you’d like to write a short one, please go
> ahead!

Well, if there's consensus I'd rather spend time on code to be reviewed, but
if others core team members would rather discuss the design at at more
abstract level before deciding whether it's a good move, then I can draft a DEP.

Cheers,
-- 
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20160624102639.GA21727%40home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding a database-agnostic JSONField into Django

[Raphael Hertzog]

Hi,

Le jeudi 23 juin 2016, Aymeric Augustin a écrit :
> On 23 Jun 2016, at 15:40, Tim Graham  wrote:
> > Marc said, "I'm happy for JSONField to be made a core field on the
> > condition that it's underlying support is more than a text blob on all
> > our main databases. It sounds like this will soon be the case.”
> 
> In order to discourage people from using JSONField on databases where it
> cannot be implemented efficiently i.e. not with a text blob, the best
> solution may be to add it to Django and raise a warning when it’s
> misused.

Why would you want to discourage people? In the recent thread "Extending
JSONField serialization", you said yourself that the use-case for a
JSONField is clearly to store and retrieve data and not to make queries on
it with smart lookups.

So while I would certainly want the generic JSONField to use the most
appropriate field type for each database, I don't think we have to wait
until all major databases have the required support. And we should
certainly not throw warnings if you use it on a database that doesn't have
any special JSON support since that is one of the selling point of this
generic module!

The generic JSONField should define the appropriate interface and have
all the logic to easily customize the behaviour for each database, but
it's also fine if it starts with using plain TEXT fields on everything
except MySQL/Postgres (which are the two databases where there are
pre-existing JSONField implementation apparently).

What would be the next logical step if I want to work on this?

Is there a need for a DEP to discuss the interface provided by this field?

Cheers,
-- 
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20160624070136.GA22662%40home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.

Adding a database-agnostic JSONField into Django

[Raphael Hertzog]

Hello,

in almost all projects I work on, I end up using a JSONField. Since
I value being able to run with any database, I'm not relying on
django.contrib.postgres.fields.JSONField. So I have been using
pypi's django-jsonfield maintained by Matthew Schinckel:
https://bitbucket.org/schinckel/django-jsonfield
(I have also packaged this for Debian)

I have recently discovered pypi's "jsonfield" maintained by Brad Jasper:
https://github.com/bradjasper/django-jsonfield

Both projects are very similar (and use the same python package name) and
both projects are actually looking for a new maintainer... since I rely on
something like this, I would be willing to try to merge the best of both
modules into a possible django.contrib.jsonfield or directly into the core.

We could use this opportunity to let the newly-integrated field use
DjangoJSONEncoder by default (see recent discussion about this) and
django.contrib.postgres could register its additionals lookups into the
generic field (assuming we use "jsonb" as underlying type for postgresql).

What do you think of this?

If inclusion into Django is not desired, then maybe we could aim to
at least merge both of those projects in a single "blessed" third-party
module that could be maintained in
https://github.com/django/django-jsonfield?

Cheers,
-- 
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20160623095656.GB22598%40home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sqlite 3.12 breaks the Django test suite

[Raphael Hertzog]

Le jeudi 07 avril 2016, Aymeric Augustin a écrit :
> This other ticket about a SQLite segfault may or may not be related:
> https://code.djangoproject.com/ticket/24080

Given how old it is, I was tempted to say it's entirely unrelated. But
maybe not. I discovered faulthandler thanks to this and the stacktrace
appears in the same place (rollback handling):

$ cd tests
$ LANG=C PYTHONPATH=.. python3 -X faulthandler ./runtests.py --verbosity=2 
--parallel=1 admin_views.tests.AdminViewBasicTest
[...]
test_popup_dismiss_related (admin_views.tests.AdminViewBasicTest) ... ok
Fatal Python error: Segmentation fault

Current thread 0x7fab40575700 (most recent call first):
  File 
"/home/rhertzog/deb/pkg/python-django/django/db/backends/sqlite3/base.py", line 
321 in execute
  File "/home/rhertzog/deb/pkg/python-django/django/db/backends/utils.py", line 
62 in execute
  File "/home/rhertzog/deb/pkg/python-django/django/db/backends/base/base.py", 
line 288 in _savepoint_rollback
  File "/home/rhertzog/deb/pkg/python-django/django/db/backends/base/base.py", 
line 328 in savepoint_rollback
  File "/home/rhertzog/deb/pkg/python-django/django/db/transaction.py", line 
243 in __exit__
  File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 
1016 in _rollback_atomics
  File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 
1073 in _fixture_teardown
  File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 
919 in _post_teardown
  File "/home/rhertzog/deb/pkg/python-django/django/test/testcases.py", line 
217 in __call__
  File "/usr/lib/python3.5/unittest/suite.py", line 122 in run
  File "/usr/lib/python3.5/unittest/suite.py", line 84 in __call__
  File "/usr/lib/python3.5/unittest/runner.py", line 176 in run
  File "/home/rhertzog/deb/pkg/python-django/django/test/runner.py", line 494 
in run_suite
  File "/home/rhertzog/deb/pkg/python-django/django/test/runner.py", line 533 
in run_tests
  File "./runtests.py", line 275 in django_tests
  File "./runtests.py", line 458 in 

Maybe SQLite 3.12 makes it more likely to trigger to the problem or something
like that. I can reproduce the crash with Python 2 and Python 3.

Cheers,
-- 
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20160407131413.GA26492%40home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.

Sqlite 3.12 breaks the Django test suite

[Raphael Hertzog]

[ CCing an upstream developer of SQlite too ]

Hello,

I did not want to open a ticket as I'm not sure if the problem
is in SQLite or in Django but the Django test suite fails really badly
with SQLite 3.12.0 that got recently released (and which is already
in Debian Unstable).

When using the default --parallel run, I get many weird
errors and sometimes even a deadlock. When I run it with --parallel=1,
I got a segfault.

You can see a log of the failure here (this is a parallel run):
https://ci.debian.net/data/packages/unstable/amd64/p/python-django/20160406_191352.autopkgtest.log.gz

There's a backtrace of the segfault (with --parallel=1):
https://bugs.debian.org/cgi-bin/bugreport.cgi?filename=gdb.txt;bug=820225;msg=27;att=1

Cheers,

PS: Tracked here in Debian too:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=820225
-- 
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20160407110012.GA18876%40home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.

Problems with cache.get_or_set behaviour

[Raphael Michel]

Hi,

I just experienced the following behaviour and would like to ask for
your opinion whether I should file this as a bug or whether this is
intended behaviour. I today had the pleasure to work with a crashed
memcached deaemon that could not restart because of a permission
problem. In this case, django, using the backend
'django.core.cache.backends.memcached.MemcachedCache'
behaved as follows:

>>> cache.set("foo", "bar")
>>> cache.get("foo")

So it fails silently and all get calls will return None. That is
sensible behaviour if one does not want cache calls to raise
exceptions. However, when using the new get_or_set() method from Django
1.9, I got

>>> cache.get_or_set('foo', default='bar')
0

This really is suprising to me. For any reasonable usage I can think of,
I would expect get_or_set() to return the specified default value.

When looking at the implementation of BaseCache.get_or_set, one finds
that in the case of the value not existing, the value is saved
and returned as follows:

if val is None:
val = self.add(key, default, timeout=timeout, version=version) 
if val:
return self.get(key, default, version)
return val

I assume that the last get() call is there to prevent a possible
race condition with other processes that is updating the key right
after our add statement.

However, if val is for some reason not evaluating to True, the result
of add() is being returned. add() is documented as returning True or
False and I don't think it makes any sense to return True/False from a
method that is expected to return a cached value or a default value.

I think we need a better behaviour of get_or_set() if add() fails but
I'm unsure how it exactly should look like, otherwise I would have
prepared a patch.

Cheers
Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20160228232625.0e613084%40kvothe.
For more options, visit https://groups.google.com/d/optout.


pgpFdwc2qwCzG.pgp
Description: Digitale Signatur von OpenPGP

Re: Channels integration plan, first draft

[Raphael Michel]

Hi,

Am Thu, 17 Dec 2015 12:27:07 -0700
> I'll admit to not being an expert on this use case at all, since I
> don't generally do it, but AFAIK SCRIPT_NAME remains pretty key for
> transparently deploying a Django site at non-root URL paths.

I used  this before and can confirm that it is currently very
conveniently possible to deploy a Django application on a non-root URL
using SCRIPT_PATH without modifying the application. It is not really
important to me, but I think it would be worth keeping it in, if it is
not too much hassle.

Cheers
Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20151217203754.4f15a4cf%40arlen.
For more options, visit https://groups.google.com/d/optout.


pgpkpxtq5I83X.pgp
Description: Digitale Signatur von OpenPGP

Re: 1.8 shipping invalid .py files in the startapp template

[Raphael Hertzog]

Hello,

I'm another Debian developer co-maintaining Django in Debian.

Le lundi 30 novembre 2015, Florian Apolloner a écrit :
> I am not against including it, but depending on how packaging works in 
> Debian it could be possible to call pycompile with a proper exclusion list 
> instead. The PR itself only works by accident, there are also files in 

We can always add many exceptions but in the end any distro packager wants
to avoid exceptions as we want to automate as much as possible
installations of Python modules (and update to new upstream versions
to make them quickly available to users).

There's a case to be made that the whole process should probably not loose the
information coming setup.py but I'd argue that this information is not
really useful information for packagers until it's properly stored in the
filesystem by the module installation process.

And I would also argue that this would then need to be fixed at the Python
level so that Data is actually stored in separate directories
(/usr/share/something) and not mixed up with Python code.

All in all, I believe our request to be reasonable and I would like
you to consider applying Chris's PR or similar code.

Thank you!

> That said, the amount of issues I have seen in #django due to the 
> debundling of pip in debian/ubuntu is not something I can count on one hand 
> anymore. Most of the time I just gave up and suggested to install pip 
> manually, all problems gone. The syntax error has popped up three times so 
> far (iirc) and requires no fixing aside from educating people.

I find it weird to have such feelings when you go to such lengths to
implement a very nice LTS support policy for Django itself.

Cheers,
-- 
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20151208082148.GB23223%40home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.

ORM difficulties

[Raphael Gaschignard]

Hi list, 

  I want to preface this by saying I’m really glad to see the ORM where it 
is in 1.8, it’s gotten really far and I now think it’s not hopeless to 
imagine writing complex things in the ORM…


   So earlier this week I wrote a bit of a rant in the bug tracker about 
how annotate is confusing (https://code.djangoproject.com/ticket/25834).


  A short summary is that annotate and values (with a spattering of order_by) 
are basically what decides grouping, but there’s a lot of trial and error 
involved in getting the right grouping with a tricky interaction between 
the two.


  A behaviour that exemplifies this is that 
queryset.annotate(foo=thing).annotate(bar=other_thing) is *not* the same as 
queryset.annotate(foo=thing, 
bar=other_thing) given certain things. This goes against the intuitive 
interpretation of the queryset API IMO.


  I think there should be some update to the API to render grouping more 
explicit. Absent that, given that the docs are 

now oriented in a sort of “You shouldn’t need extras anymore” fashion, 
there really should be a “SQL to ORM” migration guide/cookbook to point out 
explicitly how to go from SELECT stuff FROM table GROUP BY properties to 
the right annotate/values. I found what jarshwah pointed out in the ticket 
was really helpful.


  Also, some random feature requests:

   - .values(‘stuff’, my_thing=Coalesce(‘thing’, ‘stuff’)) should work
   - there should be a provided Year and Month functions to extract 
   years/months from date fields

  Anyways I wanted to share this experience so that anyone who has the 
courage to right new docs/continue evolving things can know of at least one 
team’s difficulties.


 Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8d175352-d15e-4dc2-a471-014d6b0bc3e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: annoyance with Python 3.2 support in Django 1.8

[Raphael Michel]

Hi,

Am Wed, 25 Nov 2015 16:36:52 -0800 (PST)
schrieb Tim Graham <timogra...@gmail.com>:
> b. Install the latest non-broken Python 3.2 release (3.2.5)
> "manually" (without using deadsnakes) on the newer CI servers

While it would only really hurt the people in charge with the bugfix
releases, as Django 1.8 will be around for another 2.5 years, I think
dropping CI is not a good idea, considering that is comparatively easy
to compile specific python versions by oneself.

However, we should definitely document somewhere that there are
possbile unknown problems with 3.2.6 and the django admin.

Cheers
Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20151126152618.2f4dba63%40kvothe.
For more options, visit https://groups.google.com/d/optout.


pgpBiht4aGxUy.pgp
Description: Digitale Signatur von OpenPGP

Re: Validate domains against IDNA2008 instead of IDNA2003?

[Raphael Michel]

Hi,

Am Wed, 2 Sep 2015 14:35:22 -0700 (PDT)
schrieb Tim Graham <timogra...@gmail.com>:
> However, python standard lib [
> https://docs.python.org/3/library/codecs.html?highlight=idna#module-encodings.idna
>  
> only offers IDNA2003] validation in , and one have to use [
> https://pypi.python.org/pypi/idna a 3rd party lib] to encode/validate 
> against IDNA2008. 

If I do not miss something, this is a "bug" in Python and not in Django.

However, I believe it could be good idea to have the 3rd-party idna
package as an optional dependency that Django makes use of if present
but works completely without it.

Cheers,
Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20150902235105.7718b8d1%40kvothe.
For more options, visit https://groups.google.com/d/optout.


pgpovt7EcT3MO.pgp
Description: Digitale Signatur von OpenPGP

Re: A new setting for custom gettext domains?

[Raphael Michel]

Hi,

I think the setting is worth it, it can be really useful when working
with complex translation setups.

However, I feel that LOCALE_FILENAMES is a poor choice for the setting
name, as it differs from standard gettext terminology in a very unclear
way. Why not name it LOCALE_DOMAINS?
(To me, _FILENAMES sounds as if I should pass either a path or at least
a full filename, but it is neither of both.)

Raphael

Am Tue, 28 Jul 2015 11:48:14 -0700 (PDT)
schrieb Tim Graham <timogra...@gmail.com>:

> Since adding new settings is sometimes controversial, I wanted to ask
> for thoughts about this ticket [1] / pull request [2] which proposes
> a new LOCALE_FILENAMES setting to allow using additional gettext
> domains besides "django". I don't use translations myself, so
> thoughts about whether the usefulness of the feature justifies a new
> setting or if there is some other solution to the problem described
> in the ticket would be welcome.
> 
> [1] https://code.djangoproject.com/ticket/6376
> [2] https://github.com/django/django/pull/5053
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20150820153900.300d77ec%40kvothe.
For more options, visit https://groups.google.com/d/optout.


pgpyKOcMfnmQ2.pgp
Description: Digitale Signatur von OpenPGP

Re: Should migrations have their imports sorted?

[Raphael Michel]

Am Sat, 25 Jul 2015 16:32:21 -0700 (PDT)
schrieb Tim Graham <timogra...@gmail.com>:
> I think it's been addressed in 1.9 to at least some extent. See 
> https://github.com/django/django/commit/7f20041bca43ca33f0a9617793f2af7ab07c3fab

Oh, I didn't find that. Wonderful, sorry for the noise :)

Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20150726110721.557235cc%40kvothe.
For more options, visit https://groups.google.com/d/optout.


pgpA742XfliXB.pgp
Description: Digitale Signatur von OpenPGP

Should migrations have their imports sorted?

[Raphael Michel]

Hello,

some time ago, Django started to use isort to make sure imports are
formatted and sorted consistently throughout the codebase. I like this
very much, so I started adopting this practice for my own codebases.

However, today I noticed that when Django generates a migration, the
imports in the outputted python file are not necessarily sorted,
causing my checks to fail. In my case it was about these imports:

-from django.db import models, migrations
+from django.db import migrations, models

I had a short look at the source code, but being a novice to the
django.db subtree, I had no idea where exactly to look and I wanted to
ask here: Is this something trivial to fix or is it something that
cannot be solved without changing things very deeply?

This is, of course, absolutely not important, if not completely
irrelevant. However, if it would be a really easy fix, it could be a
nice bonus feature.

Cheers,
Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20150726005423.74fa6353%40kvothe.
For more options, visit https://groups.google.com/d/optout.


pgpyStu8Idok4.pgp
Description: Digitale Signatur von OpenPGP

Re: Why deprecate the ability to reverse using the Python path?

[Raphael Michel]

Am Thu, 16 Apr 2015 01:21:04 -0700 (PDT)
schrieb guettli <h...@tbz-pariv.de>:
> We never jumped on the "give the URLs a name" wagon.

Why don't you go with Tim's idea to implement a custom url() method to
automatically give the views their path as URL names? This should be
rather easy and you should be able to do the migration in half an our
without any drawbacks.

Best regards,
Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20150416103313.79d5f25d%40kvothe.
For more options, visit https://groups.google.com/d/optout.


pgpBXxZAuLhmG.pgp
Description: Digitale Signatur von OpenPGP

Re: Two phase cleaning of models/forms

[Raphael Michel]

Hi,

Am Thu, 2 Apr 2015 09:15:42 -0700 (PDT)
schrieb Preston Timmons <prestontimm...@gmail.com>:
> One use case is for validating address forms. We deal with a lot of
> them with varying levels of validation based on country, state, zip
> code, etc. Sometimes, multiple sets of address fields appear on the
> same form. We can't simply reuse the fields without worrying about
> disparate validation routines specified on the form in addition to the
> fields. This leads to a meticulous set of mixins. It gets the job
> done, but I don't think that's great api.

While Thomas' use case (issue due dates that are required for
certain issue states) can be solved using clean() without major
drawbacks, for your use case (re-usable address fields), I'd go for
MultiValueFields[1] -- this logic does not really belong to the form.

Raphael

[1] https://docs.djangoproject.com/en/1.8/ref/forms/fields/#multivaluefield

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20150408204959.6e08dc20%40kvothe.
For more options, visit https://groups.google.com/d/optout.


pgpO3ul2YEB_7.pgp
Description: Digitale Signatur von OpenPGP

Re: Password validation in Django revisited

[Raphael Michel]

Hi Erik,

Am Sun, 8 Mar 2015 15:47:27 +0100
schrieb Erik Romijn <erom...@solidlinks.nl>:
> I've taken another stab at 16860[1]: having a password
> validation/policy in Django. I've made an initial simple PR[2] to
> show the approach I'd like to use - no tests or documentation yet,
> the example validators are not very nice, possibly bad naming, etc.
> But this should be sufficient to show how I would like to tackle
> this. There's quite a few decisions to take, influencing the later
> possibilities, which I'll try to outline below.

I really like this a lot, both the idea of the ticket and the way
you're proposing to implement it.

However, I'm not quite sure whether I like the style you chose to pass
constructor arguments in the PASSWORD_VALIDATORS setting you proposed. I
believe it might be worth considering to design the PASSWORD_VALIDATORS
setting in a similar fashion to the new TEMPLATES setting.

I understand why you chose this style and it might really be the best
way to do this one thing, but I fear it adds complexity because it
behaves differently than all other settings variables and thus makes it
harder to remember how Django's settings work.

Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20150308172841.3775a4df%40kvothe.
For more options, visit https://groups.google.com/d/optout.


pgpI3LsGgFZcH.pgp
Description: Digitale Signatur von OpenPGP

Re: Psycopg2 version support

[Raphael Michel]

Am Mon, 16 Feb 2015 01:15:53 -0800 (PST)
schrieb Thomas Stephenson <ovan...@gmail.com>:
> The postgres-*.*-dev libraries are a requirement for both 2.0.14 and
> 2.5, so if it's possible to install 2.0.14 via the package manager
> then it's possible to pip install 2.5.

No, I believe this is not correct. The system package manager might (or
in most cases, will) distribute a _compiled_ version of 2.0.14 where you
do not need the -dev libraries to execute them, while pip will download
_sources_ and compile them for you and therefore needs the -dev
libraries.

Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20150216133103.2c3600e5%40kvothe.
For more options, visit https://groups.google.com/d/optout.


pgpZgwRpUl_Xd.pgp
Description: Digitale Signatur von OpenPGP

Re: Feature proposal: Conditional block tags

[Raphael Michel]

Hi,

Am Sat, 31 Jan 2015 22:36:05 +0100
schrieb Aymeric Augustin <aymeric.augus...@polytechnique.org>:
> I suspect it has to do with {% if %} being interpreted at runtime
> while {% block %} is interpreted at compile time.
> 
> I never investigated this fully. If someone does, I’m interested :-)

This was my guess as well. However, if this indeed is the reason, then
a {% block … if … %} syntax would make it impossible to interpret
blocks at compile time, and I don't think Markus' proposal (or the {%
if %}{% block %} syntax, which I do like more than {% block if %}) is
worth that effort.

Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20150201145722.76fae6d0%40arlen.
For more options, visit https://groups.google.com/d/optout.


pgpE_TMS5x3Re.pgp
Description: Digitale Signatur von OpenPGP

Re: Feature proposal: Conditional block tags

[Raphael Michel]

Hi,

Am Sat, 31 Jan 2015 18:12:57 +
schrieb Unai Zalakain <u...@gisa-elkartea.org>:
> Correct me if I'm wrong but the same exact behaviour can be achieved
> by:
> 
> {% block myblock %}
> {% if myvariable %}
> my content
> {% else %}
> {{ block.super }}
> {% endif %}
> {% endblock myblock %}

I guess you're right, but that's exactly Markus' point. This is
inconvenient -- it is exactly one of those tradeoffs Aymeric
mentioned, and seems not well balanced to me, as does not look good in
the template code and I don't suppose that every Django developers
knows about {{ block.super }}.

What is the technical reason for {% if … %}{% block %} not being
possible (or, not behaving as expected)? Enabling this syntax would
improve readability a lot and satisfy Markus' request as well
without introducing new syntax.
(Although from my uneducated guesses of how the blocks work, this might
be very hard and not worthy to implement.)

Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20150131221221.574c4180%40arlen.
For more options, visit https://groups.google.com/d/optout.


pgp0X_QE5nl3g.pgp
Description: Digitale Signatur von OpenPGP

Re: Django documentation doesn't show correctly in any browser

[Raphael Michel]

Am Tue, 13 Jan 2015 20:36:45 +0100
schrieb Aymeric Augustin <aymeric.augus...@polytechnique.org>:
> Well, there are 10k subscribers to this list. Who else is seing this?

I don't, and I run pretty much the same configuration (Chrome on Arch
Linux on multiple computers). I think it pretty obviously is an issue
with the Fira Mono webfont, so maybe some weird font cache
configuration issues. Does this web page work for you?
https://mozilla.github.io/Fira/

Best regards,
Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20150113211107.5439856e%40arlen.
For more options, visit https://groups.google.com/d/optout.


pgpJRQIYJSl_y.pgp
Description: Digitale Signatur von OpenPGP

Re: ANN: Django website redesign launched

[Raphael Michel]

Hi,

I agree with Tino in most points.
Thanks to the volunteers who did this and I really love the fact,
that I can finally browse the docs on a mobile device. Great!

But there certainly is a loss of both originality and the readability.
Both of these things are no blockers and don't make the new design
really bad, but both are things worth improving.

For originality, bringing back some of the old colors could help. They
suited Django very well and looked less like a million other pages out
there. I can't really help here, as I am no talented designer.

For readability, I like Tino's suggestion as well as the tickets [1]
and [2]. I think with minor improvements like a modified line height,
more bold headlines and more contrast in boxes and code, this
documentation could be both as modern as the new design and as readable
as the old one :)

Am Wed, 17 Dec 2014 12:58:37 +0100
schrieb Tino de Bruijn <tin...@gmail.com>:
> Does anybody agree on that and should I create a pull request?

Absolutely, sounds good!

Cheers,
Raphael


[1] https://github.com/django/djangoproject.com/issues/216
[2] https://github.com/django/djangoproject.com/issues/198

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20141217162141.215d6d1d%40arlen.
For more options, visit https://groups.google.com/d/optout.


pgpUY8xpnNQF4.pgp
Description: Digitale Signatur von OpenPGP

Re: Storage engine aliases?

[Raphael Michel]

Hi,

Am Thu, 2 Oct 2014 08:46:26 +0200
schrieb Jannis Leidel <lei...@gmail.com>:
> If something like that were be tried I would also recommend to fix
> the ambiguity with regard to “media” and “static” by renaming the
> first to “uploads”.
> 
> What do others think about that idea? Too much effort for little
> gain? Too late?

No, I can tell this IS a big source of confusion not only for
beginners, so it is worth a change to me.

Cheers,
Raphael


signature.asc
Description: PGP signature

Re: Problem migrating from South to Django migrations for Linux distributions

[Raphael Hertzog]

Hi Brian,

On Fri, 25 Jul 2014, Brian May wrote:
> I can't help but think this might be unrealistic (?). Changes required for
> new versions of Django often break compatibility with old versions, and
> 1.4.5 is really old now. Just because many packages don't have versioned
> dependencies on Django (or a versioned dependency on a really old version)
> doesn't necessarily mean they will work with any 1.4.5. Anyone want to test
> every Django package in Debian sid against the Django in wheezy?

I agree with you that this is not realistic. A better alternate solution
is to document the process of how to execute the South migrations with
Django 1.6 in a virtualenv after having done the upgrade to Debian 8.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20140725074344.GA15591%40x230-buxy.home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.

Re: Problem migrating from South to Django migrations for Linux distributions

[Raphael Hertzog]

Hi Andrew, 

thanks for your quick answer.

On Thu, 24 Jul 2014, Andrew Godwin wrote:
> There is no way around this; it's unfortunate that the packaging situation
> means that Django will get auto-upgraded as part of a distribution upgrade;
> I'm surprised that Debian hasn't had this with packages before? (Version
> upgrades that break installed but non-packaged things)

We probably had this kind of things before and the best we can do for
non-packaged things is usally to document this in the release notes.

But for packaged things, we try usually hard to get things to just work
without any human intervention. Hence my question.

> Neither of your suggested ways to go forward will work; the two history
> models are very different, so the tagging of positions isn't going to work,
> and Django 1.7 has changed substantially enough internally that porting
> South 1.x up to it would be a very large amount of work.

OK.

> Also, what are the applications in particular that this will be a problem
> for? I'm curious to know what Django + South things Debian is shipping
> these days.

Applications that depend on South and have different upstream versions
in Debian 7 and Debian 8 are:
http://tracker.debian.org/pkg/python-django-voting
http://tracker.debian.org/pkg/python-django-threadedcomments
http://tracker.debian.org/pkg/python-django-reversion
http://tracker.debian.org/pkg/python-django-picklefield
http://tracker.debian.org/pkg/bcfg2

Given the package names, it probably means only a single end-user application.
The others are Django "extensions" for use in non-packaged applications.

And looking more closely the case of bcfg2, the package in Debian 7 does
not use South, it started using South in the version in Jessie so it
should be easy to deal with.

For the 4 others, they should provide some NEWS.Debian entry warning
users of the potential upgrade problem.

(Bccing the 5 relevant bug reports to keep a record of this)

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Discover the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20140725080457.GC15591%40x230-buxy.home.ouaza.com.
For more options, visit https://groups.google.com/d/optout.

Re: Is using version 2 of the pickle protocol in {DB,FileBased}Cache

[Raphael Kubo da Costa]

Paul McMillan <p...@mcmillan.ws> writes:

> Ah, this does sound like a pretty nasty issue with our code then.
> Unfortunately, the test suite doesn't get run as often as it should
> with the various cache backends enabled, and I imagine that may be how
> this cropped up. There have been a number of similar bugs in the past
> - caching is hard! I'd appreciate it if you could open a ticket for
> this and reference the mailing list discussion. If you could set the
> owner to me (PaulM), I'll work on it when I get a chance. Thanks for
> taking the time to track down how this is happening.

Hey there, Paul,

Thanks for the interest, I've filed bug #15863.

In case you have some ideas on the directions to take, I could try
working on a patch to tackle the issue -- from what I see, using
pickle.HIGHEST_PROTOCOL is causing the bug, however using a value <
pickle.HIGHEST_PROTOCOL is also unwanted due to the serialized string
being much larger?

In case I work on a patch to improve the unit tests for this part,
should I file a new ticket? Should this mail have actually been sent as
a comment to the ticket? :)

-- 
Raphael Kubo da Costa
ProFUSION embedded systems
http://profusion.mobi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Is using version 2 of the pickle protocol in {DB,FileBased}Cache

[Raphael Kubo da Costa]

Paul McMillan <p...@mcmillan.ws> writes:

> Yes, SimpleCookie is known to be an unpickleable class. We shouldn't
> be directly pickling it anywhere in Django. In your code, you should
> probably turn the cookie into a string before caching it. I'm not
> clear if the bug you're experiencing is happening in Django's code or
> something your application is doing directly with SimpleCookie.

[snip]

> I think that your provided test case is trying to do something that is
> explicitly not supported, but I'm unclear on whether or not there is
> an issue in Django-provided code. Could you provide a little more
> information?

Hi Paul,

I am not trying to pickle SimpleCookie directly -- in fact, I enabled
the cache middlewares in settings.py and then set CACHE_BACKEND to
'file:///some/directory'.

I then had a view with no specific cache decorators, but since the
session backend is also on it added the `Vary: Cookie' header.

After that, I started noticing that a login page including the
`csrf_token' tag started repeating the token when I used curl to access
it without providing any cookies or login credentials. And after the
first time I accessed it, the Set-Cookie header started misbehaving like
it did in the test case I attached -- instead of looking like

  Set-Cookie: foo=bar; other-parameters;

it was looking like

  Set-Cookie: foo="Set-Cookie: foo=bar; other-parameters;"

and the value in the csrf tag was being expanded to something along the
lines of "Set-Cookie: foo=bar; other-parameters;", so validation failed
later.

Some investigation led me to find the problem in the cache backend I was
using, as the cookies inside the cached HttpResponse were being
serialized incorrectly and later picked by FetchFromCacheMiddleware.

-- 
Raphael Kubo da Costa
ProFUSION embedded systems
http://profusion.mobi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Is using version 2 of the pickle protocol in {DB,FileBased}Cache

[Raphael Kubo da Costa]

Hello there,

I was experiencing some problems with Django's caching system on 1.2.X
(1.2.5, to be more specific) when using either the database or the
file-based backends.

Both use pickle and call pickle.dumps(..., pickle.HIGHEST_PROTOCOL) when
serializing the data after being called by UpdateCacheMiddleware.

However, it looks like pickle does not play nice with SimpleCookie-based
classes [1]. In the first request (still not cached), the csrf token is
set correctly as follows:

  Set-Cookie:  csrftoken=XX; Max-Age=31449600; Path=/

When the same view is requested again, though, the cookie is retrieved
incorrectly from the cache by FetchFromCacheMiddleware:

  Set-Cookie:  csrftoken="Set-Cookie: csrftoken=XX Max-Age=31449600\073 Path=/"

The locmem, dummy and memcached backends do not present this problem:
locmem does not specify the protocol version when calling pickle.dumps,
which means protocol version 0 will be used; dummy does not do anything
and memcached does not use pickle. Pickle protocol versions 0 and 1 work
fine.

The following patch to the unit tests should break both
FileBasedCacheTests and DBCacheTests. I only tested it against the 1.2.X
git branch, but 1.3.X should present the same behaviour, and both Python
2.7.1 and Python 3.2 fail.

diff --git a/tests/regressiontests/cache/tests.py 
b/tests/regressiontests/cache/tests.py
index 0581e4e..5611eef 100644
--- a/tests/regressiontests/cache/tests.py
+++ b/tests/regressiontests/cache/tests.py
@@ -285,6 +285,22 @@ class BaseCacheTests(object):
 self.assertEqual(self.cache.get("expire2"), "newvalue")
 self.assertEqual(self.cache.has_key("expire3"), False)
 
+def test_cookie_caching(self):
+try:
+from Cookie import SimpleCookie
+except ImportError:
+from http.cookies import SimpleCookie
+
+test_cookie = SimpleCookie()
+test_cookie['key'] = 'some value'
+
+self.cache.set('some_cookie', test_cookie)
+
+cached_cookie = self.cache.get('some_cookie')
+
+self.assertEqual(cached_cookie['key'].value,
+ test_cookie['key'].value)
+
 def test_unicode(self):
 # Unicode values can be cached
 stuff = {

[1] http://bugs.python.org/issue826897

-- 
Raphael Kubo da Costa
ProFUSION embedded systems
http://profusion.mobi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Sprints DjangoCon Europe

[Raphael Passini Diniz]

I live in Brazil, Belo Horizonte to be more precise, but far from RJ, about
6 hours by car.
If i can help in anything, plz, let me know.

2011/4/8 Wim Feijen <wimfei...@gmail.com>

> More news about the sprints!
>
> During the sprints we will have access to the Dialogue Cafe in de Waag
> building (see: http://www.dialoguecafe.org/). Dialogue Cafe offers
> facilities for video conferencing. Wouldn't it be great if we could
> sprint all over the world? Actually, up to now, two locations outside
> of Amsterdam have been opened, namely Lisbon (Portugal) and Rio de
> Janeiro (Brazil). In addition, Ramallah (India) will probably open in
> june, so maybe in time maybe not for the sprints.
>
> My question right now is: do you know any programmers in (the vicinity
> of) Lisbon, Rio de Janeiro or Ramallah? Who would be willing to attend
> to the sprints and possibly help in organizing the local communities?
>
> Please let me know. You can respond here or by private mail as well.
>
> Thanks!
>
> Wim
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>


-- 
Atenciosamente Raphael Passini Diniz

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Benevolent Dicta^H^H^H Designer for Life

[Raphael Passini Diniz]

Hi Idan,

Good luck! I'll expecting some exciting news! :D

2011/3/22 Wim Feijen <wimfei...@gmail.com>

> Hi Idan,
>
> Those are two great topics, congratulations to you and to us :) and
> good luck!
>
> Wim
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>


-- 
Atenciosamente Raphael Passini Diniz

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Summer of Code 2011: students wanted!

[Raphael Passini Diniz]

et you work on this
> for the
> summer" as much as "convince us you understand the problem you're
> proposing to work on and that your solution has a chance of working
> and being accepted by the core body of developers".
>
> Our goal this year is for *every single project* to result in code that
> gets
> committed back to Django. We'll accomplish this goal by *rejecting*
> applications that don't show us good odds of success.
>
> What a good proposal looks like
> ---
> Once you've got some discussion done, and have a handle on the task,
> you'll need to submit your proposal on the GSoC website
> (http://www.google-melange.com/).
>
> We'll be looking for a few things in your proposal:
>
>- A concise but complete description of the problem.
>
>- A concrete, as detailed as you can, proposal of how you plan to
> modify
>  Django to fix said problem. This is where you'll include links to
>  your previous discussions on django-developers.
>
>- A timeline, ideally broken into week or two-week chunks, showing the
>  steps you plan to take as you work on the problem. This is obviously
>  subject to change, but it'll show us that you have a handle on the
> scope
>  of the problem. You should also indicate roughly how much time
>  (hours/week) you can devote to the problem. Some folks work full-time
>  (40+ hours/week) on SoC. Less is OK, but if you've only got weekends
>  free we'd like to know about that.
>
>- Some information about yourself, including why you think you're
>  qualified to work on this problem. We don't need or want a resume,
> but
>  include any relevant information about you and your background
> that'll
>  help us get a feel for who you are.
>
> What a bad proposal looks like
> --
>
> Just to round things out, a couple of things that happen in applications
> that have absolutely no chance:
>
>- Somebody we've never heard of proposing something *really* ambitious.
>  It's not too hard to guess roughly at the required amount of work,
> for
>  those of use who maintain Django or work regularly on the
> internals, so
>  if somebody proposes something that's too hard, we can tell. And
> if we
>  don't know them at all, it's hard to trust they'll get things done.
>
>- However, the, the problem shouldn't be too small. It's called
> Summer of
>  Code, not Weekend of Code.
>
>- A proposed change without any evidence on django-dev or
> django-users (or
>  elsewhere) that the change is needed. We make changes because
> there are
>  use-cases for them, not because we can. So any proposal should be
> driven
>  by trying to fix some existing problem, not creating a "wouldn't
> it be
>  nice if...?" situation.
>
>- This will sound clichéd, but a badly written application is a real
>  turn-off. If you can't write an important document like that and use
>  spelling, grammar and sentences with at least some punctuation, we'll
>  worry about your ability to communicate effectively with your
> mentor and
>  the community.
>
>  There's a lot of people in our community who don't speak English as a
>  first language. *That's* not a problem; we're not going to be grading
>  you! However, we need to see that you can communicate clearly, and so
>  basic proofreading is a must.
>
>
> Next steps
> --
>
> If you've read all that -- and congratulations, by the way! -- the next
> things
> to do are:
>
>- Start looking for a project, and draw up some rough plans.
>
>- Get an initial proposal together, and post it to django-developers.
>
> Good luck, and we look forward to your proposals!
>
> Andrew
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>


-- 
Atenciosamente Raphael Passini Diniz

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: auto_now_add overwrites a given date

[Noam Raphael]

Thanks!

Ok, I see that the problem is more complicated than I thought, so I'll
leave it that way.

I just want to add that if you decide that you don't like the current
behaviour, you can always use FutureWarning, to first warn if a value
was set and is about to be discarded, and then change the behaviour.

Have a good day,
Noam

On 7/5/07, Russell Keith-Magee <[EMAIL PROTECTED]> wrote:
>
> On 7/5/07, Noam <[EMAIL PROTECTED]> wrote:
> >
> > The conclusion is that auto_now_add should set the date only if it is
> > None. Or, another solution: it should set the date upon instance
> > creation, not upon saving.
>
> FYI - this is a known problem, with scope and ramifications much
> larger than the problem you have identified. It also affects any model
> with a customized save() method, and is a major problem for accurate
> data serialization, which in turn is a problem for testing.
>
> The problem is logged as #4459; I'm hoping to spend some time fixing
> this in the near future.
>
> Neither of the solutions you propose are viable - as others have
> noted, the first is backwards incompatible, the second would make
> auto_now_add equivalent to auto_now.
>
> The best solution I can think of (and the solution that will probably
> get implemented, unless a better solution is proposed) is to add a
> 'preprocess=True' default argument to the save() method of a model.
> Default behaviour will be unaffected; however, if you call
> model.save(preprocess=False),  Django will skip the pre-processing
> steps that modify dates and do other forms of database coercion. This
> will allow serialization to force a literal interpretation of input
> data, and would fix your problem as well.
>
> Yours
> Russ Magee %-)
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Unicode field names

[Noam Raphael]

Thanks a lot - I switched to the unicode branch and it works very
nice! I'll report any bugs I encounter.

Noam

On 13/06/07, Malcolm Tredinnick <[EMAIL PROTECTED]> wrote:
>
> On Tue, 2007-06-12 at 21:16 +, Noam wrote:
> > Hello,
> >
> > First of all, I apologize if I post this in the wrong forum. It just
> > seems to me like something which requires design decisions, and not a
> > simple bug. If I am wrong, please correct me.
> >
> > I'm making a site which will be in Hebrew. I have a model with lines
> > like these:
> >
> > telephone = CharField(maxlength=30)
> >
> > I changed my settings so that the admin interface is in Hebrew, and it
> > looks very nice, but of course "Telephone" is displayed in English. So
> > I made the obvious change:
> >
> > telephone = CharField(u'טלפון', maxlength=30)
> >
> > However, it causes the admin app to fail with a nasty
> > UnicodeDecodeError.
>
> There have been a lot of posts about unicode and Django on the users
> list lately. The short version is this: unicode on trunk is pretty much
> unsupported in general and if it works for any particular case, you are
> most fortunate. The Unicode branch fixes these problems.
>
> Probably best to check out the Unicode branch (see the wiki or search on
> Google) and then post questions to django-users.
>
> Regards,
> Malcolm
>
>
>
> >
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Allowing multiple {% block %}s in one template

[Noam Raphael]

Ok, I posted my patch at http://code.djangoproject.com/ticket/4529

On 10/06/07, Malcolm Tredinnick <[EMAIL PROTECTED]> wrote:
> By the way, one advantage of writing the functionality you want as an
> extra tag (something not called "block") is that you (or anybody else
> who wants to work this way) can easily use it as a third-party tag, so
> it doesn't need to be part of core.

I think that if it's not accepted, the easiest way to accomplish what
I want would be my hack. This way I won't have to use a different name
for blocks, and I won't have to maintain an external tag system. It's
still a hack, though.
>
> > (By the way, the current SVN doesn't load the loader_tags module, at
> > least for me. I fixed it (or so it seems) by moving the line
> > "add_to_builtins('django.template.loader_tags')" from
> > template/loaders.py to template/__init__.py. Did I do something wrong
> > or is it a bug?)
>
> The {% block %} tag seems to work currently (it's tested as part of the
> test suite), so that would seem to suggested loader_tags is being
> imported correctly. Note that the import of that file happens as part of
> template.get_library(), which is called by add_to_builtins(). So the
> loader_tags file is imported, just not explicitly. No changes seem to be
> needed there, unless I'm misunderstanding what you are saying.
>
I did a fresh SVN checkout, and made a new project, and run
./manage.py shell, and got:

[EMAIL PROTECTED]:/tmp/mysite$ ./manage.py shell
Python 2.5.1 (r251:54863, May  2 2007, 16:56:35)
[GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.template import Template, Context
>>> t = Template('''{% block title %}Hello{% end block %}''')
Traceback (most recent call last):
  File "", line 1, in 
  File "/home/noam/lib/python/django/template/__init__.py", line 162,
in __init__
self.nodelist = compile_string(template_string, origin)
  File "/home/noam/lib/python/django/template/__init__.py", line 178,
in compile_string
return parser.parse()
  File "/home/noam/lib/python/django/template/__init__.py", line 284, in parse
self.invalid_block_tag(token, command)
  File "/home/noam/lib/python/django/template/__init__.py", line 328,
in invalid_block_tag
raise self.error( token, "Invalid block tag: '%s'" % command)
TemplateSyntaxError: Invalid block tag: 'block'

Thanks,
Noam

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Allowing multiple {% block %}s in one template

[Noam Raphael]

Ok, I've implemented my suggestion.

For those concerned about the complexity of implementation, it turns
out that all that's needed is a simple change in the block parser, to
make it return the same object for all blocks with the same name in a
template. The rendering code wasn't touched at all.

(In order to make sure that all blocks of the same name really had the
same content, I added an optional argument to the parse() method which
allows it to return the list of tokens from which a block was made.)

Where should I post the diff?

Noam

(By the way, the current SVN doesn't load the loader_tags module, at
least for me. I fixed it (or so it seems) by moving the line
"add_to_builtins('django.template.loader_tags')" from
template/loaders.py to template/__init__.py. Did I do something wrong
or is it a bug?)

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Allowing multiple {% block %}s in one template

[Noam Raphael]

> Aside from the "urgh!" factor from writing the content in multiple
> times, if you are going to do this, why not just create a "reuse-block"
> tag that re-inserts the block value the template parser has already
> worked out previously? Doesn't need any change to the behaviour of
> "block" then.
>
As I see it, because you:
1. Add another tag that users should be aware of.
2. Encourage users to do what James was afraid that they will do.

I don't see this as "changing the behavior of blocks". I think my
suggestion is merely to relax the restriction - multiple blocks with
the same name are not disallowed altogether - they are allowed, as
long as they all have the same content.

Noam

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Allowing multiple {% block %}s in one template

[Noam Raphael]

On 08/06/07, Marty Alchin <[EMAIL PROTECTED]> wrote:
>
> On 6/8/07, Noam <[EMAIL PROTECTED]> wrote:
> > Let me improve my suggestion, so that it mostly solves James' concern
> > and can be explained in one sentence:
> >
> > -
> > Multiple blocks with the same name would be allowed, as long as all of
> > them have the same content.
> > -
>
> I'm not sure I like that as it stands. In your example, you provide
> content for both of the blocks, which clearly violates DRY, since (by
> virtue of your new rule) they're always going to be identical. If a
> template designer ever has to update one, they have to update both.
>
If the time comes when you need a page with a different title in the
window and in the heading, you can always extend your base template
like this:



{% block windowtitle %}Bug System - {% block title %}Page
Title{% endblock %}{% endblock %}


{% block headingtitle %}{% block title %}Page Title{% endblock
%}{% endblock %}
{% block body %}Page Body{% endblock %}



With that template, the original specific template would continue to work:

{% extends "base.html" %}
{% block title %}Post a New Bug{% endblock %}
{% block body %}...{% endblock %}

But you will also be able to write a template like this:

{% extends "base.html" %}
{% block windowtitle %}Bug 12345{% endblock %}
{% block headingtitle %}Bug 12345 - Allow multiple blocks with the
same name{% endblock %}
{% block body %}...{% endblock %}

Does it help?

> I'm still -0 on this, but I might be +0 (yeah, big change) if the rule
> stated that in order to have multiple blocks with the same name, all
> must be EMPTY. This would allow a child template to fill in multiple
> blocks, while not giving template designers a reason to get confused
> by duplicate content.
>
I don't think it's confusing that both {% block title %} have the same
content - it just shows that they will get the same contents in the
rendered page.

> I'm still not sold on the idea as a whole though, it just seems like
> any support for it would be quite a mess, whether for the
> implementation or the users.

About the implementation - I think that it is best to try to implement
and see how complicated is it. About the users - I don't think it will
be a mess. People with a need like mine will just write two empty
blocks and everything will work as they expected, or will write one
empty block and one full block, will get an error stating that
multiple blocks with the same name in the same template must have the
same content, and fix it. I don't see what further complications
should arise.
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---