Re: Migrations, commands and syncdb

[Andrew Godwin]

Yes, the large number of weird things people need custom SQL for is why I
want to push for migrations much more overall - especially for cases like
post_syncdb (should be replaced by data migrations) and the arbitrary
initial SQL file support (should be replaced by custom migrations).

I don't plan to add in an equivalent to "syncdb --all" - the squashing
should be enough to make things nice and fast for testing, and will have
the added benefit where your test database matches your production one more
closely. Squash (what was previously called rebase) is designed to let you
throw away old migrations, especially in a case where you tightly control
all installs.

There might be some scope for a version of the migrate command that does a
one-shot, in-memory squash-then-execute - essentially a very quick way of
doing initial migrations - but that depends if it turns out to be useful
and how fast the migration optimiser/compressor that runs on the squashes
will be.

Andrew


On Fri, May 31, 2013 at 1:38 PM, Marc Tamlyn  wrote:

> I have generally not kept my migrations so they always work from scratch
> for similar reasons Luke said - Data migrations are often dependent on the
> data in the site - this is particularly relevant to content-driven (CMSy)
> sites where I may need to move a load of page about as a data migration at
> some point in the history, but this data will not exist for the tests.
> These "throwaway" data migrations are IMO pretty common. As a consequence
> (and given the speed requirements) I've generally never run a site without
> SOUTH_TESTS_MIGRATE set.
>
> Also, when using an older library which has a long migration history
> included, when we start a new site using it we generally use `syncdb --all`
> so as not to worry about the history. Will this command still be supported
> by the new `migrate` command?
>
> I can see the argument for making sure the migrations work, and they may
> be necessary for migrations containing custom SQL the site depends on, but
> I would suggest that many developers effectively treat migrations as things
> that can be thrown away once they've been run against all the relevant
> databases. Perhaps the rebase command needs to consider this - how can we
> throw away old migrations (especially data migrations).
>
> (PS: +1 to option 3 in principle)
>
>
> On Friday, 31 May 2013 12:44:20 UTC+1, Anssi Kääriäinen wrote:
>>
>>  On 05/31/2013 02:08 PM, Andrew Godwin wrote:
>>
>>   One problem with this is that you have to be careful to write
>>> migrations
>>>  that will always work from scratch. This is best practice, but I have on
>>> occasion used data migrations that were for specific problems, and may
>>> have depended on specific data in the database. I've also used them for
>>> populating parts of the database at specific upgrade points. Usually I
>>> wouldn't want this data to be installed for my tests - I want my tests
>>> to explicitly do the setup they need, and not be complicated by data
>>> from data migrations.
>>>
>>> For these reasons, and because of the slowness of running all the
>>> migrations, there are some projects where I have a settings file for
>>> running the tests that excludes South from INSTALLED_APPS, which solves
>>> the problem nicely.
>>>
>>> The slowness problem can be fixed by squashing, but it seems a shame to
>>> require that simply for the sake of running tests. And it doesn't solve
>>> the other problems. I would like to be able to turn off migrations for
>>> tests, and just skip to the final schema.
>>>
>>
>>   Yes, this has traditionally been a problem, and it's difficult to say
>> what to do here.
>>
>>  I personally have the opinion that stuff you add in data migrations
>> _should_ be part of the tests - after all, it's usually crucial data that
>> the site won't work without, and you're never going to test an installation
>> without them.
>>
>>  However, I understand people don't like doing this. Problem is, the
>> alternative is to reserve the old syncdb method for running tests, and then
>> to introduce a new setting (which in South is SOUTH_TESTS_MIGRATE) that
>> lets users turn it on and off, which seems like a bad road to go down.
>>
>>  I'd rather keep it so you must run migrations for tests - as I believe
>> we should be making people do this as a general case - and if you want,
>> having the data migration operations have an option you can pass where they
>> no-op during tests, something like:
>>
>>  operations = [
>> Python(
>> """
>> for author in Author.objects.all():
>> author.fix()
>> """,
>> skip_during_tests=True,
>> ),
>> ]
>>
>>
>> If test database is set up by syncdb and you have some SQL that isn't
>> model migrations, then you will likely need two sets of migrations. One
>> that applies to production database, and one that applies to fresh database
>> from syncdb. You will need to constantly keep the "applies to syncdb
>> database" migrations updated, a

Re: Is file based cache is safe for concurrent process?

[Ivan Smirnoff]

> During cache body creation, concurrent processes can also make useless
cache bodies in case usage tmp file (not locking).

-- sorry, during cache writing to tmp file...


2013/5/31 Ivan Smirnoff 

> I was also created the backend, but did not have time to make a fork.
> See
> https://bitbucket.org/evotech/cache-tagging/src/d26adf9f66581923f03ca2d89261568b521eb82e/cache_tagging/django_cache_tagging/backends.py?at=default
>
> Also, I think, it will be better to move directory clearing to Thread for
> async, and run it not every time (see link above)
>
> By the way, file locking is better than tmp file for resource
> consumption. During cache body creation, concurrent processes can also make
> useless cache bodies in case usage tmp file (not locking).
>
>
> 2013/5/31 
>
> I've created a ticket with a script that demonstrates the issue and
>> provides an implementation that uses werkzeug's tempfile approach
>> https://code.djangoproject.com/ticket/20536
>>
>> Before I start forking on github I'd like some feedback on the ticket.
>> Thanks!
>>
>> Op donderdag 28 maart 2013 01:23:52 UTC+1 schreef Russell Keith-Magee het
>> volgende:
>>>
>>>
>>> On Thu, Mar 28, 2013 at 7:13 AM, Ivan  wrote:
>>>

 Yes, you are right. But I asked for another reason.
 For example, I see, that werkzeug's cache write cache trough temporary
 file https://github.com/mitsuhiko/**werkzeug/blob/master/werkzeug/**
 contrib/cache.py#L660
 But Django cache do not use neither, tmp file or file locking.
 So, multiple processes can write a file simultaneously. Also, other
 processes can obtain incomplete data.

 I think it's a bug, so I asked a question in this group
 (django-developers).

>>>
>>> Ok - *that* is a different matter. For future reference, if you're
>>> suggesting that something might be a bug, it's helpful to make that clear
>>> in your question.
>>>
>>> As I said in my response on django-users, the file based cache is
>>> primarily there as a proof of concept that the caching API can be targeted
>>> at multiple backends. However, if someone were to propose a patch to add
>>> file locking or temp file write-throughs, I don't imagine it would be
>>> turned down.
>>>
>>> So - feel free to open a ticket for this; and if you want to try your
>>> hand at a patch, this should be a relatively simple patch to create
>>> (although tests could be a little bit complex). As a helper, the
>>> django.core.files.locks module already contains some OS-abstracted code for
>>> file locking.
>>>
>>> Yours,
>>> Russ Magee %-)
>>>
>>  --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Django developers" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/django-developers/PCgYCInrL0g/unsubscribe?hl=en
>> .
>> To unsubscribe from this group and all its topics, send an email to
>> django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at
>> http://groups.google.com/group/django-developers?hl=en.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Is file based cache is safe for concurrent process?

[Ivan Smirnoff]

I was also created the backend, but did not have time to make a fork.
See
https://bitbucket.org/evotech/cache-tagging/src/d26adf9f66581923f03ca2d89261568b521eb82e/cache_tagging/django_cache_tagging/backends.py?at=default

Also, I think, it will be better to move directory clearing to Thread for
async, and run it not every time (see link above)

By the way, file locking is better than tmp file for resource consumption.
During cache body creation, concurrent processes can also make useless
cache bodies in case usage tmp file (not locking).


2013/5/31 

> I've created a ticket with a script that demonstrates the issue and
> provides an implementation that uses werkzeug's tempfile approach
> https://code.djangoproject.com/ticket/20536
>
> Before I start forking on github I'd like some feedback on the ticket.
> Thanks!
>
> Op donderdag 28 maart 2013 01:23:52 UTC+1 schreef Russell Keith-Magee het
> volgende:
>>
>>
>> On Thu, Mar 28, 2013 at 7:13 AM, Ivan  wrote:
>>
>>>
>>> Yes, you are right. But I asked for another reason.
>>> For example, I see, that werkzeug's cache write cache trough temporary
>>> file https://github.com/mitsuhiko/**werkzeug/blob/master/werkzeug/**
>>> contrib/cache.py#L660
>>> But Django cache do not use neither, tmp file or file locking.
>>> So, multiple processes can write a file simultaneously. Also, other
>>> processes can obtain incomplete data.
>>>
>>> I think it's a bug, so I asked a question in this group
>>> (django-developers).
>>>
>>
>> Ok - *that* is a different matter. For future reference, if you're
>> suggesting that something might be a bug, it's helpful to make that clear
>> in your question.
>>
>> As I said in my response on django-users, the file based cache is
>> primarily there as a proof of concept that the caching API can be targeted
>> at multiple backends. However, if someone were to propose a patch to add
>> file locking or temp file write-throughs, I don't imagine it would be
>> turned down.
>>
>> So - feel free to open a ticket for this; and if you want to try your
>> hand at a patch, this should be a relatively simple patch to create
>> (although tests could be a little bit complex). As a helper, the
>> django.core.files.locks module already contains some OS-abstracted code for
>> file locking.
>>
>> Yours,
>> Russ Magee %-)
>>
>  --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/PCgYCInrL0g/unsubscribe?hl=en
> .
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers?hl=en
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Is file based cache is safe for concurrent process?

[jaap]

I've created a ticket with a script that demonstrates the issue and 
provides an implementation that uses werkzeug's tempfile approach 
https://code.djangoproject.com/ticket/20536

Before I start forking on github I'd like some feedback on the ticket. 
Thanks!

Op donderdag 28 maart 2013 01:23:52 UTC+1 schreef Russell Keith-Magee het 
volgende:
>
>
> On Thu, Mar 28, 2013 at 7:13 AM, Ivan >wrote:
>
>>
>> Yes, you are right. But I asked for another reason.
>> For example, I see, that werkzeug's cache write cache trough temporary 
>> file 
>> https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/contrib/cache.py#L660
>> But Django cache do not use neither, tmp file or file locking.
>> So, multiple processes can write a file simultaneously. Also, other 
>> processes can obtain incomplete data.
>>
>> I think it's a bug, so I asked a question in this group 
>> (django-developers).
>>
>
> Ok - *that* is a different matter. For future reference, if you're 
> suggesting that something might be a bug, it's helpful to make that clear 
> in your question. 
>
> As I said in my response on django-users, the file based cache is 
> primarily there as a proof of concept that the caching API can be targeted 
> at multiple backends. However, if someone were to propose a patch to add 
> file locking or temp file write-throughs, I don't imagine it would be 
> turned down.
>
> So - feel free to open a ticket for this; and if you want to try your hand 
> at a patch, this should be a relatively simple patch to create (although 
> tests could be a little bit complex). As a helper, the 
> django.core.files.locks module already contains some OS-abstracted code for 
> file locking.
>
> Yours,
> Russ Magee %-)
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Testing generated SQL under different backends

[Anssi Kääriäinen]

On 31 touko, 15:51, Marc Tamlyn  wrote:
> In particular I'd be quite strongly against the main Django codebase
> shipping with potentially broken drivers - this will reflect very badly on
> the project for new users.

The thing is, this is *already* true. Oracle GIS was badly broken in
1.4 release for a long time. Many GIS tests are still failing (1.4,
1.5 and master), but at least saving objects is now possible. If
nobody cares enough to fix this, then I don't see a big problem. What
we should likely do is add some warnings somewhere about potentially
broken state of Oracle GIS (import time + docs maybe).

We could also throw the Oracle GIS code away, but I don't see much
point in doing that. The code is working well enough now that Oracle
GIS is used by some users.

Of course the best case scenario is that somebody is motivated enough
to fix the tests.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Testing generated SQL under different backends

[Anssi Kääriäinen]

On 31 touko, 16:58, Michael Manfre  wrote:
> On Fri, May 31, 2013 at 8:51 AM, Marc Tamlyn  wrote:
> > In terms of non-core backends (and I include Oracle in this) - my
> > preferred option would not be to include them all in core Django, but to
> > collate them in and manage them under the Django project - i.e.
> > github.com/django/django-mssql. These are "looked after" by the Django
> > project to ensure that there is one place to find that code base, but are
> > externally maintained. This is effectively similar to the way that
> > localflavor is progressing, although we should wait and see how that plays
> > out first - there are some teething issues there at the moment.
>
> Django-mssql can be installed with pip and the project's pages (code, docs,
> etc) are the first several hits on the popular search engines. Moving a
> repo for the sake of moving it is a non starter for django-mssql. I'm
> guessing that most of the other 3rd party database backends would object as
> well, especially those not currently hosted on github. Relocating a repo is
> disruptive to the developer(s) and also the users. I've gone through this
> once when I moved the project from google code to bitbucket, which had the
> huge benefit of making it easier and more enjoyable for me to work on the
> project.
>
> The disruption associated with relocating the project in to the core would
> at least be balanced by many benefits. Just to name a few, easier test
> integration, backend's issues in trac, no more django version/feature
> checking in the backend, and more eyes watching commits (even if no one
> else works on the backend). There is also the potential of easily fixing
> the poorly named flags on DatabaseFeature.

>From ORM coding perspective the issue is this: I want the ability to
alter backends API at will. Of course, we have already changed the API
when needed (transaction handling changes, connection initialization
changes for example), and we will likely do this in future, too
(support for database schemas for example).

Of course, this doesn't mean the API should be changed without reason.
But if there is a reason, then we must be able to change the API.
(Same goes for other internal APIs, too).

If unstable APIs are OK for 3rd party backend devs, then all is OK. I
don't think there is point in moving code anywhere for the sake of
moving it. The reason must be making development easier for both 3rd
party devs and core devs. I don't see much difference for users, pip
install is easy enough to do.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Testing generated SQL under different backends

[Michael Manfre]

On Fri, May 31, 2013 at 8:51 AM, Marc Tamlyn  wrote:

> In terms of non-core backends (and I include Oracle in this) - my
> preferred option would not be to include them all in core Django, but to
> collate them in and manage them under the Django project - i.e.
> github.com/django/django-mssql. These are "looked after" by the Django
> project to ensure that there is one place to find that code base, but are
> externally maintained. This is effectively similar to the way that
> localflavor is progressing, although we should wait and see how that plays
> out first - there are some teething issues there at the moment.


Django-mssql can be installed with pip and the project's pages (code, docs,
etc) are the first several hits on the popular search engines. Moving a
repo for the sake of moving it is a non starter for django-mssql. I'm
guessing that most of the other 3rd party database backends would object as
well, especially those not currently hosted on github. Relocating a repo is
disruptive to the developer(s) and also the users. I've gone through this
once when I moved the project from google code to bitbucket, which had the
huge benefit of making it easier and more enjoyable for me to work on the
project.

The disruption associated with relocating the project in to the core would
at least be balanced by many benefits. Just to name a few, easier test
integration, backend's issues in trac, no more django version/feature
checking in the backend, and more eyes watching commits (even if no one
else works on the backend). There is also the potential of easily fixing
the poorly named flags on DatabaseFeature.

Regards,
Michael Manfre

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Proposal/Discussion: url and template search using the directory structure of apps within django sites

[Jorge C . Leitão]

Hi.

I don't have any more arguments, and I agree with most of your criticisms.
However, I still think that the change of 

get_template(template_name)
...
to
get_template(template_name*, dirs=None*)
...
template, origin = find_template(template_name*, dirs*)
(and similars in that module)

is a very simple way of giving developers the possibility to code their own 
template loader that can pass the argument dirs to the get_template... In 
my case, I only changed the django code because I had no other option when 
doing what I was doing, and so I made it such that dirs is an optional 
argument.

In conclusion, you convinced me that all other ideas should not be part of 
Django's code. If the developer wants, it should be particular to its own 
code. Nevertheless, I still think this small modification on the loader 
should be implemented, as it opens the door to developers develop their own 
loader that can, if they need, get a specific template from a specific dir.

Regards,
Jorge



On Friday, May 31, 2013 2:45:27 PM UTC+2, Marc Tamlyn wrote:
>
> I'm really quite unconvinced by your arguments I'm afraid. In particular 
> one of the strengths of the normal template loader is that it does *not* 
> start locally - so I can use a pluggable application and override templates 
> very easily. The normal pattern for pluggable applications is that they 
> namespace all of their templates (say for an application called `myapp` the 
> template paths would be `myapp/secondary_header.html`). This means these 
> kind of name collisions are far less likely to occur - only if the app name 
> matches. If that happens then you have all manner of issues - you'd need to 
> be incredibly careful about relative imports everywhere etc to be able to 
> rename the top level name of a package and not break it - I think having to 
> update template paths as well is not the biggest issue here.
>
> In respect to URLs, URL namespacing is already a feature in Django to 
> allow the same URLs to be deployed multiple times and to namespace the 
> reverses. (
> https://docs.djangoproject.com/en/dev/topics/http/urls/#url-namespaces)
>
> On Thursday, 30 May 2013 13:13:55 UTC+1, Jorge C. Leitão wrote:
>>
>>  I must say that reading this into Russel (and other)'s replies is very 
>>> far fetched (language as well as content), uncalled for, and seems to 
>>> expose an assumption that the people here prefer the easy path of swatting 
>>> suggestions with a quick "nay".  I 'd say the opposite is much closer to 
>>> the truth.
>>
>>
>> I'm sorry, this was a joke; not something to be taken seriously. On the 
>> contrary, I assume people here are actually very well intended, and that's 
>> why I feel confident enough to put the joke here. Anyway, I'm not here to 
>> offend anyone or anything, so, I'm sorry for the sentence, I would retract 
>> it if I could.
>>
>> I generally accept your criticisms and agree with your point of view that 
>> it is not so strait-forward in django's spirit. Let's them try to reach 
>> some specific points that would be worth implementing, even without the 
>> whole idea I'm proposing.
>>
>> It seems you could achieve what you're after (at least for templates) 
>>> with a custom template loader. A variation on the app directories loader 
>>> [0] can impose a different search order than the default one that simply 
>>> follows the order in INSTALLED_APPS. Am I wrong?
>>
>>
>> YES!! Definitely this. However, the function get_template does not accept 
>> a directory to search. It is not possible to pass the directory of the 
>> search to the get_template: django.templates.loader.get_template only 
>> accepts one argument, i.e. you cannot tell which directory you want the 
>> template to be searched from the call. The two of the 10 lines of code I 
>> was referring to was change in that function:
>>
>> from
>> get_template(template_name)
>> ...
>> to
>> get_template(template_name*, dirs=None*)
>> ...
>> template, origin = find_template(template_name*, dirs*)
>>
>> this is compatible with the find_template function, which also has two 
>> arguments:
>>
>> find_template(name, dirs=None)
>>
>> For consistency, this should be implemented on the other two functions:
>>
>> render_to_string
>> select_template
>>
>> which should also receive dirs as arguments, with default=None and 
>> respective passing in the their calls of get_template.
>>
>> This allows the developer to actually *choose* which template it is 
>> refering to when it uses render("secondary_header.html"), and works with 
>> the 
>>
>> app_directories.Loader because it already has a optional argument 
>> "template_dirs".
>>
>> So, yes, this is actually one of my proposals... change these 4 or 5 
>> lines.
>>
>> Cheers,
>> Jorge
>>
>>
>>
>> On Thursday, May 30, 2013 12:50:34 PM UTC+2, Yishai Beeri wrote:
>>>
>>> Hello Jorge,
>>>
>>> On Thu, 30 May 2013 12:42:20 +0300, Jorge C. Leitão <
>>> jorgeca...@gmail.com> wrote:
>>>
>>> Hi Russell Keith-Magee.
>>>
>>> 

Re: Testing generated SQL under different backends

[Marc Tamlyn]

In terms of non-core backends (and I include Oracle in this) - my preferred 
option would not be to include them all in core Django, but to collate them 
in and manage them under the Django project - i.e. 
github.com/django/django-mssql. These are "looked after" by the Django 
project to ensure that there is one place to find that code base, but are 
externally maintained. This is effectively similar to the way that 
localflavor is progressing, although we should wait and see how that plays 
out first - there are some teething issues there at the moment.

In particular I'd be quite strongly against the main Django codebase 
shipping with potentially broken drivers - this will reflect very badly on 
the project for new users.

On Wednesday, 29 May 2013 08:42:19 UTC+1, Anssi Kääriäinen wrote:
>
> On 28 touko, 17:46, Michael Manfre  wrote: 
> > > I don't know how to make this actually work so that Django's test 
> > > runner finds the correct test class from the backend's test module 
> > > instead of the generic backends test class. Optimally a backend could 
> > > subclass any test class to specialize or skip tests. 
> > 
> > django.tests.runtests.setup already has special handling for adding gis 
> > test apps. Something similar could be used to for database backends. Not 
> > 100% sure what would be a good way of flagging core tests as needing to 
> be 
> > skipped or wrapped to expect failure, but we might be able to avoid that 
> > issue entirely with a combination of database feature flags and moving 
> the 
> > vendor specific problem cases to the respective backends. 
>
> An example of what I think would be the ultimate solution: 
>
> class TestWeirdJoinConditions(TestCase): 
> def test_something(self): 
> ... 
> If it turns out MySQL is having problems with this test, you could 
> then have a class somewhere in the backend: 
>
> from queries.tests import TestWeirdJoinConditions 
>
> class TestWeirdJoinConditionsMySQL(TestWeirdJoinConditions): 
>@skip 
>def test_something(self): 
>... 
>
> Assuming this could technically work (that is, there is a way to use 
> only the MySQL version of the class) it should be possible for a 
> backend to alter everything about the test class if need be, and the 
> test class doesn't need to know anything about the backend. So, no 
> dependency from queries tests to possible 3rd party backends. OTOH 
> this adds a dependency from backend to arbitrary test classes... 
>
> Just moving some tests to backends is a possibility. The problem is 
> that if some third party backend has problems with a test we will need 
> to move that test to backends without having anything in core 
> requiring that. This creates a weird dependency between Django and 3rd 
> party backends. Of course, we have that dependency already. For 
> example some tests use IntegerField(primary_key=True) instead of 
> AutoField, the reason is that SQL Server doesn't like manually setting 
> AutoField values. 
>
> > > For 3rd party backend support in general: My opinion is that we should 
> > > include them all in core. The rules would be: 
> > 
> > By "them", do you mean 3rd party backend tests or the entire 3rd party 
> > backend? 
>
> Whole backend. This could work if the idea is that core doesn't 
> guarantee that all backends are working all the time. The only 
> guarantee is that core tries to find a maintainers for backends. If 
> nobody is found, then the backend simply doesn't work. 
>
> This is the way device driver development is done in Linux. Basically, 
> if nobody cares enough to fix a broken driver, then it isn't a 
> problem. This turns out to work really, really well. 
>
> Interestingly something like this has already happened to Oracle GIS. 
> It was broken for a long time in 1.4 release (you couldn't save 
> anything IIRC). Then Jani Tiainen needed the GIS functionality, wrote 
> a patch and now GIS is in mostly working state. It still doesn't pass 
> the GIS tests, but that doesn't seem to be a problem in practice. 
>
> Based on how things work in Linux I think similar arrangement should 
> work in Django, too. The crucial point is that adding a new backend to 
> Django's code base must not add any significant amount of work to core 
> committers. If the idea is to add backends to Django's code base and 
> have any sort of core-committer guarantee that all backends work all 
> the time (that is, you need to run tests against all backends before 
> commit) then there is no way more backends in Django could work. 
>
> I don't know what others in core think about this... 
>
> Of course, the most important question is if maintaining backends in 
> 3rd party projects is painful enough to justify this discussion. I 
> don't know. If there isn't a problem in practice, then lets not solve 
> imaginary problems... 
>
>  - Anssi 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group

Re: Proposal/Discussion: url and template search using the directory structure of apps within django sites

[Marc Tamlyn]

I'm really quite unconvinced by your arguments I'm afraid. In particular 
one of the strengths of the normal template loader is that it does *not* 
start locally - so I can use a pluggable application and override templates 
very easily. The normal pattern for pluggable applications is that they 
namespace all of their templates (say for an application called `myapp` the 
template paths would be `myapp/secondary_header.html`). This means these 
kind of name collisions are far less likely to occur - only if the app name 
matches. If that happens then you have all manner of issues - you'd need to 
be incredibly careful about relative imports everywhere etc to be able to 
rename the top level name of a package and not break it - I think having to 
update template paths as well is not the biggest issue here.

In respect to URLs, URL namespacing is already a feature in Django to allow 
the same URLs to be deployed multiple times and to namespace the reverses. 
(https://docs.djangoproject.com/en/dev/topics/http/urls/#url-namespaces)

On Thursday, 30 May 2013 13:13:55 UTC+1, Jorge C. Leitão wrote:
>
>  I must say that reading this into Russel (and other)'s replies is very 
>> far fetched (language as well as content), uncalled for, and seems to 
>> expose an assumption that the people here prefer the easy path of swatting 
>> suggestions with a quick "nay".  I 'd say the opposite is much closer to 
>> the truth.
>
>
> I'm sorry, this was a joke; not something to be taken seriously. On the 
> contrary, I assume people here are actually very well intended, and that's 
> why I feel confident enough to put the joke here. Anyway, I'm not here to 
> offend anyone or anything, so, I'm sorry for the sentence, I would retract 
> it if I could.
>
> I generally accept your criticisms and agree with your point of view that 
> it is not so strait-forward in django's spirit. Let's them try to reach 
> some specific points that would be worth implementing, even without the 
> whole idea I'm proposing.
>
> It seems you could achieve what you're after (at least for templates) with 
>> a custom template loader. A variation on the app directories loader [0] can 
>> impose a different search order than the default one that simply follows 
>> the order in INSTALLED_APPS. Am I wrong?
>
>
> YES!! Definitely this. However, the function get_template does not accept 
> a directory to search. It is not possible to pass the directory of the 
> search to the get_template: django.templates.loader.get_template only 
> accepts one argument, i.e. you cannot tell which directory you want the 
> template to be searched from the call. The two of the 10 lines of code I 
> was referring to was change in that function:
>
> from
> get_template(template_name)
> ...
> to
> get_template(template_name*, dirs=None*)
> ...
> template, origin = find_template(template_name*, dirs*)
>
> this is compatible with the find_template function, which also has two 
> arguments:
>
> find_template(name, dirs=None)
>
> For consistency, this should be implemented on the other two functions:
>
> render_to_string
> select_template
>
> which should also receive dirs as arguments, with default=None and 
> respective passing in the their calls of get_template.
>
> This allows the developer to actually *choose* which template it is 
> refering to when it uses render("secondary_header.html"), and works with 
> the 
>
> app_directories.Loader because it already has a optional argument 
> "template_dirs".
>
> So, yes, this is actually one of my proposals... change these 4 or 5 lines.
>
> Cheers,
> Jorge
>
>
>
> On Thursday, May 30, 2013 12:50:34 PM UTC+2, Yishai Beeri wrote:
>>
>> Hello Jorge,
>>
>> On Thu, 30 May 2013 12:42:20 +0300, Jorge C. Leitão  
>> wrote:
>>
>> Hi Russell Keith-Magee.
>>
>> Thanks for your criticisms. Let's see if I can answer some of them. 
>>  
>>
>>> I'd turn this question back onto you -- why *should* the search start 
>>> locally? What collisions are you seeing between apps that makes a global 
>>> search impractical?
>>
>>
>> Very good question! I think this question is related with your point
>>
>> Well, no - I can see how you've managed to construct an example where 
>>> local search might be useful (although I still don't concede that it 
>>> couldn't be answered with global lookups). However, what I need to be 
>>> convinced is to see an actual use case where this is an issue -- not a 
>>> synthetic example, but a real world problem.
>>
>>  
>> I.e. in my poor example I was not able to explain myself on the reason 
>> for using local search.
>>
>> I'm sorry for the extension, but the reason is not so obvious.
>>
>> Consider the case where in a an app you have an header, with the website 
>> logo and stuff, which you want to be consistent within your website, and 
>> you have smaller secondary header, (secondary_header.html) which you want 
>> to populate according to a specific app rendering the template.
>>
>>
>> The "a specific app rendering 

Re: Migrations, commands and syncdb

[Marc Tamlyn]

I have generally not kept my migrations so they always work from scratch 
for similar reasons Luke said - Data migrations are often dependent on the 
data in the site - this is particularly relevant to content-driven (CMSy) 
sites where I may need to move a load of page about as a data migration at 
some point in the history, but this data will not exist for the tests. 
These "throwaway" data migrations are IMO pretty common. As a consequence 
(and given the speed requirements) I've generally never run a site without 
SOUTH_TESTS_MIGRATE set.

Also, when using an older library which has a long migration history 
included, when we start a new site using it we generally use `syncdb --all` 
so as not to worry about the history. Will this command still be supported 
by the new `migrate` command?

I can see the argument for making sure the migrations work, and they may be 
necessary for migrations containing custom SQL the site depends on, but I 
would suggest that many developers effectively treat migrations as things 
that can be thrown away once they've been run against all the relevant 
databases. Perhaps the rebase command needs to consider this - how can we 
throw away old migrations (especially data migrations).

(PS: +1 to option 3 in principle)

On Friday, 31 May 2013 12:44:20 UTC+1, Anssi Kääriäinen wrote:
>
>  On 05/31/2013 02:08 PM, Andrew Godwin wrote:
>  
>   One problem with this is that you have to be careful to write migrations
>>  that will always work from scratch. This is best practice, but I have on
>> occasion used data migrations that were for specific problems, and may
>> have depended on specific data in the database. I've also used them for
>> populating parts of the database at specific upgrade points. Usually I
>> wouldn't want this data to be installed for my tests - I want my tests
>> to explicitly do the setup they need, and not be complicated by data
>> from data migrations.
>>
>> For these reasons, and because of the slowness of running all the
>> migrations, there are some projects where I have a settings file for
>> running the tests that excludes South from INSTALLED_APPS, which solves
>> the problem nicely.
>>
>> The slowness problem can be fixed by squashing, but it seems a shame to
>> require that simply for the sake of running tests. And it doesn't solve
>> the other problems. I would like to be able to turn off migrations for
>> tests, and just skip to the final schema.
>>
>
>   Yes, this has traditionally been a problem, and it's difficult to say 
> what to do here.
>
>  I personally have the opinion that stuff you add in data migrations 
> _should_ be part of the tests - after all, it's usually crucial data that 
> the site won't work without, and you're never going to test an installation 
> without them.
>
>  However, I understand people don't like doing this. Problem is, the 
> alternative is to reserve the old syncdb method for running tests, and then 
> to introduce a new setting (which in South is SOUTH_TESTS_MIGRATE) that 
> lets users turn it on and off, which seems like a bad road to go down.
>
>  I'd rather keep it so you must run migrations for tests - as I believe 
> we should be making people do this as a general case - and if you want, 
> having the data migration operations have an option you can pass where they 
> no-op during tests, something like:
>
>  operations = [
> Python(
> """
> for author in Author.objects.all():
> author.fix()
> """,
> skip_during_tests=True,
> ),
> ]
>   
>
> If test database is set up by syncdb and you have some SQL that isn't 
> model migrations, then you will likely need two sets of migrations. One 
> that applies to production database, and one that applies to fresh database 
> from syncdb. You will need to constantly keep the "applies to syncdb 
> database" migrations updated, and there is no guarantee that you will 
> actually arrive to similar state than what you have in production database 
> (in other words, migrations should be part of your tests like Andrew said).
>
> For example the SQL could be trigger or view creation SQL. I know, not the 
> most typical setup. But there are enough users who want to use traditional 
> SQL for schema alterations that this use case should be part of design 
> goals.
>
> Dump + load of production schema is surprisingly good way to attack this 
> problem - you are testing against exact copy of your production schema, and 
> only "in development" migrations need to be applied. This is not a good 
> default, but very useful if you happen to need it. My understanding is that 
> I can actually do this somewhat easily with the planned design.
>
>  - Anssi
>  

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django

Re: Migrations, commands and syncdb

[Anssi Kääriäinen]

On 05/31/2013 02:08 PM, Andrew Godwin wrote:


One problem with this is that you have to be careful to write
migrations
that will always work from scratch. This is best practice, but I
have on
occasion used data migrations that were for specific problems, and may
have depended on specific data in the database. I've also used
them for
populating parts of the database at specific upgrade points. Usually I
wouldn't want this data to be installed for my tests - I want my tests
to explicitly do the setup they need, and not be complicated by data
from data migrations.

For these reasons, and because of the slowness of running all the
migrations, there are some projects where I have a settings file for
running the tests that excludes South from INSTALLED_APPS, which
solves
the problem nicely.

The slowness problem can be fixed by squashing, but it seems a
shame to
require that simply for the sake of running tests. And it doesn't
solve
the other problems. I would like to be able to turn off migrations for
tests, and just skip to the final schema.


 Yes, this has traditionally been a problem, and it's difficult to say 
what to do here.


I personally have the opinion that stuff you add in data migrations 
_should_ be part of the tests - after all, it's usually crucial data 
that the site won't work without, and you're never going to test an 
installation without them.


However, I understand people don't like doing this. Problem is, the 
alternative is to reserve the old syncdb method for running tests, and 
then to introduce a new setting (which in South 
is SOUTH_TESTS_MIGRATE) that lets users turn it on and off, which 
seems like a bad road to go down.


I'd rather keep it so you must run migrations for tests - as I believe 
we should be making people do this as a general case - and if you 
want, having the data migration operations have an option you can pass 
where they no-op during tests, something like:


operations = [
Python(
"""
for author in Author.objects.all():
author.fix()
""",
skip_during_tests=True,
),
]


If test database is set up by syncdb and you have some SQL that isn't 
model migrations, then you will likely need two sets of migrations. One 
that applies to production database, and one that applies to fresh 
database from syncdb. You will need to constantly keep the "applies to 
syncdb database" migrations updated, and there is no guarantee that you 
will actually arrive to similar state than what you have in production 
database (in other words, migrations should be part of your tests like 
Andrew said).


For example the SQL could be trigger or view creation SQL. I know, not 
the most typical setup. But there are enough users who want to use 
traditional SQL for schema alterations that this use case should be part 
of design goals.


Dump + load of production schema is surprisingly good way to attack this 
problem - you are testing against exact copy of your production schema, 
and only "in development" migrations need to be applied. This is not a 
good default, but very useful if you happen to need it. My understanding 
is that I can actually do this somewhat easily with the planned design.


 - Anssi

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Migrations, commands and syncdb

[Andrew Godwin]

>
> One problem with this is that you have to be careful to write migrations
> that will always work from scratch. This is best practice, but I have on
> occasion used data migrations that were for specific problems, and may
> have depended on specific data in the database. I've also used them for
> populating parts of the database at specific upgrade points. Usually I
> wouldn't want this data to be installed for my tests - I want my tests
> to explicitly do the setup they need, and not be complicated by data
> from data migrations.
>
> For these reasons, and because of the slowness of running all the
> migrations, there are some projects where I have a settings file for
> running the tests that excludes South from INSTALLED_APPS, which solves
> the problem nicely.
>
> The slowness problem can be fixed by squashing, but it seems a shame to
> require that simply for the sake of running tests. And it doesn't solve
> the other problems. I would like to be able to turn off migrations for
> tests, and just skip to the final schema.
>

 Yes, this has traditionally been a problem, and it's difficult to say what
to do here.

I personally have the opinion that stuff you add in data migrations
_should_ be part of the tests - after all, it's usually crucial data that
the site won't work without, and you're never going to test an installation
without them.

However, I understand people don't like doing this. Problem is, the
alternative is to reserve the old syncdb method for running tests, and then
to introduce a new setting (which in South is SOUTH_TESTS_MIGRATE) that
lets users turn it on and off, which seems like a bad road to go down.

I'd rather keep it so you must run migrations for tests - as I believe we
should be making people do this as a general case - and if you want, having
the data migration operations have an option you can pass where they no-op
during tests, something like:

operations = [
Python(
"""
for author in Author.objects.all():
author.fix()
""",
skip_during_tests=True,
),
]

Andrew

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Migrations, commands and syncdb

[Luke Plant]

On 30/05/13 20:55, Andrew Godwin wrote:

> The way South works, and the way this will work (for new installs as
> well as tests) is that if migrations are present for an app, it will
> always create new databases by running through them from the very first
> migration to the latest one.

One problem with this is that you have to be careful to write migrations
that will always work from scratch. This is best practice, but I have on
occasion used data migrations that were for specific problems, and may
have depended on specific data in the database. I've also used them for
populating parts of the database at specific upgrade points. Usually I
wouldn't want this data to be installed for my tests - I want my tests
to explicitly do the setup they need, and not be complicated by data
from data migrations.

For these reasons, and because of the slowness of running all the
migrations, there are some projects where I have a settings file for
running the tests that excludes South from INSTALLED_APPS, which solves
the problem nicely.

The slowness problem can be fixed by squashing, but it seems a shame to
require that simply for the sake of running tests. And it doesn't solve
the other problems. I would like to be able to turn off migrations for
tests, and just skip to the final schema.

Luke

-- 
"I washed a sock. Then I put it in the dryer. When I took it out,
it was gone."  (Steven Wright)

Luke Plant || http://lukeplant.me.uk/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Digest for django-developers@googlegroups.com - 17 Messages in 3 Topics

[Andrew Godwin]

That's the nasty edge case - currently it just fails, in the new system
it'll generate three migrations (add A with no FK; add B with FK; add FK to
A). As for the previous post, the syncdb-like step will always run before
the migration step; thus, dependencies are allowed from a migrated to a
non-migrated app, but not vice-versa.

Andrew


On Fri, May 31, 2013 at 8:22 AM, Anssi Kääriäinen
wrote:

> On 05/31/2013 09:55 AM, Mantas wrote:
>
>> Quoting 
>> django-developers@**googlegroups.com(2013-05-31
>>  07:39:24)
>>
>>> From: Andrew Godwin 
>>> Date: May 30 07:03PM +0100
>>> Url: http://groups.google.com/**group/django-developers/msg/**
>>> 137ab14f361fd052
>>>
>>> 2. Leave syncdb as it is, like South does, and have everything happen
>>> through a "migrate" command. Leads to weird interactions where each
>>> command
>>> knows about the other, and must be run in a certain order, but which
>>> isn't
>>> immediately obvious.
>>>
>> Order in which syncdb and migrate should be run also depends on
>> dependencies between apps.
>>
>> For example, if you have two apps, A and B:
>>
>>  # A/models.py
>>
>>  class A(models.Model):
>>  name = models.CharField(max_length=**128)
>>
>>
>>  # B/models.py
>>
>>  class B(models.Model):
>>  name = models.CharField(max_length=**128)
>>  aref = models.ForeignKey(A)
>>
>> Then, first you must run syncdb/migrations for A app, and *only* then
>> for app B.
>>
>> So to fully fix migrations, Django should be aware about dependencies
>> between apps, which is very important not only for migrations.
>>
>
> Good point. How does this work with migrations:
>
> app_a:
>   class A(models.Model):
> b = models.ForeignKey("app_b.b")
>
> app_b:
>   class B(models.Model):
> a = models.ForeignKey("app_a.a")
>
> In this case you can't have self-contained migrations for app_a and app_b
> separately if you want to run first app_a's migrations, then app_b's (or
> vice versa).
>
>  - Anssi
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to 
> django-developers+unsubscribe@**googlegroups.com
> .
> To post to this group, send email to 
> django-developers@**googlegroups.com
> .
> Visit this group at http://groups.google.com/**
> group/django-developers?hl=en
> .
> For more options, visit 
> https://groups.google.com/**groups/opt_out
> .
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Digest for django-developers@googlegroups.com - 17 Messages in 3 Topics

[Anssi Kääriäinen]

On 05/31/2013 09:55 AM, Mantas wrote:

Quoting django-developers@googlegroups.com (2013-05-31 07:39:24)

From: Andrew Godwin 
Date: May 30 07:03PM +0100
Url: http://groups.google.com/group/django-developers/msg/137ab14f361fd052

2. Leave syncdb as it is, like South does, and have everything happen
through a "migrate" command. Leads to weird interactions where each command
knows about the other, and must be run in a certain order, but which isn't
immediately obvious.

Order in which syncdb and migrate should be run also depends on
dependencies between apps.

For example, if you have two apps, A and B:

 # A/models.py

 class A(models.Model):
 name = models.CharField(max_length=128)


 # B/models.py

 class B(models.Model):
 name = models.CharField(max_length=128)
 aref = models.ForeignKey(A)

Then, first you must run syncdb/migrations for A app, and *only* then
for app B.

So to fully fix migrations, Django should be aware about dependencies
between apps, which is very important not only for migrations.


Good point. How does this work with migrations:

app_a:
  class A(models.Model):
b = models.ForeignKey("app_b.b")

app_b:
  class B(models.Model):
a = models.ForeignKey("app_a.a")

In this case you can't have self-contained migrations for app_a and 
app_b separately if you want to run first app_a's migrations, then 
app_b's (or vice versa).


 - Anssi

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.