Test reordering and TransactionTestCase cleanup

[Andreas Pelme]

I am trying to run my Django test suite with an alternative test runner 
(py.test), and found some issues with test isolation.

TransactionTestCase does currently not clean up after itself (i.e. flush the 
database), but instead assumes that the next test will flush the database. It 
is generally a good idea to restore the state after the test run, and let other 
tests start out with a known state.

Djangos default testrunner reorders the test suite to run all TestCase tests 
before TransactionTestCases, which avoids this problem. I cannot find this 
reordering documented anywhere, or even commented anywhere in the 
implementation (test/simple.py: reorder_suite and 
DjangoTestSuiteRunner.build_suite).

I propose to move the flush command out of _fixture_setup to _post_teardown in 
TransactionTestCase. This makes it possible for arbitrary execution order of 
the test suite. Performance could potentially be an issue, since it is "lazy" 
today and flushes the database just right before it is needed. That would 
however only affect the last test in the run (i.e. it will truncate the tables, 
and then the database itself will be teared down).

Are there any objections or anything that might break with such a change? Are 
there any specific reasons it works the way it does?

I will be happy to create a ticket and patch for this, I would get to input in 
case I miss something obvious.

Cheers
Andreas

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Test reordering and TransactionTestCase cleanup

[Andreas Pelme]

On Thursday 3 May 2012 at 21:17, Karen Tracey wrote:
> On Thu, May 3, 2012 at 12:29 PM, Andreas Pelme <andr...@pelme.se 
> (mailto:andr...@pelme.se)> wrote:
> > 
> > Djangos default testrunner reorders the test suite to run all TestCase 
> > tests before TransactionTestCases, which avoids this problem. I cannot find 
> > this reordering documented anywhere,
> 
> It is documented:
> https://docs.djangoproject.com/en/1.4/topics/testing/#django.test.TransactionTestCase
> 
> Mentioned in both the 3rd paragraph and the note.
Thanks -- I'm not sure how I missed that. I'll make sure my patch updates the 
docs too.

Andreas


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Test reordering and TransactionTestCase cleanup

[Andreas Pelme]

On Thursday 3 May 2012 at 22:14, Ramiro Morales wrote:
> On Thu, May 3, 2012 at 1:29 PM, Andreas Pelme <andr...@pelme.se 
> (mailto:andr...@pelme.se)> wrote:
> > I am trying to run my Django test suite with an alternative test runner 
> > (py.test), and found some issues with test isolation.
> > 
> > TransactionTestCase does currently not clean up after itself (i.e. flush 
> > the database), but instead assumes that the next test will flush the 
> > database.
> > [...]
> > 
> > Djangos default testrunner reorders the test suite to run all TestCase 
> > tests before TransactionTestCases, which avoids this problem.
> 
> Just a quick related note
> 
> There is a [1]proposal to extend this reordering to be:
> 
> * TestCases
> * doctests
> * TransactionTestCases
> 
> So doctests aren't affected either.
> 
> I intend to commit a fix for it soon.
> 
> -- 
> Ramiro Morales
> 
> 1. https://code.djangoproject.com/ticket/12408

This won't be an issue if TransactionTestCase resets the database after itself.

Andreas 


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Test reordering and TransactionTestCase cleanup

[Andreas Pelme]

On Thursday 3 May 2012 at 19:23, Anssi Kääriäinen wrote:
> On May 3, 7:29 pm, Andreas Pelme <andr...@pelme.se (http://pelme.se)> wrote:
> > I am trying to run my Django test suite with an alternative test runner 
> > (py.test), and found some issues with test isolation.
> >  
> > TransactionTestCase does currently not clean up after itself (i.e. flush 
> > the database), but instead assumes that the next test will flush the 
> > database. It is generally a good idea to restore the state after the test 
> > run, and let other tests start out with a known state.
> >  
> > Djangos default testrunner reorders the test suite to run all TestCase 
> > tests before TransactionTestCases, which avoids this problem. I cannot find 
> > this reordering documented anywhere, or even commented anywhere in the 
> > implementation (test/simple.py: reorder_suite and 
> > DjangoTestSuiteRunner.build_suite).
> >  
> > I propose to move the flush command out of _fixture_setup to _post_teardown 
> > in TransactionTestCase. This makes it possible for arbitrary execution 
> > order of the test suite. Performance could potentially be an issue, since 
> > it is "lazy" today and flushes the database just right before it is needed. 
> > That would however only affect the last test in the run (i.e. it will 
> > truncate the tables, and then the database itself will be teared down).
> >  
> > Are there any objections or anything that might break with such a change? 
> > Are there any specific reasons it works the way it does?
>  
> I don't know if anything will break...
>  
> I think this is a good idea. In addition to the above issues this
> would allow the test ran tell the flushing which tables are dirty,
> which not. Currently the transaction_regress for example uses just a
> single table. But after each test, all tables will be truncated, and
> the contenttypes and permissions will be reloaded. I bet we could
> speed up the testing _a lot_ if the cleaning up could be made more
> intelligent. Making it robust is somewhat hard but should not be
> impossible. Of course, the "StateTrackingTestCase" should not be done
> in the same patch at all... just a possibility for future work.


There are indeed some work that can be done to speed up the test running. :)

Here's the ticket:
  
https://code.djangoproject.com/ticket/18271  

… and an initial patch:

https://github.com/django/django/pull/45

Is there any sane way to test this except to run the test suite itself?

Andreas


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Test reordering and TransactionTestCase cleanup

[Andreas Pelme]

On Friday 4 May 2012 at 13:30, Karen Tracey wrote:
> On Fri, May 4, 2012 at 4:46 AM, Andreas Pelme <andr...@pelme.se 
> (mailto:andr...@pelme.se)> wrote:
> > 
> > That's a good question. Anyone who wrote to original 
> > TransactionTestCase/reordering implementation that wants to chime in? :-)
> 
> I worked on the test speedups that introduced TransactionTestCase and
> I added the re-ordering, but the behavior of database flush being done
> at the beginning of a test existed before that work. The conversation
> that led to addition of the reordering is here:
> 
> http://groups.google.com/group/django-developers/browse_thread/thread/1e4f4c840b180895/
> 
> doctests (which we have no more?) play a prominent role in that
> discussion. While we have gotten rid of doctests in Django's own
> suite, we still support apps which may use doctests in their code, so
> anything we do to change when the DB is cleared needs to take that
> into account. On the table then was the idea of adding cleanup after
> doctests, possibly that would need to be re-considered if you want to
> move the database clearing to the end of everything rather than the
> beginning of TransactionTestCase.


#12408 is related here, which is caused by TransactionTestCase's not cleaning 
up after themselves. Rather than changing the test order to TestCase -> 
doctests -> TransactionTestCase I think the solution below is better.

Karens proposed a fix for this (#2) in the above thread [1], it basically says:

1) Make TransactionTestCase clean up after itself
2) Order the test suite to make sure doctests are always run *last*

That would run all unittest-style test cases first (the order of TestCase / 
TransactionTestCase will not be important), and last, all doctests.

A test suite that have doctests that *expects* state to be left from a 
TransactionTestCase before it would be broken by this change. In my opinion, a 
test suite with such expectations is already very broken and needs to be fixed 
any way, and should not be considered a blocker to this change (in that case, 
#12408 is a blocker too, since that effect will be the same).

I have updated my pull request to make unittest.TestCase subclasses to run 
first (i.e. doctests will run after all unit tests):
https://github.com/django/django/pull/45

Am I missing something? This seems like the most reliable way to solve this to 
me.

(I originally discovered this when I tried to run my tests with an alternative 
test runner, which does not order the tests in any particular way. I will still 
not be able to run doctests that does not clean up after themselves in 
arbitrary order, but as long as I document that's not supported with my test 
runner, I don't consider that to be a problem.)

Andreas

[1] https://groups.google.com/d/msg/django-developers/Hk9MhAsYCJU/fB4rj7F5SXEJ

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Test reordering and TransactionTestCase cleanup

[Andreas Pelme]

On Saturday 5 May 2012 at 00:39, Anssi Kääriäinen wrote:
> On May 4, 2:30 pm, Karen Tracey  wrote:
> Thanks for the link. While reading the previous threads I spotted at
> one blocker issue: the first TransactionTestCase will not start with
> zeroed database sequence values. To prevent this one would need to
> flush the DB before the first TransactionTestCase. In addition, if the
> goal is to get rid of test ordering, then one would need to flush
> before every test case which isn't good. Getting rid of the
> requirement to reset sequences between test cases would be good.
> Oracle doesn't reset the sequences reliably currently just for
> example. But I don't think we can change that for backwards
> compatibility reasons. While tests that rely on sequence values are
> IMHO broken, changing this would result in numerous broken tests for
> users upgrading Django. Notably normal TestCases do not do sequence
> resetting at all currently.
>  
> This is a blocker: why change the flushing from pre-test to post-test
> if the end result is that you can't run the tests in arbitrary order
> anyways? Or if you want to be able to run them in arbitrary order you
> will need to run at least the sequence resets pre-test anyways.
> Resetting sequences can be really expensive, so there goes any
> potential savings in speed.
>  
> Ideas?

Agreed, it is indeed backward incompatible. However, I think we can make the 
upgrade for those affected very anyways:

 * Document this change properly in the release notes and in the testing docs
 * Document that tests should not depend on hard coded primary key values (this 
is already hinted in the testing docs where TestCase/TransactionTestCase is 
described)
 * Add a reset_sequences attribute on TransactionTestCase, that can be applied 
to existing TransactionTestCase tests that already uses hard coded primary key 
value - which will reset sequences *before* the test run, just like before. 
(And those tests will also flush the database after they are run)

I have updated the pull request with the reset_sequences attribute and an 
initial draft for the release notes and documentation:
https://github.com/django/django/pull/45

1.1 introduced a far greater change to testing behavior (the TestCase with 
transaction/rollback), if that change was considered acceptable from a backward 
compatibility perspective, this change should be too. When that happened, the 
same problem appeared since TestCase does not reset sequences. (Yeah, just 
because it was done in the past does not make it OK now, but that should give 
some perspective).

I guess it boils down to where to draw the line for backward compatibility 
here. :-) I personally think it is acceptable since we can offer a very easy 
fix for the existing (already broken) tests out there.


Andreas


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Attach a technincal html response to email on server error

[Andreas Pelme]

On 10 nov 2011, at 22:03, Ric wrote:
> hi, i receive django email on 500 error.
> 
> with that kind of mail is difficult to debug, because i don't have a
> lot of informations.
> 
> it's much more simple to debug when i set DEBUG = True and i receive
> an html technical response.
> with that response i can read a lot of informations (like vars,
> request information, server settings and so on)
> 
> my idea is to attach an html file, that is the very same response you
> got on error with DEBUG = True.
> 
> this response has got no external dependencies (css and js are inline
> in the html), so it can be attached as a file in the email that is
> sended on server error.
> 
> this is very simple to archieve, and developers can open it with the
> browser and see a lot of information.



This is already possible since Django 1.3 with the logging framework and 
AdminEmailHandler:

https://docs.djangoproject.com/en/dev/topics/logging/#django.utils.log.AdminEmailHandler


-Andreas

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Model fields in localflavor

[Andreas Pelme]

[This messages is related to
http://groups.google.com/group/django-developers/browse_thread/thread/64e4d42590145347
]

On Jul 27, 11:39 pm, Malcolm Tredinnick <[EMAIL PROTECTED]> wrote:
> On Sun, 2008-07-27 at 16:39 +0200, Andreas Pelme wrote:
> > Hello
>
> > I have started writing a patch for adding Swedish (se) local flavor.
>
> > I looked at the code for other countries, and noticed that they only
> > contains form fields, not model fields.
>
> > Why is that? E.g. a postal code is very likely to be stored in a
> > database. Why not add default model fields (with appropriate
> > formfield(), validate() etc)?
>
> Well, right now, validation at the model field level isn't really
> supported. The old-style validators were only used by oldforms (and
> old-admin) and the new work that Honza Kral is doing isn't finished yet
> (although there'll probably be something done in time for the 1.0-beta.
>
> Even when that's done however, it won't require adding model fields for
> every type of localflavor. That would be overkill. Instead, it will be
> appropriate to add some validators for those particular types (and
> there'll be a fair bit of sharing with existing form validators).
>
> Realise that (a) almost everything is going to be stored in the database
> in a character field anyway, so different explicit Field subclasses
> doesn't add a lot and (b) you often aren't going to want a separate
> field column in the database for each locale. If somebody is entering an
> address, it will usually be stored in the same fields, regardless of
> whether they're entering an address in Sweden, Argentina or Australia.
>
> Regards,
> Malcolm

I finally got a ticket and patch created for the Swedish localflavor:

http://code.djangoproject.com/ticket/9289
http://code.djangoproject.com/attachment/ticket/9289/django_localflavor_se.diff

However, Swedish personal identity numbers are a little tricky, and can
be represented in a couple of different formats.

The appropriate thing would be to save the fields as their "long"
representation MMDD in the database. However, it is really
common to use them formatted as YYMMDD-, which creates some issues
with people older than 100 years. I wont go into more details about how
the numbers works, however, it would be really useful to do something
like this:

Having a model field with appropriate formfield(), that has the methods
long() and short() for outputting the identity number in the different
formats. You most likely want to store the data in the long format, and
mostly you are interested in the short format. It could also have
birth_day() that returns a datetime.date object.

My questions:
1) Would this be a good use case for introducing a model field in
localflavor(.se)?
2) If so, where should the tests live?
3) If not, with just the form field, the long(), short() and birth_day()
functions would still be very useful when dealing with these numbers.
Where should they live then? contrib.localflavor.se.utils?


Cheers
-- 
Andreas Pelme <[EMAIL PROTECTED]>
PGP/GPG key: 1024D/2F6D209F

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Swedish localflavor

[Andreas Pelme]

The ticket/patch for Swedish localflavor[1] needs review. It has been
discussed[2] on the Swedish user group and everyone seems to be happy
with the features and implementation. The patch contains tests and docs.
I have been using it on several production sites for the last couple of
months without any issues.

Is there any chance we could get this into 1.1? What is currently
keeping it from not being committed to trunk? If there is something I
can do to help getting it into trunk, please let me know.

Best regards
Andreas Pelme

[1] http://code.djangoproject.com/ticket/9289
[2]
http://groups.google.com/group/django-se/browse_thread/thread/bfa40d7637607f2c

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: App-loading reloaded - running code at startup

[Andreas Pelme]

Aymeric,

Your work is amazing and has improved Django so much. Your work on timezone
support, the transaction overhaul and now this app changes makes my and a
lot of other peoples experience with Django so much better. Thank you!

2013/12/30 Aymeric Augustin <aymeric.augus...@polytechnique.org>

> ## Solution 2.a — Require users to call django.setup() before they use the
> ORM. setup() would configure the settings and populate the app registry.
>

I think this approach is the only proper solution. We have hit multiple
bugs within the import mechanism in the ORM with models not finding related
models (#20143 for instance). To workaround this, we always explicitly call
get_models() in the web app servers, celery workers and scripts before
anything else. Whenever we forget the get_models() call, things break in a
lot of interesting ways. :-)

Being explicit with an exception when trying to use the ORM in a
non-initialised state would be very very useful in situations like this,
and would have saved us many hours of confusion.

--
Andreas Pelme

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CA%2B8bEMHrH10OLxQ-50UeRM08YDpAiRhXGq6LhTpOG7JsRp_M7w%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Re: Feature request: post/pre commit/rollback signals

[Andreas Pelme]

On 10 jan 2014, at 09:19, Jesús Espino <jespi...@gmail.com> wrote:

> Hi folks!
> 
> The propose is add post and pre signals for commit and rollback actions (as 
> sqlalchemy orm events). This allows attach some code when a transaction is 
> committed or rolled back.
> 
> I have some problem, such as send email only when a transaction is committed 
> successfully, and it's can be done with simple monkey patching the db 
> backend. But it not appear to be an isolated problem, and would be awesome if 
> these signals are included in core.
> 
> If a purpose is accepted, I can take care of making the issue + pull-request.
> 
> Greetings.
> Jesus.


Hi,

This is a tricky issue that has been discussed earlier, you will probably want 
to check out ticket #14051:

https://code.djangoproject.com/ticket/14051

—
Andreas Pelme

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/948D5F1E-F86D-44A0-9EEA-F1E9A6A4A382%40pelme.se.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [GSOC] Shifting to Py.Test and Improving the Test Suite

[Andreas Pelme]

Hi Chris,

It looks like you invoke nosetests and not py.test, therefore you do not get 
the results one would expect with py.test:

On 7 apr 2014, at 11:52, Chris Wilson  wrote:
> OK, here is one.
> 
> chris@lap-x201:~/projects/2014/webassets$ .ve/bin/tox -e py27 -- 
> tests.test_filters:TestPyScss.test_search_path

...

> py27 runtests: commands[0] | nose tests 
> tests.test_filters:TestPyScss.test_search_path

...

>  File 
> "/home/chris/projects/2014/webassets/.tox/py27/local/lib/python2.7/site-packages/nose/case.py",
>  line 197, in runTest


This is what the output from a similar test with pytest would result in: 
(Screenshot at http://pelme.se/~andreas/private/pytest_assertion.png to give 
the correct colors/formatting.)

$ py.test test_foo.py
= test session starts 
==
platform darwin -- Python 2.7.5 -- pytest-2.5.2.dev1
plugins: xdist
collected 1 items

test_foo.py F

=== FAILURES 
===
 test_something 


def test_something():
foo = {'out.css': 'something'}
>   assert foo.get('out.css') == 'h1 {\n  color: #ff;\n}\na {\n color: 
> #ff8000;\n}\n'
E   assert 'something' == 'h1 {\n  color: #ff...\n color: #ff8000;\n}\n'
E - something
E + h1 {
E +   color: #ff;
E + }
E + a {
E +  color: #ff8000;
E + }

test_foo.py:4: AssertionError
=== 1 failed in 0.01 seconds 
===



Cheers,
Andreas

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/38638CE1-A189-4954-A412-FE77908436AA%40pelme.se.
For more options, visit https://groups.google.com/d/optout.

Re: Allow deferral of ModelSignal callback invocation until after transaction commit

[Andreas Pelme]

Hi,

> On 30 apr 2015, at 18:42, Carl Meyer  wrote:
> 
> transaction-hooks is actually fairly small and understandable too. And I
> don't think it's hard to use for this situation, either; you'd just need
> to use `connection.on_commit` in your signal handler if you wanted to
> delay some action until post-commit.
> 
>> Unless it's going to be super easy to port transaction-hooks over, I
>> think it might be nice to get this in and sealed (it's not adding too
>> much more complexity). If transaction-hooks would land in core well,
>> let's do that.
> 
> I don't think landing transaction-hooks in core for 1.9 would be hard at
> all, and no-one has objected to the idea (AFAIK). I (or anyone really)
> just need to get around to it. Motivation has been low so far mostly
> because it works fine as an external add-on.


I did an initial port of django-transaction-hooks, it was pretty 
straightforward. All the hard work has already been done. :-)

Here is the PR: https://github.com/django/django/pull/4593

The patch is not yet finished (there’s a todo-list at the PR with some missing 
pieces). Let me know what you think and I’ll be happy to continue working on a 
proper patch to get it into a merge-able state.

Cheers,
Andreas

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5A9C5A84-A28C-4556-BD7F-7F1699A166AA%40pelme.se.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Allow deferral of ModelSignal callback invocation until after transaction commit

[Andreas Pelme]

> On 3 maj 2015, at 15:44, Christopher Adams  
> wrote:
> 
> So unless anyone has objections, I'm going to put my branch on hold for now. 
> If anyone still wants me to see if there's a way it can work I'm willing to 
> give it a bit more work, however I think it should probably be passed over 
> for the time being. Glad my post may have kicked off the conversation to get 
> a eventual solution in.
> 
> Next steps: is everyone agreed that attempting to merge Carl's 
> django-transaction-hooks is the way forward? Happy to help on that if you 
> guys need help. Pretty excited to get a post-commit hook into core in 1.9, no 
> matter how that ends up happening.

Christopher: Thank you very much for your efforts on this, I am also very 
excited to get some post-commit hook into Django. This has been a long standing 
issue for me personally, having run into this in different projects in 
different situations over the years. Lately I’ve been happily using 
django-transaction-hooks, but at the same time been sad that it wasn’t part of 
Django. You’re post made me jump in and try to port django-transaction-hooks.

There is some work/discussions/progress going on in the PR at 
https://github.com/django/django/pull/4593. Currently most things seems to work 
and the docs is written. The most tricky issue (related to sqlite autocommit 
semantics) has hopefully been solved.

Any review of the code/tests/docs and input on the open questions would be very 
welcome and helpful!

Cheers,
Andreas

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/10E04DB3-15D0-4BBE-9087-B0C3F00633F2%40pelme.se.
For more options, visit https://groups.google.com/d/optout.

Re: Discussion related to ticket #26822 (new migrations, --keepdb and --parallel option)

[Andreas Pelme]

Hello,

> On 5 juli 2016, at 00:22, Romain Garrigues  
> wrote:
> After some investigation, I have seen that, in case of keepdb context, in 
> django/db/backends/*/creation.py, if the cloned databases already exist, we 
> don't touch them, which leads to this new field not created in cloned ones.
> 
> I have proposed in the PR to rebuild the cloned databases, even with keepdb 
> option, to be sure that we always have the cloned databases with the latest 
> migration state.
> 
> The problem with this method is that it will increase test database 
> initialization time, as we will now systematically copy all cloned databases, 
> even with --keepdb option (except the default one).

We’ve been doing similar things in pytest-django (with the --reuse-db option 
and pytest-xdist) and faces similar problems. Currently you have to 
force-recreate the databases and then all processes will run migrations and it 
is very slow.

I’ve been playing around with a solution to this: In my own project I create a 
template database and call it `test_myproject_`. 
Whenever a migration file changes (an existing file or a new migration file) - 
a new database is created and all clones are recreated.

Currently this lives as a hack in my own code base, but I would like to explore 
this further and it could be a way forward. Here is my scripts that calculates 
the hash and creates the databases:
https://gist.github.com/pelme/4b3dac475cd6b1dec4fd67d25d2e7cdc
https://gist.github.com/pelme/4a3ad3a62b6244068ff63736342f9509

This method could be refined: It is not necessary to create a database with a 
new name every time migrations change. I.e. we could create a private table 
with a single row that contains the hash.

This approach hashes only the migration files directly involved in migrations, 
if you are using a 3rd-party library that’s imported, that will not trigger a 
new migration run.

As an end user of this the experience is quite nice: You only experience the 
migration/cloning slowness whenever migrations actually changed, otherwise 
everything is fast. You don’t have to remember any special command line options.

Cheers,
Andreas

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5A553E81-F6FF-4A85-ADB3-01D66AE072B9%40pelme.se.
For more options, visit https://groups.google.com/d/optout.

Pluggable secret key backend

[Andreas Pelme]

Hi,

settings.SECRET_KEY can be used for sessions, password resets, form wizards and
other cryptographic signatures via the signing APIs. Changing SECRET_KEY means
that all of those will be invalidated and the users will be affected in weird
ways without really knowing what happened. (Why am I logged out? Where did my
form submission go? Why does not this password reset link work?). This is
desirable in case the key is compromised and swift action must be taken.

There are other situations when it would be nice to change the SECRET_KEY when
this sudden invalidation is not desirable:

- When someone leaves a project/company that had access to the production
  system. After SSH keys/login credentials is revoked the developer could
  potentially have a copy of the secret key. It is essentially a backdoor with
  full remote access. It would be wise to rotate the key in those cases.

- Periodic and automatic rotations of keys to make it less useful in the
  future.

The current situation of a single SECRET_KEY makes key rotation impractical. If
you run a busy site with active users 24/7, there is never a nice time to
change the SECRET_KEY.

A solution for this problem would be sign new secrets with a new key while
still allow signatures made with the old key to be considered valid at the same
time. Changing keys and having a couple of hours of overlap where signatures
from both keys are accepted would mitigate most of the user facing problems
with invalidating sessions, password reset links and form wizard progress.

You could do this today by implementing your own session backend, message
storage backend and password reset token generator but that is cumbersome and
does not work across reusable apps that directly use low level Django signing
APIs unless they too provide hooks to provide your own secret.

I propose a pluggable project wide secret key backend
(settings.SECRET_KEY_BACKEND maybe?) with an API something like:

class SecretKeyBackend:
  def get_signing_key(self): …
  def get_verification_keys(self): ...

The default (and backward compatible) backend would then be implemented as
something like:

class SecretKeySettingsBackend:
  def get_signing_key(self):
return settings.SECRET_KEY
  def get_verification_keys(self):
return [settings.SECRET_KEY]

django.core.signing.Signer.{sign,unsign} would need to be updated to use this
backend instead of directly using settings.SECRET_KEY.

That would solve the problem project wide and work across any third party
application that uses django.core.signing directly.

This would open the door for third party secrets backend packages that
retrieves keys from systems such as Hashicorp Vault, AWS Secrets Manager,
Google Cloud KMS, Docker Secrets etc.

Having a method that retrieves the key would allow changes to secret key during
run time instead of relying on a hard coded setting would allow the key to
change without restarting the server process.

Would something like this be worth pursuing? Could it be designed in som other
way? I could not find any previous discussion/tickets on this and thought it
would be a good idea to discuss it here before opening a ticket or making an
attempt at a PR. :)

Cheers,

Andreas


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20D8A2BD-BC9C-4F02-9038-044687165DE9%40pelme.se.
For more options, visit https://groups.google.com/d/optout.

Re: Pluggable secret key backend

[Andreas Pelme]

On 10 Nov 2018, at 13:29, Adam Johnson  wrote:
> 
> Hi Andreas
> 
> I like your proposal, moving to a backend is an elegant way of solving both 
> the immediate problem and opening up the other possibilities you mentioned.

Thanks Adam, I am glad you like the proposal. :)

> I think it would also be nice to have an "out of the box" way of rotating the 
> key, without needing to implement a custom backend. Perhaps a second setting 
> OLD_SECRET_KEYS that may contain a list of old keys that are returned for 
> verification too? Or we could allow SECRET_KEY to be a list/tuple, and if so, 
> sign with the first and verify with all of them.

Agreed, I will add something like that then! :)

Cheers,
Andreas

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/A16A11DF-1439-46EF-BF0D-85C483F53608%40pelme.se.
For more options, visit https://groups.google.com/d/optout.

Re: Pluggable secret key backend

[Andreas Pelme]

> On 10 Nov 2018, at 13:00, ludovic coues  wrote:
> 
> I don't see how this would work.
> 
> For example the session. You take the user cookie. You try to validate with 
> your secret key. That doesn't work because the current key is the new one.
> 
> With a custom cookie backend, you could check if the old secret could 
> validate the cookie. But you need to change your cookie backend to handle the 
> case of multiple secret key. And all third party session backend need to 
> update.


I propose that we make the low level django.core.signing aware of multiple 
keys. Everything that is already using django.core.signing such as signed 
cookies, sessions and password reset tokens would need *not* need to change.

Cheers,
Andreas

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5ED0D5A2-AC77-4231-824C-2EDDD7F2A903%40pelme.se.
For more options, visit https://groups.google.com/d/optout.

Re: django.utils.dateparse

[Andreas Pelme]

On 4 Feb 2019, at 15:04, Giuseppe De Marco  wrote:
> 
> python3 -m timeit -s   "import sys, os; sys.path.append(os.getcwd()); from 
> datetime_heuristic_parser import datetime_heuristic_parser; 
> print(datetime_heuristic_parser('04/12/2018 09:7:4Z'))"


That command is not correct. timeit -s takes two arguments: setup code and 
benchmark code. This command just executes the setup code and does not run any 
code at all for the actual benchmark.

The correct command would be something like this (I did not run this command 
myself but you get the idea):
python3 -m timeit -s   "import sys, os; sys.path.append(os.getcwd()); from 
datetime_heuristic_parser import datetime_heuristic_parser” 
"datetime_heuristic_parser('04/12/2018 09:7:4Z’)"

Cheers,
Andreas

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0DCE461D-F01F-48C6-B5DD-CBCEFE2895FC%40pelme.se.
For more options, visit https://groups.google.com/d/optout.