Re: Development GUI

[Alec Taylor]

Hi Timothy,

Not sure where you're going with this… but sounds interesting.

[watching] -> looking forward to some sample code + whatever in the Readme
:)

On Fri, Sep 7, 2012 at 11:38 AM, Timothy Clemans
wrote:

> Hi,
>
> I'm developing a web-based development GUI for Django. See
> https://github.com/timothyclemans/djangomodelcg for a simple code
> generator for models I wrote today.
>
> Should I develop this for inclusion in Django, fork of Django, or third
> party library?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/django-developers/-/5UM2SQPw8U0J.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Improve Django markdown rendering.

[ptone]

On Thursday, September 6, 2012 10:48:30 PM UTC-4, waylan wrote:
>
> If instead, improvements are only going to be made to the markdown 
> filter, then I would suggest a complete overhaul allowing access to 
> all of markdown's features [2].
>

In fact the plan is to deprecate the markup contrib module entirely

https://code.djangoproject.com/ticket/18054 

finishing and getting landed the patch is on my todo list for the sprints 
at Djangocon.

The overhaul you suggest is a great opportunity for someone to offer an 
improved replacement version available outside of Django.

-Preston

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/7kMd5U9gv6UJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Improve Django markdown rendering.

[Donald Stufft]

https://bitbucket.org/ionata/django-bleach 


On Thursday, September 6, 2012 at 10:48 PM, Waylan Limberg wrote:

> On Thu, Sep 6, 2012 at 8:22 PM, Thomas Purchas  (mailto:tpurc...@gmail.com)> wrote:
> > I have submitted a path to improve the way Django handles html in markdown.
> > Specifically how it escapes it.
> > 
> > Ticket 6526 has all of the detail, could someone please review my patch.
> As the maintainer of the Python-Markdown library, I'll weigh in here.
> 
> I'll start by noting that safe-mode was a poorly chosen name for the
> feature (which was added before I joined the project). Really, it is a
> means of restricting raw html and IMO, should only always "escape" raw
> html. However, because of the word "safe" in the name, the feature has
> grown to support other so-called "safety" features to avoid XSS (for
> example injecting malicious JavaScript into a markdown style link)
> etc. That said, I can make no claims that it is actually "safe".
> 
> The "replace" option is supported for backward compatibility reasons
> (also why it is still the default) and "remove" is really just a
> shortcut for "replace" with the "html_replacement_text" set to a blank
> string. Not sure why anyone would want either of those options. Remove
> is too surprising to the document author and replace is a lousy (IMO)
> attempt at an explanation. While escape might not be expected by the
> author, once noticed, it not nearly as surprising.
> 
> If someone wants "safe" output from Markdown, I recommend using a
> library specific to that purpose like bleach [1]. In fact, it would
> make more sense to me to create a separate "clean" filter (perhaps
> called "clean_html"?) which calls `bleach.clean()` and provide it as a
> filter that can be used with any of the markup languages offered by
> Django.
> 
> If instead, improvements are only going to be made to the markdown
> filter, then I would suggest a complete overhaul allowing access to
> all of markdown's features [2]. For example, markdown outputs xhtml by
> default. Some may want html - which the markdown lib supports - but
> the Django filter does not.
> 
> To me, the markdown filter in its current state is completely useless.
> I have always had to re-implement my own, more powerful solution.
> 
> My recommendation is to do one of the following (in order of preference):
> 
> 1) Remove contrib.markup (per Django's depreciation policy) and leave
> it to third party apps to support.
> 2) Completely refactor the markdown filter to support all of the
> markdown library's features except for "safe_mode" and add a new
> "clean" filter which can wrap any markup filter.
> 3) Add a new "clean" filter and simply drop support for Markdown's
> safe_mode - leaving the rest as is.
> 
> I do not recommend the approach of the current patch. It leaves a bad
> taste in my mouth. Also note that I do not recommend supporting
> Markdown's "safe_mode" in any form. Of course, the Django team will
> need to make whatever decision will better serve the community - not
> me.
> 
> [1]: https://github.com/jsocol/bleach
> [2]: http://packages.python.org/Markdown/reference.html#markdown
> 
> -- 
> 
> \X/ /-\ `/ |_ /-\ |\|
> Waylan Limberg
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com 
> (mailto:django-developers@googlegroups.com).
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com 
> (mailto:django-developers+unsubscr...@googlegroups.com).
> For more options, visit this group at 
> http://groups.google.com/group/django-developers?hl=en.
> 
> 


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Development GUI

[ptone]

This falls pretty squarely in the category of 3rd party project in the 
Django ecosphere.

While it is a tool oriented for Django, it would not be part of what Django 
itself does as a web framework.

-Preston


On Thursday, September 6, 2012 9:38:24 PM UTC-4, Timothy Clemans wrote:
>
> Hi,
>
> I'm developing a web-based development GUI for Django. See 
> https://github.com/timothyclemans/djangomodelcg for a simple code 
> generator for models I wrote today.
>
> Should I develop this for inclusion in Django, fork of Django, or third 
> party library?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/bCc1FF-CBHQJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Improve Django markdown rendering.

[Thomas Purchas]

I have submitted a path to improve the way Django handles html in 
markdown. Specifically how it escapes it.

Ticket 6526  has all of the 
detail, could someone please review my patch.

Thanks in advance. 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/pVIPk8QvRusJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Backporting some Python 3 support features to 1.4.x?

[Luke Plant]

On 06/09/12 19:09, Aymeric Augustin wrote:

> Now we have two alternatives:
> 1) tell them to roll their own compat or wait until they can target 1.5
> + 1.6 — quite a downer,
> 2) add some forwards compatibility tools to 1.4.
> 
> We generally don't add features to minor releases, but there's a
> precedent (csrf_noop), and the patch is as harmless as they come (see
> attachment).
> 
> What do you think?

This was something I was going to bring up myself, since it is a major
obstacle to re-usable django apps actually providing Python 3
compatibility. I'm +1 on adding forwards compatibility to 1.4.X,
assuming a safe patch.

Luke

-- 
"In your presence there is fullness of joy; at your right hand are
pleasures forevermore" Psalm 16:11

Luke Plant || http://lukeplant.me.uk/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Testing multidb with TEST_MIRROR

[Anssi Kääriäinen]

On 7 syys, 01:10, Jeremy Dunck  wrote:
> Well that is pretty damn awesome.  It also doesn't fix my problem.
> Let's ignore my problem for now, though.  I'd like to look at the
> diffs.  I doubt this is the real 
> diff:https://github.com/akaariai/django/compare/master...fast_tests_merged
>
> What's the base branch for the fast_tests_merged comparison?

I updated the master branch of my fork, so the above link should now
contain the correct comparison. My master branch was only about 500
commits stale... :)

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Testing multidb with TEST_MIRROR

[Anssi Kääriäinen]

On 7 syys, 00:35, Andrew Godwin  wrote:
> That's an incredible speedup - I've had a quick look over the patch, and it
> looks to be doing all the right things, so I'd definitely be behind merging
> this in. Have you tried running the test runner over some third-party apps'
> tests to make sure it works? I suspect South's/migrations' might get weird,
> but then I do nasty things and that's my problem to fix.

I haven't ran any tests except the Django's test suite. How things
should work is that if you use full_flush everything works as always,
and if you don't then the only problem should be raw SQL queries (or
in South's case probably also queries done through the migrations
ORM). If things actually work that way is still an open question :)

For raw SQL queries and other changes which aren't done through
Django's ORM you can use either self.changed_models.add(some_model) or
django.db.models.signals.model_changed.send(some_model) to mark that
model's table as dirty.

Testing this on real world apps would be really useful!

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Testing multidb with TEST_MIRROR

[Donald Stufft]

On Thursday, September 6, 2012 at 6:10 PM, Jeremy Dunck wrote:
> What's the base branch for the fast_tests_merged comparison? 
https://github.com/akaariai/django/compare/django:master...fast_tests_merged 

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Testing multidb with TEST_MIRROR

[Jeremy Dunck]

On Thu, Sep 6, 2012 at 12:16 PM, Anssi Kääriäinen
 wrote:
> On 3 syys, 07:40, Anssi Kääriäinen  wrote:
>> I would like to make the TransactionTestCase faster. Currently when
>> running Django's test suite, for every test ran you will truncate
>> around 1000 tables, then create around 4000 objects (permissions +
>> content types). Likely you will write to one or two tables in the
>> test, and then do the truncate/recreate dance again. There is room for
>> improvement. Track which tables have been modified and refresh only
>> those tables.
>
> I now have a pretty good WIP approach of tracking changes in testing.
> The changes can be found from here: [https://github.com/akaariai/
> django/tree/fast_tests_merged]. The approach relies on existing
> signals + a new "model_changed" signal, which is used to signal
> bulk_create and .update() changes, and could be used for raw SQL
> changes, too.
>
> The results are promising:
>   - PostgreSQL with selenium tests, master 33m4s, patched: 9m51s
>   - SQLite, no selenium tests, master: 7m35s, patched: 3m6s.

Well that is pretty damn awesome.  It also doesn't fix my problem.
Let's ignore my problem for now, though.  I'd like to look at the
diffs.  I doubt this is the real diff:
https://github.com/akaariai/django/compare/master...fast_tests_merged

What's the base branch for the fast_tests_merged comparison?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Testing multidb with TEST_MIRROR

[Andrew Godwin]

That's an incredible speedup - I've had a quick look over the patch, and it
looks to be doing all the right things, so I'd definitely be behind merging
this in. Have you tried running the test runner over some third-party apps'
tests to make sure it works? I suspect South's/migrations' might get weird,
but then I do nasty things and that's my problem to fix.

Andrew

On Thu, Sep 6, 2012 at 4:35 PM, Anssi Kääriäinen wrote:

> On 6 syys, 19:17, Anssi Kääriäinen  wrote:
> > I now have a pretty good WIP approach of tracking changes in testing.
> > The changes can be found from here: [https://github.com/akaariai/
> > django/tree/fast_tests_merged]. The approach relies on existing
> > signals + a new "model_changed" signal, which is used to signal
> > bulk_create and .update() changes, and could be used for raw SQL
> > changes, too.
> >
> > The results are promising:
> >   - PostgreSQL with selenium tests, master 33m4s, patched: 9m51s
> >   - SQLite, no selenium tests, master: 7m35s, patched: 3m6s.
>
> I did a little bit more tuning, and now the runtime for SQLite is
> 2m34s (of which 30s is spent in time.sleep()). I can't see any more
> easy targets for performance improvements. The only thing that comes
> to mind is making fixture_loading use bulk_insert, but that has some
> problems (signals and multitable models aren't well supported by
> bulk_create).
>
> The changes done compared to the last try:
> - Made validation skip already validated models
> - Made contenttype fetching in permission creation more efficient
> - Skip permission creation when there is nothing to create
>
> I also tested MySQL and Oracle, MySQL was around 9m30s without
> selenium tests and Oracle ran in 16m, no selenium tests. On Oracle the
> database creation takes 7m, actually running the tests 9m.
>
> For those interested there is a pyprofile at:
> http://users.tkk.fi/~akaariai/sqlite.pyprofile.gz
>
> As said, there doesn't seem to be anything too easy to squash any
> more.
>
>  - Anssi
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Backporting some Python 3 support features to 1.4.x?

[Andrew Godwin]

I'm definitely +1 on this - I have a few codebases I want to start
converting but also want to keep running on 1.4, and the patch looks
sensible to me. There is precedent for this, and even if there wasn't, this
is a nice way to get the migration cycle started.

Andrew

On Thu, Sep 6, 2012 at 3:05 PM, Donald Stufft wrote:

>  Just as an additional aside, the apps can also depend on the actual six
> library itself instead of Django's embedded version (It could be an
> optional dependency on Django < 1.5). The major things I think would be
> anything Django specific that don't come from six.
>
> On Thursday, September 6, 2012 at 2:09 PM, Aymeric Augustin wrote:
>
> Hello,
>
> Several people have started porting their apps for Python 3, but they're
> running into trouble when they try to support both Django 1.4 and
> master. They end up with regrettable patterns such as:
>
> try:
> from django.utils.six import PY3
> except ImportError:
> PY3 = False
>
> Authors of pluggable apps generally chose to support the current and the
> previous version of Django. Thus their users can alternatively upgrade
> Django and their apps, which makes upgrades less scary.
>
> Now we have two alternatives:
> 1) tell them to roll their own compat or wait until they can target 1.5 +
> 1.6 — quite a downer,
> 2) add some forwards compatibility tools to 1.4.
>
> We generally don't add features to minor releases, but there's a precedent
> (csrf_noop), and the patch is as harmless as they come (see attachment).
>
> What do you think?
>
> --
> Aymeric.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
> Attachments:
>  - python3-forwards-compat-for-django-14.diff
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Testing multidb with TEST_MIRROR

[Anssi Kääriäinen]

On 6 syys, 19:17, Anssi Kääriäinen  wrote:
> I now have a pretty good WIP approach of tracking changes in testing.
> The changes can be found from here: [https://github.com/akaariai/
> django/tree/fast_tests_merged]. The approach relies on existing
> signals + a new "model_changed" signal, which is used to signal
> bulk_create and .update() changes, and could be used for raw SQL
> changes, too.
>
> The results are promising:
>   - PostgreSQL with selenium tests, master 33m4s, patched: 9m51s
>   - SQLite, no selenium tests, master: 7m35s, patched: 3m6s.

I did a little bit more tuning, and now the runtime for SQLite is
2m34s (of which 30s is spent in time.sleep()). I can't see any more
easy targets for performance improvements. The only thing that comes
to mind is making fixture_loading use bulk_insert, but that has some
problems (signals and multitable models aren't well supported by
bulk_create).

The changes done compared to the last try:
- Made validation skip already validated models
- Made contenttype fetching in permission creation more efficient
- Skip permission creation when there is nothing to create

I also tested MySQL and Oracle, MySQL was around 9m30s without
selenium tests and Oracle ran in 16m, no selenium tests. On Oracle the
database creation takes 7m, actually running the tests 9m.

For those interested there is a pyprofile at: 
http://users.tkk.fi/~akaariai/sqlite.pyprofile.gz

As said, there doesn't seem to be anything too easy to squash any
more.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Backporting some Python 3 support features to 1.4.x?

[Donald Stufft]

Just as an additional aside, the apps can also depend on the actual six library 
itself instead of Django's embedded version (It could be an optional dependency 
on Django < 1.5). The major things I think would be anything Django specific 
that don't come from six.


On Thursday, September 6, 2012 at 2:09 PM, Aymeric Augustin wrote:

> Hello,
>  
> Several people have started porting their apps for Python 3, but they're 
> running into trouble when they try to support both Django 1.4 and master. 
> They end up with regrettable patterns such as:  
>  
> try:
> from django.utils.six import PY3
> except ImportError:
> PY3 = False
>  
> Authors of pluggable apps generally chose to support the current and the 
> previous version of Django. Thus their users can alternatively upgrade Django 
> and their apps, which makes upgrades less scary.  
>  
> Now we have two alternatives:
> 1) tell them to roll their own compat or wait until they can target 1.5 + 1.6 
> — quite a downer,
> 2) add some forwards compatibility tools to 1.4.
>  
> We generally don't add features to minor releases, but there's a precedent 
> (csrf_noop), and the patch is as harmless as they come (see attachment).
>  
> What do you think?  
>  
> --  
> Aymeric.
> --  
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com 
> (mailto:django-developers@googlegroups.com).
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com 
> (mailto:django-developers+unsubscr...@googlegroups.com).
> For more options, visit this group at 
> http://groups.google.com/group/django-developers?hl=en.
>  
> Attachments:  
> - python3-forwards-compat-for-django-14.diff
>  


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Backporting some Python 3 support features to 1.4.x?

[Mark Lavin]

Hi,

I am one such author and I wanted to share my experience. I have ported my 
code and test suites for 3 applications to run on Django 1.3-1.5dev and 
Python 2.6, 2.7 and 3.2 (1.5dev only).
Those are https://github.com/mlavin/django-responsive, 
https://github.com/mlavin/django-ad-code and 
https://github.com/mlavin/django-scribbler. There are fairly new apps and 
don't
have many users/active deployments and so made good candidates for this 
work.

For the most part these were fairly easy to port. Adding unicode_literals 
everywhere was most of the work. There were a few places where I needed to 
make use of django.utils.six.
The largest pain point was trying to follow the recommendation for models 
__str__/__unicode__ in 
https://docs.djangoproject.com/en/dev/topics/python3/#str-and-unicode-methods
If you are trying to support Django 1.4 then you can't really use this 
recommendation without backporting the decorator because of the way it was 
written to be no-op for
Python 3 rather than Python 2. I understand why this decision was made and 
in the long run it is the right decision. For now I think app maintainers 
will have a bit of pain without
some of these features being backported to a 1.4.X release. You do have the 
option of using the hack you referenced (which I am doing) or a similar 
conditional import 
along with conditional __str__/__unicode__ which this decorator was meant 
to avoid. It's certainly not hard to do but it feels dirty.

My work now has been focused on getting the tests to pass and now they do. 
Another piece will be to actually deploy a project using one of these app.
I'll be continuing to do this work while at DjangoCon and I'll happily 
chime in any other stumbling blocks I run into while I port some other 
larger applications.

Best,

Mark

On Thursday, September 6, 2012 2:10:18 PM UTC-4, Aymeric Augustin wrote:
>
> Hello,
>
> Several people have started porting their apps for Python 3, but they're 
> running into trouble when they try to support both Django 1.4 and 
> master. They end up with regrettable patterns such as:
>
> try:
> from django.utils.six import PY3
> except ImportError:
> PY3 = False
>
> Authors of pluggable apps generally chose to support the current and the 
> previous version of Django. Thus their users can alternatively upgrade 
> Django and their apps, which makes upgrades less scary.
>
> Now we have two alternatives:
> 1) tell them to roll their own compat or wait until they can target 1.5 + 
> 1.6 — quite a downer,
> 2) add some forwards compatibility tools to 1.4.
>  
> We generally don't add features to minor releases, but there's a precedent 
> (csrf_noop), and the patch is as harmless as they come (see attachment).
>
> What do you think?
>  
> -- 
> Aymeric.
>  

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/goU624tDqlkJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Backporting some Python 3 support features to 1.4.x?

[Aymeric Augustin]

Hello,

Several people have started porting their apps for Python 3, but they're
running into trouble when they try to support both Django 1.4 and
master. They end up with regrettable patterns such as:

try:
from django.utils.six import PY3
except ImportError:
PY3 = False

Authors of pluggable apps generally chose to support the current and the
previous version of Django. Thus their users can alternatively upgrade
Django and their apps, which makes upgrades less scary.

Now we have two alternatives:
1) tell them to roll their own compat or wait until they can target 1.5 +
1.6 — quite a downer,
2) add some forwards compatibility tools to 1.4.

We generally don't add features to minor releases, but there's a precedent
(csrf_noop), and the patch is as harmless as they come (see attachment).

What do you think?

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.



python3-forwards-compat-for-django-14.diff
Description: Binary data

Re: Testing multidb with TEST_MIRROR

[Anssi Kääriäinen]

On 3 syys, 07:40, Anssi Kääriäinen  wrote:
> I would like to make the TransactionTestCase faster. Currently when
> running Django's test suite, for every test ran you will truncate
> around 1000 tables, then create around 4000 objects (permissions +
> content types). Likely you will write to one or two tables in the
> test, and then do the truncate/recreate dance again. There is room for
> improvement. Track which tables have been modified and refresh only
> those tables.

I now have a pretty good WIP approach of tracking changes in testing.
The changes can be found from here: [https://github.com/akaariai/
django/tree/fast_tests_merged]. The approach relies on existing
signals + a new "model_changed" signal, which is used to signal
bulk_create and .update() changes, and could be used for raw SQL
changes, too.

The results are promising:
  - PostgreSQL with selenium tests, master 33m4s, patched: 9m51s
  - SQLite, no selenium tests, master: 7m35s, patched: 3m6s.

So, more than 3x speedup and more than 2x speedup. PostgreSQL without
Selenium tests is taking 7m14s, so it is slightly faster than SQLite
on master.

The current approach uses two flags in TransactionTestCase to control
state tracking: flush_all, and detect_leaks. The first instructs the
tester to do a full DB flush after the test, the latter can be used to
check if dirty state is leaked. It however tracks only object counts
currently, so updates could be missed. Making it more intelligent will
be somewhat hard as we can't compare the objects directly - the
autoincrement primary keys will not match.

The default for flush_all is False in the patched version. Defaulting
to that isn't of course acceptable if this was included in core.
Existing applications would break. IMO it should be a setting which
defaults to True.

The detect_leaks flag in TransactionTestCase should instead be a flag
to manage.py test, this way if there are mysterious failures it should
be somewhat straightforward to detect them by just giving that flag to
the test runner.

The patch also contains a cleanup to loaddata command. In master the
command is slow and complex. Now it is faster (around 50s removed by
the loaddata cleanup alone IIRC), and somewhat less complex. The patch
isn't 100% ready. Once ready I am planning to commit this separate of
the state tracking work.

Still, there is the __deepcopy__ removal patch from #16759 in there.
It removes just 15 seconds of runtime compared to loaddata + state
tracking, but I like to keep bringing this patch up... :)

I also tried to track installed fixture state across tests. We often
repeatedly load the same fixtures. This however got too complex, as
now we need to track state between tests instead of inside one test.
For my needs a better approach would be a "base_data" fixture which is
loaded for all tests, and a test that dirties the base data would be
responsible for cleaning up and reloading the fixture. One could
already implement this using the syncdb signal, as this is basically
what is done for permissions and contenttypes. The use case for me
would be mass-loading external data (medical ICD codes for example)
before testing. Reloading such codes for each test case is
horrendously slow.

I would like to get feedback on the suggested API
(settings.TRACK_TEST_STATE, --detect-state-leaks flag for manage.py
test), and if we want something like this at all in Django. This could
work as external test class, too, if we added some of the needed hooks
to Django. This would of course mean the benefits would not be there
for Django core testing. The ability to restrict flushing to subset of
models and the new signal would at least need to be in core.

 - Anssi

Raw test data:

Patched, Postgres, Selenium:
Ran 4850 tests in 520.710s

OK (skipped=145, expected failures=4)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...

real9m50.534s
user4m3.283s
sys 0m19.497s

---
Master, Postgres, Selenium:
Ran 4848 tests in 1932.700s

OK (skipped=145, expected failures=4)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...

real33m3.587s
user17m55.659s
sys 0m50.519s

---
Patched, Postgres, no selenium:
Ran 4850 tests in 389.648s

OK (skipped=145, expected failures=4)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...

real7m14.441s
user4m4.699s
sys 0m19.853s

-
Patched, SQLite, no selenium:
Ran 4850 tests in 176.807s

OK (skipped=173, expected failures=4)
Destroying test database for alias 'default'...
Destroying test database for alias 'other'...

real3m6.210s
user2m21.041s
sys 0m12.521s


Master, SQLite, no selenium:
Ran 4848 tests in 444.018s

OK (skipped=173, expected failures=4)
Destroying test database for alias 

Re: Moving forward with Serialization.

[Tom Christie]

Thanks Piotr,

> But when serializing also Field is saved with data so it's not so clean.

I'm not sure whether it's an issue or not.  As with Django's form fields, 
the fields are copied off of base_fields, so saving state 
isn't necessarily a problem.  I'd agree that it's possibly not the cleanest 
part of the API, but it might be good enough.

> For each format there is Serializer class which is made 
from NativeSerializer ... and FormatSerializer

That would work.
I was planning on taking a slightly different approach in order to merge 
the work into Django.
In my opinion the `SERIALIZATION_MODULES` setting wouldn't make much sense 
once serialization and parsing/rendering are decoupled.  Furthermore it 
places some very specific, slightly odd, and 
undocumented constraints on the interface that serialization modules must 
provide.  Eg.  They must include a class named 'Serializer' and a class 
named 'Deserializer'.

My preference would be to put that setting on the road to deprecation, 
replacing it with something more appropriate, eg:

SERIALIZER_CLASS='django.core.serializers.FixtureSerializer'
RENDERER_CLASSES=(...)
PARSER_CLASSES=(...)

It'd be possible to do that in a way that'd be consistent with Django's 
deprecation policy, if the initial release continued to check 
if SERIALIZATION_MODULES is set, and use that in preference to the new 
style settings.

The `serialize`, `deserialize` and `get_serializer` functions continue to 
exist and would be light backward-compatible wrappers around the core of 
the new-style serialization.
 
Having said that I could probably live with either approach.

I guess it would probably make sense for me to pull django-serializers into 
a fork of django at some point, to better demonstrate what I'm suggesting.

> * IMO there is bug related to xml. All model fields must be transform to 
> text before xml serialization. In current django serialization framework 
> field's method value_to_string is responsible for this.

Thanks.  I've not looked into that yet.   It sounds feasible, though I 
think it'd have to only be affecting custom model fields in xml, and should 
be easy enough to fix.  If it's easy for you to provide an example case 
where django-serializer's output differs from the existing fixtures it'd be 
much appreciated.

Regards,

  Tom


On Saturday, 1 September 2012 12:39:32 UTC+1, Piotr Grabowski wrote:
>
> W dniu 31.08.2012 10:25, Tom Christie pisze: 
> > > I personally think that Forms are already the place that should 
> > handle (de)serialisation. They already serialise to HTML: why should 
> > they not be able to serialise to other stream types? 
> > 
> > Conceptually I agree.  As it happens django-serializers is perfectly 
> > capable of rendering into HTML forms, I just haven't yet gotten around 
> > to writing a form renderer, since it was out-of-scope of the fixture 
> > serialization functionality. 
> > 
> > Pragmatically, I'm not convinced it'd work very well.  The existing 
> > Forms implementation is tightly coupled to form-data input and HTML 
> > output, and I think trying to address that without breaking 
> > backwards compatibility would be rather difficult.  It's maybe easy 
> > enough to do for flat representations, and pk relationships, but 
> > extending it to deal with nested representations, being able to use a 
> > Form as a field on another Form, and representing custom relationships 
> > would all take some serious hacking.  My personal opinion is that 
> > whatever benefits you'd gain in DRYness, you'd lose in code 
> > complexity.  Having said that, if someone was able to hack together a 
> > Forms-based fixture serialization/deserialization implementation that 
> > passes the Django test suite, and didn't look too kludgy, I'd be 
> > perfectly willing to revise my opinion. 
>
> I am not quite sure but I think Forms should be build based on some 
> serialization API not opposite. Forms are more precise way of models 
> serialization - they are models serialized to html (specific format) 
> with some validation (specific actions) when deserializing. 
>
>
> I like Tom's django-serialziers but there are some things that I want to 
> mention: 
>
> * Process of serialization is split to two parts - transformation to 
> python native datatype (serializer) and next to specific text format 
> (renderer). But when serializing also Field is saved with data so it's 
> not so clean. I also have an issues with this but I resolve it in 
> different way (not say better :) 
>
> * In master branch Serializer is closely tied to Renderer so if there is 
> different Renderer class than new Serializer is needed. In forms branch 
> it is done in __init__ serialize method and this must be rewrite for 
> backward compatibility if django-serializers goes to core. I want to 
> propose my solution [1]: 
> For each format there is Serializer class which is made from 
> NativeSerializer ( from models to python native datatype) and 
> FormatSerializer 

Re: RawQuerySet as subquery when used with an __in filter

[Alex Hill]

Hi Anssi,

Thanks for your feedback! Responses inline.

  1. It should not be assumed that the primary key is the field needed 
> in the inner query. We do have foreign keys to other fields, and one 
> can use this approach without foreign keys at all.


Hmm, yes. Thinking about it, this seems like the biggest obstacle to 
implementing the query-wrapping approach.

The reason I proposed just selecting the PK was to mirror the behaviour of 
QuerySet when passed to __in, as happens here:

https://github.com/django/django/blob/master/django/db/models/query.py#L947 

But since we're talking about raw SQL, we'd want maximum flexibility - so 
would we need to introduce a ValuesRawQuerySet in order to select arbitrary 
columns? It seems that would be the approach that most closely mirrors 
QuerySet.

  2. If the query is of the form NOT IN, then we need to also filter 
> out null values, otherwise no results are returned if the raw query 
> contains a single null (NOT IN is weird...)
>

Yep.
 

>   3. Does the wrapping of the subquery into (select field from 
> (raw_query) where field is not null) cause performance problems? It 
> could be possible that this prevents the DB's optimizer from working 
> correctly.
>

This would need more testing, but Postgres at least optimises the inner 
query away nicely. These two queries have the same execution plan:

SELECT ... WHERE foo NOT IN (SELECT foo FROM (SELECT * FROM table WHERE 
...) WHERE foo IS NOT NULL)
SELECT ... WHERE foo NOT IN (SELECT foo FROM table WHERE ... AND foo IS NOT 
NULL)
 
Same goes for the equivalent IN queries.

  4. If the query doesn't select the required column (that is, it is 
> deferred), we don't have any way to know that. This seems like a 
> documentation issue, but worth mentioning.
>

Yes, definitely. The docs page where .raw() is introduced will have to 
explain this feature with a big warning.
 

> It would be useful to have this feature, so a +1 to this idea. The 
> above issues do not seem too severe, so I believe it is possible to 
> make this work.
>

Me too!

Thanks,
Alex

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To view this discussion on the web visit 
https://groups.google.com/d/msg/django-developers/-/S11TF_KH5XkJ.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.