Re: Capturing faked migrations in django_migrations table

2015-09-12 Thread Markus Holtermann
Hey there,

here's a pull request for this 
feature: https://github.com/django/django/pull/5279 . I'd appreciate some 
review.

/Markus

On Wednesday, May 20, 2015 at 9:25:00 AM UTC+10, steve byerly wrote:
>
> Awesome, thank you very much for the detailed answer.
> -Steve
>
> On Tue, May 19, 2015 at 4:16 PM, Shai Berger  > wrote:
>
>> Hi Steve,
>>
>> On Wednesday 20 May 2015 01:50:15 steve byerly wrote:
>> >
>> > I'm also unclear what the argument against storing in the migrations 
>> table
>> > is vs logging them - honest question. Since I have 4 web servers, the
>> > information would be logged to any one of them - depending on which
>> > executed the migration during deploy.
>> >
>>
>> The argument is, basically, that logging solves a more general use-case, 
>> and
>> so it is a better solution for the problem.
>>
>> Your use case, as you described it:
>>
>> > During a deploy, it's really nice to make sure you faked the right
>> > migrations/didn't fake the wrong migrations
>>
>> fits in the general pattern of "I want to know what happened in this 
>> system".
>> Logging, as far as we understand, solves that problem. The migrations 
>> table
>> solves a different problem -- it records which migrations the system 
>> considers
>> to have been executed, to support the decision of which migrations to run
>> (when asked to). One of the outstanding "conflicts" between the two goals 
>> is
>> encountered when a migration is rolled back: For "decision support", the
>> simplest handling is to delete the migration's record from the table (and 
>> this
>> is what Django does, AFAIK). For "forensics", this behavior is quite
>> unacceptable.
>>
>> The most natural solution for knowing what happened in a system is 
>> logging.
>> When you have 4 servers, you want some federated logging anyway -- 
>> migrations
>> are probably not the only case where something that happens on one server
>> affects the behavior of others. Be that as it may, Python's logging 
>> library
>> allows you to set different handlers for different loggers -- so, assuming
>> migrations get their own logger, it shouldn't be very hard to set up 
>> logging
>> so that all migration logging goes to some special place. That place can 
>> even
>> be the database -- see e.g. https://pypi.python.org/pypi/django-logdb (I 
>> have
>> no personal experience with that package, I just searched for it).
>>
>> HTH,
>> Shai.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d1a9dd7e-86c8-402f-bd5a-8c2a674bf63e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Improving MSSQL and Azure SQL support on Django

2015-09-12 Thread Tim Allen
Sorry to have missed meeting you at DjangoCon, Meet, but I'll add my 
findings here to the record.

By way of background, I work at The Wharton School, where we're a 
Python/Django (on RHEL) and SQL Server shop. I was responsible for 
implementing a working configuration for Django, starting with version 1.6, 
including building Vagrant boxes for developers which had to be 'vagrant up 
plug-and-play' ready to go. We've had pretty good success with the stack 
we're now using, but hit quite a few pain points and found many points 
where things can very much be improved.

First, drivers at the Linux level.

   - We've had our best luck with FreeTDS and unixODBC for reliability. 
   We've been using it successfully throughout the process.
   - We found the MS provided driver more performant, specifically when 
  dealing with large numbers of inserts.
  - The MS driver also offered features like bulk-loading of data, 
  which while not Django specific, would have still been a big win for our 
  team using Python without Django.
  - FreeTDS + unixODBC runs under any Linux distro we tested 
  (RHEL/CentOS, Ubuntu, Debian) and Solaris.
   - As mentioned, we found the MSODBC Driver for RedHat 
   (http://www.microsoft.com/en-us/download/details.aspx?id=36437) to be more 
   performant, however, ran into quite a few issues starting with Django 1.7.
   - There seem to be issues with multi-threading. Starting with Django 
  1.7, we had to run Django's runserver with the '--nothreading' option.
  - We ran into further issues with SQLRowCount returns from the 
  driver. These rendered it unusable.
  - Only supports RedHat, but also works on CentOS. We'd prefer one 
  that at least also works on Ubuntu/Debian (a definite must for the Django 
  community).
  - The driver is closed-source.
   - A note about FreeTDS: while FreeTDS 0.95 support TDS Version 7.3, as 
   does pyodbc, we had to stick with TDS Version 7.2, as any stack we tried 
   mapped to incorrect column types when switching to 7.3, which supports new 
   SQL Server 2008 column types. I haven't had the time to dig deeper, map 
   correctly, and issue a P.R. on this one. Additionally, TDS version numbers 
   are very confusing to users (see my Stack Overflow responses, heh). At one 
   point in time, FreeTDS has named the new TDS version "8.0" before Microsoft 
   made an official declaration; Microsoft then called it "7.2". While often 
   this will not affect configuration, now that 7.3 is support in FreeTDS 
   0.95, if "8.0" is entered in configuration, it is now default to "7.3" 
   which causes issues, and the configuration in Django, odbc.ini, and 
   freetds.conf must all explicitly state "7.2".

Regardless of the driver chosen (FreeTDS+unixODBC or MSODBC), we ended up 
having to use pyodbc instead of pymssql:

   - We have a test suite performing table creates / destroys, basic CRUD 
   operations, stored procedure execution, and more against both pyodbc and 
   pymssql.
   - pymssql outperforms pyodbc significantly against SQL Server, 
  especially on SELECT and INSERT operations.
   - pymssql on Linux offers no stable Django options, as noted throughout 
   this thread.
   - pyodbc offers several options.
  - we initially started using django-pyodbc (lionheart on GitHub), 
  which worked but required quite a few tweaks to the settings.
  - we moved to django-pyodbc-azure, which we found a much easier 
  install / Django DATABASES {} configuration, and is kept up to date in a 
  timely fashion.
   
To summarize, here's what we now use:

   - FreeTDS 0.91 - 0.95 (dependent on RedHat/CentOS version)
   - unixODBC (dependent on RedHat/CentOS version)
   - pyodbc
   - django-pyodbc-azure
   
It works well for us, but we've had to make compromises, and the promise of 
better performance we've seen in certain scenarios is tempting. If I were 
building a wish list, here's what I'd like to see, for performance and 
ease-of-installation:

   - Native driver to replace FreeTDS + unixODBC for SQL Server connections 
   that is supported and runs on more than just RedHat/CentOS, preferably open 
   source!
   - Easy, prompt free option for install: I had to hack to install to 
   avoid having to respond to interactive prompts in Vagrantfiles, Chef 
   recipes, etc.
   - Eventual inclusion in EPEL, etc, for yum or apt-get installs.
   - Python package (to replace pyodbc with one that supports SQL Server 
   functionality and performance)
   - Django-Python package (to replicate django-pyodbc-azure's mappings to 
   the pyodbc replacement)
   - Support bcp, SSIS, DATE data type, FILTERED and SPATIAL (GeoDjango, 
   anyone?) index types, easy Stored Procedure calls, OUTPUT variables, and 
   more I can't remember off the top of my head.

These would also be big wins for users of other languages / frameworks, 
such as PHP and Ruby web frameworks, Flask, etc, who use SQL Server.

So there's 

Re: status of 1.9 release blockers

2015-09-12 Thread Tim Graham
With 1 week to go until alpha:

I haven't heard anything from Marc or Preston on the major features from 
the last mail.

Jani continues to give updates in #django-dev about his progress on getting 
the Oracle GIS backend working. Things seem to be on track for finishing up 
by alpha.

On Friday, September 4, 2015 at 3:55:22 PM UTC-4, Tim Graham wrote:
>
> Status with 2 weeks and a weekend to go until alpha (Monday, September 21).
>
> Planned major features for 1.9:
>
> PostgreSQL Full Text Search (Marc Tamlyn)
>
> I haven't seen any recent activity on the pull request, nor have I heard 
> anything from Marc about it.
> https://github.com/django/django/pull/4726
>
> Custom indexes (Marc Tamlyn)
>
> I haven't see any work on revising the DEP, so this seem out for 1.9.
>
> https://github.com/django/deps/pull/6
>
> Template-based widget rendering (Preston Timmons)
> I think the code is in fairly good shape. It seems the main work left to 
> is to write documentation.
> https://github.com/django/django/pull/4848
>
> Release blockers:
>
> - Add Oracle support for new-style GIS functions
> https://code.djangoproject.com/ticket/24688
> Owner: Jani Tiainen
> Status: In progress (from recent IRC discussion, it sounds like Jani is 
> making good progress)
>
> On Monday, August 24, 2015 at 3:40:22 PM UTC-4, Tim Graham wrote:
>>
>> Time to kickoff the progress tracker for the next major release!
>>
>> Here's the status with 4 weeks to go until alpha (September 21).
>>
>> Release blockers:
>>
>> - Add Oracle support for new-style GIS functions
>> https://code.djangoproject.com/ticket/24688
>> Owner: Jani Tiainen
>> Status: In progress (test suite runs on Oracle now, which is an 
>> improvement from last week)
>>
>>
>> Other tickets tagged for 1.9:
>>
>> - No way to create tables for apps without migrations
>> https://code.djangoproject.com/ticket/25144
>> Status: Awaiting replies to "Keeping apps without migrations?" thread for 
>> a decision on what to do
>> Mailing list thread: 
>> https://groups.google.com/d/topic/django-developers/qHP4ZQSK9xM/discussion
>> Owner: pending discussion
>>
>> - Replace admin icons images by inline SVG
>> https://code.djangoproject.com/ticket/20597
>> Owner: elky
>> Status: In progress
>>
>> - Update tutorial screenshots for new admin theme
>> https://code.djangoproject.com/ticket/25200
>> Status: to be done after previous item is completed
>>
>>
>> Relevant ticket filters:
>>
>> Release blockers affecting master:
>>
>> https://code.djangoproject.com/query?status=assigned=new=master=Release+blocker
>>
>> Tickets tagged 1.9:
>>
>> https://code.djangoproject.com/query?status=assigned=new=~1.9
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f702794b-2886-415d-988c-71c267765efa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Making the test suite run faster

2015-09-12 Thread Tim Graham
Aymeric merged this on Thursday. Big thanks to him!

Now it's time to have a competition to build the machine that will run the 
Django test suite in the fastest time. :-)

On Wednesday, September 9, 2015 at 3:06:30 AM UTC-4, Aymeric Augustin wrote:
>
> Le 8 sept. 2015 à 18:42, Michael Manfre  
> a écrit : 
> > 
> > I agree with Shai. The database backend needs to be able to control this 
> feature. 
>
> Yes, I'm implementing that. The feature will be opt-in for database 
> backends. 
>
> -- 
> Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7d4b63cd-53ff-4129-8ed8-85ca9db40b97%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Fellow Report - September 12, 2015

2015-09-12 Thread Tim Graham


Report for week ending September 12, 2015:

Triaged

---

https://code.djangoproject.com/ticket/25366 - Changing the type of a 
primary_key field in a migration does not update FK columns pointing to it 
(duplicate)

https://code.djangoproject.com/ticket/25354 - related_query_name doesn't 
support class/app_label interpolation (accepted)

https://code.djangoproject.com/ticket/25337 - Validate domains against 
IDNA2008 instead of IDNA2003 (needsinfo)

https://code.djangoproject.com/ticket/25344 - Document that 
prefetch_related() cache doesn't change unless it's refetched from the 
database (accepted)

https://code.djangoproject.com/ticket/25368 - Rename X_CSRFTOKEN to 
X-CSRFTOKEN (invalid)

https://code.djangoproject.com/ticket/25346 - collectstatic --clear doesn't 
delete broken symlinks from the STATIC_ROOT (accepted)

https://code.djangoproject.com/ticket/25352 - 
django.views.generic.dates.WeekArchiveView use of _date_from_string results 
on off-by-one weeks handling (needsinfo)

https://code.djangoproject.com/ticket/25372 - TypeError on manage.py's 
autocomplete for subcommands that do not use argparse (fixed)

https://code.djangoproject.com/ticket/25373 - Add logging for {% include %} 
exceptions when template.debug = False (accepted)

https://code.djangoproject.com/ticket/25379 - Remove obsolete PostgreSQL 
steps from GeoDjango tutorial (fixed)

https://code.djangoproject.com/ticket/25380 - Add Postgres.app to the 
suggested options for PostGIS on OS X (fixed)

https://code.djangoproject.com/ticket/25381 - "default_app_config" causes 
custom commands to not be found (invalid)

https://code.djangoproject.com/ticket/25389 - SimpleLazyObject doesn't 
pickle correctly when wrapping a Model (accepted)

https://code.djangoproject.com/ticket/25387 - ModelAdmin actions don't get 
access to the ActionForm (accepted)

Authored



https://github.com/django/django/pull/5243 - Fixed #25356 -- Removed 
default_app_config from startapp template.

https://github.com/django/django/pull/5266 - Fixed a GeoIP2 test failure 
with the latest GeoIP2 database.

Reviewed/committed

--

https://github.com/django/django/pull/5240 - Fixed #11544 -- Removed 
!important rules in contrib.admin styles.

https://github.com/django/django/pull/5241 - Fixed #25200 -- Updated 
tutorial screenshots for new admin theme.

https://github.com/django/django/pull/4585 - Fixed #24706 -- Made 
ModelForm._post_clean() handle a ValidationError raised when constructing 
the model instance.

https://github.com/django/django/pull/4939 - Fixed #24917 -- Made admindocs 
display model methods that take arguments.

https://github.com/django/django/pull/4588 - Fixed #24857 -- Added "python 
-m django" entry point.

https://github.com/django/django/pull/4423 - Refs #24215 -- Improved error 
message for unhandled lazy model operations.

https://github.com/django/django/pull/5242 - Fixed #25350 -- Added alias 
--no-input for --noinput to management commands.

https://github.com/django/django/pull/5215 - Fixed #25135 -- Deprecated the 
contrib.admin allow_tags attribute.

https://github.com/django/django/pull/5256 - Fixed #25374 -- Made 
ModelAdmin checks work on instances instead of classes.

https://github.com/django/django/pull/5269 - Fixed #25200 -- Updated admin 
screenshots in docs.

https://github.com/django/django/pull/5271 - Fixed #25203 -- Documented how 
to pass Apache environment variables to Django.

https://github.com/django/django/pull/5264 - Fixed #24765 -- Allowed 
template context updates to flatten a Context.

https://github.com/django/django/pull/5224 - Fixed #23395 -- Limited line 
lengths to 119 characters.

Reviews of core dev work



https://github.com/django/django/pull/4761 - Fixed #20461 -- Allowed 
running tests in parallel.
https://github.com/django/django/pull/5255 - Fixed #24919 -- Allowed 
disabling of migrations on a per app basis

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e19d6494-65cb-490e-9ab1-5403b81d9292%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Django Admin - ModelAdmin exclude

2015-09-12 Thread Luke Plant

My initial reaction:

The docs, with Tim's correction, will look like:

The ``get_exclude`` method is given the ``HttpRequest`` and the ``obj`` 
being edited and is expected to return a list of fields, as described 
above in the :attr:`ModelAdmin.exclude` section.


However, the default implementation of ``get_exclude`` does not return a 
list at all, it returns None. And, it has to do this, otherwise it would 
actually break things - currently None is used as a sentinel value and 
it is different from an empty list 
(https://github.com/django/django/pull/5214/files#diff-3c42de3e53aba87b32c494f995a728dfL1801)


So, an overridden `get_exclude` method that calls `super` will get 
surprises, and will have to deal with them.


I would expect a method like `get_exclude` to hide these kind of warts, 
but for historic reasons it can't.  By adding this method, we're adding 
another wart - a method that is documented as returning a list but 
doesn't. Of course, we could document all the gory details, but the 
longer the documentation, the more I think "this just smells". Django 
has enough warts and smells without adding more.


I realise that `get_fields` already behaves in the same way as the 
proposed `get_exclude`, but it's not very nice. The behaviour of 
`get_fieldsets` is much better.


However, having said all this, I realise there is a reasonable use case 
for this (like Peter Farrell's). Also, the current asymmetry between 
'fields' and 'exclude' is also a wart. There must be a lesson for the 
future in all of this, I'm not sure what exactly...



Luke


On 03/09/15 17:28, Tim Graham wrote:
A pull request has been offered: 
https://github.com/django/django/pull/5214


It seems simple enough to me. Marc or Luke, would you withdraw your 
objection after seeing the implementation?


On Wednesday, September 2, 2015 at 7:09:18 AM UTC-4, Ramez Ashraf wrote:

I agree that there are a lot of of hooks in the admin and we might
need to better document them or remove unneeded or maybe make a
revolution :)
But, with the current api and scheme, having get_exclude is more
of the expected behavior of Django and i think it should be
implemented.

What we should certainly do -i believe- is more documenting the
admin flow & how it's constructed, best practices to achieve
certain repeated goals etc..

Regards,
Ramez


On Saturday, June 6, 2015 at 3:43:11 PM UTC+2, Marc Tamlyn wrote:

I don't think we should add this. Exclude is passed to the
form, and we already have a way of changing the form based on
the request. I'd rather see changes made to reduce some of the
many many ways to select fields in the admin. By my count at
present we have:

ModelAdmin.fields
ModelAdmin.get_fields()
ModelAdmin.exclude
ModelAdmin.fieldsets
ModelAdmin.get_fieldsets()
ModelAdmin.readonly_fields
ModelAdmin.get_readonly_fields()
ModelAdmin.form concrete fields
ModelAdmin.form.Meta.fields
ModelAdmin.form.Meta.exclude
ModelAdmin.get_form()

There's an argument the only one missing here is
get_exclude(), but I think the current API is silly.
Personally, I'd like to see us moving towards us encouraging
doing more work in the form (and defining the form) rather
than doing so much in the ModelAdmin class. This may well
require better support for fieldsets in django.forms.

Marc

On 6 June 2015 at 05:06, Peter Farrell
 wrote:

We are writing a custom admin for CleanerVersion that
subclasses ModelAdmin. Just about every attribute has a
hook method which makes extension easier. For example,
list_display has get_list_display(). However, exclude
doesn't have one implemented and it seems like an
oversight. I'm proposing we add one.

Our current work seeking is to write a property descriptor
for exclude but then the admin check fails with it not
being a tuple or list. Then you have to create a custom
admin checks class to suppress the exclude check error
because it's not a tuple or list (but really a descriptor).

If this is ok, I'Ill write a PR.

--
You received this message because you are subscribed to
the Google Groups "Django developers  (Contributions to
Django itself)" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to
django-d...@googlegroups.com.
Visit this group at
http://groups.google.com/group/django-developers
.
To view this