Re: queryset caching note in docs

[Marco Paolini]

On 28/10/2011 15:55, Tom Evans wrote:

On Fri, Oct 28, 2011 at 1:05 PM, Marco Paolini  wrote:

it's a bit more complex: there are basically two phases:
  1) row fetching from db using cursor.fetchmany()
  2) model instance creation in queryset

both are delayed as much as possible (queryset lazyness)

phase two (model instance creation) depends on how you evaluate the
queryset:
  - testing for truth will only create enough instances to fill the cache
  - calling .exists() will try to fetch a single object and leave the cache
alone
  - iterating will create instances in batches
  - .iterator() will create the instances one at the time (but on the same db
cursos using cursor.fetchmany()
  - many other options...



You are probably more informed about the internals than me, but isn't
it irrelevant about when the instances are created; they are created
from the same query, even if the instances are not instantiated until
iterated to. Any changes to the database that happened after that
query is executed will not be reflected in the instances that are
created from that query.

IE the critical point is when the query is executed, not when the data
is fetched from the db cursor.

Yep, you're right

while digging into django and sqlite3 backend internals I discovered:

sqlite cursors from the same connection are NOT isoltated:


curs1 = conn.cursor()
curs2 = conn.cursor()
curs1.execute('select * from entry')
curs1.fetchmany(100)

[(1, 'hey'), (2, 'hey',)...

curs2.execute('update entry set name =\'\'')
curs1.fetchmany(100)

[(100, ''), (101, '',)...

to overcome this issue django fetches all cursor rows in memory in a single 
list while iterating
a resultset (see django/db/models/sql/compiler.py SqlCompiler.execute_sql)

so if you do this:

for obj in Entry.objects.all():
 pass

django does this:
 - creates a cursor
 - then calls fetchmany(100) until ALL rows are fetched
 - creates a list containing ALL fetched rows
 - passes this list to queryset instance for lazy model instance creation

I didn't know that. (maybe we should document it somewhere...)

Now that I do, I also know it's time to move to postgresql...

cheers,

Marco

--
You received this message because you are subscribed to the Google Groups "Django 
developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Request for review / Multiple time zone support for datetime representation

[Aymeric Augustin]

Hello,

I'm glad to announce that the time zone support branch is ready for review.

Review the changes: 
https://bitbucket.org/aaugustin/django/compare/..django/django
Clone the branch: hg clone https://bitbucket.org/aaugustin/django
Play with the demo app: hg clone https://bitbucket.org/aaugustin/django-tz-demo

The docs are supposed to explain everything you need, and I described many 
design decisions in my weekly check-ins, so I'm purposefully keeping this email 
short on details :)

Finally here's a teaser: http://myks.org/stuff/django-tz-demo-2.png



During this project, most things went according to the initial plan.

There is only one outstanding issue that I know of. `QuerySet.dates()` operates 
in the database timezone, ie. UTC, while an end user would expect it to take 
into account its current time zone. This affects `date_hierarchy` in the admin 
and the date based generic views. The same problem also exists for the `year`, 
`month`, `day`, and `week_day` lookups.

Under PostgreSQL, provided the current time zone has a valid name (e.g. it's 
provided by pytz), we could temporarily set the connection's timezone to this 
time zone and restore it to UTC afterwards. With other backends, I can't think 
of a simpler solution than fetching all values and doing the aggregation in 
Python. Neither of these  thrills me, so for the time being, I've documented 
this as a known limitation.

Best regards,

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.