I've been working on a migration script, and turned to my trusty friend the
WindowedRangeQuery recipe:
https://bitbucket.org/zzzeek/sqlalchemy/wiki/UsageRecipes/WindowedRangeQuery
I ran into a small problem and hoping someone else may have done this and
saved me some work.
I'm trying to work out some edge cases on a large table, and the default
performance is pretty bad in Postgres.
The two big issues are:
1. The initial query to create an iterable range is painful. It takes me
about 2 minutes. This seems to be due to:
a - loading a lot of data into memory
b - sequentially scanning the 'id' column. this function won't use the
primary key, or any other index, for id. it seq scans the whole table.
2. Window ranges are created for all ids in the table. If I only want to
work on half the records, with a min/max, the 'iterable' query doesn't use
the min/max filter values; it generates 100% of potential ranges based on
the id, and the filters are only used for the 'where' clause.
Has anyone worked out an improved version of this technique, or am I
cobbling something together myself?
--
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
