Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
On 4/22/15 12:45 AM, Oliver Palmer wrote: We're using a Flask extension to work with sqlalchemy called flask-sqlalchemy. The engine is usually not directly exposed but echo can be enabled using a configuration var https://github.com/pyfarm/pyfarm-master/commit/5d0abc03273f0fcce3c7d2cf44ef8981dd31aa41 which should have the same impact in terms of logging the sql statements. This time around I only ran the tests for one module because we hit Travis's 4MB log limit in a couple of seconds when running all the tests at once. I can run the full test suite locally and upload that log output somewhere if you need it. Otherwise, here's the failures when using 3e80d628bd133d0fd0687e35b8d13abd1d31d6df (search for 'IntegrityError'): https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494875/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494876/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494877/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494878/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494880/log.txt OK can you please share: 1. What is the system by which these table names are being generated, as they appear to be dynamic?Are Table objects being copied, are there events within the execution process, or how is that working? 2. Have you tried running this test suite locally? Even just this one test? Does it pass? Can you please set the environment variable PYTHONHASHSEED=random (see https://docs.python.org/2/using/cmdline.html#envvar-PYTHONHASHSEED for use) and then run this single test in a local environment repeatedly? 3. Share the mapping and/or Table metadata that is used within the drop_all() for these tables: CREATE TABLE test14222015041441_users ( id INTEGER NOT NULL, active BOOLEAN, username VARCHAR(254) NOT NULL, password VARCHAR(64), email VARCHAR(255), expiration DATETIME, onetime_code VARCHAR(64), last_login DATETIME, PRIMARY KEY (id), CHECK (active IN (0, 1)), UNIQUE (username), UNIQUE (email) ) CREATE TABLE test14222015041441_user_roles ( user_id INTEGER, role_id INTEGER, FOREIGN KEY(user_id) REFERENCES test14222015041441_users (id), FOREIGN KEY(role_id) REFERENCES test14222015041441_roles (id) ) CREATE TABLE test14222015041441_roles ( id INTEGER NOT NULL, active BOOLEAN, name VARCHAR(128) NOT NULL, expiration DATETIME, description TEXT, PRIMARY KEY (id), CHECK (active IN (0, 1)), UNIQUE (name) ) 4. Can you confirm that these tables are in fact defined with MetaData() and Table(), and are **not** using autoload=True in any way? This includes no use of automap, SQLSoup, etc. 5. Is this the same test that fails every time (or at least when things fail)? Or do different tests of a similar nature fail randomly? But, it also passed once too (like the test I did yesterday): https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494879/log.txt All of the above are from this Travis job: https://travis-ci.org/pyfarm/pyfarm-master/builds/59494874 And with sqlalchemy 0.9.9. Same tests and logging configuration just consistently passing compared to the above: https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494606/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494607/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494609/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494610/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494611/log.txt Here's the original Travis job for those: https://travis-ci.org/pyfarm/pyfarm-master/builds/59494605 By the way, thanks a bunch helping us taking a look at this. On Tuesday, April 21, 2015 at 11:42:30 AM UTC-4, Michael Bayer wrote: On 4/21/15 11:19 AM, Guido Winkelmann wrote: On Tuesday 21 April 2015 09:43:51 Mike Bayer wrote: On 4/21/15 6:45 AM, Guido Winkelmann wrote: On Monday 20 April 2015 21:57:40 Oliver Palmer wrote: [...] So I got to thinking about what we're doing differently with sqlite and this bit of code comes to mind: # sqlite specific configuration for development if db.engine.name http://db.engine.name == sqlite: @event.listens_for(Engine, connect) def set_sqlite_pragma(dbapi_connection, connection_record): cursor = dbapi_connection.cursor() cursor.execute(PRAGMA foreign_keys=ON) cursor.execute(PRAGMA synchronous=OFF) cursor.execute(PRAGMA journal_mode=MEMORY) cursor.close() If I comment the above out in our application.py https://github.com/pyfarm/pyfarm-master/blob/f22912cd7d89b93c146801fd1575 https://github.com/pyfarm/pyfarm-master/blob/f22912cd7d89b93c146801fd1575 ff0
Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
On 4/22/15 12:45 AM, Oliver Palmer wrote: We're using a Flask extension to work with sqlalchemy called flask-sqlalchemy. The engine is usually not directly exposed but echo can be enabled using a configuration var https://github.com/pyfarm/pyfarm-master/commit/5d0abc03273f0fcce3c7d2cf44ef8981dd31aa41 which should have the same impact in terms of logging the sql statements. This time around I only ran the tests for one module because we hit Travis's 4MB log limit in a couple of seconds when running all the tests at once. I can run the full test suite locally and upload that log output somewhere if you need it. oddly enough, using just the schema you have this issue seems to reproduce. On SQLite, and not Postgresql. Which is completely weird because the sorting of tables has nothing to do with the backend. always a mystery how something that looks about as possible as all the buildings being upside down one day, will become something very simple. Otherwise, here's the failures when using 3e80d628bd133d0fd0687e35b8d13abd1d31d6df (search for 'IntegrityError'): https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494875/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494876/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494877/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494878/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494880/log.txt But, it also passed once too (like the test I did yesterday): https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494879/log.txt All of the above are from this Travis job: https://travis-ci.org/pyfarm/pyfarm-master/builds/59494874 And with sqlalchemy 0.9.9. Same tests and logging configuration just consistently passing compared to the above: https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494606/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494607/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494609/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494610/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494611/log.txt Here's the original Travis job for those: https://travis-ci.org/pyfarm/pyfarm-master/builds/59494605 By the way, thanks a bunch helping us taking a look at this. On Tuesday, April 21, 2015 at 11:42:30 AM UTC-4, Michael Bayer wrote: On 4/21/15 11:19 AM, Guido Winkelmann wrote: On Tuesday 21 April 2015 09:43:51 Mike Bayer wrote: On 4/21/15 6:45 AM, Guido Winkelmann wrote: On Monday 20 April 2015 21:57:40 Oliver Palmer wrote: [...] So I got to thinking about what we're doing differently with sqlite and this bit of code comes to mind: # sqlite specific configuration for development if db.engine.name http://db.engine.name == sqlite: @event.listens_for(Engine, connect) def set_sqlite_pragma(dbapi_connection, connection_record): cursor = dbapi_connection.cursor() cursor.execute(PRAGMA foreign_keys=ON) cursor.execute(PRAGMA synchronous=OFF) cursor.execute(PRAGMA journal_mode=MEMORY) cursor.close() If I comment the above out in our application.py https://github.com/pyfarm/pyfarm-master/blob/f22912cd7d89b93c146801fd1575 https://github.com/pyfarm/pyfarm-master/blob/f22912cd7d89b93c146801fd1575 ff0 6f4883724/pyfarm/master/application.py#L208 module the second nosetests example above works without issues. This looks to me like you are fixing the problem by just not enabling foreign key support in sqlite. Since the problem was a foreign key violation, telling sqlite to not bother enforcing those will make it so we don't see the problem in the tests anymore, but it doesn't fix whatever is going on here... what is needed here is actual logging of the tables as they are being dropped. The claim here is that the ordering of the tables is wrong in 1.0.0. So can we please see the full list of DROP statements logged for both the 0.9.9 version and the 1.0.0 version? Can you help me with that? I don't know how to make SQLAlchemy log all its DROP statements. logging is through the standard Python logging system, or alternatively the echo=True flag on create_engine() as a shortcut, but for a travis build I'd imagine that logging and stdout might already be routed around. http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html?highlight=logging#configuring-logging http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html?highlight=logging#configuring-logging Guido W. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe
Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
On 4/22/15 12:45 AM, Oliver Palmer wrote: We're using a Flask extension to work with sqlalchemy called flask-sqlalchemy. The engine is usually not directly exposed but echo can be enabled using a configuration var https://github.com/pyfarm/pyfarm-master/commit/5d0abc03273f0fcce3c7d2cf44ef8981dd31aa41 which should have the same impact in terms of logging the sql statements. This time around I only ran the tests for one module because we hit Travis's 4MB log limit in a couple of seconds when running all the tests at once. I can run the full test suite locally and upload that log output somewhere if you need it. Otherwise, here's the failures when using 3e80d628bd133d0fd0687e35b8d13abd1d31d6df (search for 'IntegrityError'): can you please try version 39978060b0d81bd470aade97e608. All issues should be resolved at this point, please let me know ASAP, thanks! -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
On Wednesday 22 April 2015 14:17:02 Mike Bayer wrote: On 4/22/15 12:45 AM, Oliver Palmer wrote: We're using a Flask extension to work with sqlalchemy called flask-sqlalchemy. The engine is usually not directly exposed but echo can be enabled using a configuration var https://github.com/pyfarm/pyfarm-master/commit/5d0abc03273f0fcce3c7d2cf44e f8981dd31aa41 which should have the same impact in terms of logging the sql statements. This time around I only ran the tests for one module because we hit Travis's 4MB log limit in a couple of seconds when running all the tests at once. I can run the full test suite locally and upload that log output somewhere if you need it. Otherwise, here's the failures when using 3e80d628bd133d0fd0687e35b8d13abd1d31d6df (search for 'IntegrityError'): can you please try version 39978060b0d81bd470aade97e608. All issues should be resolved at this point, please let me know ASAP, thanks! I just tested, all our tests pass now. Thanks again! Guido W. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
On Monday 20 April 2015 13:24:49 Michael Bayer wrote:
On Apr 20, 2015, at 12:56 PM, Guido Winkelmann guido@ambient-
entertainment.de wrote:
On Monday 20 April 2015 11:23:06 Mike Bayer wrote:
On 4/20/15 8:09 AM, Guido Winkelmann wrote:
On MySQL/PostgreSQL, this line fails:
Apparently, sqlalchemy will use symbol('NEVER_SET') where the id of
the model used for filtering should be.
this is a known regression and is fixed in 1.0.1:
http://docs.sqlalchemy.org/en/latest/changelog/changelog_10.html#change-1.
0.1
if you can confirm with current master that this is fixed I can release
today or tomorrow as this particular regression is fairly severe.
I just tested, the problem is still present in the current master
(bd61e7a3287079cf742f4df698bfe3628c090522 from github).
Oh, read your text, while you haven't provided a code sample it sounds like
you are possibly saying filter(Foo.relationship == some_transient_object)
and expecting that all the None values come out. Yes? That is just the
kind of example of just happened to work I'm talking about. Can you
confirm this is what you are doing please ?
Yes, basically.
Hopefully can find a fix for
that.There is an entry detailing the behavioral change here but these
effects were unanticipated (hence there were five betas, to little avail).
Guido W.
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
On Tuesday 21 April 2015 09:43:51 Mike Bayer wrote: On 4/21/15 6:45 AM, Guido Winkelmann wrote: On Monday 20 April 2015 21:57:40 Oliver Palmer wrote: [...] So I got to thinking about what we're doing differently with sqlite and this bit of code comes to mind: # sqlite specific configuration for development if db.engine.name == sqlite: @event.listens_for(Engine, connect) def set_sqlite_pragma(dbapi_connection, connection_record): cursor = dbapi_connection.cursor() cursor.execute(PRAGMA foreign_keys=ON) cursor.execute(PRAGMA synchronous=OFF) cursor.execute(PRAGMA journal_mode=MEMORY) cursor.close() If I comment the above out in our application.py https://github.com/pyfarm/pyfarm-master/blob/f22912cd7d89b93c146801fd1575 ff0 6f4883724/pyfarm/master/application.py#L208 module the second nosetests example above works without issues. This looks to me like you are fixing the problem by just not enabling foreign key support in sqlite. Since the problem was a foreign key violation, telling sqlite to not bother enforcing those will make it so we don't see the problem in the tests anymore, but it doesn't fix whatever is going on here... what is needed here is actual logging of the tables as they are being dropped. The claim here is that the ordering of the tables is wrong in 1.0.0. So can we please see the full list of DROP statements logged for both the 0.9.9 version and the 1.0.0 version? Can you help me with that? I don't know how to make SQLAlchemy log all its DROP statements. Guido W. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
On 4/21/15 6:45 AM, Guido Winkelmann wrote: On Monday 20 April 2015 21:57:40 Oliver Palmer wrote: [...] So I got to thinking about what we're doing differently with sqlite and this bit of code comes to mind: # sqlite specific configuration for development if db.engine.name == sqlite: @event.listens_for(Engine, connect) def set_sqlite_pragma(dbapi_connection, connection_record): cursor = dbapi_connection.cursor() cursor.execute(PRAGMA foreign_keys=ON) cursor.execute(PRAGMA synchronous=OFF) cursor.execute(PRAGMA journal_mode=MEMORY) cursor.close() If I comment the above out in our application.py https://github.com/pyfarm/pyfarm-master/blob/f22912cd7d89b93c146801fd1575ff0 6f4883724/pyfarm/master/application.py#L208 module the second nosetests example above works without issues. This looks to me like you are fixing the problem by just not enabling foreign key support in sqlite. Since the problem was a foreign key violation, telling sqlite to not bother enforcing those will make it so we don't see the problem in the tests anymore, but it doesn't fix whatever is going on here... what is needed here is actual logging of the tables as they are being dropped. The claim here is that the ordering of the tables is wrong in 1.0.0. So can we please see the full list of DROP statements logged for both the 0.9.9 version and the 1.0.0 version? then we will confirm that, then we look at the table metadata to see that it should in fact enforce the 0.9.9 ordering, then we've confirmed that's the issue and can produce a reproduction case. Guido W. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
On Monday 20 April 2015 19:22:36 Mike Bayer wrote: On 4/20/15 12:56 PM, Guido Winkelmann wrote: I just tested, the problem is still present in the current master (bd61e7a3287079cf742f4df698bfe3628c090522 from github). Guido W. can you please try current master at least as of a3af638e1a95d42075e25e874746, thanks. I just updated to 3e80d628bd133d0fd0687e35b8d13abd1d31d6df (which comes after a3af638e1a95d42075e25e874746). The problem has gone away with those changes. Thanks for looking into this! (We're still going to merge the patch where we drop the query entirely if the object we compare the relationship to is transient...) Guido W. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
On 4/21/15 11:19 AM, Guido Winkelmann wrote: On Tuesday 21 April 2015 09:43:51 Mike Bayer wrote: On 4/21/15 6:45 AM, Guido Winkelmann wrote: On Monday 20 April 2015 21:57:40 Oliver Palmer wrote: [...] So I got to thinking about what we're doing differently with sqlite and this bit of code comes to mind: # sqlite specific configuration for development if db.engine.name == sqlite: @event.listens_for(Engine, connect) def set_sqlite_pragma(dbapi_connection, connection_record): cursor = dbapi_connection.cursor() cursor.execute(PRAGMA foreign_keys=ON) cursor.execute(PRAGMA synchronous=OFF) cursor.execute(PRAGMA journal_mode=MEMORY) cursor.close() If I comment the above out in our application.py https://github.com/pyfarm/pyfarm-master/blob/f22912cd7d89b93c146801fd1575 ff0 6f4883724/pyfarm/master/application.py#L208 module the second nosetests example above works without issues. This looks to me like you are fixing the problem by just not enabling foreign key support in sqlite. Since the problem was a foreign key violation, telling sqlite to not bother enforcing those will make it so we don't see the problem in the tests anymore, but it doesn't fix whatever is going on here... what is needed here is actual logging of the tables as they are being dropped. The claim here is that the ordering of the tables is wrong in 1.0.0. So can we please see the full list of DROP statements logged for both the 0.9.9 version and the 1.0.0 version? Can you help me with that? I don't know how to make SQLAlchemy log all its DROP statements. logging is through the standard Python logging system, or alternatively the echo=True flag on create_engine() as a shortcut, but for a travis build I'd imagine that logging and stdout might already be routed around. http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html?highlight=logging#configuring-logging Guido W. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
We're using a Flask extension to work with sqlalchemy called flask-sqlalchemy. The engine is usually not directly exposed but echo can be enabled using a configuration var https://github.com/pyfarm/pyfarm-master/commit/5d0abc03273f0fcce3c7d2cf44ef8981dd31aa41 which should have the same impact in terms of logging the sql statements. This time around I only ran the tests for one module because we hit Travis's 4MB log limit in a couple of seconds when running all the tests at once. I can run the full test suite locally and upload that log output somewhere if you need it. Otherwise, here's the failures when using 3e80d628bd133d0fd0687e35b8d13abd1d31d6df (search for 'IntegrityError'): https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494875/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494876/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494877/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494878/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494880/log.txt But, it also passed once too (like the test I did yesterday): https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494879/log.txt All of the above are from this Travis job: https://travis-ci.org/pyfarm/pyfarm-master/builds/59494874 And with sqlalchemy 0.9.9. Same tests and logging configuration just consistently passing compared to the above: https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494606/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494607/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494609/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494610/log.txt https://s3.amazonaws.com/archive.travis-ci.org/jobs/59494611/log.txt Here's the original Travis job for those: https://travis-ci.org/pyfarm/pyfarm-master/builds/59494605 By the way, thanks a bunch helping us taking a look at this. On Tuesday, April 21, 2015 at 11:42:30 AM UTC-4, Michael Bayer wrote: On 4/21/15 11:19 AM, Guido Winkelmann wrote: On Tuesday 21 April 2015 09:43:51 Mike Bayer wrote: On 4/21/15 6:45 AM, Guido Winkelmann wrote: On Monday 20 April 2015 21:57:40 Oliver Palmer wrote: [...] So I got to thinking about what we're doing differently with sqlite and this bit of code comes to mind: # sqlite specific configuration for development if db.engine.name == sqlite: @event.listens_for(Engine, connect) def set_sqlite_pragma(dbapi_connection, connection_record): cursor = dbapi_connection.cursor() cursor.execute(PRAGMA foreign_keys=ON) cursor.execute(PRAGMA synchronous=OFF) cursor.execute(PRAGMA journal_mode=MEMORY) cursor.close() If I comment the above out in our application.py https://github.com/pyfarm/pyfarm-master/blob/f22912cd7d89b93c146801fd1575 ff0 6f4883724/pyfarm/master/application.py#L208 module the second nosetests example above works without issues. This looks to me like you are fixing the problem by just not enabling foreign key support in sqlite. Since the problem was a foreign key violation, telling sqlite to not bother enforcing those will make it so we don't see the problem in the tests anymore, but it doesn't fix whatever is going on here... what is needed here is actual logging of the tables as they are being dropped. The claim here is that the ordering of the tables is wrong in 1.0.0. So can we please see the full list of DROP statements logged for both the 0.9.9 version and the 1.0.0 version? Can you help me with that? I don't know how to make SQLAlchemy log all its DROP statements. logging is through the standard Python logging system, or alternatively the echo=True flag on create_engine() as a shortcut, but for a travis build I'd imagine that logging and stdout might already be routed around. http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html?highlight=logging#configuring-logging Guido W. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
On Monday 20 April 2015 21:57:40 Oliver Palmer wrote: [...] So I got to thinking about what we're doing differently with sqlite and this bit of code comes to mind: # sqlite specific configuration for development if db.engine.name == sqlite: @event.listens_for(Engine, connect) def set_sqlite_pragma(dbapi_connection, connection_record): cursor = dbapi_connection.cursor() cursor.execute(PRAGMA foreign_keys=ON) cursor.execute(PRAGMA synchronous=OFF) cursor.execute(PRAGMA journal_mode=MEMORY) cursor.close() If I comment the above out in our application.py https://github.com/pyfarm/pyfarm-master/blob/f22912cd7d89b93c146801fd1575ff0 6f4883724/pyfarm/master/application.py#L208 module the second nosetests example above works without issues. This looks to me like you are fixing the problem by just not enabling foreign key support in sqlite. Since the problem was a foreign key violation, telling sqlite to not bother enforcing those will make it so we don't see the problem in the tests anymore, but it doesn't fix whatever is going on here... Guido W. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
On 4/20/15 12:56 PM, Guido Winkelmann wrote: I just tested, the problem is still present in the current master (bd61e7a3287079cf742f4df698bfe3628c090522 from github). Guido W. can you please try current master at least as of a3af638e1a95d42075e25e874746, thanks. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
Hey original developer pyfarm-master here, Guido pointed me at this thread. I've run a test with a3af638e1a95d42075e25e874746 and the sqlite tests are still failing to drop the tables: https://travis-ci.org/pyfarm/pyfarm-master/builds/59341150 Since Guido commented however I merged a PR he proposed and that seems to have fixed most of our other failures (everything except for the drop tables issue with sqlite). I started to think there's something different with respect to sqlite and wrote this which does almost exactly what the tests do (and it always works): import os os.environ.update( PYFARM_DATABASE_URI=sqlite:///:memory: ) from pyfarm.master.application import db for i in range(5): # setUp from pyfarm.models.agent import Agent from pyfarm.models.job import Job from pyfarm.models.jobtype import JobType from pyfarm.models.software import ( Software, SoftwareVersion, JobSoftwareRequirement, JobTypeSoftwareRequirement) from pyfarm.models.tag import Tag from pyfarm.models.task import Task from pyfarm.models.user import User from pyfarm.models.jobqueue import JobQueue from pyfarm.models.gpu import GPU db.create_all() # execute tests # tearDown db.session.remove() db.drop_all() There's a gist of the above here for those who have issues displaying the above properly: https://gist.github.com/opalmer/0850879794e81198c3a0 If you run a test individually, it also always works: env PYFARM_DATABASE_URI=sqlite:///:memory: nosetests tests/test_models/test_model_users.py:UserTest.test_user_auth_token However if you run something that requires a few test iterations it almost always fails when dropping the tables: env PYFARM_DATABASE_URI=sqlite:///:memory: nosetests tests/test_models/test_model_users.py:UserTest I say almost always fails because on occasion the above will pass too. I'm only using the above test case as an example but other tests seem to have the same problem. So I got to thinking about what we're doing differently with sqlite and this bit of code comes to mind: # sqlite specific configuration for development if db.engine.name == sqlite: @event.listens_for(Engine, connect) def set_sqlite_pragma(dbapi_connection, connection_record): cursor = dbapi_connection.cursor() cursor.execute(PRAGMA foreign_keys=ON) cursor.execute(PRAGMA synchronous=OFF) cursor.execute(PRAGMA journal_mode=MEMORY) cursor.close() If I comment the above out in our application.py https://github.com/pyfarm/pyfarm-master/blob/f22912cd7d89b93c146801fd1575ff06f4883724/pyfarm/master/application.py#L208 module the second nosetests example above works without issues. Here's a test on travis with sqlalchemy 1.0.0 and the above code commented out: https://travis-ci.org/pyfarm/pyfarm-master/builds/59345110 And from the latest master (didn't expect a difference here but wanted to be sure): https://travis-ci.org/pyfarm/pyfarm-master/builds/59346659 And here's the same tests with the same code above commented out with sqlalchemy 0.9.9 https://travis-ci.org/pyfarm/pyfarm-master/builds/59346146 So I'm not sure yet if this is a bug in sqlalchemy 1.0.0+ or not because I didn't dive that deeply into the code changes for event handling. Regardless, I think this is probably something we should update in our tests anyway since we could couple the execution of those pragma statements more closely with the tests as they run to avoid the issue in the future. I would be curious to know if this is actually a bug in event handling though. ---Oliver On Monday, April 20, 2015 at 7:22:47 PM UTC-4, Michael Bayer wrote: On 4/20/15 12:56 PM, Guido Winkelmann wrote: I just tested, the problem is still present in the current master (bd61e7a3287079cf742f4df698bfe3628c090522 from github). Guido W. can you please try current master at least as of a3af638e1a95d42075e25e874746, thanks. -- You received this message because you are subscribed to the Google Groups sqlalchemy group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at http://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
[sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
Hi,
Have there been any non-backwards-compatible changes in SQLAlchemy 1.0
compared to 0.9.9?
We are seeing a lot of sudden breakage in our unit tests when switching to
SQLAlchemy 1.0 from 0.9.9. Tests that worked fine before suddenly fail
across the board.
Here's a an example of a test build that suddenly failed on 1.0:
https://travis-ci.org/pyfarm/pyfarm-master/builds/58860924
If you compare the builds on sqlite with those MySQL/PostgreSQL, you will
see there are two different, seemingly unrelated things going wrong:
On sqlite, drop_all() seems to fail to get the order of table drops right,
and consequently runs into a referential integrity error.
On MySQL/PostgreSQL, this line fails:
association = TaskTaskLogAssociation.query.filter_by(task=task,
log=task_log, attempt=attempt).first()
In this context, log is a relationship in the
model TaskTaskLogAssociation to model TaskLog. task_log is an object of
type TaskLog, but one that has never been written to the database and has
no set id. That leads to this error message in the logs:
nose.proxy.ProgrammingError: (psycopg2.ProgrammingError) function
symbol(unknown) does not exist
LINE 3: ...72015052936_task_log_associations.attempt = 1 AND symbol('NE...
^
HINT: No function matches the given name and argument types. You might
need to add explicit type casts.
[SQL: 'SELECT test29172015052936_task_log_associations.task_log_id AS
test29172015052936_task_log_associations_task_log_id,
test29172015052936_task_log_associations.task_id AS
test29172015052936_task_log_associations_task_id,
test29172015052936_task_log_associations.attempt AS
test29172015052936_task_log_associations_attempt,
test29172015052936_task_log_associations.state AS
test29172015052936_task_log_associations_state \nFROM
test29172015052936_task_log_associations \nWHERE
test29172015052936_task_log_associations.attempt = %(attempt_1)s AND
%(param_1)s = test29172015052936_task_log_associations.task_log_id AND
%(param_2)s = test29172015052936_task_log_associations.task_id \n LIMIT
%(param_3)s'] [parameters: {'param_1': symbol('NEVER_SET'), 'attempt_1': 1,
'param_2': 1, 'param_3': 1}]
Apparently, sqlalchemy will use symbol('NEVER_SET') where the id of the
model used for filtering should be.
It may be a bit questionable to filter by a model that doesn't even exist
in the database, but, again, this used to work fine in 0.9.9.
Regards,
Guido W.
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
On Apr 20, 2015, at 12:56 PM, Guido Winkelmann
gu...@ambient-entertainment.de wrote:
On Monday 20 April 2015 11:23:06 Mike Bayer wrote:
On 4/20/15 8:09 AM, Guido Winkelmann wrote:
[...]
On sqlite, drop_all() seems to fail to get the order of table drops
right, and consequently runs into a referential integrity error.
If you can post a reproducible issue, that's what I can work with.
I'm afraid the best I can offer right now is the current state of the pyfarm-
master code base. It's 100% reproducible there, but it's not exactly a
reduced test case...
There are changes to how tables are sorted in the absence of foreign key
dependency, where this ordering was previously undefined, it is now
determinstic; see
http://docs.sqlalchemy.org/en/latest/changelog/changelog_10.html#change-aab33
2eedafc8e090f42b89ac7a67e6c.
On MySQL/PostgreSQL, this line fails:
Apparently, sqlalchemy will use symbol('NEVER_SET') where the id of
the model used for filtering should be.
this is a known regression and is fixed in 1.0.1:
http://docs.sqlalchemy.org/en/latest/changelog/changelog_10.html#change-1.0.1
if you can confirm with current master that this is fixed I can release
today or tomorrow as this particular regression is fairly severe.
I just tested, the problem is still present in the current master
(bd61e7a3287079cf742f4df698bfe3628c090522 from github).
Oh, read your text, while you haven't provided a code sample it sounds like you
are possibly saying filter(Foo.relationship == some_transient_object) and
expecting that all the None values come out. Yes? That is just the kind of
example of just happened to work I'm talking about. Can you confirm this is
what you are doing please ? Hopefully can find a fix for that.There is an
entry detailing the behavioral change here but these effects were unanticipated
(hence there were five betas, to little avail).
Guido W.
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an
email to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
On Monday 20 April 2015 11:23:06 Mike Bayer wrote:
On 4/20/15 8:09 AM, Guido Winkelmann wrote:
[...]
On sqlite, drop_all() seems to fail to get the order of table drops
right, and consequently runs into a referential integrity error.
If you can post a reproducible issue, that's what I can work with.
I'm afraid the best I can offer right now is the current state of the pyfarm-
master code base. It's 100% reproducible there, but it's not exactly a
reduced test case...
There are changes to how tables are sorted in the absence of foreign key
dependency, where this ordering was previously undefined, it is now
determinstic; see
http://docs.sqlalchemy.org/en/latest/changelog/changelog_10.html#change-aab33
2eedafc8e090f42b89ac7a67e6c.
On MySQL/PostgreSQL, this line fails:
Apparently, sqlalchemy will use symbol('NEVER_SET') where the id of
the model used for filtering should be.
this is a known regression and is fixed in 1.0.1:
http://docs.sqlalchemy.org/en/latest/changelog/changelog_10.html#change-1.0.1
if you can confirm with current master that this is fixed I can release
today or tomorrow as this particular regression is fairly severe.
I just tested, the problem is still present in the current master
(bd61e7a3287079cf742f4df698bfe3628c090522 from github).
Guido W.
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
On 4/20/15 8:09 AM, Guido Winkelmann wrote:
Hi,
Have there been any non-backwards-compatible changes in SQLAlchemy 1.0
compared to 0.9.9?
Most behavioral changes are listed out at
http://docs.sqlalchemy.org/en/rel_1_0/changelog/migration_10.html; I've
urged everyone to please read through this document. None of the
behavioral changes are backwards incompatible at face value, however,
the nature of SQLAlchemy is necessarily one where there's lots of
behaviors that applications can find themselves relying upon, and when
we improve these behaviors, applications which relied upon bugs,
inconsistencies, or things that just happened to work a certain way can
break when we make things more consistent or apply definitions to
behaviors that were previously not defined.
There have also been five beta releases put out. In particular the
NEVER_SET issue you are receiving is a known regression that is now
fixed, but unfortunately not enough people were interested in trying out
any of these five beta releases in order to find this fairly common
condition, so it is only fixed for 1.0.1.
If you compare the builds on sqlite with those MySQL/PostgreSQL, you
will see there are two different, seemingly unrelated things going wrong:
On sqlite, drop_all() seems to fail to get the order of table drops
right, and consequently runs into a referential integrity error.
If you can post a reproducible issue, that's what I can work with.
There are changes to how tables are sorted in the absence of foreign key
dependency, where this ordering was previously undefined, it is now
determinstic; see
http://docs.sqlalchemy.org/en/latest/changelog/changelog_10.html#change-aab332eedafc8e090f42b89ac7a67e6c.
On MySQL/PostgreSQL, this line fails:
Apparently, sqlalchemy will use symbol('NEVER_SET') where the id of
the model used for filtering should be.
this is a known regression and is fixed in 1.0.1:
http://docs.sqlalchemy.org/en/latest/changelog/changelog_10.html#change-1.0.1
if you can confirm with current master that this is fixed I can release
today or tomorrow as this particular regression is fairly severe.
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
Re: [sqlalchemy] Non backwards-compatible changes in 1.0? Lots of suddenly failing tests here.
On 4/20/15 8:09 AM, Guido Winkelmann wrote:
Hi,
Have there been any non-backwards-compatible changes in SQLAlchemy 1.0
compared to 0.9.9?
We are seeing a lot of sudden breakage in our unit tests when
switching to SQLAlchemy 1.0 from 0.9.9. Tests that worked fine before
suddenly fail across the board.
Here's a an example of a test build that suddenly failed on 1.0:
https://travis-ci.org/pyfarm/pyfarm-master/builds/58860924
If you compare the builds on sqlite with those MySQL/PostgreSQL, you
will see there are two different, seemingly unrelated things going wrong:
On sqlite, drop_all() seems to fail to get the order of table drops
right, and consequently runs into a referential integrity error.
On MySQL/PostgreSQL, this line fails:
association = TaskTaskLogAssociation.query.filter_by(task=task,
log=task_log, attempt=attempt).first()
In this context, log is a relationship in the
model TaskTaskLogAssociation to model TaskLog. task_log is an object
of type TaskLog, but one that has never been written to the database
and has no set id. That leads to this error message in the logs:
nose.proxy.ProgrammingError: (psycopg2.ProgrammingError) function
symbol(unknown) does not exist
LINE 3: ...72015052936_task_log_associations.attempt = 1 AND symbol('NE...
^
HINT: No function matches the given name and argument types. You
might need to add explicit type casts.
[SQL: 'SELECT test29172015052936_task_log_associations.task_log_id AS
test29172015052936_task_log_associations_task_log_id,
test29172015052936_task_log_associations.task_id AS
test29172015052936_task_log_associations_task_id,
test29172015052936_task_log_associations.attempt AS
test29172015052936_task_log_associations_attempt,
test29172015052936_task_log_associations.state AS
test29172015052936_task_log_associations_state \nFROM
test29172015052936_task_log_associations \nWHERE
test29172015052936_task_log_associations.attempt = %(attempt_1)s AND
%(param_1)s = test29172015052936_task_log_associations.task_log_id AND
%(param_2)s = test29172015052936_task_log_associations.task_id \n
LIMIT %(param_3)s'] [parameters: {'param_1': symbol('NEVER_SET'),
'attempt_1': 1, 'param_2': 1, 'param_3': 1}]
Apparently, sqlalchemy will use symbol('NEVER_SET') where the id of
the model used for filtering should be.
It may be a bit questionable to filter by a model that doesn't even
exist in the database, but, again, this used to work fine in 0.9.9.
This is odd. What was working fine in 0.9.9 doing exactly? Was it
coming out with NULL =
test29172015052936_task_log_associations.task_log_id ? Looking in
0.9, there is no logic in this case to convert the = to IS in this
case as the parameter from the object is not evaluated til after the
query is generated.This query will *always* return False, because
NULL cannot be compared with =.
I guess that's whats desired here, that the query returns nothing, but
this is a lot like the idea of x IN (), e.g. it's useless to emit this
query, and it relies upon kind of a weird quirk of SQL. I almost wonder
if this should emit a warning. Because if we do eventually make it so
that IS NULL comes out, the results can change for more complex
relationships that explicitly want to compare some columns to NULL.
Regards,
Guido W.
--
You received this message because you are subscribed to the Google
Groups sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send
an email to sqlalchemy+unsubscr...@googlegroups.com
mailto:sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com
mailto:sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
sqlalchemy group.
To unsubscribe from this group and stop receiving emails from it, send an email
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at http://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
