On Wed, Aug 29, 2018 at 2:35 AM, Mohit Agarwal <[email protected]> wrote:
> Hi Mike, > Thanks for replying. > > The code snipped as mentioned before is - > try: > obj = Obj() > session.add(obj) > session.commit() -> this will take a lot of time in some cases (since we > have a trigger on insert of this entity which is suboptimal for certain use > cases) > except Exception as e: > session.rollback() > logger.exception(e) > raise e > > > > Unfortunately before we log the exception trace, we do session.rollback() > which leads to a simple AssertionError on the line "*assert > self._is_transaction_boundary" *in sqlalchemy library > > > From my server logs though, i can provide high level logs - > "time": 1535505375696, "line": "[2018-08-29 01:16:15 +0000] [1] > [CRITICAL] WORKER TIMEOUT (pid:12)", "host": "logentries-d0rqp" } > "time": 1535505375698, "line": "/usr/local/lib/python2.7/ > dist-packages/sqlalchemy/orm/session.py:434: SAWarning: Session's state > has been changed on a non-active transaction - this state will be > discarded.", "host": "logentries-d0rqp" } > "time": 1535505375699, "line": " \"Session's state has been changed on > \"", "host": "logentries-d0rqp" } > > > "time": 1535505375700, "line": do_create\\n db.session.rollback()\\n File > \"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/scoping.py\", > line 150, in do\\n return getattr(self.registry(), name)(*args, > **kwargs)\\n File > \"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py\", > line 754, in rollback\\n self.transaction.rollback()\\n File > \"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py\", > line 437, in rollback\\n > boundary._restore_snapshot(dirty_only=boundary.nested)\\n > File \"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py\", > line 273, in _restore_snapshot\\n assert > self._is_transaction_boundary\\nAssertionError\\n'", > "host": "logentries-d0rqp" } > > So this could have happened - > 1. We have gunicorn worker timeout set at 30 seconds. So worker was killed > by gunicorn after waiting for 30 seconds for api call to respond. > 2. this may trigger some session teardown on sqlalchemy. We use > Flask-SQLAlchemy==2.0 (this is just a theory, i am trying to find out > documents around this) > 3. some exception happened in the ongoing query because of that. (not able > to find out the exception trace, since it is logged after > session.rollback()) > 4. a rollback in our except block then lead to this error. > > Hope this helps. > > Meanwhile, I am working on adding logs and also optimizing our trigger. > are you using gevent monkeypatching in order to be compatible with gunicorn? if a gevent worker is killed, SQLAlchemy definitely does not clean up correctly at all, you at least need to be on 1.1 because you need this: http://docs.sqlalchemy.org/en/latest/changelog/migration_11.html#change-3803 > > Thanks > Mohit > > On Tue, Aug 28, 2018 at 6:44 PM Mike Bayer <[email protected]> > wrote: > >> >> >> On Tue, Aug 28, 2018 at 7:56 AM, Mohit Agarwal <[email protected]> >> wrote: >> >>> I will also like to know is there anytime out setting to sql alchemy >>> query because of which it is raising an exception after 30 seconds and >>> going into the except block. We have not explicitly passed >>> statement_timeout in our implementation. >>> >> >> >> I would need to see the complete error message you are getting so I can >> google it. SQLAlchemy itself has no concept of timeouts, this is >> something that happens either at the driver level or in the server side >> configuration of your database. >> >> >> >> >> >> >>> >>> On Tuesday, August 28, 2018 at 5:20:39 PM UTC+5:30, Mohit Agarwal wrote: >>>> >>>> Hi, >>>> We are seeing exceptions:AssertionError being raised when of our APIs >>>> has a long running query. In code we are rolling back transaction if any >>>> error is received while committing. Basically we have this general wrapper >>>> >>>> try: >>>> session.commit() >>>> except Exception as e: >>>> session.rollback() >>>> >>> raise e >>> >>>> >>>> >>>> >>>> Our sql alchemy version - 1.0.6 >>>> Our database - Azure SQL (sql server) >>>> >>>> >>>> Stack trace - >>>> File "/code/api/named_location/resources.py", line 258, in >>>> create_named_locations_dataclass >>>> File "/code/api/named_location/operations.py", line 94, in do_create >>>> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/scoping.py", >>>> line 150, in do >>>> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", >>>> line 754, in rollback >>>> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", >>>> line 437, in rollback >>>> File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", >>>> line 273, in _restore_snapshot >>>> >>>> From the code it looks like it fails here - >>>> def _restore_snapshot(self, dirty_only=False): >>>> *assert self._is_transaction_boundary* >>>> What does it mean, why rollback is failing ? >>>> >>>> >>>> >>>> Thanks >>>> Mohit >>>> >>> -- >>> SQLAlchemy - >>> The Python SQL Toolkit and Object Relational Mapper >>> >>> http://www.sqlalchemy.org/ >>> >>> To post example code, please provide an MCVE: Minimal, Complete, and >>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full >>> description. >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "sqlalchemy" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/sqlalchemy. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> SQLAlchemy - >> The Python SQL Toolkit and Object Relational Mapper >> >> http://www.sqlalchemy.org/ >> >> To post example code, please provide an MCVE: Minimal, Complete, and >> Verifiable Example. See http://stackoverflow.com/help/mcve for a full >> description. >> --- >> You received this message because you are subscribed to the Google Groups >> "sqlalchemy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/sqlalchemy. >> For more options, visit https://groups.google.com/d/optout. >> > -- > SQLAlchemy - > The Python SQL Toolkit and Object Relational Mapper > > http://www.sqlalchemy.org/ > > To post example code, please provide an MCVE: Minimal, Complete, and > Verifiable Example. See http://stackoverflow.com/help/mcve for a full > description. > --- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/sqlalchemy. > For more options, visit https://groups.google.com/d/optout. > -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
