Automap was, at the most regular, only ever used upon connection creation
to a given database. (*obvious implication is that DB changes within a
connection might be problematic? not an applicable scenario thus far in our
code*)
Subsequent queries through that connection did *not* automap.
Still, my application would build new connections somewhat regularly.
Because of that expense you referred to I decided to cache the mapping
results in the global space to avoid performing that operation over and
over. As far as I know I don't thin that's a component of this problem.
Here's the gist of my connection initiation code that's run upon
instantiation of the DB class.
Mind you, a prior coworker wrote this and there's so many ways of
interfacing with SQLAlchemy that I'm not even sure where he got it from.
db_url = engine.url.URL(drivername = self.drivername,
> username = self.username,
> password = self.password,
> host = self.host,
> database = self.database,
> query = {'charset':'utf8'})
> self.engine =
> create_engine(db_url,encoding='utf8',convert_unicode=True)
> self.session = Session(self.engine)
> self.connection = self.engine.connect()
>
# Automap functionality has a lot of overhead, so cache the
> results on a per-host/database basis
> id = (self.drivername,self.host,self.database)
> if id not in DB.tables:
> base = automap_base()
> base.prepare(self.engine, reflect=True,
> name_for_scalar_relationship=name_for_scalar_relationship)
> DB.tables[id] = base.classes
> self.tables = DB.tables[id]
On Thu, May 24, 2018 at 5:50 PM, Mike Bayer <[email protected]>
wrote:
> On Thu, May 24, 2018 at 5:40 PM, Dave Mittner <[email protected]>
> wrote:
> > Also posted here:
> > https://stackoverflow.com/questions/50123090/
> application-process-unusable-after-cant-proceed-with-initialization-of-o
> >
> >
> > I have a multithreaded application that runs various jobs in threads.
> One of
> > these jobs goes out to various data sources to query for data. On
> occasion
> > the mapping process fails and an exception is thrown.
> >
> > That on its own isn't a big deal; my system is designed to compensate for
> > periodically failing jobs.
> >
> > The problem is that that mapping failure seems to be recorded in a global
> > space that then prevents all future mapping attempts to be aborted. Even
> > attempts on completely different threads using completely different
> > databases. This renders my entire application effectively broken from
> that
> > point on.
> >
> > After looking in SQLAlchemy's code, mappers are stored in a
> _mapper_registry
> > global space variable and once any mapper in the registry errors out, any
> > attempt to configure a new mapper will fail.
> >
> > Mapping failures of this nature may be rare -- and indeed it only rarely
> > happens on the connection I'm having a problem with -- but this complete
> > locking behavior of all future mapping seems very odd to me. If there
> isn't
> > a way around this I might have no choice but to have my process
> completely
> > exit when the exception is encountered, even if that means killing other
> > running threads.
>
> are you creating mappers on the fly or on a per-request basis? You'd
> want to ideally have mappings created just once at the module import
> level. Then when your application is ready to start up, call
> configure_mappers() and everything will be set up.
>
> if those are not patterns you're able to use, then please provide more
> specifics. from your stack trace on SO, it seems like you are using
> automap. When is that running? If per request, this very expensive
> and will have problems.
>
> The mapping process *is* guarded by a mutex so it is difficult to
> produce an issue with mappings failing - the stack trace you post
> almost appears like there is some kind of naming issue happening where
> a particular mapper has been garbage collected or something like that
> yet still being referred towards by other mappers that are being
> configured. need to see details of how your code works.
>
>
>
> >
> > Any ideas?
> >
> > --
> > SQLAlchemy -
> > The Python SQL Toolkit and Object Relational Mapper
> >
> > http://www.sqlalchemy.org/
> >
> > To post example code, please provide an MCVE: Minimal, Complete, and
> > Verifiable Example. See http://stackoverflow.com/help/mcve for a full
> > description.
> > ---
> > You received this message because you are subscribed to the Google Groups
> > "sqlalchemy" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [email protected].
> > To post to this group, send email to [email protected].
> > Visit this group at https://groups.google.com/group/sqlalchemy.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> SQLAlchemy -
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
> description.
> ---
> You received this message because you are subscribed to the Google Groups
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at https://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.
>
--
SQLAlchemy -
The Python SQL Toolkit and Object Relational Mapper
http://www.sqlalchemy.org/
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable
Example. See http://stackoverflow.com/help/mcve for a full description.
---
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
