On Sun, May 27, 2018 at 1:34 PM, Dave Mittner <[email protected]> wrote:
>> so the stack trace shows that your application has *other* mappings in
>> it that are *not* coming from this automap block, which I assumed was
>> not the case. I continue to have a very vague idea of the scope and
>> design of this application so I have to guess a lot.
>
>
> How are you figuring that?
> Every MySQL connection is coming strictly through the DB class
> I also have an Oracle connection that's established separately, but it's not
> doing any mapping at all (or shouldn't be) and only has a single SELECT to a
> stored procedure.
Upon further thought you're right, this is the segment of the trace in question:
File "/home/dmittner/loci2/app/api/mod_system/models.py", line 2281,
in update_organizations_from_oracle
party_name=data['party_name']
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/session.py",
line 1909, in merge
_resolve_conflict_map=_resolve_conflict_map)
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/session.py",
line 2019, in _merge
for prop in mapper.iterate_properties:
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/mapper.py",
line 1920, in iterate_properties
configure_mappers()
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/mapper.py",
line 3013, in configure_mappers
mapper._post_configure_properties()
File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/mapper.py",
line 1811, in _post_configure_properties
prop.init()
which said to me, "configure_mappers() is finding unconfigured mappers
to configure", but that's the bug, that automap is still working while
this runs so a configure_mappers() call that should find nothing finds
unfinished work.
the final step is to confirm that the _CONFIGURE_MUTEX fix I gave you,
around the .prepare() step, fixes the problem. The other mutex
logic that is there is not strictly necessary but still helpful to
prevent your program from running multple automaps against the same
cache id.
>
> I'll try the provided code.
>
> On Sun, May 27, 2018 at 9:31 AM, Mike Bayer <[email protected]>
> wrote:
>>
>> so the stack trace shows that your application has *other* mappings in
>> it that are *not* coming from this automap block, which I assumed was
>> not the case. I continue to have a very vague idea of the scope and
>> design of this application so I have to guess a lot.
>>
>> What we will do is absolultely prevent automap from having a
>> non-mapped class in play while any part of the process might call
>> configure mappers. This could also be patched into automap.
>>
>> Add this import to the top:
>>
>> from sqlalchemy.orm.mapperlib import _CONFIGURE_MUTEX
>>
>>
>> Change the prepare line to be like this:
>>
>>
>> _CONFIGURE_MUTEX.acquire()
>> try:
>> base.prepare(
>>
>> name_for_scalar_relationship=name_for_scalar_relationship)
>> finally:
>> _CONFIGURE_MUTEX.release()
>>
>>
>> This will prevent any configuration from occurring while automap
>> creates classes and maps them. if that fixes the problem I can add
>> this to automap.prepare().
>>
>>
>>
>>
>>
>> On Sun, May 27, 2018 at 2:59 AM, Dave Mittner <[email protected]>
>> wrote:
>> > Well, thus far the locking/collecting solution doesn't seem to be doing
>> > the
>> > trick.
>> >
>> > Here's my revised code. Note that I haven't instituted any specified
>> > table
>> > exclusion, and my caching is temporarily disabled. Also note I made the
>> > automap_mutex a global as a static attribute on the DB class. If it's
>> > defined per-thread then the lock only applies for that thread, which
>> > kind of
>> > defeated the purpose.
>> >
>> >> db_url = engine.url.URL(drivername = self.drivername,
>> >> username = self.username,
>> >> password = self.password,
>> >> host = self.host,
>> >> database = self.database,
>> >> query = {'charset':'utf8'})
>> >> self.engine =
>> >> create_engine(db_url,encoding='utf8',convert_unicode=True)
>> >> self.session = Session(self.engine)
>> >> self.connection = self.engine.connect()
>> >> id = (self.drivername,self.host,self.database)
>> >> app.logger.debug('Locking')
>> >> DB.automap_mutex.acquire()
>> >> app.logger.debug('Locked')
>> >> try:
>> >> if id not in DB.tables:
>> >> base = automap_base()
>> >> try:
>> >> base.prepare(
>> >> self.engine,
>> >> reflect = True,
>> >> name_for_scalar_relationship =
>> >> name_for_scalar_relationship
>> >> )
>> >> configure_mappers()
>> >> except:
>> >> del base
>> >> gc.collect()
>> >> raise
>> >> else:
>> >> self.tables = base.classes
>> >> #DB.tables[id] = base.classes
>> >> finally:
>> >> app.logger.debug('Unlocking')
>> >> DB.automap_mutex.release()
>> >> app.logger.debug('Unlocked')
>> >> #self.tables = DB.tables[id]
>> >
>> >
>> > The locking does appear to be working. I can tell that much from the log
>> > outputs. Here are dual threads trying to establish connections within
>> > close
>> > proximity of each other.
>> >
>> > [2018-05-26 23:16:00,024] [23223/139778217465600] [DEBUG] CORE Locking
>> > [2018-05-26 23:16:00,025] [23223/139778217465600] [DEBUG] CORE Locked
>> > [2018-05-26 23:16:00,026] [23223/139777630271232] [DEBUG] CORE Locking
>> > [2018-05-26 23:16:00,392] [23223/139778217465600] [DEBUG] CORE Unlocking
>> > [2018-05-26 23:16:00,392] [23223/139778217465600] [DEBUG] CORE Unlocked
>> > [2018-05-26 23:16:00,392] [23223/139777630271232] [DEBUG] CORE Locked
>> > [2018-05-26 23:16:00,800] [23223/139777630271232] [DEBUG] CORE Unlocking
>> > [2018-05-26 23:16:00,800] [23223/139777630271232] [DEBUG] CORE Unlocked
>> >
>> > But the locking isn't preventing the initial "UnmappedClassError", nor
>> > the
>> > subsequent and repeating "InvalidRequestError: One or more mappers
>> > failed to
>> > initialize" exceptions.
>> >
>> >> Traceback (most recent call last):
>> >> File "/home/dmittner/loci2/app/jobs.py", line 507, in start
>> >> result = self.callable(self,**self.parameters)
>> >> File "/home/dmittner/loci2/app/jobs.py", line 406, in
>> >> job_update_organizations
>> >> Core(db=job.db).update_organizations_from_oracle()
>> >> File "/home/dmittner/loci2/app/api/mod_system/models.py", line 2281,
>> >> in
>> >> update_organizations_from_oracle
>> >> party_name=data['party_name']
>> >> File
>> >> "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/session.py",
>> >> line 1909, in merge
>> >> _resolve_conflict_map=_resolve_conflict_map)
>> >> File
>> >> "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/session.py",
>> >> line 2019, in _merge
>> >> for prop in mapper.iterate_properties:
>> >> File
>> >> "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/mapper.py",
>> >> line 1920, in iterate_properties
>> >> configure_mappers()
>> >> File
>> >> "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/mapper.py",
>> >> line 3013, in configure_mappers
>> >> mapper._post_configure_properties()
>> >> File
>> >> "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/mapper.py",
>> >> line 1811, in _post_configure_properties
>> >> prop.init()
>> >> File
>> >> "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/interfaces.py",
>> >> line
>> >> 184, in init
>> >> self.do_init()
>> >> File
>> >>
>> >> "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/relationships.py",
>> >> line 1655, in do_init
>> >> self._process_dependent_arguments()
>> >> File
>> >>
>> >> "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/relationships.py",
>> >> line 1712, in _process_dependent_arguments
>> >> self.target = self.mapper.mapped_table
>> >> File
>> >>
>> >> "/usr/local/lib/python3.4/dist-packages/sqlalchemy/util/langhelpers.py",
>> >> line 767, in __get__
>> >> obj.__dict__[self.__name__] = result = self.fget(obj)
>> >> File
>> >>
>> >> "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/relationships.py",
>> >> line 1634, in mapper
>> >> configure=False)
>> >> File "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/base.py",
>> >> line 426, in class_mapper
>> >> raise exc.UnmappedClassError(class_)
>> >> sqlalchemy.orm.exc.UnmappedClassError: Class
>> >> 'sqlalchemy.ext.automap.geo_country_map_data' is not mapped
>> >
>> >
>> >
>> >> Traceback (most recent call last):
>> >> File "/home/dmittner/loci2/app/jobs.py", line 507, in start
>> >> result = self.callable(self,**self.parameters)
>> >> File "/home/dmittner/loci2/app/jobs.py", line 434, in
>> >> job_update_geo_database
>> >> Core(db=job.db).update_geo()
>> >> File "/home/dmittner/loci2/app/api/mod_system/models.py", line 4018,
>> >> in
>> >> update_geo
>> >> database = app.config['GEO_SQL_DB']
>> >> File "/home/dmittner/loci2/app/api/mod_system/models.py", line 790,
>> >> in
>> >> __init__
>> >> configure_mappers()
>> >> File
>> >> "/usr/local/lib/python3.4/dist-packages/sqlalchemy/orm/mapper.py",
>> >> line 3010, in configure_mappers
>> >> raise e
>> >> sqlalchemy.exc.InvalidRequestError: One or more mappers failed to
>> >> initialize - can't proceed with initialization of other mappers.
>> >> Triggering
>> >> mapper: 'Mapper|geo_country_label_data|geo_country_label_data'.
>> >> Original
>> >> exception was: Class 'sqlalchemy.ext.automap.geo_country_map_data' is
>> >> not
>> >> mapped
>> >
>> >
>> > Based on the above code I really don't think I'd have any references to
>> > anything persisting. My only thought was that maybe the engine is being
>> > given a reference when it's passed into prepare(), but I didn't notice
>> > anything like that when I glanced at the source code.
>> >
>> >
>> > On Sat, May 26, 2018 at 3:47 PM, Mike Bayer <[email protected]>
>> > wrote:
>> >>
>> >>
>> >>
>> >> On Sat, May 26, 2018, 1:49 PM Dave Mittner <[email protected]>
>> >> wrote:
>> >>>>
>> >>>> # 3. configure now, instead of waiting to do it during a query.
>> >>>
>> >>>
>> >>> That right there is the reason this problem seems so hard to track
>> >>> down.
>> >>> I wasn't seeing any error at connection-time, but rather at query
>> >>> time. And
>> >>> that query could be triggered on any connection, in any area of the
>> >>> code. In
>> >>> theory it wouldn't even necessarily happen at the same time that
>> >>> connections
>> >>> are being established. So I'd get an error at Point A when the root
>> >>> cause
>> >>> actually tracks back to Point B, and this was made possibly because of
>> >>> the
>> >>> use of the global _mapper_registry; it enabled a problem based on one
>> >>> connection to trigger an error in another connection.
>> >>>
>> >>> So by locking threading and triggering the error via
>> >>> configure_mappers(),
>> >>> I'm able to see the error within the proper connection context.
>> >>>
>> >>> I guess what I don't understand is why you do the mapping at
>> >>> query-time
>> >>> and not connection-time.
>> >>
>> >>
>> >> Well this is not "the mapping", this is the point at which the
>> >> disparate
>> >> mappers all over the application are configured against each other,
>> >> namely
>> >> the relationships defined between them as well as any event handlers
>> >> that
>> >> work post-configure. The the configure_mappers() step is called
>> >> automatically both when a query is built up as well as when a mapped
>> >> object
>> >> is constructed. Both of these things happen before any database
>> >> connection
>> >> is necessarily present.
>> >>
>> >> It is not uncommon that an application that has a well-defined point at
>> >> which things start up would call configure_mappers explicitly, which
>> >> fits
>> >> with what you're doing. SQLAlchemy could also have decided to make
>> >> this
>> >> explicitness required, but we've made it automatic and usually with no
>> >> problems. But the case I think you've found here warrants some
>> >> consideration, since automap does create mappings within a function,
>> >> rather
>> >> than at module import time as is more typical.
>> >>
>> >>
>> >>
>> >>> With the "base.classes" attribute being immediately available after
>> >>> base.prepare, I believed the mapping was already being done at
>> >>> connection-time, and therefor there should have been an exception
>> >>> throwing
>> >>> then, and this problem wouldn't exist. Of course you might also need
>> >>> to
>> >>> detect if threading is being used and lock it, but this could all be
>> >>> behavior triggered by a threading=True parameter on the "base.prepare"
>> >>> call.
>> >>>
>> >>> Anyway, I'll apply these conventions and see how it goes.
>> >>>
>> >>> On Sat, May 26, 2018 at 9:44 AM, Mike Bayer <[email protected]>
>> >>> wrote:
>> >>>>
>> >>>> On Sat, May 26, 2018 at 12:19 AM, Dave Mittner
>> >>>> <[email protected]>
>> >>>> wrote:
>> >>>> >> Can you elaborate on this system? There's some open source code
>> >>>> >> that
>> >>>> >> creates SQLAlchemy mappings which you have no control over ?
>> >>>> >> What
>> >>>> >> do
>> >>>> >> you mean "on the database" ?
>> >>>> >
>> >>>> >
>> >>>> > I mean something must be happening on the database server, itself,
>> >>>> > causing
>> >>>> > SQLAlchemy's automap process to fail. Or there's a network problem.
>> >>>> > Or
>> >>>> > somewhere else in the line. I can say with certainty that there are
>> >>>> > no
>> >>>> > conditions in my application that vary from connection to
>> >>>> > connection.
>> >>>> > Every
>> >>>> > attempt to connect in my code is identical, yet sometimes it fails.
>> >>>> >
>> >>>> > And even if there were a race condition causing a mapping to fail,
>> >>>> > that
>> >>>> > still shouldn't prevent all future attempts by the application to
>> >>>> > automap
>> >>>> > other databases. This is the inherent problem of using the global
>> >>>> > space --
>> >>>> > it affects the entire application and not just one instantiation of
>> >>>> > a
>> >>>> > class.
>> >>>> > Each connection/engine isn't isolated from each other, so a failure
>> >>>> > on
>> >>>> > any
>> >>>> > one can impact others. A bug in one module of an application can
>> >>>> > shut
>> >>>> > down
>> >>>> > the entire application indefinitely.
>> >>>> >
>> >>>> > That brings us to this comment you made in one of your examples:
>> >>>> >
>> >>>> >> the program is not expected to be able to run with bad mappings in
>> >>>> >> it.
>> >>>> >
>> >>>> >
>> >>>> > But why not just allow exceptions to serve their purpose? If you
>> >>>> > throw
>> >>>> > an
>> >>>> > exception when a mapping fails, rather than keeping it stored in
>> >>>> > _mapper_registry in a failed state, then other connections are free
>> >>>> > to
>> >>>> > function. A failed mapping will only cause issues with the one
>> >>>> > connection
>> >>>> > and only with other mappings that reference it. My application, for
>> >>>> > example,
>> >>>> > has no such mappings that would fail, yet my application is unable
>> >>>> > to
>> >>>> > proceed with other functions due to how the mapping failure is
>> >>>> > handled.
>> >>>>
>> >>>>
>> >>>> I understand the temptation to see that this part of the mapping
>> >>>> system uses a global registry and then jump ahead to the common
>> >>>> advice
>> >>>> that "global variables are bad", and therefore this is an
>> >>>> inappropriate architecture. However, there is a context, a history,
>> >>>> and a purpose for why the system is architected this way, and I've
>> >>>> shared an example program I wrote to illustrate one particular reason
>> >>>> why this is, however I didn't provide a full narrative explanation.
>> >>>>
>> >>>> I'll also note again that automap itself is not the standard way
>> >>>> SQLAlchemy was intended to be used, and if the ORM were entirely
>> >>>> focused on automap from the ground up, the mapper configuration
>> >>>> process would likely work differently. In particular, your
>> >>>> references to "connections/engines not isolated from each other"
>> >>>> illustrates a misunderstanding that I am assuming originates from
>> >>>> looking at things in an "automap" perspective, because there is not
>> >>>> actually any relationship between a SQLAlchemy engine and a class
>> >>>> mapping in any way whatsoever. A network error cannot cause the
>> >>>> problem you describe, for example, because the reflection process
>> >>>> completes fully before any mappers are constructed. automap itself
>> >>>> does not anticipate a concurrency use case as you are doing, so at
>> >>>> the
>> >>>> very least the documentation needs to include examples on how to
>> >>>> accommodate for this use case which likely would have prevented this
>> >>>> issue from happening in the first place.
>> >>>>
>> >>>> I'll reiterate as well that the reflection process here can be
>> >>>> tailored to exclude tables you don't care about, and that you can
>> >>>> remove failed mappers from _mapper_registry as demonstrated in a
>> >>>> second example program I've shared, however I perhaps have not stated
>> >>>> clearly enough that this should be within the scope of a cleanup
>> >>>> process that you take when mappings fail. That your program starts
>> >>>> to work eventually is very likely because Python's garbage collector
>> >>>> eventually runs. My suggestion is to run it immediately in the
>> >>>> failure case.
>> >>>>
>> >>>> Within the scope of suggestions I've made like using mutexes,
>> >>>> limiting
>> >>>> the reflection process, and using garbage collection, these are all
>> >>>> efforts to provide things you can try to help solve your problem. In
>> >>>> an effort to make this as easy as possible, below is the snippet of
>> >>>> code you've given me with these techniques integrated, including
>> >>>> comments with my latest thinking on what exactly the problem you are
>> >>>> seeing likely is. I hope that you can work with these techniques to
>> >>>> see if it solves your issue.
>> >>>>
>> >>>> from sqlalchemy.orm import configure_mappers
>> >>>> import threading
>> >>>> import gc
>> >>>>
>> >>>> automap_mutex = threading.Lock()
>> >>>>
>> >>>> def automap_mapping_thing():
>> >>>> db_url = engine.url.URL(drivername = self.drivername,
>> >>>> username = self.username,
>> >>>> password = self.password,
>> >>>> host = self.host,
>> >>>> database = self.database,
>> >>>> query = {'charset':'utf8'})
>> >>>> self.engine =
>> >>>> create_engine(db_url,encoding='utf8',convert_unicode=True)
>> >>>> self.session = Session(self.engine)
>> >>>>
>> >>>> self.connection = self.engine.connect()
>> >>>>
>> >>>> # Automap functionality has a lot of overhead, so cache the
>> >>>> results on a per-host/database basis
>> >>>> id = (self.drivername,self.host,self.database)
>> >>>>
>> >>>> try:
>> >>>> # look in the dictionary. 99% of the time the id will be
>> >>>> # present, since this is a cache, so don't waste time with
>> >>>> 'in'
>> >>>> check
>> >>>> self.tables = DB.tables[id]
>> >>>> except KeyError:
>> >>>> # more expensive KeyError catch for the 1% of the time the id
>> >>>> # isn't there. also allows the check for the key to be more
>> >>>> or less atomic
>> >>>>
>> >>>> # step 1, lock. The race you are getting is, one automap
>> >>>> process
>> >>>> # has created a class but not mapped it yet, while some other
>> >>>> thread
>> >>>> # calls configure_mappers() which hits it. automap does not
>> >>>> # anticipate this use case right now.
>> >>>> automap_mutex.acquire()
>> >>>>
>> >>>> # try/finally block for the mutex
>> >>>> try:
>> >>>>
>> >>>> # 1. since we are mutexing, multiple threads may have
>> >>>> reached
>> >>>> # here, and we might not be the first, so, check if
>> >>>> someone
>> >>>> already
>> >>>> # did the work:
>> >>>>
>> >>>> if id in DB.tables:
>> >>>> return
>> >>>>
>> >>>> # 2. reflect tables, exclude tables you don't need
>> >>>> def exclude_tables_we_dont_need(table_name, metadata):
>> >>>> return table_name not in (
>> >>>> 'geo_state_map_data', 'other_table', 'etc')
>> >>>>
>> >>>> # or, inclusive
>> >>>> # return table_name in (
>> >>>> # 'table_i_care_about',
>> >>>> 'other_table_i_care_about')
>> >>>>
>> >>>> # do the reflection
>> >>>> base = automap_base()
>> >>>> base.metadata.reflect(
>> >>>> self.connection, only=exclude_tables_we_dont_need)
>> >>>>
>> >>>> # try/except block for ensuring bad mappers are removed
>> >>>> try:
>> >>>> # 2. prepare mappings from reflected tables.
>> >>>> base.prepare(
>> >>>>
>> >>>> name_for_scalar_relationship=name_for_scalar_relationship)
>> >>>>
>> >>>> # 3. configure now, instead of waiting to do it
>> >>>> during a
>> >>>> query.
>> >>>> configure_mappers()
>> >>>> except:
>> >>>>
>> >>>> # 4. something went wrong w/ the prepare or
>> >>>> configure.
>> >>>> # clean up
>> >>>>
>> >>>> log.error("Error occurred", exc_info=True)
>> >>>>
>> >>>> # remove all the new mappers we just made from memory
>> >>>> del base
>> >>>>
>> >>>> # gc collect. _mapper_registry will be emptied of
>> >>>> these
>> >>>> # mappers
>> >>>> gc.collect()
>> >>>>
>> >>>> # re-raise, or not. whatever you want to do.
>> >>>> raise
>> >>>> else:
>> >>>> # fully mapped classes ready to go.
>> >>>> DB.tables[id] = base.classes
>> >>>> finally:
>> >>>> # leave critical section
>> >>>> automap_mutex.release()
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> >
>> >>>> >
>> >>>> > In any case, I have my application attempting the mapping anew
>> >>>> > every
>> >>>> > minute
>> >>>> > and it's gone for a few hours now without the problem being
>> >>>> > triggered.
>> >>>> > I'm
>> >>>> > really just shooting in the dark here, though. I'm hoping that I
>> >>>> > can
>> >>>> > find
>> >>>> > something else going on at the same time as the failure to explain
>> >>>> > what's
>> >>>> > triggering it, but I'm not hopeful. And without knowing how it's
>> >>>> > being
>> >>>> > triggered, it's nigh impossible to intentionally trigger it to test
>> >>>> > fixes or
>> >>>> > bypasses. That's why my focus has been on the underlying conceptual
>> >>>> > issue of
>> >>>> > SQLAlchemy's error handling method blocking things altogether. And
>> >>>> > conceptually, I don't think it should matter what causes an error;
>> >>>> > as
>> >>>> > a
>> >>>> > matter of principle I think the existing mechanism is presumptuous
>> >>>> > in
>> >>>> > not
>> >>>> > allowing any subsequent mappers from working.
>> >>>> >
>> >>>> >
>> >>>> > On Fri, May 25, 2018 at 2:54 PM, Mike Bayer
>> >>>> > <[email protected]>
>> >>>> > wrote:
>> >>>> >>
>> >>>> >> On Fri, May 25, 2018 at 3:21 PM, Dave Mittner
>> >>>> >> <[email protected]>
>> >>>> >> wrote:
>> >>>> >> >> This statement does not make sense without more information.
>> >>>> >> >> automap
>> >>>> >> >> either succeeds against a given database, or it fails. It is
>> >>>> >> >> not
>> >>>> >> >> easy
>> >>>> >> >> to produce a situation where it fails "only rarely". Things
>> >>>> >> >> that
>> >>>> >> >> could cause this are: 1. the database you are automapping
>> >>>> >> >> against
>> >>>> >> >> is
>> >>>> >> >> constantly changing, 2. the mapping is relying upon some
>> >>>> >> >> subtle
>> >>>> >> >> hash
>> >>>> >> >> ordering or 3. automap is being used in a multi-threaded
>> >>>> >> >> context
>> >>>> >> >> where
>> >>>> >> >> more than one automap tries to hit the same tables at the same
>> >>>> >> >> time.
>> >>>> >> >> When something happens "only rarely", that points to race
>> >>>> >> >> conditions
>> >>>> >> >> or sometimes hash ordering issues. To rule out #2, you
>> >>>> >> >> would
>> >>>> >> >> create a test script that does your automap by itself and try
>> >>>> >> >> to
>> >>>> >> >> run
>> >>>> >> >> automap hundreds of times against the particular database that
>> >>>> >> >> caused
>> >>>> >> >> a failure, if under Python 2 you'd want to set
>> >>>> >> >> PYTHONHASHSEED='random'
>> >>>> >> >> in the environment. But the error you showed looks more like
>> >>>> >> >> a
>> >>>> >> >> race
>> >>>> >> >> of two automaps hitting the same table.
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > That's my point though, if there's a condition causing the
>> >>>> >> > failure,
>> >>>> >> > it's
>> >>>> >> > not
>> >>>> >> > in my code. It's on the database. A third party system.
>> >>>> >>
>> >>>> >> Can you elaborate on this system? There's some open source code
>> >>>> >> that
>> >>>> >> creates SQLAlchemy mappings which you have no control over ?
>> >>>> >> What
>> >>>> >> do
>> >>>> >> you mean "on the database" ?
>> >>>> >>
>> >>>> >>
>> >>>> >> >
>> >>>> >> > It comes down to this: SQLAlchemy doesn't handle that kind of
>> >>>> >> > problem
>> >>>> >> > gracefully.
>> >>>> >>
>> >>>> >> I love to support new use cases if I can get a complete definition
>> >>>> >> of
>> >>>> >> what it is you're trying to do and why my suggestions can't be
>> >>>> >> taken.
>> >>>> >> You showed me the code that is causing the problem, and I
>> >>>> >> suggested
>> >>>> >> using a mutex and I can show you specifically how to fix what
>> >>>> >> might
>> >>>> >> be
>> >>>> >> causing the problem. Your description of this code is "Here's
>> >>>> >> the
>> >>>> >> gist of my connection initiation code that's run upon
>> >>>> >> instantiation
>> >>>> >> of
>> >>>> >> the DB class. Mind you, a prior coworker wrote this and there's so
>> >>>> >> many ways of interfacing with SQLAlchemy that I'm not even sure
>> >>>> >> where
>> >>>> >> he got it from." - now you're saying this is third party code
>> >>>> >> you
>> >>>> >> can't change?
>> >>>> >>
>> >>>> >>
>> >>>> >> > If something happens in an external system that causes problems
>> >>>> >> > in mine, that's something I should be able to handle and move
>> >>>> >> > on.
>> >>>> >> > If the
>> >>>> >> > same problem happens 10,000 times, every time I try it, that's
>> >>>> >> > fine.
>> >>>> >> > I'll
>> >>>> >> > catch the exception, log it, abort that thread, and other
>> >>>> >> > processes
>> >>>> >> > will
>> >>>> >> > continue on. Connections to other databases will still function.
>> >>>> >> >
>> >>>> >> > But that isn't the case here. One failure prevents all future
>> >>>> >> > mapping
>> >>>> >> > operations, even to other databases that would work.
>> >>>> >>
>> >>>> >> if you pull in third party code that places an event within
>> >>>> >> SQLAlchemy
>> >>>> >> against all new mappings, and that code is broken and fails, it
>> >>>> >> will
>> >>>> >> break your entire application. It is not controversial that
>> >>>> >> installing broken code in your applciation in the same process
>> >>>> >> space
>> >>>> >> will break that process altogether. Perhaps I have not been
>> >>>> >> clear,
>> >>>> >> but it is not normal for automap to "fail occasionally". If used
>> >>>> >> correctly, it will not fail. I've offered to help you repair that
>> >>>> >> system.
>> >>>> >>
>> >>>> >>
>> >>>> >> >
>> >>>> >> > Regardless, I don't want the development overhead of having to
>> >>>> >> > maintain
>> >>>> >> > static mappings and I don't have the time or manpower to go back
>> >>>> >> > and
>> >>>> >> > replace
>> >>>> >> > ORM use with more direct query text building,
>> >>>> >>
>> >>>> >> This has not been suggested as your only option.
>> >>>> >>
>> >>>> >> > so I'll probably just dig in
>> >>>> >> > and find a way to undo what SQLAlchemy is doing in the global
>> >>>> >> > space
>> >>>> >> > --
>> >>>> >> > maybe
>> >>>> >> > remove any mappers in _mapper_registry that have
>> >>>> >> > _configure_failed
>> >>>> >> > as
>> >>>> >> > true,
>> >>>> >> > when these exceptions occur, since that seems to be precisely
>> >>>> >> > what's
>> >>>> >> > preventing subsequent attempts from even being tried.
>> >>>> >>
>> >>>> >> That's a system that's been that way for about 9 years, and like
>> >>>> >> any
>> >>>> >> system it certainly can be changed if a new use case is introduced
>> >>>> >> which has no reasonable alternative, but to undertake such a
>> >>>> >> change
>> >>>> >> would need a well defined rationale which I haven't gotten here.
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >> >
>> >>>> >> > On Fri, May 25, 2018 at 6:44 AM, Mike Bayer
>> >>>> >> > <[email protected]>
>> >>>> >> > wrote:
>> >>>> >> >>
>> >>>> >> >> Here's a second demo why _mapper_registry exists and has to be
>> >>>> >> >> global:
>> >>>> >> >>
>> >>>> >> >> from sqlalchemy import *
>> >>>> >> >> from sqlalchemy.orm import *
>> >>>> >> >> from sqlalchemy.ext.declarative import declarative_base
>> >>>> >> >> from sqlalchemy import inspect
>> >>>> >> >>
>> >>>> >> >> Base = declarative_base()
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >> class A(Base):
>> >>>> >> >> __tablename__ = 'a'
>> >>>> >> >>
>> >>>> >> >> id = Column(Integer, primary_key=True)
>> >>>> >> >> data = Column(String)
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >> class B(Base):
>> >>>> >> >> __tablename__ = 'b'
>> >>>> >> >>
>> >>>> >> >> id = Column(Integer, primary_key=True)
>> >>>> >> >> aid = Column(ForeignKey('a.id'))
>> >>>> >> >> data = Column(String)
>> >>>> >> >> a = relationship("A", backref="bs")
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >> e = create_engine("sqlite://", echo=True)
>> >>>> >> >> Base.metadata.create_all(e)
>> >>>> >> >>
>> >>>> >> >> # let's assume SQLAlchemy did not keep a global list of all
>> >>>> >> >> mappers.
>> >>>> >> >> # let's take away "configure_mappers" and instead only
>> >>>> >> >> configure
>> >>>> >> >> mappers
>> >>>> >> >> # as the program refers to them. Patch out configure_mappers
>> >>>> >> >> to
>> >>>> >> >> do
>> >>>> >> >> nothing.
>> >>>> >> >> from sqlalchemy.orm import mapperlib
>> >>>> >> >> mapperlib.configure_mappers = lambda: None
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >> # here's the specific steps configure_mappers() needs to do for
>> >>>> >> >> a
>> >>>> >> >> particular
>> >>>> >> >> # mapper.
>> >>>> >> >> def configure_mapper(cls):
>> >>>> >> >> a_mapper = inspect(cls).mapper
>> >>>> >> >> a_mapper._post_configure_properties()
>> >>>> >> >> a_mapper._expire_memoizations()
>> >>>> >> >> a_mapper.dispatch.mapper_configured(
>> >>>> >> >> a_mapper, a_mapper.class_)
>> >>>> >> >>
>> >>>> >> >> # about to use "A". Let's configure:
>> >>>> >> >> configure_mapper(A)
>> >>>> >> >> a1 = A()
>> >>>> >> >>
>> >>>> >> >> # check the a1.bs collection. Except. There isn't one :)
>> >>>> >> >> mappers
>> >>>> >> >> often
>> >>>> >> >> # contain instructions to add attributes to other mappers. if
>> >>>> >> >> the
>> >>>> >> >> program
>> >>>> >> >> # refers to those dependent mappers first, we need to have
>> >>>> >> >> configured
>> >>>> >> >> # all mappers. if one of the mappers is failing, that's a bug
>> >>>> >> >> in
>> >>>> >> >> the
>> >>>> >> >> program.
>> >>>> >> >> # the program is not expected to be able to run with bad
>> >>>> >> >> mappings
>> >>>> >> >> in
>> >>>> >> >> it.
>> >>>> >> >> assert a1.bs == []
>> >>>> >> >>
>> >>>> >> >> # program has failed
>> >>>> >> >>
>> >>>> >> >> # maybe we could use the "Base" here as the collection, not
>> >>>> >> >> global
>> >>>> >> >> _mapper_registry.
>> >>>> >> >> # but we support relationships and other linkages between
>> >>>> >> >> mappers
>> >>>> >> >> that
>> >>>> >> >> have different
>> >>>> >> >> # "bases", and declarative is only an extension to the core
>> >>>> >> >> mapping
>> >>>> >> >> system in any case.
>> >>>> >> >> # perhaps a "SQLAlchemy 2" can tighten and modernize this
>> >>>> >> >> system
>> >>>> >> >> so
>> >>>> >> >> that all mappers
>> >>>> >> >> # are truly segmented into independent namespaces but this can
>> >>>> >> >> also
>> >>>> >> >> cause more confusion
>> >>>> >> >> # too. Overall, programs just aren't expected to have invalid
>> >>>> >> >> mappings.
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >> a1bs = a1.bs
>> >>>> >> >>
>> >>>> >> >> # about to use "B". Let's configure:
>> >>>> >> >> configure_mapper(B)
>> >>>> >> >> a1bs.append(B())
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >>
>> >>>> >> >> On Fri, May 25, 2018 at 9:24 AM, Mike Bayer
>> >>>> >> >> <[email protected]>
>> >>>> >> >> wrote:
>> >>>> >> >> > Here's a demo, you want to also call gc_collect() when you
>> >>>> >> >> > dispose of
>> >>>> >> >> > your failed mappers:
>> >>>> >> >> >
>> >>>> >> >> > from sqlalchemy import *
>> >>>> >> >> > from sqlalchemy.orm import *
>> >>>> >> >> > from sqlalchemy.ext.declarative import declarative_base
>> >>>> >> >> > from sqlalchemy import exc
>> >>>> >> >> >
>> >>>> >> >> > Base = declarative_base()
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> > class A(Base):
>> >>>> >> >> > __tablename__ = 'a'
>> >>>> >> >> >
>> >>>> >> >> > id = Column(Integer, primary_key=True)
>> >>>> >> >> > data = Column(String)
>> >>>> >> >> > bs = relationship("B")
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> > class B(Base):
>> >>>> >> >> > __tablename__ = 'b'
>> >>>> >> >> >
>> >>>> >> >> > id = Column(Integer, primary_key=True)
>> >>>> >> >> > aid = Column(ForeignKey('a.id'))
>> >>>> >> >> > data = Column(String)
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> > # bad mapper
>> >>>> >> >> > class C(Base):
>> >>>> >> >> > __tablename__ = 'c'
>> >>>> >> >> >
>> >>>> >> >> > id = Column(Integer, primary_key=True)
>> >>>> >> >> >
>> >>>> >> >> > # nonexistent relationship
>> >>>> >> >> > q = relationship("q")
>> >>>> >> >> >
>> >>>> >> >> > try:
>> >>>> >> >> > configure_mappers()
>> >>>> >> >> > except exc.InvalidRequestError as err:
>> >>>> >> >> > print("First exception: %s" % err)
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> > class D(Base):
>> >>>> >> >> > __tablename__ = 'd'
>> >>>> >> >> > id = Column(Integer, primary_key=True)
>> >>>> >> >> > aid = Column(ForeignKey('a.id'))
>> >>>> >> >> > a = relationship("A")
>> >>>> >> >> >
>> >>>> >> >> > # other mappers are blocked
>> >>>> >> >> > try:
>> >>>> >> >> > Session().query(D)
>> >>>> >> >> > except exc.InvalidRequestError as err:
>> >>>> >> >> > print("Second exception: %s" % err)
>> >>>> >> >> >
>> >>>> >> >> > # delete offending class
>> >>>> >> >> > del C
>> >>>> >> >> >
>> >>>> >> >> > # garbage collect, as mappings contain cycles
>> >>>> >> >> > import gc
>> >>>> >> >> > gc.collect()
>> >>>> >> >> >
>> >>>> >> >> > # mapper is gone
>> >>>> >> >> > print(Session().query(D))
>> >>>> >> >> > print("it worked!")
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> > On Fri, May 25, 2018 at 9:16 AM, Mike Bayer
>> >>>> >> >> > <[email protected]>
>> >>>> >> >> > wrote:
>> >>>> >> >> >> On Fri, May 25, 2018 at 7:11 AM, Dave Mittner
>> >>>> >> >> >> <[email protected]>
>> >>>> >> >> >> wrote:
>> >>>> >> >> >>>> So that is already a new fact (which i sort of guessed
>> >>>> >> >> >>>> might
>> >>>> >> >> >>>> be in
>> >>>> >> >> >>>> play) that this is a multi-tenant system. How many
>> >>>> >> >> >>>> databases
>> >>>> >> >> >>>> are
>> >>>> >> >> >>>> we
>> >>>> >> >> >>>> talking about that are accessed by a single application?
>> >>>> >> >> >>>> What do
>> >>>> >> >> >>>> the
>> >>>> >> >> >>>> tables in these databases look like, are they all the same
>> >>>> >> >> >>>> across
>> >>>> >> >> >>>> all
>> >>>> >> >> >>>> DBs (in which case use fixed mappings) or are they totally
>> >>>> >> >> >>>> different?
>> >>>> >> >> >>>> if all the DBs have the same table structure then you
>> >>>> >> >> >>>> should
>> >>>> >> >> >>>> use
>> >>>> >> >> >>>> only
>> >>>> >> >> >>>> a single table/mapping structure for each DB.
>> >>>> >> >> >>>
>> >>>> >> >> >>>
>> >>>> >> >> >>> There's probably something like 5 or 6 distinct MySQL
>> >>>> >> >> >>> databases I
>> >>>> >> >> >>> connect
>> >>>> >> >> >>> to, each on a separate host, each unique. That's why I
>> >>>> >> >> >>> cache
>> >>>> >> >> >>> the
>> >>>> >> >> >>> automapped
>> >>>> >> >> >>> classes on a per-server-per-host basis..
>> >>>> >> >> >>
>> >>>> >> >> >> the critical questions are: 1. do each of these databases
>> >>>> >> >> >> have the
>> >>>> >> >> >> same table structures? or if not 2. are these *fixed*
>> >>>> >> >> >> structures
>> >>>> >> >> >> that you could map statically without using reflection?
>> >>>> >> >> >>
>> >>>> >> >> >>
>> >>>> >> >> >>
>> >>>> >> >> >>>
>> >>>> >> >> >>>> I've never seen that before but I might guess that you
>> >>>> >> >> >>>> have
>> >>>> >> >> >>>> multiple threads reflecting tables and creating classes of
>> >>>> >> >> >>>> the
>> >>>> >> >> >>>> identical name in different threads at the same time?
>> >>>> >> >> >>>> You
>> >>>> >> >> >>>> definitely
>> >>>> >> >> >>>> can't do that without modifying how the
>> >>>> >> >> >>>> sqlalchemy.orm.mapper._mapper_registry works. You need
>> >>>> >> >> >>>> to
>> >>>> >> >> >>>> either
>> >>>> >> >> >>>> ensure these names are unique at all times, and if you
>> >>>> >> >> >>>> expect
>> >>>> >> >> >>>> multiple
>> >>>> >> >> >>>> threads to access the same names, you need to use a mutex
>> >>>> >> >> >>>> to
>> >>>> >> >> >>>> prevent
>> >>>> >> >> >>>> them from doing so concurrently.
>> >>>> >> >> >>>
>> >>>> >> >> >>>
>> >>>> >> >> >>> I'm not entirely sure I understand what you're getting at
>> >>>> >> >> >>> here.
>> >>>> >> >> >>> Before
>> >>>> >> >> >>> I
>> >>>> >> >> >>> even added the caching, I was fully able to automap the
>> >>>> >> >> >>> same
>> >>>> >> >> >>> database
>> >>>> >> >> >>> over
>> >>>> >> >> >>> and over again on each new connection without any issue.
>> >>>> >> >> >>> And
>> >>>> >> >> >>> it
>> >>>> >> >> >>> functions
>> >>>> >> >> >>> perfectly fine now with multiple threads running multiple
>> >>>> >> >> >>> connections
>> >>>> >> >> >>> to the
>> >>>> >> >> >>> same database. The only problem I'm having is after an
>> >>>> >> >> >>> automap
>> >>>> >> >> >>> fails
>> >>>> >> >> >>> to
>> >>>> >> >> >>> properly read a database structure and map it to classes,
>> >>>> >> >> >>> which
>> >>>> >> >> >>> happens only
>> >>>> >> >> >>> rarely.
>> >>>> >> >> >>
>> >>>> >> >> >> This statement does not make sense without more information.
>> >>>> >> >> >> automap
>> >>>> >> >> >> either succeeds against a given database, or it fails. It
>> >>>> >> >> >> is
>> >>>> >> >> >> not
>> >>>> >> >> >> easy
>> >>>> >> >> >> to produce a situation where it fails "only rarely".
>> >>>> >> >> >> Things
>> >>>> >> >> >> that
>> >>>> >> >> >> could cause this are: 1. the database you are automapping
>> >>>> >> >> >> against is
>> >>>> >> >> >> constantly changing, 2. the mapping is relying upon some
>> >>>> >> >> >> subtle
>> >>>> >> >> >> hash
>> >>>> >> >> >> ordering or 3. automap is being used in a multi-threaded
>> >>>> >> >> >> context
>> >>>> >> >> >> where
>> >>>> >> >> >> more than one automap tries to hit the same tables at the
>> >>>> >> >> >> same
>> >>>> >> >> >> time.
>> >>>> >> >> >> When something happens "only rarely", that points to race
>> >>>> >> >> >> conditions
>> >>>> >> >> >> or sometimes hash ordering issues. To rule out #2, you
>> >>>> >> >> >> would
>> >>>> >> >> >> create a test script that does your automap by itself and
>> >>>> >> >> >> try
>> >>>> >> >> >> to run
>> >>>> >> >> >> automap hundreds of times against the particular database
>> >>>> >> >> >> that
>> >>>> >> >> >> caused
>> >>>> >> >> >> a failure, if under Python 2 you'd want to set
>> >>>> >> >> >> PYTHONHASHSEED='random'
>> >>>> >> >> >> in the environment. But the error you showed looks more
>> >>>> >> >> >> like
>> >>>> >> >> >> a
>> >>>> >> >> >> race
>> >>>> >> >> >> of two automaps hitting the same table.
>> >>>> >> >> >>
>> >>>> >> >> >>
>> >>>> >> >> >>
>> >>>> >> >> >> What you're suggesting seems to indicate I should be having
>> >>>> >> >> >> problems
>> >>>> >> >> >>> constantly, but I'm not.
>> >>>> >> >> >>
>> >>>> >> >> >> the prevalence of a race condition is proportional to how
>> >>>> >> >> >> unlikely
>> >>>> >> >> >> the
>> >>>> >> >> >> race is in the first place and how much concurrency is in
>> >>>> >> >> >> play
>> >>>> >> >> >> on a
>> >>>> >> >> >> given basis. Whether it happens every minute or once a
>> >>>> >> >> >> month
>> >>>> >> >> >> doesn't
>> >>>> >> >> >> really matter.
>> >>>> >> >> >>
>> >>>> >> >> >>>
>> >>>> >> >> >>>> How does your program know exactly how to interact with
>> >>>> >> >> >>>> these
>> >>>> >> >> >>>> automap
>> >>>> >> >> >>>> databases if it knows nothing of what tables are present
>> >>>> >> >> >>>> or
>> >>>> >> >> >>>> their
>> >>>> >> >> >>>> structure? If your application *does* know these
>> >>>> >> >> >>>> things,
>> >>>> >> >> >>>> then
>> >>>> >> >> >>>> you
>> >>>> >> >> >>>> should tell automap about it.
>> >>>> >> >> >>>
>> >>>> >> >> >>>
>> >>>> >> >> >>> Well, the entire point of using automap is to make the data
>> >>>> >> >> >>> easily
>> >>>> >> >> >>> accessible through the mapped objects, so the higher level
>> >>>> >> >> >>> code
>> >>>> >> >> >>> certainly
>> >>>> >> >> >>> knows what it's doing. This low-level DB class, however, is
>> >>>> >> >> >>> only
>> >>>> >> >> >>> meant
>> >>>> >> >> >>> to
>> >>>> >> >> >>> expedite the connection configuration process. Ostensibly
>> >>>> >> >> >>> the
>> >>>> >> >> >>> higher
>> >>>> >> >> >>> level
>> >>>> >> >> >>> code could pass more information into the lower level class
>> >>>> >> >> >>> in
>> >>>> >> >> >>> terms
>> >>>> >> >> >>> of
>> >>>> >> >> >>> tables it'll be using and so forth, but that's a pretty
>> >>>> >> >> >>> expensive
>> >>>> >> >> >>> way
>> >>>> >> >> >>> to
>> >>>> >> >> >>> bypass this problem.
>> >>>> >> >> >>
>> >>>> >> >> >> If the schemas of these databases are fixed, it's typical to
>> >>>> >> >> >> have
>> >>>> >> >> >> them
>> >>>> >> >> >> mapped up front. Since you have a finite, small number of
>> >>>> >> >> >> databases,
>> >>>> >> >> >> you should look into having them mapped either up front or
>> >>>> >> >> >> within a
>> >>>> >> >> >> section that is mutexed by a threading.Lock, and in either
>> >>>> >> >> >> case
>> >>>> >> >> >> you
>> >>>> >> >> >> call configure_mappers() as soon as they are all set up.
>> >>>> >> >> >>
>> >>>> >> >> >>
>> >>>> >> >> >>
>> >>>> >> >> >>>
>> >>>> >> >> >>>> As far as legitimate mappings failing afterwards, that's
>> >>>> >> >> >>>> only
>> >>>> >> >> >>>> if
>> >>>> >> >> >>>> you
>> >>>> >> >> >>>> don't get rid of these failed mappers. If you "del" a
>> >>>> >> >> >>>> mapper
>> >>>> >> >> >>>> that
>> >>>> >> >> >>>> failed to configure and make sure it is garbage collected,
>> >>>> >> >> >>>> it
>> >>>> >> >> >>>> will
>> >>>> >> >> >>>> not
>> >>>> >> >> >>>> interfere with subsequent mappings. so you probably want
>> >>>> >> >> >>>> to
>> >>>> >> >> >>>> call
>> >>>> >> >> >>>> configure_mappers(), then if it fails, make sure you lose
>> >>>> >> >> >>>> references
>> >>>> >> >> >>>> to those mappers that failed.
>> >>>> >> >> >>>
>> >>>> >> >> >>>
>> >>>> >> >> >>> As I indicated in my original post, it's not my code that's
>> >>>> >> >> >>> holding
>> >>>> >> >> >>> on
>> >>>> >> >> >>> to
>> >>>> >> >> >>> the fact that a mapping failed. It's being done in the
>> >>>> >> >> >>> global
>> >>>> >> >> >>> space
>> >>>> >> >> >>> within
>> >>>> >> >> >>> SQLAlchemy, itself. That's what I find utterly mind
>> >>>> >> >> >>> boggling.
>> >>>> >> >> >>> I
>> >>>> >> >> >>> don't
>> >>>> >> >> >>> know
>> >>>> >> >> >>> why it would hold on to anything in the global space and
>> >>>> >> >> >>> certainly
>> >>>> >> >> >>> not
>> >>>> >> >> >>> trigger failures on subsequent mapping attempts if a past
>> >>>> >> >> >>> attempt
>> >>>> >> >> >>> failed,
>> >>>> >> >> >>> but that's exactly what it's doing.
>> >>>> >> >> >>>
>> >>>> >> >> >>> You can see the code here:
>> >>>> >> >> >>>
>> >>>> >> >> >>>
>> >>>> >> >> >>>
>> >>>> >> >> >>>
>> >>>> >> >> >>> https://github.com/zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/orm/mapper.py#L3017
>> >>>> >> >> >>
>> >>>> >> >> >> SQLAlchemy does not hold onto mappers that your program does
>> >>>> >> >> >> not
>> >>>> >> >> >> refer
>> >>>> >> >> >> towards. I can demonstrate this if you like. The code at
>> >>>> >> >> >> 3017 is
>> >>>> >> >> >> only invoked if the mapper is still in the _mapper_registry,
>> >>>> >> >> >> which
>> >>>> >> >> >> is
>> >>>> >> >> >> a WeakKeyDictionary. SQLAlchemy only maintains weak
>> >>>> >> >> >> references
>> >>>> >> >> >> to a
>> >>>> >> >> >> particular mapper.
>> >>>> >> >> >>
>> >>>> >> >> >> Now, there are a lot of challenges in getting your program
>> >>>> >> >> >> to
>> >>>> >> >> >> no
>> >>>> >> >> >> longer refer to a particular mapper, depending on what
>> >>>> >> >> >> you're
>> >>>> >> >> >> doing.
>> >>>> >> >> >> If other mappers extend from your mapper, that's a strong
>> >>>> >> >> >> reference.
>> >>>> >> >> >> If other mappers refer to your mapper via relationship(),
>> >>>> >> >> >> thats
>> >>>> >> >> >> a
>> >>>> >> >> >> strong reference. But here, if you are keeping these
>> >>>> >> >> >> per-connection
>> >>>> >> >> >> mappings all independent of each other, they should get
>> >>>> >> >> >> garbage
>> >>>> >> >> >> collected. But if there's a bug in the weak referencing
>> >>>> >> >> >> in
>> >>>> >> >> >> SQLAlchemy or there's a particularly difficult mapping
>> >>>> >> >> >> pattern
>> >>>> >> >> >> in
>> >>>> >> >> >> your
>> >>>> >> >> >> program you need help releasing, I'd need to see specific
>> >>>> >> >> >> scripts
>> >>>> >> >> >> that
>> >>>> >> >> >> illustrate this.
>> >>>> >> >> >>
>> >>>> >> >> >>
>> >>>> >> >> >>>
>> >>>> >> >> >>> On Thu, May 24, 2018 at 8:11 PM, Mike Bayer
>> >>>> >> >> >>> <[email protected]>
>> >>>> >> >> >>> wrote:
>> >>>> >> >> >>>>
>> >>>> >> >> >>>> On Thu, May 24, 2018 at 10:34 PM, Dave Mittner
>> >>>> >> >> >>>> <[email protected]>
>> >>>> >> >> >>>> wrote:
>> >>>> >> >> >>>> > Automap was, at the most regular, only ever used upon
>> >>>> >> >> >>>> > connection
>> >>>> >> >> >>>> > creation to
>> >>>> >> >> >>>> > a given database. (obvious implication is that DB
>> >>>> >> >> >>>> > changes
>> >>>> >> >> >>>> > within
>> >>>> >> >> >>>> > a
>> >>>> >> >> >>>> > connection might be problematic? not an applicable
>> >>>> >> >> >>>> > scenario
>> >>>> >> >> >>>> > thus
>> >>>> >> >> >>>> > far in
>> >>>> >> >> >>>> > our
>> >>>> >> >> >>>> > code)
>> >>>> >> >> >>>>
>> >>>> >> >> >>>> So that is already a new fact (which i sort of guessed
>> >>>> >> >> >>>> might
>> >>>> >> >> >>>> be in
>> >>>> >> >> >>>> play) that this is a multi-tenant system. How many
>> >>>> >> >> >>>> databases
>> >>>> >> >> >>>> are
>> >>>> >> >> >>>> we
>> >>>> >> >> >>>> talking about that are accessed by a single application?
>> >>>> >> >> >>>> What do
>> >>>> >> >> >>>> the
>> >>>> >> >> >>>> tables in these databases look like, are they all the same
>> >>>> >> >> >>>> across
>> >>>> >> >> >>>> all
>> >>>> >> >> >>>> DBs (in which case use fixed mappings) or are they totally
>> >>>> >> >> >>>> different?
>> >>>> >> >> >>>> if all the DBs have the same table structure then you
>> >>>> >> >> >>>> should
>> >>>> >> >> >>>> use
>> >>>> >> >> >>>> only
>> >>>> >> >> >>>> a single table/mapping structure for each DB.
>> >>>> >> >> >>>>
>> >>>> >> >> >>>> If I am reading the error you are getting over at
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>> https://stackoverflow.com/questions/50123090/application-process-unusable-after-cant-proceed-with-initialization-of-o
>> >>>> >> >> >>>> correctly, I've never seen that before but I might guess
>> >>>> >> >> >>>> that
>> >>>> >> >> >>>> you
>> >>>> >> >> >>>> have
>> >>>> >> >> >>>> multiple threads reflecting tables and creating classes of
>> >>>> >> >> >>>> the
>> >>>> >> >> >>>> identical name in different threads at the same time?
>> >>>> >> >> >>>> You
>> >>>> >> >> >>>> definitely
>> >>>> >> >> >>>> can't do that without modifying how the
>> >>>> >> >> >>>> sqlalchemy.orm.mapper._mapper_registry works. You need
>> >>>> >> >> >>>> to
>> >>>> >> >> >>>> either
>> >>>> >> >> >>>> ensure these names are unique at all times, and if you
>> >>>> >> >> >>>> expect
>> >>>> >> >> >>>> multiple
>> >>>> >> >> >>>> threads to access the same names, you need to use a mutex
>> >>>> >> >> >>>> to
>> >>>> >> >> >>>> prevent
>> >>>> >> >> >>>> them from doing so concurrently.
>> >>>> >> >> >>>>
>> >>>> >> >> >>>> >
>> >>>> >> >> >>>> > Here's the gist of my connection initiation code that's
>> >>>> >> >> >>>> > run
>> >>>> >> >> >>>> > upon
>> >>>> >> >> >>>> > instantiation of the DB class.
>> >>>> >> >> >>>> > Mind you, a prior coworker wrote this and there's so
>> >>>> >> >> >>>> > many
>> >>>> >> >> >>>> > ways
>> >>>> >> >> >>>> > of
>> >>>> >> >> >>>> > interfacing with SQLAlchemy that I'm not even sure where
>> >>>> >> >> >>>> > he
>> >>>> >> >> >>>> > got
>> >>>> >> >> >>>> > it
>> >>>> >> >> >>>> > from.
>> >>>> >> >> >>>> >
>> >>>> >> >> >>>> >> db_url = engine.url.URL(drivername =
>> >>>> >> >> >>>> >> self.drivername,
>> >>>> >> >> >>>> >> username =
>> >>>> >> >> >>>> >> self.username,
>> >>>> >> >> >>>> >> password =
>> >>>> >> >> >>>> >> self.password,
>> >>>> >> >> >>>> >> host = self.host,
>> >>>> >> >> >>>> >> database =
>> >>>> >> >> >>>> >> self.database,
>> >>>> >> >> >>>> >> query =
>> >>>> >> >> >>>> >> {'charset':'utf8'})
>> >>>> >> >> >>>> >> self.engine =
>> >>>> >> >> >>>> >>
>> >>>> >> >> >>>> >> create_engine(db_url,encoding='utf8',convert_unicode=True)
>> >>>> >> >> >>>> >> self.session = Session(self.engine)
>> >>>> >> >> >>>> >> self.connection = self.engine.connect()
>> >>>> >> >> >>>> >>
>> >>>> >> >> >>>> >>
>> >>>> >> >> >>>> >>
>> >>>> >> >> >>>> >> # Automap functionality has a lot of
>> >>>> >> >> >>>> >> overhead,
>> >>>> >> >> >>>> >> so
>> >>>> >> >> >>>> >> cache the
>> >>>> >> >> >>>> >> results on a per-host/database basis
>> >>>> >> >> >>>> >> id =
>> >>>> >> >> >>>> >> (self.drivername,self.host,self.database)
>> >>>> >> >> >>>> >> if id not in DB.tables:
>> >>>> >> >> >>>> >> base = automap_base()
>> >>>> >> >> >>>> >> base.prepare(self.engine, reflect=True,
>> >>>> >> >> >>>> >>
>> >>>> >> >> >>>> >> name_for_scalar_relationship=name_for_scalar_relationship)
>> >>>> >> >> >>>> >> DB.tables[id] = base.classes
>> >>>> >> >> >>>> >> self.tables = DB.tables[id]
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>> How does your program know exactly how to interact with
>> >>>> >> >> >>>> these
>> >>>> >> >> >>>> automap
>> >>>> >> >> >>>> databases if it knows nothing of what tables are present
>> >>>> >> >> >>>> or
>> >>>> >> >> >>>> their
>> >>>> >> >> >>>> structure? If your application *does* know these
>> >>>> >> >> >>>> things,
>> >>>> >> >> >>>> then
>> >>>> >> >> >>>> you
>> >>>> >> >> >>>> should tell automap about it. Especially if only care
>> >>>> >> >> >>>> about
>> >>>> >> >> >>>> three
>> >>>> >> >> >>>> tables, use metadata.reflect() and pass those names to
>> >>>> >> >> >>>> "only":
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>> http://docs.sqlalchemy.org/en/latest/core/metadata.html?highlight=metadata%20reflect#sqlalchemy.schema.MetaData.reflect.params.only
>> >>>> >> >> >>>> . Further, your application, assuming it isn't just
>> >>>> >> >> >>>> letting
>> >>>> >> >> >>>> users
>> >>>> >> >> >>>> query whatever they want, would need to know about
>> >>>> >> >> >>>> specific
>> >>>> >> >> >>>> fields
>> >>>> >> >> >>>> and
>> >>>> >> >> >>>> columns on these tables in order to work with queries and
>> >>>> >> >> >>>> mapped
>> >>>> >> >> >>>> objects effectively. You can map these fields and columns
>> >>>> >> >> >>>> up
>> >>>> >> >> >>>> front
>> >>>> >> >> >>>> since you know what they are, without using automap.
>> >>>> >> >> >>>> then
>> >>>> >> >> >>>> you
>> >>>> >> >> >>>> add
>> >>>> >> >> >>>> unit tests for these mappings to make sure they work.
>> >>>> >> >> >>>> that's
>> >>>> >> >> >>>> the
>> >>>> >> >> >>>> typical structure of a live production application.
>> >>>> >> >> >>>>
>> >>>> >> >> >>>> automap probably can use another green warning box at
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>> http://docs.sqlalchemy.org/en/latest/orm/extensions/automap.html#module-sqlalchemy.ext.automap
>> >>>> >> >> >>>> but the idea is that it's intended for expedient access to
>> >>>> >> >> >>>> a
>> >>>> >> >> >>>> particular database in an "offline", ad-hoc context. It
>> >>>> >> >> >>>> isn't
>> >>>> >> >> >>>> oriented towards high capacity use in a production
>> >>>> >> >> >>>> application
>> >>>> >> >> >>>> against
>> >>>> >> >> >>>> databases of unknown structure, because that's not
>> >>>> >> >> >>>> generally
>> >>>> >> >> >>>> useful
>> >>>> >> >> >>>> anyway, a high capacity production application would have
>> >>>> >> >> >>>> a
>> >>>> >> >> >>>> more
>> >>>> >> >> >>>> formalized notion of its schema. automap expects to
>> >>>> >> >> >>>> fail
>> >>>> >> >> >>>> against
>> >>>> >> >> >>>> an
>> >>>> >> >> >>>> unknown database until it is customized to work around the
>> >>>> >> >> >>>> issues
>> >>>> >> >> >>>> in
>> >>>> >> >> >>>> that DB, such as the functions for resolving naming
>> >>>> >> >> >>>> conflicts
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>> (http://docs.sqlalchemy.org/en/latest/orm/extensions/automap.html#handling-simple-naming-conflicts).
>> >>>> >> >> >>>>
>> >>>> >> >> >>>> As far as legitimate mappings failing afterwards, that's
>> >>>> >> >> >>>> only
>> >>>> >> >> >>>> if
>> >>>> >> >> >>>> you
>> >>>> >> >> >>>> don't get rid of these failed mappers. If you "del" a
>> >>>> >> >> >>>> mapper
>> >>>> >> >> >>>> that
>> >>>> >> >> >>>> failed to configure and make sure it is garbage collected,
>> >>>> >> >> >>>> it
>> >>>> >> >> >>>> will
>> >>>> >> >> >>>> not
>> >>>> >> >> >>>> interfere with subsequent mappings. so you probably want
>> >>>> >> >> >>>> to
>> >>>> >> >> >>>> call
>> >>>> >> >> >>>> configure_mappers(), then if it fails, make sure you lose
>> >>>> >> >> >>>> references
>> >>>> >> >> >>>> to those mappers that failed.
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>>
>> >>>> >> >> >>>> >
>> >>>> >> >> >>>> >
>> >>>> >> >> >>>> >
>> >>>> >> >> >>>> >
>> >>>> >> >> >>>> >
>> >>>> >> >> >>>> >
>> >>>> >> >> >>>> >
>> >>>> >> >> >>>> >
>> >>>> >> >> >>>> > On Thu, May 24, 2018 at 5:50 PM, Mike Bayer
>> >>>> >> >> >>>> > <[email protected]>
>> >>>> >> >> >>>> > wrote:
>> >>>> >> >> >>>> >>
>> >>>> >> >> >>>> >> On Thu, May 24, 2018 at 5:40 PM, Dave Mittner
>> >>>> >> >> >>>> >> <[email protected]>
>> >>>> >> >> >>>> >> wrote:
>> >>>> >> >> >>>> >> > Also posted here:
>> >>>> >> >> >>>> >> >
>> >>>> >> >> >>>> >> >
>> >>>> >> >> >>>> >> >
>> >>>> >> >> >>>> >> >
>> >>>> >> >> >>>> >> >
>> >>>> >> >> >>>> >> >
>> >>>> >> >> >>>> >> > https://stackoverflow.com/questions/50123090/application-process-unusable-after-cant-proceed-with-initialization-of-o
>> >>>> >> >> >>>> >> >
>> >>>> >> >> >>>> >> >
>> >>>> >> >> >>>> >> > I have a multithreaded application that runs various
>> >>>> >> >> >>>> >> > jobs in
>> >>>> >> >> >>>> >> > threads.
>> >>>> >> >> >>>> >> > One of
>> >>>> >> >> >>>> >> > these jobs goes out to various data sources to query
>> >>>> >> >> >>>> >> > for
>> >>>> >> >> >>>> >> > data.
>> >>>> >> >> >>>> >> > On
>> >>>> >> >> >>>> >> > occasion
>> >>>> >> >> >>>> >> > the mapping process fails and an exception is thrown.
>> >>>> >> >> >>>> >> >
>> >>>> >> >> >>>> >> > That on its own isn't a big deal; my system is
>> >>>> >> >> >>>> >> > designed
>> >>>> >> >> >>>> >> > to
>> >>>> >> >> >>>> >> > compensate
>> >>>> >> >> >>>> >> > for
>> >>>> >> >> >>>> >> > periodically failing jobs.
>> >>>> >> >> >>>> >> >
>> >>>> >> >> >>>> >> > The problem is that that mapping failure seems to be
>> >>>> >> >> >>>> >> > recorded
>> >>>> >> >> >>>> >> > in
>> >>>> >> >> >>>> >> > a
>> >>>> >> >> >>>> >> > global
>> >>>> >> >> >>>> >> > space that then prevents all future mapping attempts
>> >>>> >> >> >>>> >> > to
>> >>>> >> >> >>>> >> > be
>> >>>> >> >> >>>> >> > aborted.
>> >>>> >> >> >>>> >> > Even
>> >>>> >> >> >>>> >> > attempts on completely different threads using
>> >>>> >> >> >>>> >> > completely
>> >>>> >> >> >>>> >> > different
>> >>>> >> >> >>>> >> > databases. This renders my entire application
>> >>>> >> >> >>>> >> > effectively
>> >>>> >> >> >>>> >> > broken
>> >>>> >> >> >>>> >> > from
>> >>>> >> >> >>>> >> > that
>> >>>> >> >> >>>> >> > point on.
>> >>>> >> >> >>>> >> >
>> >>>> >> >> >>>> >> > After looking in SQLAlchemy's code, mappers are
>> >>>> >> >> >>>> >> > stored
>> >>>> >> >> >>>> >> > in a
>> >>>> >> >> >>>> >> > _mapper_registry
>> >>>> >> >> >>>> >> > global space variable and once any mapper in the
>> >>>> >> >> >>>> >> > registry
>> >>>> >> >> >>>> >> > errors
>> >>>> >> >> >>>> >> > out,
>> >>>> >> >> >>>> >> > any
>> >>>> >> >> >>>> >> > attempt to configure a new mapper will fail.
>> >>>> >> >> >>>> >> >
>> >>>> >> >> >>>> >> > Mapping failures of this nature may be rare -- and
>> >>>> >> >> >>>> >> > indeed it
>> >>>> >> >> >>>> >> > only
>> >>>> >> >> >>>> >> > rarely
>> >>>> >> >> >>>> >> > happens on the connection I'm having a problem with
>> >>>> >> >> >>>> >> > --
>> >>>> >> >> >>>> >> > but
>> >>>> >> >> >>>> >> > this
>> >>>> >> >> >>>> >> > complete
>> >>>> >> >> >>>> >> > locking behavior of all future mapping seems very odd
>> >>>> >> >> >>>> >> > to
>> >>>> >> >> >>>> >> > me.
>> >>>> >> >> >>>> >> > If
>> >>>> >> >> >>>> >> > there
>> >>>> >> >> >>>> >> > isn't
>> >>>> >> >> >>>> >> > a way around this I might have no choice but to have
>> >>>> >> >> >>>> >> > my
>> >>>> >> >> >>>> >> > process
>> >>>> >> >> >>>> >> > completely
>> >>>> >> >> >>>> >> > exit when the exception is encountered, even if that
>> >>>> >> >> >>>> >> > means
>> >>>> >> >> >>>> >> > killing
>> >>>> >> >> >>>> >> > other
>> >>>> >> >> >>>> >> > running threads.
>> >>>> >> >> >>>> >>
>> >>>> >> >> >>>> >> are you creating mappers on the fly or on a per-request
>> >>>> >> >> >>>> >> basis?
>> >>>> >> >> >>>> >> You'd
>> >>>> >> >> >>>> >> want to ideally have mappings created just once at the
>> >>>> >> >> >>>> >> module
>> >>>> >> >> >>>> >> import
>> >>>> >> >> >>>> >> level. Then when your application is ready to start
>> >>>> >> >> >>>> >> up,
>> >>>> >> >> >>>> >> call
>> >>>> >> >> >>>> >> configure_mappers() and everything will be set up.
>> >>>> >> >> >>>> >>
>> >>>> >> >> >>>> >> if those are not patterns you're able to use, then
>> >>>> >> >> >>>> >> please
>> >>>> >> >> >>>> >> provide
>> >>>> >> >> >>>> >> more
>> >>>> >> >> >>>> >> specifics. from your stack trace on SO, it seems like
>> >>>> >> >> >>>> >> you are
>> >>>> >> >> >>>> >> using
>> >>>> >> >> >>>> >> automap. When is that running? If per request, this
>> >>>> >> >> >>>> >> very
>> >>>> >> >> >>>> >> expensive
>> >>>> >> >> >>>> >> and will have problems.
>> >>>> >> >> >>>> >>
>> >>>> >> >> >>>> >> The mapping process *is* guarded by a mutex so it is
>> >>>> >> >> >>>> >> difficult
>> >>>> >> >> >>>> >> to
>> >>>> >> >> >>>> >> produce an issue with mappings failing - the stack
>> >>>> >> >> >>>> >> trace
>> >>>> >> >> >>>> >> you
>> >>>> >> >> >>>> >> post
>> >>>> >> >> >>>> >> almost appears like there is some kind of naming issue
>> >>>> >> >> >>>> >> happening
>> >>>> >> >> >>>> >> where
>> >>>> >> >> >>>> >> a particular mapper has been garbage collected or
>> >>>> >> >> >>>> >> something
>> >>>> >> >> >>>> >> like
>> >>>> >> >> >>>> >> that
>> >>>> >> >> >>>> >> yet still being referred towards by other mappers that
>> >>>> >> >> >>>> >> are
>> >>>> >> >> >>>> >> being
>> >>>> >> >> >>>> >> configured. need to see details of how your code
>> >>>> >> >> >>>> >> works.
>> >>>> >> >> >>>> >>
>> >>>> >> >> >>>> >>
>> >>>> >> >> >>>> >>
>> >>>> >> >> >>>> >> >
>> >>>> >> >> >>>> >> > Any ideas?
>> >>>> >> >> >>>> >> >
>> >>>> >> >> >>>> >> > --
>> >>>> >> >> >>>> >> > SQLAlchemy -
>> >>>> >> >> >>>> >> > The Python SQL Toolkit and Object Relational Mapper
>> >>>> >> >> >>>> >> >
>> >>>> >> >> >>>> >> > http://www.sqlalchemy.org/
>> >>>> >> >> >>>> >> >
>> >>>> >> >> >>>> >> > To post example code, please provide an MCVE:
>> >>>> >> >> >>>> >> > Minimal,
>> >>>> >> >> >>>> >> > Complete,
>> >>>> >> >> >>>> >> > and
>> >>>> >> >> >>>> >> > Verifiable Example. See
>> >>>> >> >> >>>> >> > http://stackoverflow.com/help/mcve
>> >>>> >> >> >>>> >> > for a
>> >>>> >> >> >>>> >> > full
>> >>>> >> >> >>>> >> > description.
>> >>>> >> >> >>>> >> > ---
>> >>>> >> >> >>>> >> > You received this message because you are subscribed
>> >>>> >> >> >>>> >> > to
>> >>>> >> >> >>>> >> > the
>> >>>> >> >> >>>> >> > Google
>> >>>> >> >> >>>> >> > Groups
>> >>>> >> >> >>>> >> > "sqlalchemy" group.
>> >>>> >> >> >>>> >> > To unsubscribe from this group and stop receiving
>> >>>> >> >> >>>> >> > emails
>> >>>> >> >> >>>> >> > from
>> >>>> >> >> >>>> >> > it,
>> >>>> >> >> >>>> >> > send
>> >>>> >> >> >>>> >> > an
>> >>>> >> >> >>>> >> > email to [email protected].
>> >>>> >> >> >>>> >> > To post to this group, send email to
>> >>>> >> >> >>>> >> > [email protected].
>> >>>> >> >> >>>> >> > Visit this group at
>> >>>> >> >> >>>> >> > https://groups.google.com/group/sqlalchemy.
>> >>>> >> >> >>>> >> > For more options, visit
>> >>>> >> >> >>>> >> > https://groups.google.com/d/optout.
>> >>>> >> >> >>>> >>
>> >>>> >> >> >>>> >> --
>> >>>> >> >> >>>> >> SQLAlchemy -
>> >>>> >> >> >>>> >> The Python SQL Toolkit and Object Relational Mapper
>> >>>> >> >> >>>> >>
>> >>>> >> >> >>>> >> http://www.sqlalchemy.org/
>> >>>> >> >> >>>> >>
>> >>>> >> >> >>>> >> To post example code, please provide an MCVE: Minimal,
>> >>>> >> >> >>>> >> Complete,
>> >>>> >> >> >>>> >> and
>> >>>> >> >> >>>> >> Verifiable Example. See
>> >>>> >> >> >>>> >> http://stackoverflow.com/help/mcve
>> >>>> >> >> >>>> >> for a
>> >>>> >> >> >>>> >> full
>> >>>> >> >> >>>> >> description.
>> >>>> >> >> >>>> >> ---
>> >>>> >> >> >>>> >> You received this message because you are subscribed to
>> >>>> >> >> >>>> >> the
>> >>>> >> >> >>>> >> Google
>> >>>> >> >> >>>> >> Groups
>> >>>> >> >> >>>> >> "sqlalchemy" group.
>> >>>> >> >> >>>> >> To unsubscribe from this group and stop receiving
>> >>>> >> >> >>>> >> emails
>> >>>> >> >> >>>> >> from
>> >>>> >> >> >>>> >> it,
>> >>>> >> >> >>>> >> send
>> >>>> >> >> >>>> >> an
>> >>>> >> >> >>>> >> email to [email protected].
>> >>>> >> >> >>>> >> To post to this group, send email to
>> >>>> >> >> >>>> >> [email protected].
>> >>>> >> >> >>>> >> Visit this group at
>> >>>> >> >> >>>> >> https://groups.google.com/group/sqlalchemy.
>> >>>> >> >> >>>> >> For more options, visit
>> >>>> >> >> >>>> >> https://groups.google.com/d/optout.
>> >>>> >> >> >>>> >
>> >>>> >> >> >>>> >
>> >>>> >> >> >>>> > --
>> >>>> >> >> >>>> > SQLAlchemy -
>> >>>> >> >> >>>> > The Python SQL Toolkit and Object Relational Mapper
>> >>>> >> >> >>>> >
>> >>>> >> >> >>>> > http://www.sqlalchemy.org/
>> >>>> >> >> >>>> >
>> >>>> >> >> >>>> > To post example code, please provide an MCVE: Minimal,
>> >>>> >> >> >>>> > Complete,
>> >>>> >> >> >>>> > and
>> >>>> >> >> >>>> > Verifiable Example. See
>> >>>> >> >> >>>> > http://stackoverflow.com/help/mcve
>> >>>> >> >> >>>> > for a
>> >>>> >> >> >>>> > full
>> >>>> >> >> >>>> > description.
>> >>>> >> >> >>>> > ---
>> >>>> >> >> >>>> > You received this message because you are subscribed to
>> >>>> >> >> >>>> > the
>> >>>> >> >> >>>> > Google
>> >>>> >> >> >>>> > Groups
>> >>>> >> >> >>>> > "sqlalchemy" group.
>> >>>> >> >> >>>> > To unsubscribe from this group and stop receiving emails
>> >>>> >> >> >>>> > from
>> >>>> >> >> >>>> > it,
>> >>>> >> >> >>>> > send
>> >>>> >> >> >>>> > an
>> >>>> >> >> >>>> > email to [email protected].
>> >>>> >> >> >>>> > To post to this group, send email to
>> >>>> >> >> >>>> > [email protected].
>> >>>> >> >> >>>> > Visit this group at
>> >>>> >> >> >>>> > https://groups.google.com/group/sqlalchemy.
>> >>>> >> >> >>>> > For more options, visit
>> >>>> >> >> >>>> > https://groups.google.com/d/optout.
>> >>>> >> >> >>>>
>> >>>> >> >> >>>> --
>> >>>> >> >> >>>> SQLAlchemy -
>> >>>> >> >> >>>> The Python SQL Toolkit and Object Relational Mapper
>> >>>> >> >> >>>>
>> >>>> >> >> >>>> http://www.sqlalchemy.org/
>> >>>> >> >> >>>>
>> >>>> >> >> >>>> To post example code, please provide an MCVE: Minimal,
>> >>>> >> >> >>>> Complete,
>> >>>> >> >> >>>> and
>> >>>> >> >> >>>> Verifiable Example. See
>> >>>> >> >> >>>> http://stackoverflow.com/help/mcve
>> >>>> >> >> >>>> for a
>> >>>> >> >> >>>> full
>> >>>> >> >> >>>> description.
>> >>>> >> >> >>>> ---
>> >>>> >> >> >>>> You received this message because you are subscribed to
>> >>>> >> >> >>>> the
>> >>>> >> >> >>>> Google
>> >>>> >> >> >>>> Groups
>> >>>> >> >> >>>> "sqlalchemy" group.
>> >>>> >> >> >>>> To unsubscribe from this group and stop receiving emails
>> >>>> >> >> >>>> from
>> >>>> >> >> >>>> it,
>> >>>> >> >> >>>> send an
>> >>>> >> >> >>>> email to [email protected].
>> >>>> >> >> >>>> To post to this group, send email to
>> >>>> >> >> >>>> [email protected].
>> >>>> >> >> >>>> Visit this group at
>> >>>> >> >> >>>> https://groups.google.com/group/sqlalchemy.
>> >>>> >> >> >>>> For more options, visit
>> >>>> >> >> >>>> https://groups.google.com/d/optout.
>> >>>> >> >> >>>
>> >>>> >> >> >>>
>> >>>> >> >> >>> --
>> >>>> >> >> >>> SQLAlchemy -
>> >>>> >> >> >>> The Python SQL Toolkit and Object Relational Mapper
>> >>>> >> >> >>>
>> >>>> >> >> >>> http://www.sqlalchemy.org/
>> >>>> >> >> >>>
>> >>>> >> >> >>> To post example code, please provide an MCVE: Minimal,
>> >>>> >> >> >>> Complete,
>> >>>> >> >> >>> and
>> >>>> >> >> >>> Verifiable Example. See http://stackoverflow.com/help/mcve
>> >>>> >> >> >>> for
>> >>>> >> >> >>> a
>> >>>> >> >> >>> full
>> >>>> >> >> >>> description.
>> >>>> >> >> >>> ---
>> >>>> >> >> >>> You received this message because you are subscribed to the
>> >>>> >> >> >>> Google
>> >>>> >> >> >>> Groups
>> >>>> >> >> >>> "sqlalchemy" group.
>> >>>> >> >> >>> To unsubscribe from this group and stop receiving emails
>> >>>> >> >> >>> from
>> >>>> >> >> >>> it,
>> >>>> >> >> >>> send
>> >>>> >> >> >>> an
>> >>>> >> >> >>> email to [email protected].
>> >>>> >> >> >>> To post to this group, send email to
>> >>>> >> >> >>> [email protected].
>> >>>> >> >> >>> Visit this group at
>> >>>> >> >> >>> https://groups.google.com/group/sqlalchemy.
>> >>>> >> >> >>> For more options, visit https://groups.google.com/d/optout.
>> >>>> >> >>
>> >>>> >> >> --
>> >>>> >> >> SQLAlchemy -
>> >>>> >> >> The Python SQL Toolkit and Object Relational Mapper
>> >>>> >> >>
>> >>>> >> >> http://www.sqlalchemy.org/
>> >>>> >> >>
>> >>>> >> >> To post example code, please provide an MCVE: Minimal,
>> >>>> >> >> Complete,
>> >>>> >> >> and
>> >>>> >> >> Verifiable Example. See http://stackoverflow.com/help/mcve
>> >>>> >> >> for a
>> >>>> >> >> full
>> >>>> >> >> description.
>> >>>> >> >> ---
>> >>>> >> >> You received this message because you are subscribed to the
>> >>>> >> >> Google
>> >>>> >> >> Groups
>> >>>> >> >> "sqlalchemy" group.
>> >>>> >> >> To unsubscribe from this group and stop receiving emails from
>> >>>> >> >> it,
>> >>>> >> >> send
>> >>>> >> >> an
>> >>>> >> >> email to [email protected].
>> >>>> >> >> To post to this group, send email to
>> >>>> >> >> [email protected].
>> >>>> >> >> Visit this group at https://groups.google.com/group/sqlalchemy.
>> >>>> >> >> For more options, visit https://groups.google.com/d/optout.
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > --
>> >>>> >> > SQLAlchemy -
>> >>>> >> > The Python SQL Toolkit and Object Relational Mapper
>> >>>> >> >
>> >>>> >> > http://www.sqlalchemy.org/
>> >>>> >> >
>> >>>> >> > To post example code, please provide an MCVE: Minimal, Complete,
>> >>>> >> > and
>> >>>> >> > Verifiable Example. See http://stackoverflow.com/help/mcve for a
>> >>>> >> > full
>> >>>> >> > description.
>> >>>> >> > ---
>> >>>> >> > You received this message because you are subscribed to the
>> >>>> >> > Google
>> >>>> >> > Groups
>> >>>> >> > "sqlalchemy" group.
>> >>>> >> > To unsubscribe from this group and stop receiving emails from
>> >>>> >> > it,
>> >>>> >> > send
>> >>>> >> > an
>> >>>> >> > email to [email protected].
>> >>>> >> > To post to this group, send email to
>> >>>> >> > [email protected].
>> >>>> >> > Visit this group at https://groups.google.com/group/sqlalchemy.
>> >>>> >> > For more options, visit https://groups.google.com/d/optout.
>> >>>> >>
>> >>>> >> --
>> >>>> >> SQLAlchemy -
>> >>>> >> The Python SQL Toolkit and Object Relational Mapper
>> >>>> >>
>> >>>> >> http://www.sqlalchemy.org/
>> >>>> >>
>> >>>> >> To post example code, please provide an MCVE: Minimal, Complete,
>> >>>> >> and
>> >>>> >> Verifiable Example. See http://stackoverflow.com/help/mcve for a
>> >>>> >> full
>> >>>> >> description.
>> >>>> >> ---
>> >>>> >> You received this message because you are subscribed to the Google
>> >>>> >> Groups
>> >>>> >> "sqlalchemy" group.
>> >>>> >> To unsubscribe from this group and stop receiving emails from it,
>> >>>> >> send an
>> >>>> >> email to [email protected].
>> >>>> >> To post to this group, send email to [email protected].
>> >>>> >> Visit this group at https://groups.google.com/group/sqlalchemy.
>> >>>> >> For more options, visit https://groups.google.com/d/optout.
>> >>>> >
>> >>>> >
>> >>>> > --
>> >>>> > SQLAlchemy -
>> >>>> > The Python SQL Toolkit and Object Relational Mapper
>> >>>> >
>> >>>> > http://www.sqlalchemy.org/
>> >>>> >
>> >>>> > To post example code, please provide an MCVE: Minimal, Complete,
>> >>>> > and
>> >>>> > Verifiable Example. See http://stackoverflow.com/help/mcve for a
>> >>>> > full
>> >>>> > description.
>> >>>> > ---
>> >>>> > You received this message because you are subscribed to the Google
>> >>>> > Groups
>> >>>> > "sqlalchemy" group.
>> >>>> > To unsubscribe from this group and stop receiving emails from it,
>> >>>> > send
>> >>>> > an
>> >>>> > email to [email protected].
>> >>>> > To post to this group, send email to [email protected].
>> >>>> > Visit this group at https://groups.google.com/group/sqlalchemy.
>> >>>> > For more options, visit https://groups.google.com/d/optout.
>> >>>>
>> >>>> --
>> >>>> SQLAlchemy -
>> >>>> The Python SQL Toolkit and Object Relational Mapper
>> >>>>
>> >>>> http://www.sqlalchemy.org/
>> >>>>
>> >>>> To post example code, please provide an MCVE: Minimal, Complete, and
>> >>>> Verifiable Example. See http://stackoverflow.com/help/mcve for a
>> >>>> full
>> >>>> description.
>> >>>> ---
>> >>>> You received this message because you are subscribed to the Google
>> >>>> Groups "sqlalchemy" group.
>> >>>> To unsubscribe from this group and stop receiving emails from it,
>> >>>> send
>> >>>> an email to [email protected].
>> >>>> To post to this group, send email to [email protected].
>> >>>> Visit this group at https://groups.google.com/group/sqlalchemy.
>> >>>> For more options, visit https://groups.google.com/d/optout.
>> >>>
>> >>>
>> >>> --
>> >>> SQLAlchemy -
>> >>> The Python SQL Toolkit and Object Relational Mapper
>> >>>
>> >>> http://www.sqlalchemy.org/
>> >>>
>> >>> To post example code, please provide an MCVE: Minimal, Complete, and
>> >>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
>> >>> description.
>> >>> ---
>> >>> You received this message because you are subscribed to the Google
>> >>> Groups
>> >>> "sqlalchemy" group.
>> >>> To unsubscribe from this group and stop receiving emails from it, send
>> >>> an
>> >>> email to [email protected].
>> >>> To post to this group, send email to [email protected].
>> >>> Visit this group at https://groups.google.com/group/sqlalchemy.
>> >>> For more options, visit https://groups.google.com/d/optout.
>> >>
>> >> --
>> >> SQLAlchemy -
>> >> The Python SQL Toolkit and Object Relational Mapper
>> >>
>> >> http://www.sqlalchemy.org/
>> >>
>> >> To post example code, please provide an MCVE: Minimal, Complete, and
>> >> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
>> >> description.
>> >> ---
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "sqlalchemy" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an
>> >> email to [email protected].
>> >> To post to this group, send email to [email protected].
>> >> Visit this group at https://groups.google.com/group/sqlalchemy.
>> >> For more options, visit https://groups.google.com/d/optout.
>> >
>> >
>> > --
>> > SQLAlchemy -
>> > The Python SQL Toolkit and Object Relational Mapper
>> >
>> > http://www.sqlalchemy.org/
>> >
>> > To post example code, please provide an MCVE: Minimal, Complete, and
>> > Verifiable Example. See http://stackoverflow.com/help/mcve for a full
>> > description.
>> > ---
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "sqlalchemy" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to [email protected].
>> > To post to this group, send email to [email protected].
>> > Visit this group at https://groups.google.com/group/sqlalchemy.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> SQLAlchemy -
>> The Python SQL Toolkit and Object Relational Mapper
>>
>> http://www.sqlalchemy.org/
>>
>> To post example code, please provide an MCVE: Minimal, Complete, and
>> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
>> description.
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "sqlalchemy" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To post to this group, send email to [email protected].
>> Visit this group at https://groups.google.com/group/sqlalchemy.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> SQLAlchemy -
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
> description.
> ---
> You received this message because you are subscribed to the Google Groups
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To post to this group, send email to [email protected].
> Visit this group at https://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.
--
SQLAlchemy -
The Python SQL Toolkit and Object Relational Mapper
http://www.sqlalchemy.org/
To post example code, please provide an MCVE: Minimal, Complete, and Verifiable
Example. See http://stackoverflow.com/help/mcve for a full description.
---
You received this message because you are subscribed to the Google Groups
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.
