On Tue, Aug 22, 2017 at 3:43 AM, Антонио Антуан <a.ch....@gmail.com> wrote: > Hi guys > I tried to implement horizontal sharding in my project. Everything is ok, > except bulk_inserts. > When I run tests, I got this error: > File "/home/anton/Projects/proj/core/model/messages.py", line 210, in > create > Session.bulk_insert_mappings(MessageEntity, to_commit) > File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/scoping.py", line 157, > in do > return getattr(self.registry(), name)(*args, **kwargs) > File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 2345, > in bulk_insert_mappings > mapper, mappings, False, False, return_defaults, False) > File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 2416, > in _bulk_save_mappings > transaction.rollback(_capture_exception=True) > File "build/bdist.linux-x86_64/egg/sqlalchemy/util/langhelpers.py", line > 60, in __exit__ > compat.reraise(exc_type, exc_value, exc_tb) > File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/session.py", line 2411, > in _bulk_save_mappings > mapper, mappings, transaction, isstates, return_defaults) > File "build/bdist.linux-x86_64/egg/sqlalchemy/orm/persistence.py", line > 35, in _bulk_insert > "connection_callable / per-instance sharding " > NotImplementedError: connection_callable / per-instance sharding not > supported in bulk_insert() > > I do not understand what 'connection_callable' does, but I really need to > implement it. Is there any ways to do it?
The idea of bulk_insert() is strictly one of performance with some degree of convenience; you don't need it to actually accomplish anything as there are many other ways to achieve what it does. the "connection_callable" thing is specifically so that an ORM persistence operation can get at the appropriate database connection for a particular INSERT/UPDATE/DELETE, however this adds complexity so is not part of bulk_insert(). In the case of application side horizontal sharding, you'd want to take the data you're passing to it and pre-sort it into individual per-shard sets, then use a Session per shard to run the inserts. Also, application side horizontal sharding is not a great way to do sharding in any case. If you're on Postgresql I'd strongly recommend using table inheritance instead. > > If it can help: I ALWAYS perform UPDATE/INSERT/DELETE and most of SELECT > queries in only one database. So, I wrote such subclass: > > > class ShardedSessionWithDefaultBind(ShardedSession): > def get_bind(self, mapper, shard_id=None, instance=None, clause=None, > **kw): > if shard_id is None: > shard_id = default_shard_id > return super(ShardedSessionWithDefaultBind, self).get_bind(mapper, > shard_id, instance, clause, **kw) > > > Maybe I can override "__init__" for my class and write > "self.connnection_callable = None"? > My research of sqlalchemy code didn't make me sure that it is safe enough. > But I see, that "connection_callable" used only for checking into > "_bulk_insert" and "_bulk_update" functions in sqlalchemy.orm.persistence > module. > So, if I 100% sure, that I ALWAYS perform INSERT/UPDATE/DELETE queries in > only one database, can I make it? > > -- > SQLAlchemy - > The Python SQL Toolkit and Object Relational Mapper > > http://www.sqlalchemy.org/ > > To post example code, please provide an MCVE: Minimal, Complete, and > Verifiable Example. See http://stackoverflow.com/help/mcve for a full > description. > --- > You received this message because you are subscribed to the Google Groups > "sqlalchemy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sqlalchemy+unsubscr...@googlegroups.com. > To post to this group, send email to sqlalchemy@googlegroups.com. > Visit this group at https://groups.google.com/group/sqlalchemy. > For more options, visit https://groups.google.com/d/optout. -- SQLAlchemy - The Python SQL Toolkit and Object Relational Mapper http://www.sqlalchemy.org/ To post example code, please provide an MCVE: Minimal, Complete, and Verifiable Example. See http://stackoverflow.com/help/mcve for a full description. --- You received this message because you are subscribed to the Google Groups "sqlalchemy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy+unsubscr...@googlegroups.com. To post to this group, send email to sqlalchemy@googlegroups.com. Visit this group at https://groups.google.com/group/sqlalchemy. For more options, visit https://groups.google.com/d/optout.
