Re: [sqlalchemy] horizontal sharding and bulk_insert

2017-08-28 Thread Антонио Антуан 
Great!
Thank you, Mike.
I really hope I can find time to figure out sqlalchemy source code and 
contribute into it. Currently I have not succeeded in this :(
But the library is amazing :)

среда, 23 августа 2017 г., 16:58:20 UTC+3 пользователь Mike Bayer написал:
>
> On Wed, Aug 23, 2017 at 2:01 AM, Антонио Антуан  > wrote: 
> > 
> > 
>
> > 
> > So, and now we always make queries without shard_id chosing: we always 
> make 
> > queries on 'default' database. One exception: when we run app on master 
> and 
> > read statistics from geo-slaves, we point it explicitly: 
> > "Session.query(...).set_shard_id(NODE_PREFIX)". So, yes, we use sharding 
> > mechanizm only for that. 
> > 
> > My question still the same: can I override "connection_callable" member 
> with 
> > None on my "ShardedSessionWithDefaultBind" class? 
>
> Sure, the ShardedSession is supposed to be more of an example class 
> that you can hack on and connection_callable is a simple None/ 
> not-None thing that is used when you flush() to get a list of 
> connections per row, where each connection is decided based on the 
> sharding conditionals. 
>
>
> > 
> > Also, if you can advise any other mechanism to make things as simple as 
> we 
> > have them now (with horizontal_sharding mechanism), it would be great, 
> our 
> > team will appreciate it :) 
>
> My suggestion was that bulk_insert_mappings() / bulk_update_mappings / 
> bulk_save_objects don't actually make the objects as any part of the 
> session's state, nor does the existing object state of the session 
> have any bearing on the operation of bulk_insert/bulk_update other 
> than being part of the same transaction.   So if you don't want to 
> tinker with the state of your sharded_session, just use a different 
> Session. 
>
> If you have code like this: 
>
>  some_stuff = my_sharded_session.query(Stuff).filter(...).all() 
>
>  my_sharded_session.bulk_save_objects(objects) 
>
> you could change it to this: 
>
>  some_stuff = my_sharded_session.query(Stuff).filter(...).all() 
>
>  ad_hoc_session = 
> Session(bind=my_sharded_session.connection(shard_id='default')) 
>  ad_hoc_session.bulk_save_objects(objects) 
>
> that way you don't have to change the state of my_sharded_session. 
> It probably doesn't matter much either way, e.g. 
> session.connection_callable=None or this, assuming there is no 
> concurrent sharing of the sharded_session instance. 
>
>
>
> > 
> >> 
> >> > 
> >> > If it can help: I ALWAYS perform UPDATE/INSERT/DELETE and most of 
> SELECT 
> >> > queries in only one database. So, I wrote such subclass: 
> >> > 
> >> > 
> >> > class ShardedSessionWithDefaultBind(ShardedSession): 
> >> > def get_bind(self, mapper, shard_id=None, instance=None, 
> >> > clause=None, 
> >> > **kw): 
> >> > if shard_id is None: 
> >> > shard_id = default_shard_id 
> >> > return super(ShardedSessionWithDefaultBind, 
> >> > self).get_bind(mapper, 
> >> > shard_id, instance, clause, **kw) 
> >> > 
> >> > 
> >> > Maybe I can override "__init__" for my class and write 
> >> > "self.connnection_callable = None"? 
> >> > My research of sqlalchemy code didn't make me sure that it is safe 
> >> > enough. 
> >> > But I see, that "connection_callable" used only for checking into 
> >> > "_bulk_insert" and "_bulk_update" functions in 
> >> > sqlalchemy.orm.persistence 
> >> > module. 
> >> > So, if I 100% sure, that I ALWAYS perform INSERT/UPDATE/DELETE 
> queries 
> >> > in 
> >> > only one database, can I make it? 
> >> > 
> >> > -- 
> >> > SQLAlchemy - 
> >> > The Python SQL Toolkit and Object Relational Mapper 
> >> > 
> >> > http://www.sqlalchemy.org/ 
> >> > 
> >> > To post example code, please provide an MCVE: Minimal, Complete, and 
> >> > Verifiable Example. See http://stackoverflow.com/help/mcve for a 
> full 
> >> > description. 
> >> > --- 
> >> > You received this message because you are subscribed to the Google 
> >> > Groups 
> >> > "sqlalchemy" group. 
> >> > To unsubscribe from this group and stop receiving emails from it, 
> send 
> >> > an 
> >> > email to sqlalchemy+...@googlegroups.com. 
> >> > To post to this group, send email to sqlal...@googlegroups.com. 
> >> > Visit this group at https://groups.google.com/group/sqlalchemy. 
> >> > For more options, visit https://groups.google.com/d/optout. 
> > 
> > -- 
> > SQLAlchemy - 
> > The Python SQL Toolkit and Object Relational Mapper 
> > 
> > http://www.sqlalchemy.org/ 
> > 
> > To post example code, please provide an MCVE: Minimal, Complete, and 
> > Verifiable Example. See http://stackoverflow.com/help/mcve for a full 
> > description. 
> > --- 
> > You received this message because you are subscribed to the Google 
> Groups 
> > "sqlalchemy" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an 
> > email to sqlalchemy+...@googlegroups.com . 
> > To post to this group, send email to sqlal...@googlegroups.com 
> . 
> > Visit

[sqlalchemy] Re: Poor example for foreign() and remote() relationships

2017-08-28 Thread Martino Io 
Ok my bad, after reading more I've realized that I don't need to specify 
remote for custom relationships between 2 distinct classes, using 
primaryjoin() should be enough...


Il giorno lunedì 28 agosto 2017 17:27:41 UTC+2, Martino Io ha scritto:
>
> I'm trying to add a simple relationship into 2 different ORM classes which 
> are not bound by DB foreign keys.
>
> The linked example 
> http://docs.sqlalchemy.org/en/latest/orm/join_conditions.html#creating-custom-foreign-conditions
>  
> provides a self referential class which passes the column attribute to 
> remote() however in a more real-world scenario I think it's most likely 
> that you'll be referencing 2 distinct classes, and since the second class 
> isn't created yet i cannot pass a column reference to remote() and neither 
> use a string reference like "second_class.column"... any suggestions?
>
> --
> Marcin
>
>

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.

[sqlalchemy] Poor example for foreign() and remote() relationships

2017-08-28 Thread Martino Io 
I'm trying to add a simple relationship into 2 different ORM classes which 
are not bound by DB foreign keys.

The linked 
example 
http://docs.sqlalchemy.org/en/latest/orm/join_conditions.html#creating-custom-foreign-conditions
 
provides a self referential class which passes the column attribute to 
remote() however in a more real-world scenario I think it's most likely 
that you'll be referencing 2 distinct classes, and since the second class 
isn't created yet i cannot pass a column reference to remote() and neither 
use a string reference like "second_class.column"... any suggestions?

--
Marcin

-- 
SQLAlchemy - 
The Python SQL Toolkit and Object Relational Mapper

http://www.sqlalchemy.org/

To post example code, please provide an MCVE: Minimal, Complete, and Verifiable 
Example.  See  http://stackoverflow.com/help/mcve for a full description.
--- 
You received this message because you are subscribed to the Google Groups 
"sqlalchemy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to sqlalchemy+unsubscr...@googlegroups.com.
To post to this group, send email to sqlalchemy@googlegroups.com.
Visit this group at https://groups.google.com/group/sqlalchemy.
For more options, visit https://groups.google.com/d/optout.