Re: [sr-dev] [SR-Users] RFC: db_redisusrloc module - review and how to deal with
Hi, just sharing my experience from last year while adding some features to db_postgres and db_sqlite On Wed, Feb 21, 2018 at 1:33 AM, Daniel-Constantin Mierla wrote: > Hello, > > (cross-posting being something I want to get developers and users > express their opinions). > > We were recently in a (lucky) situation to have two new modules > submissions targeting more or less the same purpose: allowing to use > Kamailio with a Redis backend via database API. > > One was submitted by Andreas Granig (Sipwise) and already merged with > the name db_redis, because it was designed from the beginning as a > generic DB connector, so the module can be used with auth_db, acc, > usrloc, ... > > * https://www.kamailio.org/docs/modules/devel/modules/db_redis.html > > The second one was submitted by Surendra Tiwari (Plivo), initially > having a naming conflict with db_redis, but renamed to db_redisusrloc, > see the pull request at: > > * https://github.com/kamailio/kamailio/pull/1446 > > Now, this email is about deciding the way to go forward with the second > module. > > It was designed to be used only for usrloc in the first phase, with many > specific usrloc attributes hard coded inside db_redisusrloc. Surendra > said (in a private chat) that the long term plan is to make it work for > other modules. Anyhow, at this moment is very tied to usrloc, therefore > the name of the module. > > Given that the backend (Redis) is not an SQL engine, mapping over > Kamailio's DB API needs some schema definition (see the readme of > db_redis) in order to make it generic and work for all our modules that > use a DB backend. > > On the other hand, to squeeze the best of the backend, specially in > no-SQL cases, having a dedicated DB connector module optimized for a > specific module might help to get more > performances/high-availability/scalability from the backend itself. > > In this case, for example the expires value for a contact record can be > set inside redis, so kamailio module doesn't have to run a timer routine > to clean up (it doesn't mean db_redisusrloc does it right now, I didn't > have time for a proper review, just giving an example). Surendra said > they use it in production for couple of months now and it is several > times faster than using usrloc with db_postgres (iirc, not db_mysql) for > db_mode=3 (database only mode). > > But of course, the reverse of the medal with a dedicated db connector > for a module: it adds overhead to code maintenance (besides generic > updates due to external library changes, I expect changing something > relevant in usrloc, like adding new columns, would require updates in > this module as well). > > So, there are few things I want feedback on: > > 1) how do you fill about splitting from a generic-only DB connectors to > have also some dedicated ones? This is more from confusion point of > view, as a general rule so far, we do not deny contributions if there > are other options for same kind of feature (e.g., many lcr or nat > traversal options). As long as the contributor is willing to maintain > the code, we were fine. > Generic is adding a lot of value in reusability but dedicated ones may also be needed, especialy with NoSQL. Even with SQL DB abstraction we are sometimes facing limitations. Since the db abstraction is generic it may prevent the use of some back-end specific features, even with real SQL back-end when something is *statement specific*, one example : when adding the postgres upsert last year, introduced in Postgres 9.5 the postgres team decided that the conflict/constraints must be specified explicitely in the statement, in this case I decided to do automatic constraint detections so that db_postgres could automaticaly always do upsert when insert_update is used, this was working fine with mysql because insert for update is generic. In such case may still have several options like extendeding the DB abstraction or bypassing it by presseting specific values/decisions in the db_x module (per connection, table, statement type) but if it is per statement I beleive the only option is to extend the db abstraction. > > 2) I guess usrloc, presence and dialog modules would be the main > suspects that would benefit for such dedicated connectors, in other > cases might not worth adding dedicated connectors. Missing any other > module one would like to squeeze more performances with a dedicated > connector? > I agree, dedicated ones may be valuable, Kamailio is able to do many things but it seems valuable to also be specialised in doing some tasks extremely well. > 3) should we set a different naming policy for such modules, for > example: use *dbs_* prefix instead *db_*, to suggest better it is a > DataBase Specific connector? > > 4) Andreas said he plans to do some performance testing of usrloc module > with the two modules and see the differences. Anyone else that wants to > do it? It can be a good metric to see if it worth going one way or another? > > 5)
Re: [sr-dev] [SR-Users] RFC: db_redisusrloc module - review and how to deal with
Opinion of a Kamailio novice, take with a grain of salt: Would it be appropriate to think about this general problem more akin to how some modules support either a file or a db backend? Imagine this world: - To be called a DB in Kamailio parlance, a backend needs to conform to the DB API, be generically usable, etc – all the good stuff we have now. - Some module (foo) has 3 modes of operation: - file – the module uses a file on disk - db – the module uses the DB api - redis – a very foo-specialized application of Redis and its features In my mind, having a specialized Redis backend for a module is no different than having a file backend in addition to a DB backend (see the dispatcher module for example). Just because something (redis) is database-like, doesn't necessarily mean that it can/should only be accessed as a DB (above definition), especially if the tool supports features that shouldn't be in the DB API, and those features are very valuable to a specific module. This would imply that there is a db_redis module which allows Redis to participate in anything demanding a DB (as defined above), as well as a foo_redis (name not important) module, which leverages Redis in a very foo-specific way. It would of course be the responsibility of module owners to decide when it is truly appropriate to have this sort of specialized support, or when the DB api is sufficient (including the potential additional maintenance burden, attractiveness to clients, etc). On Wed, Feb 21, 2018 at 1:35 AM Daniel-Constantin Mierla wrote: > Hello, > > (cross-posting being something I want to get developers and users > express their opinions). > > We were recently in a (lucky) situation to have two new modules > submissions targeting more or less the same purpose: allowing to use > Kamailio with a Redis backend via database API. > > One was submitted by Andreas Granig (Sipwise) and already merged with > the name db_redis, because it was designed from the beginning as a > generic DB connector, so the module can be used with auth_db, acc, > usrloc, ... > > * https://www.kamailio.org/docs/modules/devel/modules/db_redis.html > > The second one was submitted by Surendra Tiwari (Plivo), initially > having a naming conflict with db_redis, but renamed to db_redisusrloc, > see the pull request at: > > * https://github.com/kamailio/kamailio/pull/1446 > > Now, this email is about deciding the way to go forward with the second > module. > > It was designed to be used only for usrloc in the first phase, with many > specific usrloc attributes hard coded inside db_redisusrloc. Surendra > said (in a private chat) that the long term plan is to make it work for > other modules. Anyhow, at this moment is very tied to usrloc, therefore > the name of the module. > > Given that the backend (Redis) is not an SQL engine, mapping over > Kamailio's DB API needs some schema definition (see the readme of > db_redis) in order to make it generic and work for all our modules that > use a DB backend. > > On the other hand, to squeeze the best of the backend, specially in > no-SQL cases, having a dedicated DB connector module optimized for a > specific module might help to get more > performances/high-availability/scalability from the backend itself. > > In this case, for example the expires value for a contact record can be > set inside redis, so kamailio module doesn't have to run a timer routine > to clean up (it doesn't mean db_redisusrloc does it right now, I didn't > have time for a proper review, just giving an example). Surendra said > they use it in production for couple of months now and it is several > times faster than using usrloc with db_postgres (iirc, not db_mysql) for > db_mode=3 (database only mode). > > But of course, the reverse of the medal with a dedicated db connector > for a module: it adds overhead to code maintenance (besides generic > updates due to external library changes, I expect changing something > relevant in usrloc, like adding new columns, would require updates in > this module as well). > > So, there are few things I want feedback on: > > 1) how do you fill about splitting from a generic-only DB connectors to > have also some dedicated ones? This is more from confusion point of > view, as a general rule so far, we do not deny contributions if there > are other options for same kind of feature (e.g., many lcr or nat > traversal options). As long as the contributor is willing to maintain > the code, we were fine. > > 2) I guess usrloc, presence and dialog modules would be the main > suspects that would benefit for such dedicated connectors, in other > cases might not worth adding dedicated connectors. Missing any other > module one would like to squeeze more performances with a dedicated > connector? > > 3) should we set a different naming policy for such modules, for > example: use *dbs_* prefix instead *db_*, to suggest better it is a > DataBase Specific connector? > > 4) Andreas said h
Re: [sr-dev] [SR-Users] RFC: db_redisusrloc module - review and how to deal with
Hello, for fairness and a more complete picture, I should have mentioned that I am somehow the first that broke into module specific backend with topos and topos_redis. In that case, I was also guided by the goals of squeezing what ever can be exploited in terms of performances from Redis backend, given that topology hiding with stripping/adding back headers for all SIP traffic needs be be (ideally) very fast. The topos module had DB API support and can be used with mysql, postgres, etc ... topos_redis was added as an alternative to use directly a Redis server. However, topis_redis is not on top of DB API, therefore a name not related to it at all. Cheers, Daniel On 21.02.18 10:48, Alex Balashov wrote: > Hi, > > I have not dived into the technical specifics deeply or examined the > innards of db_redis, so this is really shooting from the hip: > > I can appreciate the very legitimate dilemma here. However, in general I > would advocate for keeping all DB connectors generic. > > Although I understand that there are some punishing generalisations and > contortions required to shoehorn particular modules' schematic > requirements into the more general DB API, there is a lot of value in > the highly consistent database agnosticism afforded by doing this. This > is similar to the trade-offs involved in using ORMs a lot of the time. > As things currently stand, it is possible to use any db_* backend with > Kamailio in a highly interchangeable way, and not many DB-backed systems > can truly deliver on that promise. Kamailio shines as a rare exception. > > The other problem is that the leaky / inconsistent abstractions involved > in more Byzantine DB module choices will be confusing to users, most of > whom are not software engineers presumably and will not easily > understand the difference between a module-specific DB connector and a > generic one. It will also lead to some needless duplication of effort > and possibly lopsided support for one or the other pathway to the > database in cases where a module has both generic and module-specific > connector support. > > Finally, although I understand that third parties contributing modules > are charged with maintaining them, it seems to be a practical effect > that you (Daniel) and others are dragged into maintaining orphaned or > semi-orphaned modules, or the "plumbing" dependencies that they generate > within common code. The project already has a bewildering array of > modules, all moving at different speeds, having different levels of > sophistication and so forth. Now, I know that I have always taken a more > conservative position on matters of curation of third-party > contributions relative to the implicit maintenance burden, and we may > differ on that, but I think pouring more fuel on that fire by splicing > in multiple levels of DB abstraction for some modules will create > unnecessary maintenance headaches without the corresponding pay-off. > > Whether the database is traditional relational or schemaless, I don't > think it's unreasonable to ask that any DB-backed module adhere to the > common DB API and use those mechanisms to interact with the DB. It just > leads to more consistent, maintainable code and more predictable > outcomes, while providing a level of flexibility people have come to > expect with the RDBM-backed modules. > > -- Alex > -- Daniel-Constantin Mierla www.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio Advanced Training - March 5-7, 2018, Berlin - www.asipto.com Kamailio World Conference - May 14-16, 2018 - www.kamailioworld.com ___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Re: [sr-dev] [SR-Users] RFC: db_redisusrloc module - review and how to deal with
Hi, I have not dived into the technical specifics deeply or examined the innards of db_redis, so this is really shooting from the hip: I can appreciate the very legitimate dilemma here. However, in general I would advocate for keeping all DB connectors generic. Although I understand that there are some punishing generalisations and contortions required to shoehorn particular modules' schematic requirements into the more general DB API, there is a lot of value in the highly consistent database agnosticism afforded by doing this. This is similar to the trade-offs involved in using ORMs a lot of the time. As things currently stand, it is possible to use any db_* backend with Kamailio in a highly interchangeable way, and not many DB-backed systems can truly deliver on that promise. Kamailio shines as a rare exception. The other problem is that the leaky / inconsistent abstractions involved in more Byzantine DB module choices will be confusing to users, most of whom are not software engineers presumably and will not easily understand the difference between a module-specific DB connector and a generic one. It will also lead to some needless duplication of effort and possibly lopsided support for one or the other pathway to the database in cases where a module has both generic and module-specific connector support. Finally, although I understand that third parties contributing modules are charged with maintaining them, it seems to be a practical effect that you (Daniel) and others are dragged into maintaining orphaned or semi-orphaned modules, or the "plumbing" dependencies that they generate within common code. The project already has a bewildering array of modules, all moving at different speeds, having different levels of sophistication and so forth. Now, I know that I have always taken a more conservative position on matters of curation of third-party contributions relative to the implicit maintenance burden, and we may differ on that, but I think pouring more fuel on that fire by splicing in multiple levels of DB abstraction for some modules will create unnecessary maintenance headaches without the corresponding pay-off. Whether the database is traditional relational or schemaless, I don't think it's unreasonable to ask that any DB-backed module adhere to the common DB API and use those mechanisms to interact with the DB. It just leads to more consistent, maintainable code and more predictable outcomes, while providing a level of flexibility people have come to expect with the RDBM-backed modules. -- Alex -- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ ___ Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
