Re: Roster module with custom MySQL requests
Le 01/04/2014 14:23, Tomasz Sterna a écrit : Dnia 2014-04-01, wto o godzinie 12:14 +0200, Sylvain Guglielmi pisze: The problem is : - When an user connects to jabberd2 for the first time, the active,logout... tables are empty, and the roster is already filled. - The code in dispatch.c : 130 states that if one of the user_load module fails, an unsuscribed-packet will be sent, presumably to clean the user's rosters on every session by removing this unknown contact. Just remove 'active' module from 'user-load' chain in your sm.xml. Unfortunately it does not work : When I remove 'active' module from 'user-load' chain, the auto-create in sess.c does not trigger since user_load returned sucessfully. user_create is never called for new users. This causes other issues (For example, active is never filled for the user. My search module relies on it, and you said that offline messages checked that too, and I'd not be surprised if others modules do too). It seems to me that it's related to something you stated earlier : there should be differents chains for user_load, the one when an user connects (called from sess.c) and a quicker one called from dispatch and other locations too (which should not load roster, but only plain info such as active or routing info). Sadly, I have not the time now for doing all that. I think my best option is to disable the presence unsubscribed in dispatch.c (via a config option). I'd gladly accept any better solution. When I'll have more time, I'll get back on doing the right thing. Thanks in advance for any insight. -- Sylvain Gugli Guglielmi Gamedev@Nadeo@Ubisoft
Re: Roster module with custom MySQL requests
Hello again everyone, I have some questions concerning dispatch.c, but first a short recap of the plugin I'm working on : - the plugin replaces roster and allows to write custom sql requests for accessing data (instead of the storage component). The purpose is to be able to use our already-existing buddies database, read-and-write. - I enabled auto-create/ : the server's other tables (active, logout...) get populated at the user's first connection The problem is : - When an user connects to jabberd2 for the first time, the active,logout... tables are empty, and the roster is already filled. - The code in dispatch.c : 130 states that if one of the user_load module fails, an unsuscribed-packet will be sent, presumably to clean the user's rosters on every session by removing this unknown contact. - In my case, when an user connects but one of his buddies has never been connected yet, the active module fails to user_load. The unsuscribed packet is generated, and the buddy-relationship is erased from the roster, unfortunately losing the relationship. I see three solutions here : - using the template-roster module, or adapting it, i could maybe recreate the buddies-relationships after they are deleted. But it would mean i have to keep two tables : roster-before-first-connection, and roster-after-first-connexion(=real roster), which seems weird. Also, it would mean a lot of unneeded operations on first connection. - populate all the tables of the server (active, status...) in advance (i.e before the first connection to XMPP is even possible). It makes more sense, because it avoid to have half-created sessions (with only a valid roster but nothing else). But it's also a certain amount of work for me, and server load on the future (to keep every table of the SM's database up-to-date regarding account creations). - add a config flag in sm : allow relationship to not-yet-existing buddies that would prevent the unsuscribed packet form being sent. I think the most sensible way is the last one, but I may miss some issues. Any insight would be greatly welcome, as usual. Thanks for reading me, and in advance for any answer ^_^. Le 17/01/2014 11:59, Sylvain Guglielmi a écrit : Le 10/01/2014 18:23, Tomasz Sterna a écrit You could have a separate cron component pinging 'sm' in regular intervals with special route packet, and handle this special packet in 'in-router' chain of your module. Having that it could even be done not in regular intervals, but on-demand, when your component gets triggered by web frontend. This works very well ! Thanks for the advice. My cron component is a short php script. I was unsure about how to answer disco#info queries that I get from other components, but it seems to work without answering anything. I'll read the XEP-0030 : Service Discovery RFC later and try to make a more compliant cron component. If anyone is interested in the php simple DIGEST-MD5 auth, code can be found here (It's not robust and exhaustive, but it can be useful I guess) : https://github.com/Gugli/jabberd2/blob/master/tools/sendupdatepacket.php I've been scratching my head for the last couple of days on another issue : our legacy contact system does not allow asymetrical relationships (as XMPP does). It would be very complicated to change that to a more XMPP-compliant system. The solution I came up with is to make a symmetrical mode for my roster module. In this mode : - whenever a user sends a presence subscription, the sm also consider the user allows subscription to the same contact. - whenever a user allows a contact to subscribe to his presence, the sm also consider that the user subscribe back to the same contact's presence. The session that requests the change has to be informed of this additional change with a push as any other session. Basically, item-from and item-to are linked, they always are the same value. This makes a kind of restricted XMPP protocol. It's not great, but it's the only sensible way I found to go on. Has anyone already done something approaching ? Is it bad ? Thanks very much again for all the help you provide :) -- Sylvain Gugli Guglielmi -- Sylvain Gugli Guglielmi
Re: Roster module with custom MySQL requests
Le 01/04/2014 14:23, Tomasz Sterna a écrit : Just remove 'active' module from 'user-load' chain in your sm.xml. Oh, haven't thought of that. *dumb me* (I wrongly assumed all the modules would fail to user_load without calling user_create first. Didn't think it was the purpose of the active module and that all the others should do fine xD). I found only one other module relying on the active status : roster-publish. I need to deactivate roster-publish (it relies on roster-items and roster-group that are not used when using my custom roster plugin, and i don't use ldap). Is it safe/better/not a good idea to deactivate the active plugin from every chain (user_load; user_create; user_delete) ? The sm.xml seems to indicate that the deliver plugin also use the active status, but I couldn't find any of this in the code. You may want to implement the fail-if-not-exist function in your own roster module in this chain. For now, I think it's ok : users can not connect to any c2s without existing and can not register through XMPP. But I take good note of this advice if ever I need it. Thank you very much for your time. -- Sylvain Gugli Guglielmi
Re: Roster module with custom MySQL requests
Dnia 2014-04-01, wto o godzinie 15:56 +0200, Sylvain Guglielmi pisze: Is it safe/better/not a good idea to deactivate the active plugin from every chain (user_load; user_create; user_delete) ? It's main function is to drop messages to unexisting users instead of storing them in offline messages store. If you remove it, you are potentially vulnerable to DoS attack filling your offline storage database with messages for bogus users. -- Tomasz Sterna @ http://abadcafe.pl/ @ http://www.xiaoka.com/
Re: Roster module with custom MySQL requests
Le 01/04/2014 16:40, Tomasz Sterna a écrit : Dnia 2014-04-01, wto o godzinie 15:56 +0200, Sylvain Guglielmi pisze: Is it safe/better/not a good idea to deactivate the active plugin from every chain (user_load; user_create; user_delete) ? It's main function is to drop messages to unexisting users instead of storing them in offline messages store. If you remove it, you are potentially vulnerable to DoS attack filling your offline storage database with messages for bogus users. Ok, good to know. I'll remove it form the user_load chain and let it be in the others. Thanks again. -- Sylvain Gugli Guglielmi
Re: Roster module with custom MySQL requests
Le 10/01/2014 18:23, Tomasz Sterna a écrit You could have a separate cron component pinging 'sm' in regular intervals with special route packet, and handle this special packet in 'in-router' chain of your module. Having that it could even be done not in regular intervals, but on-demand, when your component gets triggered by web frontend. This works very well ! Thanks for the advice. My cron component is a short php script. I was unsure about how to answer disco#info queries that I get from other components, but it seems to work without answering anything. I'll read the XEP-0030 : Service Discovery RFC later and try to make a more compliant cron component. If anyone is interested in the php simple DIGEST-MD5 auth, code can be found here (It's not robust and exhaustive, but it can be useful I guess) : https://github.com/Gugli/jabberd2/blob/master/tools/sendupdatepacket.php I've been scratching my head for the last couple of days on another issue : our legacy contact system does not allow asymetrical relationships (as XMPP does). It would be very complicated to change that to a more XMPP-compliant system. The solution I came up with is to make a symmetrical mode for my roster module. In this mode : - whenever a user sends a presence subscription, the sm also consider the user allows subscription to the same contact. - whenever a user allows a contact to subscribe to his presence, the sm also consider that the user subscribe back to the same contact's presence. The session that requests the change has to be informed of this additional change with a push as any other session. Basically, item-from and item-to are linked, they always are the same value. This makes a kind of restricted XMPP protocol. It's not great, but it's the only sensible way I found to go on. Has anyone already done something approaching ? Is it bad ? Thanks very much again for all the help you provide :) -- Sylvain Gugli Guglielmi
Re: Roster module with custom MySQL requests
Another issue I'm facing : I need my module to be able to detect changes in the database and report it to the connected users if necessary. (Use case : the person adds a friend with something else than a XMPP client, like a web page interface or a webservice API, while she's connected). The way I choose to do that is : - the session manager looks for a syncro='out_of_sync' field in the database - set syncro to 'syncing' for some rows - then change the memory roster items and push packets to the user's sessions with selected changes - then set syncro to 'ok'. (In order to make this case simpler, I will do as if there were no row deletions, but there are ^_^) I'm wondering when is the best way to call this function : - I can make a per-user function, looking only for changes in rows concerning a specific user, and call it at the beginning of the mod-in_sess and mod-pkt_user functions. - I can call it on a regular interval (something like 5 seconds) and look for changes for any user. Then, if the user is loaded, apply changes, and if the user has at least one opened session, send the packets. For now, I've implemented the first solution. But my feeling is that the 2nd solution would be better performance-wise (too many requests with the 1st). I'm only stopped by the regular interval thing. My question : *Should I add a timetick chain to the SM (called every second for example), and add my module to this chain (with a rate_t check) ?* I'm not thrilled by this solution, because for now, I haven't changed any code from jabberd2 except from the new module, which make it easier to test or get in production. *Is there another, better way ?* Thanks for reading, and I'm grateful in advance for any useful insight. :) On a side note : the 5s interval could lead to some race conditions, but I think we can avoid any inconststencies (by using INSERT ... ON DUPLICATE KEY UPDATE ... instead of UPDATE ... for one). The behavior being undetermined if a user issues 2 conflicting commands at the same time through differents means is not an issue, if the resulting state is the result of one of the user's commands. -- Sylvain Gugli Guglielmi Gamedev@Nadeo@Ubisoft
Re: Roster module with custom MySQL requests
Dnia 2014-01-10, pią o godzinie 13:54 +0100, Sylvain Guglielmi pisze: My question : Should I add a timetick chain to the SM (called every second for example), and add my module to this chain (with a rate_t check) ? I'm not thrilled by this solution, because for now, I haven't changed any code from jabberd2 except from the new module, which make it easier to test or get in production. Is there another, better way ? You could have a separate cron component pinging 'sm' in regular intervals with special route packet, and handle this special packet in 'in-router' chain of your module. Having that it could even be done not in regular intervals, but on-demand, when your component gets triggered by web frontend.
Re: Roster module with custom MySQL requests
Dnia 2014-01-08, śro o godzinie 03:02 +0100, Sylvain Gugli Guglielmi
pisze:
I can have something like UPDATE `roster-items` SET
`object-sequence`=`object-sequence`+1 but his break uniqueness in the
table.
I come from PostgreSQL, so explicitly handled sequences is natural to
me:
UPDATE roster-items SET object-sequence = nextval('object-sequence')
I haven't found any easy ways to do something similar with
LAST_INSERT_ID() or AUTO_INCREMENT in MySQL, hence my question.
StackOverflow [1] suggests a convoluted solution:
SELECT Auto_increment FROM information_schema.tables WHERE
table_name='the_table_you_want';
Not pretty.
I think you should be fine with manual increment or
MAX(`object-sequence`)+1
Just to check : you're talking about the pools in pool.h.
Yes.
Maybe at one point it'll be better to use pools for roster items too (item,
item-name,
item-groups and its content). I've not yet profiled anything, but this
may be a way to improve speed when loading user data for packet delivery
(without having to code 2 load-user events).
I will gladly accept code submission if you decide to do it. :-)
[1]
http://stackoverflow.com/questions/12271235/mysql-query-next-sequence-number-for-mysql-auto-incremented-field
--
Tomasz Sterna @ http://abadcafe.pl/ @ http://www.xiaoka.com/
Re: Roster module with custom MySQL requests
Dnia 2014-01-07, wto o godzinie 02:14 +0100, Sylvain Gugli Guglielmi
pisze:
As I understand it, the
object-sequence don't need to be an UNIQUE field. Am I right on
that ?
object-sequence is used mainly for sorting - to keep stanza ordering,
etc.
There is no enforcement on uniqueness, but in cases when you do care on
ordering, these should be unique.
In roster table you should be fine with just incrementing ver.
BTW, you could just write your UPDATE queries setting ver with next
sequence number for the table. It will save you incrementing in code and
keep it in line with how the original module does things.
Well, I have broken that clean separation early on. For example I
needed [...]
Sure. In case of code tied to concrete SQL it's understandable.
I was expressing my concern on changing the generic roster module which
may store data in many formats.
Also, with many more requests, I fear the jabberd2 load will be more
important when I switch our prod to this plugin, and the load is
already noticeable (15% daily peak of our server)...
My bet is that loading user data for packet delivery causes that.
There is an old item on Rob's TODO list, to have two load-user events -
one when user logs in, and other used for delivery, which loads only
necessary user data. But it is still TODO ;-)
in my experience mallocs and frees can be long, so I try to minimize
these calls as a rule. I assumed the same goes for MySQL requests).
One of the reasons we have memory pools in jabberd2.
--
Tomasz Sterna:(){ :|:};:
Instant Messaging ConsultantOpen Source Developer
http://abadcafe.pl/ http://www.xiaoka.com/portfolio
Re: Roster module with custom MySQL requests
Le 07/01/2014 13:22, Tomasz Sterna a écrit : object-sequence is used mainly for sorting - to keep stanza ordering, etc. There is no enforcement on uniqueness, but in cases when you do care on ordering, these should be unique. In roster table you should be fine with just incrementing ver. BTW, you could just write your UPDATE queries setting ver with next sequence number for the table. It will save you incrementing in code and keep it in line with how the original module does things. I can have something like UPDATE `roster-items` SET `object-sequence`=`object-sequence`+1 but his break uniqueness in the table. I haven't found any easy ways to do something similar with LAST_INSERT_ID() or AUTO_INCREMENT in MySQL, hence my question. I think I'll stick to that for now. My bet is that loading user data for packet delivery causes that. There is an old item on Rob's TODO list, to have two load-user events - one when user logs in, and other used for delivery, which loads only necessary user data. But it is still TODO Ok thanks. If performances really become an issue for me, I may get the time to look into that. One of the reasons we have memory pools in jabberd2. Just to check : you're talking about the pools in pool.h. For the record, group management in the roster do not seem to use them. If I'm correct, every time an user changes groups, there are potential mallocs/frees. Same goes for _roster_user_load. Maybe at one point it'll be better to use pools for roster items too (item, item-name, item-groups and its content). I've not yet profiled anything, but this may be a way to improve speed when loading user data for packet delivery (without having to code 2 load-user events). Then again, I'll focus on the features I need for now. Perf issues will come later. -- Sylvain Gugli Guglielmi
Re: Roster module with custom MySQL requests
Dnia 2014-01-06, pon o godzinie 18:07 +0100, Sylvain Guglielmi pisze:
Hello everyone,
To use jabberd2 with my pre-existing contacts database, I started
writing a roster module with customisable MySQL requests (I mailed this
list a while back about it, but I just started actual work). It uses
prepared statements, and config file looks like :
This is really interesting. :-D
It could really accelerate XMPP as the next social media protocol. :-)
Could you consider using libdbi instead of hardwiring to MySQL?
This would make your implementation more reusable.
- In order to make simpler custom databases, I wanted to remove the
pkt-type == pkt_S10N_UN / item-ask == 2 mechanism. According to a
comment there is no ask='unsubscribe' in RFC bis anymore . Would
anyone advise against it ?
How would that make schema simpler?
You still need to store ask='subscribe', don't you?
- something that seems weird to me : I was expecting item-ver to be
incremented each time the item is updated (for example in
_roster_save_item),
but couldn't find such code. It there something I don't grasp ?
Ahhh... Yes... There's a little trick I used there. ;-)
ver is coming from auto-incrementing field object-sequence.
As storage never issues UPDATE but always DELETE first then INSERT,
every updated roster item automagically gets new ver value.
[...] I thought it would be good to do the same
thing in the regular roster, but I couldn't find a way to remove a
specific os_object_t. Is there a way to do that ?
storage_delete() with proper filter.
Anyway here's what the code could look like, with TODOs :
The clean separation between stanza parsing to roster-item and
roster-group structures which are then stored to DB, and in reverse
makes the code more comprehensible.
I doubt that trading readability for efficiency is worthy in case of
today's microprocessor speeds and RAM availability.
Also, does your use-case covers users in many-many groups?
My experience shows that most users are in 0 groups, some in 1 group and
very few in 2+ groups. There's not many INSERTs to gain there.
Remember: Premature optimization is the root of all evil.
I would rather suggest adding 'dirty' flag, risen whenever group
membership actually changes. This would keep the separation still,
allowing for bulk drops of unneeded DELETE/INSERTs.
--
Tomasz Sterna:(){ :|:};:
Instant Messaging ConsultantOpen Source Developer
http://abadcafe.pl/ http://www.xiaoka.com/portfolio
Re: Roster module with custom MySQL requests
Le 07/01/2014 00:59, Tomasz Sterna a écrit : Dnia 2014-01-06, pon o godzinie 18:07 +0100, Sylvain Guglielmi pisze: Hello everyone, To use jabberd2 with my pre-existing contacts database, I started writing a roster module with customisable MySQL requests (I mailed this list a while back about it, but I just started actual work). It uses prepared statements, and config file looks like : This is really interesting. :-D It could really accelerate XMPP as the next social media protocol. :-) Could you consider using libdbi instead of hardwiring to MySQL? This would make your implementation more reusable. Well, the code is written and I need to go forward right now. But I'll look into it when I get the MySQL version running in production. It should not be hard provided libdbi works with prepared statements. - In order to make simpler custom databases, I wanted to remove the pkt-type == pkt_S10N_UN / item-ask == 2 mechanism. According to a comment there is no ask='unsubscribe' in RFC bis anymore . Would anyone advise against it ? How would that make schema simpler? You still need to store ask='subscribe', don't you? Yes, but I assumed it is simpler to explain in the config file the 4th parameter of the statement is a boolean meaning that the user is requiring subscription instead of a int meaning either requiring a subscription or an unsubscription. Especially if it's not a standard feature. It also cleans a few lines of code. But you're right : the benefit isn't worth bothering. I won't remove it. - something that seems weird to me : I was expecting item-ver to be incremented each time the item is updated (for example in _roster_save_item), but couldn't find such code. It there something I don't grasp ? Ahhh... Yes... There's a little trick I used there. ;-) ver is coming from auto-incrementing field object-sequence. As storage never issues UPDATE but always DELETE first then INSERT, every updated roster item automagically gets new ver value. Ok, I get it now. My custom requests however will use Update... So I'll add the increment when the item is modified. As I understand it, the object-sequence don't need to be an UNIQUE field. Am I right on that ? [...] I thought it would be good to do the same thing in the regular roster, but I couldn't find a way to remove a specific os_object_t. Is there a way to do that ? storage_delete() with proper filter. Thanks for the insight. Anyway here's what the code could look like, with TODOs : The clean separation between stanza parsing to roster-item and roster-group structures which are then stored to DB, and in reverse makes the code more comprehensible. I doubt that trading readability for efficiency is worthy in case of today's microprocessor speeds and RAM availability. Well, I have broken that clean separation early on. For example I needed to check if the max number of contacts in the roster is reached with a MySQL request, etc... So this part of readability is already lost in my code :S. Also, with many more requests, I fear the jabberd2 load will be more important when I switch our prod to this plugin, and the load is already noticeable (15% daily peak of our server)... It seemed to me a good tradeoff, I'll stick to that right now. Anyway it'll be easy to revert that later on and test the change in performance.. Also, does your use-case covers users in many-many groups? My experience shows that most users are in 0 groups, some in 1 group and very few in 2+ groups. There's not many INSERTs to gain there. Remember: Premature optimization is the root of all evil. ...until your profiling shows 1000 functions each taking 0.1% of the time, then you regret not having been careful when writing each of them. ^_^ (I lack insight on what is costly for the SessionManager, but in my experience mallocs and frees can be long, so I try to minimize these calls as a rule. I assumed the same goes for MySQL requests). I would rather suggest adding 'dirty' flag, risen whenever group membership actually changes. This would keep the separation still, allowing for bulk drops of unneeded DELETE/INSERTs. Thanks for the feedback. I'll keep everyone updated. -- Sylvain Gugli Guglielmi
