Re: [Sugar-devel] known issues with collaboration in XS 0.5.2
On Tue, Sep 7, 2010 at 17:20, Martin Langhoff mar...@laptop.org wrote: On Tue, Sep 7, 2010 at 10:44 AM, Tomeu Vizoso to...@sugarlabs.org wrote: With the script attached, some GetProperties queries take a few seconds, others more than 25 seconds (the default dbus timeout). It should be invoked like this: dbus-launch --exit-with-session python gabble_test.py Hmmm, great that we have a test script -- thanks! I don't have an XS at hand at this very moment, but I do believe what you're saying and it makes sense - There are presently 2573 registered users and the ejabberd version is ejabberd-xs-2.0.3-11.fc9.olpc.i386 , which is the last one I see in http://fedora.laptop.org/xs/testing/olpc/9/i386/ Ok - so you're on the right version of ejabberd. The datastructures get very inefficient when the number of nodes is moderately high -- some messages travel fast, others show huge latency. I thought of refactoring the handling of @all@ inside of ejabberd (if we special-case it a bit, we can avoid the mess of copied datastructures). But it's a daunting prospect. And our neighbourhood view doesn't behave well with 2.5K users either. Note that those are registered users, there's only a dozen online. So my approach has been to split up the groups in courses, via Moodle. In my (unverified) experience, this makes ejabberd _much_ happier. I guess in that case you won't accumulate lots of registered users either, so this behavior shouldn't matter in the field. Will consider purging registered users weekly or daily from public jabber servers such as jabber.sugarlabs.org. Thanks, Tomeu cheers, m -- mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] known issues with collaboration in XS 0.5.2
On Wed, Sep 8, 2010 at 5:02 AM, Tomeu Vizoso to...@sugarlabs.org wrote: Note that those are registered users, there's only a dozen online. Ah! Then no, the behaviour is 100% bogus. I guess in that case you won't accumulate lots of registered users either, so this behavior shouldn't matter in the field. Well, you do, just that they are segregated in groups. But the registered, not online users should not be in memory at all, unless they've been seen by ejabberd since it started and it's not perhaps purging its roster? Are you seeing the bug after a long run? Will consider purging registered users weekly or daily from public jabber servers such as jabber.sugarlabs.org. Does a restart of ejabberd make this better? [ My questions are a bit hazy here, as I am chasing a few matters at this moment and I cannot devote the time to repro and diagnose/debug this... apologies. ] cheers, m -- mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] known issues with collaboration in XS 0.5.2
On Wed, Sep 8, 2010 at 12:21, Martin Langhoff mar...@laptop.org wrote: On Wed, Sep 8, 2010 at 5:02 AM, Tomeu Vizoso to...@sugarlabs.org wrote: Note that those are registered users, there's only a dozen online. Ah! Then no, the behaviour is 100% bogus. I guess in that case you won't accumulate lots of registered users either, so this behavior shouldn't matter in the field. Well, you do, just that they are segregated in groups. But the registered, not online users should not be in memory at all, unless they've been seen by ejabberd since it started and it's not perhaps purging its roster? Are you seeing the bug after a long run? Will consider purging registered users weekly or daily from public jabber servers such as jabber.sugarlabs.org. Does a restart of ejabberd make this better? Nope, same slowness. Ejabberd also took all the cpu for a couple of minutes when starting up, so it probably loads all registered users to memory and does some crazy stuff with them at that point. [ My questions are a bit hazy here, as I am chasing a few matters at this moment and I cannot devote the time to repro and diagnose/debug this... apologies. ] There's no rush at all. Regards, Tomeu cheers, m -- mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] known issues with collaboration in XS 0.5.2
On Wed, Sep 8, 2010 at 12:35, Tomeu Vizoso to...@sugarlabs.org wrote:
On Wed, Sep 8, 2010 at 12:21, Martin Langhoff mar...@laptop.org wrote:
On Wed, Sep 8, 2010 at 5:02 AM, Tomeu Vizoso to...@sugarlabs.org wrote:
Note that those are registered users, there's only a dozen online.
Ah! Then no, the behaviour is 100% bogus.
I guess in that case you won't accumulate lots of registered users
either, so this behavior shouldn't matter in the field.
Well, you do, just that they are segregated in groups. But the
registered, not online users should not be in memory at all, unless
they've been seen by ejabberd since it started and it's not perhaps
purging its roster?
Are you seeing the bug after a long run?
Will consider purging registered users weekly or daily from public
jabber servers such as jabber.sugarlabs.org.
Does a restart of ejabberd make this better?
Nope, same slowness. Ejabberd also took all the cpu for a couple of
minutes when starting up, so it probably loads all registered users to
memory and does some crazy stuff with them at that point.
Some more info:
- deleting all registered users with ejabberdctl delete-older-users 0
doesn't seem to have any effect, even after restarting
- it's using mnesia, not postgresql
- while churning the CPU, strace shows it's doing this:
clock_gettime(CLOCK_MONOTONIC, {5168912, 600887090}) = 0
pread64(13, \0\0\7|\0224Vx\0\0\1\347\203h\5d\0\vpubsub_itemh\2k...,
8192, 5813560) = 8192
pread64(13, \0\0\6*\0224Vx\0\0\6\\203h\5d\0\vpubsub_itemh\2k...,
8192, 5821752) = 8192
pread64(13, \0\0\1\275\0224Vx\0\0\1\265\203h\5d\0\vpubsub_itemh\2k...,
8192, 3843384) = 8192
pread64(13, \0\0\f\202\0224Vx\0\0\6!\203h\5d\0\vpubsub_itemh\2k...,
8192, 3851576) = 8192
pread64(13, \0\0\6,\0224Vx\0\0\6$\203h\5d\0\vpubsub_itemh\2k...,
8192, 5829944) = 8192
pread64(13, \0\0\6b\0224Vx\0\0\6Z\203h\5d\0\vpubsub_itemh\2k...,
8192, 5840184) = 8192
pread64(13, \0\0\4E\0224Vx\0\0\4=\203h\5d\0\vpubsub_itemh\2k...,
8192, 3859768) = 8192
pread64(13, \0\0\10W\0224Vx\0\0\1\243\203h\5d\0\vpubsub_itemh\2k...,
8192, 3867960) = 8192
poll([{fd=3, events=POLLIN|POLLRDNORM}, {fd=5,
events=POLLIN|POLLRDNORM}, {fd=8, events=POLLIN|POLLRDNORM}, {fd=7,
events=POLLIN|POLLRDNORM}, {fd=17, events=POLLIN|POLLRDNORM}, {fd=18,
events=POLLIN|POLLRDNORM}, {fd=20, events=POLLIN|POLLRDNORM}, {fd=25,
events=POLLIN|POLLRDNORM}, {fd=21, events=POLLIN|POLLRDNORM}, {fd=28,
events=POLLIN|POLLRDNORM}, {fd=24, events=POLLIN|POLLRDNORM}, {fd=26,
events=POLLIN|POLLRDNORM}, {fd=23, events=POLLIN|POLLRDNORM}, {fd=27,
events=POLLIN|POLLRDNORM}, {fd=29, events=POLLIN|POLLRDNORM}, {fd=19,
events=POLLIN|POLLRDNORM}], 16, 0) = 0
- fd 13 is /var/lib/ejabberd/spool/pubsub_item.DAT which is 12M
- exporting the db with ejabberdctl dump /tmp/test.dump shows that
there's 8916 entries in the pubsub_item table.
So maybe something is not getting purged and maybe we have a very bad
index somewhere?
Regards,
Tomeu
[ My questions are a bit hazy here, as I am chasing a few matters at
this moment and I cannot devote the time to repro and diagnose/debug
this... apologies. ]
There's no rush at all.
Regards,
Tomeu
cheers,
m
--
mar...@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
___
Sugar-devel mailing list
Sugar-devel@lists.sugarlabs.org
http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] known issues with collaboration in XS 0.5.2
On Mon, Sep 6, 2010 at 10:27, Tomeu Vizoso to...@sugarlabs.org wrote:
On Fri, Sep 3, 2010 at 19:48, Martin Langhoff mar...@laptop.org wrote:
On Fri, Sep 3, 2010 at 12:43 PM, Tomeu Vizoso to...@sugarlabs.org wrote:
I have a local XS 0.6 that is working fine but I'm finding that the XS
0.5.2 at jabber.sugarlabs.org is not returning some results for some
of the queries that I make.
There are very important bugfixes in the final ejabberd that is in
olpcxs-testing repo for xs-0.6
I should release a 0.6.1 with the contents of olpcxs-testing, but
things haven't been kind to me lately.
You aren't telling us exactly what problems you saw on the old
ejabberd.
I'm noticing that the server never replies to some PEP queries about
buddyinfo and activity properties. The symptom on the Sugar side is
that the DBus calls timeout.
Actually, it eventually replies, it's just that it takes veeery long sometimes.
With the script attached, some GetProperties queries take a few
seconds, others more than 25 seconds (the default dbus timeout). It
should be invoked like this: dbus-launch --exit-with-session python
gabble_test.py
There are presently 2573 registered users and the ejabberd version is
ejabberd-xs-2.0.3-11.fc9.olpc.i386 , which is the last one I see in
http://fedora.laptop.org/xs/testing/olpc/9/i386/
Martin, does it match your experience that, when there are several
hundreds of registered users, PEP queries could take so much CPU?
During each query, the process 'beam' was taking 100% CPU. Postgres
didn't appeared in top.
Thanks,
Tomeu
All I can say is: that old ejabberd (+patches) had plenty of
gremlins and unexplainable behaviours. The ejabberd in olpcxs-testing
has been 100% reliable in behaviour, and I reworked the patches until
I didn't see any buggy behaviour.
Thanks, this answers perfectly my question.
Highly recommended. Should work with just a rebuild if you're not on F9.
I cannot confirm that the right patches are in the latest ejabberd on F13.
Ok, will be keeping an eye on further updates to the School Server
software and will test Sugar against its jabber server.
cheers,
m
--
mar...@laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
from functools import partial
import logging
import gobject
import dbus
import dbus.mainloop.glib
import telepathy
from dbus import PROPERTIES_IFACE
from telepathy.interfaces import ACCOUNT, \
ACCOUNT_MANAGER, \
CONNECTION, \
CHANNEL, \
CHANNEL_INTERFACE_GROUP, \
CONNECTION_INTERFACE_REQUESTS, \
CONNECTION_INTERFACE_ALIASING, \
CONNECTION_INTERFACE_CONTACTS, \
CHANNEL_TYPE_CONTACT_LIST
from telepathy.constants import HANDLE_TYPE_LIST
ACCOUNT_MANAGER_SERVICE = 'org.freedesktop.Telepathy.AccountManager'
ACCOUNT_MANAGER_PATH = '/org/freedesktop/Telepathy/AccountManager'
CONNECTION_INTERFACE_BUDDY_INFO = 'org.laptop.Telepathy.BuddyInfo'
CONNECTION_INTERFACE_ACTIVITY_PROPERTIES = \
'org.laptop.Telepathy.ActivityProperties'
dbus.mainloop.glib.DBusGMainLoop(set_as_default=True)
main_loop = gobject.MainLoop()
connection = None
def _account_property_changed_cb(account_path, properties):
if 'ConnectionStatus' not in properties:
return
if properties['ConnectionStatus'] == 0:
bus = dbus.Bus()
obj = bus.get_object(ACCOUNT_MANAGER_SERVICE, account_path)
connection_path = obj.Get(ACCOUNT, 'Connection', dbus_interface=PROPERTIES_IFACE)
connection_name = connection_path.replace('/', '.')[1:]
global connection
connection = bus.get_object(connection_name, connection_path)
properties = {
CHANNEL + '.ChannelType': CHANNEL_TYPE_CONTACT_LIST,
CHANNEL + '.TargetHandleType': HANDLE_TYPE_LIST,
CHANNEL + '.TargetID': 'subscribe',
}
properties = dbus.Dictionary(properties, signature='sv')
is_ours, channel_path, properties = \
connection.EnsureChannel(properties,
dbus_interface=CONNECTION_INTERFACE_REQUESTS)
channel = bus.get_object(connection_name, channel_path)
channel.connect_to_signal('MembersChanged', _members_changed_cb)
channel.Get(CHANNEL_INTERFACE_GROUP,
'Members',
reply_handler=_get_members_ready_cb,
error_handler=partial(_error_handler_cb,
'Connection.GetMembers'),
dbus_interface=PROPERTIES_IFACE)
def _members_changed_cb(message, added, removed, local_pending,
Re: [Sugar-devel] known issues with collaboration in XS 0.5.2
On Tue, Sep 7, 2010 at 10:44 AM, Tomeu Vizoso to...@sugarlabs.org wrote: With the script attached, some GetProperties queries take a few seconds, others more than 25 seconds (the default dbus timeout). It should be invoked like this: dbus-launch --exit-with-session python gabble_test.py Hmmm, great that we have a test script -- thanks! I don't have an XS at hand at this very moment, but I do believe what you're saying and it makes sense - There are presently 2573 registered users and the ejabberd version is ejabberd-xs-2.0.3-11.fc9.olpc.i386 , which is the last one I see in http://fedora.laptop.org/xs/testing/olpc/9/i386/ Ok - so you're on the right version of ejabberd. The datastructures get very inefficient when the number of nodes is moderately high -- some messages travel fast, others show huge latency. I thought of refactoring the handling of @all@ inside of ejabberd (if we special-case it a bit, we can avoid the mess of copied datastructures). But it's a daunting prospect. And our neighbourhood view doesn't behave well with 2.5K users either. So my approach has been to split up the groups in courses, via Moodle. In my (unverified) experience, this makes ejabberd _much_ happier. cheers, m -- mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] known issues with collaboration in XS 0.5.2
On Fri, Sep 3, 2010 at 19:48, Martin Langhoff mar...@laptop.org wrote: On Fri, Sep 3, 2010 at 12:43 PM, Tomeu Vizoso to...@sugarlabs.org wrote: I have a local XS 0.6 that is working fine but I'm finding that the XS 0.5.2 at jabber.sugarlabs.org is not returning some results for some of the queries that I make. There are very important bugfixes in the final ejabberd that is in olpcxs-testing repo for xs-0.6 I should release a 0.6.1 with the contents of olpcxs-testing, but things haven't been kind to me lately. You aren't telling us exactly what problems you saw on the old ejabberd. I'm noticing that the server never replies to some PEP queries about buddyinfo and activity properties. The symptom on the Sugar side is that the DBus calls timeout. All I can say is: that old ejabberd (+patches) had plenty of gremlins and unexplainable behaviours. The ejabberd in olpcxs-testing has been 100% reliable in behaviour, and I reworked the patches until I didn't see any buggy behaviour. Thanks, this answers perfectly my question. Highly recommended. Should work with just a rebuild if you're not on F9. I cannot confirm that the right patches are in the latest ejabberd on F13. Ok, will be keeping an eye on further updates to the School Server software and will test Sugar against its jabber server. cheers, m -- mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
[Sugar-devel] known issues with collaboration in XS 0.5.2
Hi, I have a local XS 0.6 that is working fine but I'm finding that the XS 0.5.2 at jabber.sugarlabs.org is not returning some results for some of the queries that I make. Are there any similar fixed issues in 0.6 besides what is in http://wiki.laptop.org/go/XS_Release_Notes#XS_0.6 ? Thanks, Tomeu ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] known issues with collaboration in XS 0.5.2
I noticed very strange and inconsistent behavior with j.sl.o yesterday while running two instances of sugar-jhbuild (build from git as of about 1 week ago). In neither case did I see an other XOs except the local ones on my access point. In one case, the icons appeared black and white. In the other case, in XO colors. -walter On Fri, Sep 3, 2010 at 12:43 PM, Tomeu Vizoso to...@sugarlabs.org wrote: Hi, I have a local XS 0.6 that is working fine but I'm finding that the XS 0.5.2 at jabber.sugarlabs.org is not returning some results for some of the queries that I make. Are there any similar fixed issues in 0.6 besides what is in http://wiki.laptop.org/go/XS_Release_Notes#XS_0.6 ? Thanks, Tomeu ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel -- Walter Bender Sugar Labs http://www.sugarlabs.org ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
Re: [Sugar-devel] known issues with collaboration in XS 0.5.2
On Fri, Sep 3, 2010 at 12:43 PM, Tomeu Vizoso to...@sugarlabs.org wrote: I have a local XS 0.6 that is working fine but I'm finding that the XS 0.5.2 at jabber.sugarlabs.org is not returning some results for some of the queries that I make. There are very important bugfixes in the final ejabberd that is in olpcxs-testing repo for xs-0.6 I should release a 0.6.1 with the contents of olpcxs-testing, but things haven't been kind to me lately. You aren't telling us exactly what problems you saw on the old ejabberd. All I can say is: that old ejabberd (+patches) had plenty of gremlins and unexplainable behaviours. The ejabberd in olpcxs-testing has been 100% reliable in behaviour, and I reworked the patches until I didn't see any buggy behaviour. Highly recommended. Should work with just a rebuild if you're not on F9. I cannot confirm that the right patches are in the latest ejabberd on F13. cheers, m -- mar...@laptop.org -- School Server Architect - ask interesting questions - don't get distracted with shiny stuff - working code first - http://wiki.laptop.org/go/User:Martinlanghoff ___ Sugar-devel mailing list Sugar-devel@lists.sugarlabs.org http://lists.sugarlabs.org/listinfo/sugar-devel
