Re: Future of jabberd

[Tomasz Sterna]

W dniu 30.05.2016, pon o godzinie 20∶05 +0200, użytkownik Tomasz Sterna
napisał:
> I am still not fond of the synchronous nature of storage interface,
> but changing this would require rewriting it from scratch.
> Also having an asynchronous interface for immediate in nature
> backends like file backend or BDB, would require arm twisting.

Let's have it in the open:
https://github.com/jabberd2/jabberd2/issues/120




-- 
smoku @ http://abadcafe.pl/ @ http://xiaoka.com/



signature.asc
Description: This is a digitally signed message part

Re: Future of jabberd

[Tomasz Sterna]

W dniu 31.05.2016, wto o godzinie 16∶31 +, użytkownik Shawn Debnath
napisał:
> Re 1. Merging separate daemons to one.
> I am not sure if merging them into one process is the best idea. It
> sure is convenient, but isolation is a nice thing to have. Specially,
> when you have unauthorized users hammering on C2S.

Oh. I wasn't clear on that.
You will have the option to choose which components you want to run in
process, so if you wish you can keep the current setup of having one
process for each component. Possibly on different machines.

I just want the simple setup to have the option to run all components
in one process.


-- 
smoku @ http://abadcafe.pl/ @ http://xiaoka.com/



signature.asc
Description: This is a digitally signed message part

Re: Future of jabberd

[Shawn Debnath]

I agree with Tomasz that there are changes that are definitely needed 
given how much technology has changed since jabberd2 was first released.

Re 1. Merging separate daemons to one.

I am not sure if merging them into one process is the best idea. It sure is
convenient, but isolation is a nice thing to have. Specially, when you
have unauthorized users hammering on C2S. To be honest, I believe, that
instead of going micro – go macro. Think distributed scalable server platform
rather than ease of running on one machine. In today’s services oriented world,
this approach will find a better home I believe. Which brings me to router. 
Today, it’s a bottleneck. There are several ways, as mentioned in Tomasz’s 
email, to fix this issue and something that would be great to tackle this as
part of this re-architecture. MoongooseIM is a good place to look at for
scalable XMPP platform architecture.

4. Configuration interface

Subscription based model works well and its okay to force developers
to write config change handlers. It’s 2016, time to not argue about this
point given the advantages.

5. JavaScript support.

Personally, C/C++ is fine, however, if folks insist, would be good to keep
the existing setup and allow for JS SM components as a compile
time option.

6. String handling

Something that would be nice to see is allowing SM modules or others
to pre-process the PKT to extract as much meaningful data once (in the
beginning) and then passing that object around. Currently, at each
stage of the SM module call chain, we have to re-process the PKT or NAD
to extract the same data over and over again. Unnecessary work that 
can be avoided easily. Easy to do in C++ but a bit awkward in C.


On 5/30/16, 1:31 AM, "Tomasz Sterna"  wrote:

>There are some things we already talked about on Gitter channel [1],
>but I would like to raise them on the ML for peer review.
>
>As you can see from late activity, jabberd2 project is far from dead.
>With the inclusion of new features like WebSocket support, C99 code
>compatibility, IPv6 improvements, modern TLS handling, SASL Anonymous,
>password hashing, CRAM-MD5 and more... it is not a stale codebase
>anymore.
>
>But it is far from modern too...
>There are some changes I would like to introduce in the near future and
>I would like to hear your thoughts about:
>
>1. Merging separate daemons to one.
>Current design of jabberd2 with separate router, sm, c2s, s2s processes
>is designed to allow nice separation of concerns and distribution of
>processing. Separate processes are proved to be better approach than
>threads too.
>But most installations of jabberd are not distributed, with one
>instance of each component. Especially when c2s and sm got vhost
>support and are able to handle more than one domain.
>Also, modern OS architectures are tuned for event processing rather
>than multithreading, so event based architecture is better suited for
>them. Even jabberd2 process internally is event based on MIO.
>So, it makes sense to allow for running all component instances in one
>process, especially on amateur, low load servers.
>Merging processes will allow for having one main loop only, so
>maintaining bugfixes in it will be easier (main.c of all processes is a
>copy-paste, with all the bugs, so bugs are also multiplied).
>
>2. Phasing out MIO.
>This is closely related to above. MIO used by jabberd2 does not have
>clerar main loop support, which is implemented separately in each
>component main.c and is hardly pluggable.
>Also, the way MIO is implemented (in .h file, with platform specific
>bits in .c) makes it a maintanance nightmare.
>I would really like to replace it with a modern, upstream maintained
>event library. The nicest one I know is libuv, which also gives us nice
>platform independence layer.
>I already have a working c2s port to libuv as a PoC.
>
>3. Phasing out router.
>router component is the one binding all the others.
>In current design it is the single point of failure. Other components
>already support multiple instances, but router proved to be difficult
>to multiply.
>The most radical, yet compelling solution to this problem is getting
>rid of the router at all. There are many cooked solutions for local
>packet distribution, which Local Message Bus [2] looks like most
>promising solution. I would see either Mbus [3] or NN_BUS [4] taking
>role of router component.
>The added advantage of using a Message Bus is the ability to connect to
>the bus with alternative implementations to perform own actions.
>i.e. having the ability to use CLI tools to eavesdrop and send messages
>to the bus proved to be priceless when I implemented a PoC of the Bus
>in experimental jabberd branch.
>Bus also solves the problem of distribution - it is up to the
>deployment administrator whether one sets up local, one-machine only
>bus or a network distributed one.
>
>4. Configuration interface.
>A the moment jabberd is configured with static XML files loaded at

Re: Future of jabberd

[Ed - 0x1b, Inc.]

On Mon, May 30, 2016 at 9:32 PM, Tomasz Sterna  wrote:
> W dniu 30.05.2016, pon o godzinie 10∶31 +0200, użytkownik Tomasz Sterna
> napisał:
>> 7. DBI interface to RDBM.
>
> Just one more question.
>
> Do you (ML) have a use case for having SM storage in SQL?
> Is it just for distributed SM only?
> Maybe it is not worth the effort and we should just drop it and embed
> something like LMDB [1] in?
>
> I do see value of having SQL backend for authreg, to integrate with
> existing userbase, but SM storage? Does it really need to be
> abstracted?
>
>
> [1] https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database
>
> --
>  /o__
> (_<^'  The best cure for insomnia is to get a lot of sleep. -W.C. Fields
>

Congratulations - you have Jabberd3 outlined, um.. so long as the name
isn't taken yet? In terms of "heavy", if it runs in OBSD on a
BeagleBone Black or Debian on a Rasperry Pi Zero, then I think you're
OK. Running on an Arduino is a special type of crazy - not that i
wouldn't mind seeing it done. Running Jabberd3* as part of a website
would be a new kind of distributed. If going full Javascript - this
might be interesting: http://electron.atom.io/

In any case - Thank You very much for a great XMPP server - keep having fun
*just trying it on - Jabberd11? too binary?

Re: Future of jabberd

[Tomasz Sterna]

W dniu 30.05.2016, pon o godzinie 10∶31 +0200, użytkownik Tomasz Sterna
napisał:
> 7. DBI interface to RDBM.

Just one more question.

Do you (ML) have a use case for having SM storage in SQL?
Is it just for distributed SM only?
Maybe it is not worth the effort and we should just drop it and embed
something like LMDB [1] in?

I do see value of having SQL backend for authreg, to integrate with
existing userbase, but SM storage? Does it really need to be
abstracted?


[1] https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database

-- 
 /o__ 
(_<^'  The best cure for insomnia is to get a lot of sleep. -W.C. Fields



signature.asc
Description: This is a digitally signed message part

Re: Future of jabberd

[lists]

If you can create and maintain the database through standard programs (sqlite 
being my favorite), I don't mind the caching. But in a system that uses TLS, is 
a database lookup that significant of a time sink in the whole transaction 
flow. 

This is interesting reading:
http://stackoverflow.com/questions/11216647/why-is-sqlite-faster-than-redis-in-this-simple-benchmark

I don't do a lot of database stuff, but learning on mysql and then trying 
sqlite, I'm just amazed at how it cranks. I really like sqlite3. 

  Original Message  
From: Tomasz Sterna
Sent: Monday, May 30, 2016 4:30 PM
To: jabberd2@lists.xiaoka.com
Reply To: jabberd2@lists.xiaoka.com
Subject: Re: Future of jabberd

W dniu 30.05.2016, pon o godzinie 12∶50 -0700, użytkownik
li...@lazygranch.com napisał:
> Do you really have to cache something in jabberd when the data can be
> pulled from the sql database? Sure the data has changed. But if you
> pull a fresh record each time, I don't see the issue.

Unfortunately RDBMs are notorious to be a choking point.
You just cannot fetch data over and over again and expect reasonable
preformance. This is the reason for raise of memcached, redis etc.

Also, see: https://metajack.wordpress.com/2008/08/26/choosing-an-xmpp-server/


-- 
/o__ 
(_<^' I must follow the people. Am I not their leader? -Benjamin Disraeli

Re: Future of jabberd

[Tomasz Sterna]

W dniu 30.05.2016, pon o godzinie 12∶50 -0700, użytkownik
li...@lazygranch.com napisał:
> Do you really have to cache something in jabberd when the data can be
> pulled from the sql database? Sure the data has changed. But if you
> pull a fresh record each time, I don't see the issue.

Unfortunately RDBMs are notorious to be a choking point.
You just cannot fetch data over and over again and expect reasonable
preformance. This is the reason for raise of memcached, redis etc.

Also, see: https://metajack.wordpress.com/2008/08/26/choosing-an-xmpp-server/


-- 
 /o__ 
(_<^'  I must follow the people. Am I not their leader? -Benjamin Disraeli



signature.asc
Description: This is a digitally signed message part

Re: Future of jabberd

[lists]

Do you really have to cache something in jabberd when the data can be pulled 
from the sql database? Sure the data has changed. But if you pull a fresh 
record each time, I don't see the issue.


  Original Message  
From: Tomasz Sterna
Sent: Monday, May 30, 2016 11:40 AM
To: jabberd2@lists.xiaoka.com
Reply To: jabberd2@lists.xiaoka.com
Subject: Re: Future of jabberd

W dniu 30.05.2016, pon o godzinie 10∶00 -0700, użytkownik
li...@lazygranch.com napisał:
> That is one of the beauties of programs written around standard tools
> like ‎sql. You can hook into the database and add features, or not.

The issue with this approach is that SM component caches user data and
has no way of knowing that data was changed directly in database
backend.

http://martinfowler.com/bliki/TwoHardThings.html




-- 
smoku @ http://abadcafe.pl/ @ http://xiaoka.com/

Re: Future of jabberd

[Tomasz Sterna]

W dniu 30.05.2016, pon o godzinie 15∶40 +0200, użytkownik Matěj Cepl
napisał:
>    https://metajack.wordpress.com/2008/08/26/choosing-an-xmpp-server/
>  
>    by heart, don't you? When doing large changes in the 
>    codebase, it would be probably prudent to take those 
>    objections into considertaion, especially database 
>    transaction “abuse”.

:-)
https://github.com/jabberd2/jabberd2/blob/master/etc/sm.xml.dist.in#L212

I am still not fond of the synchronous nature of storage interface, but
changing this would require rewriting it from scratch.
Also having an asynchronous interface for immediate in nature backends
like file backend or BDB, would require arm twisting.

But I do keep it in mind.



-- 
 /o__ "Zaphod grinned two manic grins, sauntered over to the bar 
(_<^' and bought most of it." 



signature.asc
Description: This is a digitally signed message part

Re: Future of jabberd

[Tomasz Sterna]

W dniu 30.05.2016, pon o godzinie 10∶00 -0700, użytkownik
li...@lazygranch.com napisał:
> That is one of the beauties of programs written around standard tools
> like ‎sql. You can hook into the database and add features, or not.

The issue with this approach is that SM component caches user data and
has no way of knowing that data was changed directly in database
backend.

http://martinfowler.com/bliki/TwoHardThings.html




-- 
smoku @ http://abadcafe.pl/ @ http://xiaoka.com/



signature.asc
Description: This is a digitally signed message part

Re: Future of jabberd

[lists]

That is one of the beauties of programs written around standard tools like 
‎sql. You can hook into the database and add features, or not. ;-)

In defense of web interfaces, you can hand off that administration task to 
someone else without granting login access to the server. Or depending on your 
setup, the web interface could be restricted to lan facing only, such as in the 
proposed ddwrt setup. 

It might pay to check out the XMPP pentest tools if what sounds like a total 
rewrite of jabberd is in the works.



  Original Message  
From: brahmann
Sent: Monday, May 30, 2016 8:26 AM
To: jabberd2@lists.xiaoka.com‎
Reply To: jabberd2@lists.xiaoka.com
Subject: Re: Future of jabberd

Agree (web).
I wrote some years ago simple web interface for jabberd2, need to review 
it and rewrite for new one later. Its will be good for those who want it 
but not in jabberd2 code inside.

Using it with mysql and postgresql - works perfectly


wbr, brahmann

On 30.05.2016 16:43, li...@lazygranch.com wrote:
> Regarding item 4, seriously, does everything ‎these days have to have a web 
> interface? It just increases the attack surface. Adding a web interface means 
> one more thing to protect against hackers, which means writing rules for the 
> WAF or adding something else for fail2ban or sshguard to watch.
>
> Most services have a "reload" and "restart" in the service command. It really 
> isn't a burden to use them. The burden is to have yet another means to 
> restart the service.
>
> Personally, I viewed the XML setup as a feature since it is self documenting.
>
> My preference would be for better sqlite support. That is add and delete 
> users from sqlite3 rather than mysql.
>

Re: Future of jabberd

[Greg Troxel]

I'd also like to see multi-user conferencing integrated.   I did a
little maintenance on the muc plugin a few years ago, but the code is
still too scary.

Re: Future of jabberd

[brahmann]

Agree (web).
I wrote some years ago simple web interface for jabberd2, need to review 
it and rewrite for new one later. Its will be good for those who want it 
but not in jabberd2 code inside.


Using it with mysql and postgresql - works perfectly


wbr, brahmann

On 30.05.2016 16:43, li...@lazygranch.com wrote:

Regarding item 4, seriously, does everything ‎these days have to have a web 
interface? It just increases the attack surface. Adding a web interface means 
one more thing to protect against hackers, which means writing rules for the 
WAF or adding something else for fail2ban or sshguard to watch.

Most services have a "reload" and "restart" in the service command. It really 
isn't a burden to use them. The burden is to have yet another means to restart the service.

Personally, I viewed the XML setup as a feature since it is self documenting.

My preference would be for better sqlite support. That is add and delete users 
from sqlite3 rather than mysql.

Re: Future of jabberd

[Matěj Cepl]

On 2016-05-30, 08:31 GMT, Tomasz Sterna wrote:
> But it is far from modern too...
> There are some changes I would like to introduce in the near future and
> I would like to hear your thoughts about:

I completely agree with these comments:

1. It would be probably wise to maintain stable jabberd2 branch 
   as long as the new jabberd2 (or dare I say jabberd3?) is 
   being developed. One of the nice things about jabberd2 is 
   that it is rather stable, so I would prefer if we still have 
   some enteprise-production-stable version of jabberd2 until 
   the jabberd3 gets to the same level of stability/maturity.

2. I really like the idea of JavaScript plugins/components, but 
   I agree with other commenter that it should be made optional 
   (or even it could be XEP-0114 external components itself?).  
   Also, I would love if this functionality was in addition not 
   as a replacement of XEP-0114 external components.

3. I guess you know 
   https://metajack.wordpress.com/2008/08/26/choosing-an-xmpp-server/ 
   by heart, don't you? When doing large changes in the 
   codebase, it would be probably prudent to take those 
   objections into considertaion, especially database 
   transaction “abuse”.

Best,

Matěj

-- 
https://matej.ceplovi.cz/blog/, Jabber: mc...@ceplovi.cz
GPG Finger: 3C76 A027 CA45 AD70 98B5  BC1D 7920 5802 880B C9D8
 
The politician attempts to remedy the evil by increasing the very
thing that caused the evil in the first place: legal plunder.
-- Frederick Bastiat

Re: Future of jabberd

[lists]

Regarding item 4, seriously, does everything ‎these days have to have a web 
interface? It just increases the attack surface. Adding a web interface means 
one more thing to protect against hackers, which means writing rules for the 
WAF or adding something else for fail2ban or sshguard to watch. 

Most services have a "reload" and "restart" in the service command. It really 
isn't a burden to use them. The burden is to have yet another means to restart 
the service.

Personally, I viewed the XML setup as a feature since it is self documenting. 

My preference would be for better sqlite support. That is add and delete users 
from sqlite3 rather than mysql.

Re: Future of jabberd

[Marek Červenka]

i see Phoenix rise from the ashes ... ;)

+1 for all

2) libuv is used by node.js - good choice
4) i'm coming from telco side - 
https://wiki.asterisk.org/wiki/display/AST/ARI+Push+Configuration

6) important is easy integration with ELK stack
7) i hope its better than unixODBC


Dne 30.5.2016 v 10:31 Tomasz Sterna napsal(a):

There are some things we already talked about on Gitter channel [1],
but I would like to raise them on the ML for peer review.

As you can see from late activity, jabberd2 project is far from dead.
With the inclusion of new features like WebSocket support, C99 code
compatibility, IPv6 improvements, modern TLS handling, SASL Anonymous,
password hashing, CRAM-MD5 and more... it is not a stale codebase
anymore.

But it is far from modern too...
There are some changes I would like to introduce in the near future and
I would like to hear your thoughts about:

1. Merging separate daemons to one.
Current design of jabberd2 with separate router, sm, c2s, s2s processes
is designed to allow nice separation of concerns and distribution of
processing. Separate processes are proved to be better approach than
threads too.
But most installations of jabberd are not distributed, with one
instance of each component. Especially when c2s and sm got vhost
support and are able to handle more than one domain.
Also, modern OS architectures are tuned for event processing rather
than multithreading, so event based architecture is better suited for
them. Even jabberd2 process internally is event based on MIO.
So, it makes sense to allow for running all component instances in one
process, especially on amateur, low load servers.
Merging processes will allow for having one main loop only, so
maintaining bugfixes in it will be easier (main.c of all processes is a
copy-paste, with all the bugs, so bugs are also multiplied).

2. Phasing out MIO.
This is closely related to above. MIO used by jabberd2 does not have
clerar main loop support, which is implemented separately in each
component main.c and is hardly pluggable.
Also, the way MIO is implemented (in .h file, with platform specific
bits in .c) makes it a maintanance nightmare.
I would really like to replace it with a modern, upstream maintained
event library. The nicest one I know is libuv, which also gives us nice
platform independence layer.
I already have a working c2s port to libuv as a PoC.

3. Phasing out router.
router component is the one binding all the others.
In current design it is the single point of failure. Other components
already support multiple instances, but router proved to be difficult
to multiply.
The most radical, yet compelling solution to this problem is getting
rid of the router at all. There are many cooked solutions for local
packet distribution, which Local Message Bus [2] looks like most
promising solution. I would see either Mbus [3] or NN_BUS [4] taking
role of router component.
The added advantage of using a Message Bus is the ability to connect to
the bus with alternative implementations to perform own actions.
i.e. having the ability to use CLI tools to eavesdrop and send messages
to the bus proved to be priceless when I implemented a PoC of the Bus
in experimental jabberd branch.
Bus also solves the problem of distribution - it is up to the
deployment administrator whether one sets up local, one-machine only
bus or a network distributed one.

4. Configuration interface.
A the moment jabberd is configured with static XML files loaded at
daemon startup. It is close to impossible to change the values in
runtime, as random places of the process are using copies of values or
direct pointers to values from config structure.
This heavily impedes implementation of features such as XEP-0133
Service Administration or Web interface.
 From my experience, the best handling of such requirements is to
provide write-only/change-subscribe interface similar to GConf/dconf.
This interface does not allow reading on-demand of random values, but
allows only subscription to change and write-value + publish change.
This approach forces programmer to write value-change handlers in
application code, which allows changing the value by anyone at any
moment.
Do you know any standalone library that implements such approach,
or do I need to implement custom solution in jabberd codebase?

5. JavaScript support.
Let's face it - JavaScript is all the hype today :-) It also is a very
good language for data processing. I think it would be a good solution
for implementation of modern XEP logic in sm component.
sm is implemented in C with all RFC required logic, and all XEPs are
loadable modules to sm and these add JEP/XEP functionality.
Having an option to implement XEP logic in JS instead of plain C,
should speed up recent and experimental XEP adoption in jabberd.
This gives concerns to jabberd2 as an embedded server though - current
jabberd2 is perfectly able to work fine on low resource machines such
as DD-WRT router. Introducing heavy JS JIT machine 

Re: Future of jabberd

[brahmann]

Hi, about all - good direction i think!

1. Event-driven single - good choice.
2. There are alot of other good libraries like libev too, but libuv - 
good, used it in my work pretty stable with 300-1000 events per loop. 
But last time we using libev coz FreeBSD releated more.
5. About js, logic on js, imho not good idea... jabberd2 good production 
class jabber server and its working on my work environments alot of 
years. Time for this will be when jabberd2 will be singleprocess and 
need to fork it to jabberd2-js port with XEP-logic on js. But not change 
good product... imho


I can help with testing and maybe this summer alittle with coding.. this 
email is jabber too for connections


Thank you for developing Jabberd2

wbr, brahmann

On 30.05.2016 11:31, Tomasz Sterna wrote:

There are some things we already talked about on Gitter channel [1],
but I would like to raise them on the ML for peer review.

As you can see from late activity, jabberd2 project is far from dead.
With the inclusion of new features like WebSocket support, C99 code
compatibility, IPv6 improvements, modern TLS handling, SASL Anonymous,
password hashing, CRAM-MD5 and more... it is not a stale codebase
anymore.

But it is far from modern too...
There are some changes I would like to introduce in the near future and
I would like to hear your thoughts about:


[SKIP]


What do you think of these ideas, guys?
Do you have any suggestions for even more disturbing improvements? :-)
I would love to hear all feedback.

You may take a look at experimental/PoC codebase at ashnazg [10] branch on 
GitHub.


[1] http://gitter.im/jabberd2/jabberd2
[2] http://en.wikipedia.org/wiki/Message-oriented_middleware
[3] http://www.mbus.org/
[4] http://nanomsg.org/v0.9/nn_bus.7.html
[5] http://duktape.org/
[6] http://log4c.sourceforge.net/
[7] http://libdbi.sourceforge.net/
[8] http://github.com/antirez/sds
[9] http://www.gnu.org/software/libunistring/
[10] http://github.com/jabberd2/jabberd2/tree/ashnazg