Let me clarify some things here,
The XMPP protocol is not here to transport binary data, we have a bunch
of other protocols for that. XMPP is only here to transport the
signalisations to create bitstreams connections between the different
end-points.
I've implemented Jingle over WebRTC threw XMPP in my project, WebRTC
re-use a lot of existing protocol (RTP, SRTP, RTPC, SDP), Jingle is
just here to do a little bit of signalisation ("someone is calling
you"…) and to convert the standard SDP (Session Description Protocol,
which is used in SIP too) to a standard XML Jingle package.
Creating a tool that convert SDP to Jingle is quite easy ( check our
PHP implementation
http://bazaar.launchpad.net/~movim/movim/trunk/view/head:/lib/SDPtoJingle.php
and
http://bazaar.launchpad.net/~movim/movim/trunk/view/head:/lib/JingletoSDP.php).
When you've established a RTP connection (RTP take place on top of UDP)
you can send 4K 60FPS 3D video, it doesn't matter.
Secondly, saying that the XML overhead is a big issue is for me a
non-sense. A couple of XMPP servers already support XML compression
(with a minimum of 30% to more than 50% of bandwidth gain), see XEP
like XEP-0138: Stream Compression
(http://xmpp.org/extensions/xep-0138.html) and XEP-0229: Stream
Compression with LZW (http://xmpp.org/extensions/xep-0229.html).
Furthermore the whole web is based on XHTML (on top of HTTP which
create way more overhead than XMPP on top of TCP).
The big issue that we have now on XMPP is definitly not the XML size
(every 3G connections and cheap smartphones can now handle this amount
of data). The issue is more on improving how the XMPP stanza are sent
threw the network and how to limit useless stanza to be send to the
client (and between the servers). For the ejabberd server with Pubsub,
we have sometime more than 50% of the stanza (XMPP packets) that are
totally useless (the client already have the information, it's a old
information…).
As far as I know, base64 heavy data are only used to send and receive
the users avatars (and for some authentications exchanges in SASL2, I
think mostly to prevent encoding issues).
Finally the Websocket are not here to improve the XMPP connections on
desktops and mobiles clients, they are already using a pure TCP
connection for that. The Websocket extension is mostly here to replace
the BOSH system (which mimic a full-duplex XMPP communication on top of
HTTP) which, yes, have a big overhead and serious performances issues.
The aim of XMPP over Websockets is just to bring XMPP to the modern
browsers.
For the XMPP clients, there's already a couple of them, with strenghts
and weaknesses. From my point of view Gajim has a quite good
implementation of XMPP but the UI is really not "user friendly". Pidgin
is quite great but the XMPP support is not as good as expected. Empathy
start to be really interesting and the integration of the client in the
OS is quite great.
You can also take a look at the HipChat proprietary client which offer
a nice user experience on top of XMPP.
For me, Google has dropped the XMPP protocol for a couple of reasons :
- Yes, the XMPP Standard Fundation can be slow to react and they are
not working as fast as Google would like.
- They are currently replacing XMPP with Hangout which is a
proprietary, centralized and a totally obscur system. They are not
replacing XMPP because its lack of features, they are building
something totally different. And because the XMPP network is
decentralized and totally open, you can't control and monitor it (or
tape/spy it) easily.
- Some clients already used the Gmail XMPP connexion for other purposes
than the standard IM stuffs.
- With Hangout, Google can now control the network, the servers but
also the clients. To create Hangout-compatible clients we have to
reverse-engineer their protocol (like we does for MSN, Skype or QQ).
I'm fighting for more than 5 years now to built a full social network
on top of XMPP, and yes I still think that this protocol is totally
underrated. With XMPP you can do way more than just IM.
If I compare your little 'requirement list' with XMPP :
- XMPP is already built as a client-server protocol on top of TCP (with
a couple of peer-to-peer extensions)
- As I said the overhead here is not the XML protocol but the way you
send data threw the network, and we are working with the XSF to
optimize theses stuffs
- Efficient encoding, XMPP is in full UTF-8 and handle perfectly any
encoding (my client is used on many differents countries and I never
had any bad feedbacks for this thing).
- Stream encryption is supported C-S and S-S, the XSF is trying to push
full encryptions between the servers
(https://xmpp.net/result.php?domain=jabber.org&type=client for
example). For the record, Goggle has never put any encryption between
the Gmail XMPP server and the rest of the network, all the data are
broadcasted in plain text. The OTR protocol is already used by a a lot
of XMPP clients (on mobile too http://www.xabber.org/).
- XMPP support "buge data" streaming (SOCKS5 bitestream
http://xmpp.org/extensions/xep-0065.html) or Jingle can be used for
this purpose.
- The core of XMPP is built with a nice presence and ressources system.
There's also extensions like XEP-0085: Chat State Notifications
(http://xmpp.org/extensions/xep-0085.html) or XEP-0224: Attention
(http://xmpp.org/extensions/xep-0224.html) to add nice features to a
chat conversation. There is also extensions to handle archiving,
message synchronisation between the clients and so on.
- The core of XMPP is built on one-to-one and one-to-many system. You
have several way to do this : chatrooms for simple messaging, you can
do CC/mailing-list/transfert of messages to one or several contacts,
and there is PubSub which add a nice stream management feature (which
we used with Atom to publish and/or retrieve posts) on the PubSub nodes.
- Just have a look of the ~300 extensions already shipped with XMPP
http://xmpp.org/xmpp-protocols/xmpp-extensions/
I'll just finish with this : http://xkcd.com/927/
Regards,
Jaussoin Timothée aka edhelas
On jeu., juil. 10, 2014 at 12:27 , Dāvis Mosāns <[email protected]>
wrote:
2014-07-08 23:31 GMT+03:00 Lodewijk andré de la porte <[email protected]>:
2014-07-08 21:53 GMT+02:00 Dāvis Mosāns <[email protected]>:
Also what about NUL bytes? I bet most parsers are implemented in
C/C++ using typical char * null-terminated string, how'll pass this
JSON to someone? because well NUL...
At the parser level you would find a "b" character that's not
between brackets, signalling a binary header is coming. A binary
header is actually just the number of bytes that follow in binary
format. The following bytes are then a binary file, to be assigned
to a string as if it were a variable. We have a binarybuffer in
javascript, that sort of thing. It would contain the NULL byte if
you like it to.
[...]
I think it's much more serious that you have to serve the BJSON
completely as a binary file. It's not like you can dump it onto a
webpage anymore. You can't slip it into your normal HTTP text
transfer bodies either, has to work with attachments. Attachments
should be secure when facing malformed transfers* etc. Inconvenient,
but not the end.
A lot of protocols are text based, often implemented using C strings
and that's what I mean, you can't embed a JSON with binary data
containing NUL there (because NUL will terminate that string), so you
handle it like typical binary file and then what's the point of JSON
to use in first place, because I don't see how it can be any better
than any other proper binary data. Such binary JSON gives only
overhead but no advantages.
2014-07-09 7:25 GMT+03:00 Bill Stewart <[email protected]>:
I haven't used it in years, but I was always quite fond of XDR
https://en.wikipedia.org/wiki/External_Data_Representation
Sun's External Data Representation coding from the 80s, RFC-1014.
Defines a bunch of variable types, and gives you tools for packing
and unpacking them.
It's actually pretty good, but there are reasons why Protobuf was
created and used instead. The main benefit of Protobuf is that it's
easily extendable and can have optional fields. If you add or remove
optional fields to server all old clients will still work like
nothing have changed. But with XDR you can't do that unless you add
another layer on top of it, but that's more work comparing to just
taking Protobuf and using it. Also currently Protobuf is much more
popular and have more libraries available for dozens of langauges.
2014-07-09 8:30 GMT+03:00 "Łukasz \"Cyber Killer\" Korpalski"
<[email protected]>:
It's really nice that so many of you got into the spirit and start
thinking about how to change xmpp to make it something new, but what
are
you achieving here? It will end up being a new protocol, incompatible
with existing xmpp, it will take a few years to finish the spec, then
another 10+ years until any meaningful applications start using it
(if
at all)... So yeah, except being "coder porn" it does nothing to help
the problem here and now.
The goal would be to create smaller overhead and thus be more
performance effective. Also it doesn't have to be incompatible. It
could be incorporated in XMPP so that new applications could use it
but other's just use same legacy XMPP and everything keeps working
fine and people wouldn't know what's happening under the hood, nor
they would care. And I think it would be trivial to convince people
to use and enable this "Binary" XMPP mode (if it's implemented in
their client) which makes their chat client app to use 100x times
less bandwidth and 50x times less CPU time (spent in parsing), thus
your phone's battery would last longer. And yes XML overhead is that
big.
A technically pretty proto won't help, today’s world has a huge
problem
with taking anything new. Better to stay with existing stuff, make it
maybe less efficient because of it, but it will be here fast, when
it's
needed. Plus being less efficient is a no issue today, with fast
machines (you can use compression on the fly, yes really :-P ),
loads of
storage, broadband connections (even the 3G data caps are getting
larger
and larger each year), etc. People are sending gigabytes of binary
files
in base64 each day in email messages, so why even care? ;-)
I guess you don't know that nothing is ever fast or good enough.
People will always want things faster. What about real-time video
call in 4k @ 60 FPS ? It's unreal to imagine this in XMPP unless some
really good binary protocol is used so that it's not your software
that creates a bottleneck, but if it does then your software is bad
and why would I use it over other that can do it, the one that was
designed for it, for example see Elemental Demonstrates 4K HEVC Video
at 60 fps in London
Anyway, I must admit that I haven't studied XMPP enough to know how
good or bad it is, but always should try to minimize any overhead,
basically you want to process as little as possible.
Here straight from wiki, weaknesses:
Does not support Quality of Service (QoS)
XMPP does not have the ability to set the timing flow of messages,
preventing XMPP from becoming practical for many embedded distributed
realtime, Machine-to-Machine, or IoT applications.
High overhead for embedded applications
As a text based protocol, XMPP has a relatively high computing and
network overhead.
In-band binary data transfer is inefficient
Binary data must be first base64 encoded before it can be transmitted
in-band. Therefore any significant amount of binary data (e.g., file
transfers) is best transmitted out-of-band, using in-band messages to
coordinate. The best example of this is the Jingle XMPP Extension
Protocol, XEP-0166. This issue are being adressed by the experimental
XEP-0322: Efficient XML Interchange (EXI) Format.
that sounds really really bad. But it's not all lost, Jingle actually
seems good as it have option to switch to Real-time Transport
Protocol (RTP) and then it's just pure binary stream with minimal
overhead. And looks like they are aware of these issues as EXI is
being developed, but still while it's a big step forward, it will
never beat pure binary protocol.
Also from wiki, this is good idea:
A perhaps more efficient transport for real-time messaging is
WebSocket, a web technology providing for bi-directional, full-duplex
communications channels over a single TCP connection. Experimental
implementations of XMPP over WebSocket exist, and a (now-expired)
Internet-Draft documenting this approach was published at the IETF
but not yet standardized.
In my opinion the bottom line is - a small addition to existing xmpp
has
a far larger chance of being widely adopted (by applications and by
the
users) than a completely new protocol. And despite how awesome coder
one
might be - you won't be able to write all those implementations
yourself
or convince the masses to switch (again!).
Maybe yes, maybe no. I think if you've written specification in very
clear and understandable way and if you've reference implementation
library which everyone could just link against and if your protocol
does it better than current existing solutions then I don't see why
it wouldn't get adapted. Besides you don't need it implemented
everywhere, you need it so that it's in application you use and you
could contribute there yourself.
2014-07-09 13:17 GMT+03:00 Lodewijk andré de la porte <[email protected]>:
[...]
There is no masses using XMPP. Masses of coders, maybe, and they
will use the best tool for the job.
All the extensions have succeeded in making any XMPP app lacking in
usability. I sure haven't found any nice XMPP clients, nice enough
to compare with native clients.
That's true indeed, currently there aren't any decent XMPP client
(atleast I'm not aware of any). I mean from user's usability point
(UX/UI). There are good either proprietary clients (eg. Skype) or
good open source clients (eq. Quassel) that doesn't support XMPP :D
In fact I'm willing to bet everyone in the western world uses FB,
Google chat and MSN (slackers and slowpokes). They all have limited
XMPP implementations, they native clients do more. And there's no
good app for interacting with XMPP.
About which Western wold you're talking about? I don't know, but I
would assume that in Europe, Skype would be one of the most popular
clients. Atleast here MSN never was a thing and everyone have always
been using Skype and almost everyone still does. FB isn't really used
that much (here we've better alternative). And about Google Talk only
some people are aware that it even exists. I know that in Russia it's
ICQ and in China it's QQ that's dominating there. Anyway I think they
don't implement XMPP because it's too much work for too small
benefit. Also then how they would compete with others if they would
be limited with XMPP if it doesn't do something that they want to do.
So own protocol might be safer bet for a lot of companies. By the way
Google Talk dropped XMPP support.
I actually really hoped that Tox would succeed so that I could drop
Skype and convince others to do same :D But now I'm not so sure if
there will be some replacement in nearest future. Actually I think
one of reason why Tox created new protocol and didn't used XMPP is
that generally XMPP consists of client-server architecture and P2P
support is only with extension which isn't widely known nor
implemented much. Also XMPP seems way too complicated than it should
be.
Pidgin really isn't good, it's just the only one out there. And it
is still in the MSN era. I've switched to Office 2013 from
Libre/OpenOffice and it really is in a different league all
together. And it sucks that it is. But what can we do?
I personally don't like Pidgin too. And about Office and LibreOffice
they both have their own advantages and disadvantages but I wouldn't
say that any of them would be significantly better or worse than
other. Also it's nothing to do with open souirce, it's just we've
more users that want everything to be perfect without any effort for
free than go and help and contribute to projects.
Ok, so anyway I've come to conclusion that I'm not sure anymore
whether it's better to try to fix XMPP or just create new much
simpler and better protocol. But in any case we really should
summarize all our points about what's bad in XMPP and send them to
XMPP group and see if they're willing to fix them and accept our
proposals.
If we want to create a new protocol, then I've few ideas about it.
Firstly it's really a shame that a lot of things are getting
reinvented over and over again. So the main goal would be don't
reinvent stuff but reuse as much as possible from existing things. It
would help a lot as there's libraries for already everything and
would just have to combine them. Next, think about high-level
differences between instant message, group message, offline chat
message, email, SMS, MMS, mailing list. What are differences?
Actually if you think about it, there are no differences at all, it's
the same message being delivered in various ways. But why single
protocol couldn't handle it all? In fact it could. We're already
using browser for all of that. It's just that there's no
specification to unify it all in single protocol. Imagine if we
wouldn't need a separate application for IRC, for email nor for other
chats. It all could be handled by one general protocol. And why stop
here, actually calls, sound and video are also exactly same. There's
no really difference between text or sound/video, it's just a data
and it depends only how you interpret it. Also security, there's only
one option, just encrypt it all before sending. It's just so simple.
This next generation protocol would be a superset of typical media
protocols and thus it wouldn't matter anymore what other people use.
You would just use some middle-ware protocol layer that would
translate that foreign protocol to this next-gen protocol and you
simply use your favorite client. From user's point of view it doesn't
matter at all which protocol is used under the hood.
So the main features and design goals of this protocol should be:
Support for peer-to-peer and client-server architectures over both
TCP and UDP
Lightweight, minimal overhead, generally be just pass-thru
Efficient encoding, basically binary streams
Encryption by default with option for OTR
Data stream itself can contain anything: text, images, sound, video,
animations, screen sharing (and even remote desktop control would be
possible)
Dozens of events: joined, left, started typing and so on
One to one and one to many streams
maybe more things
So how that all would be possible? It's extremely simple. In
client-server architecture, open connection to server, establish TLS
(for example) now send this next-gen protocol header message (to
specify what kind of data you're sending) and then send your data, it
could be for example a raw Theora stream (directly from your camera)
or just a text message. Then server just relays it to all parties.
I don't think it could be any simpler and nothing will beat this in
terms of performance. You've just some basic dependencies and what
would be the reason to complicate this? It's trivial to implement as
there's already libraries for those. In case of offline messages
server would just store them and in fact it could do same for sound
and videos too. You know Vine, Snapchat? what about YouTube,
Soundcloud, basically it could support them all. IMO this is millions
worth idea :D
It's worth looking at WebSockets, WebRTC and probably other standards.
I really don't understand why currently it all have been made so
complicated because it isn't.
