Re: [Standards] Support for stickers (custom emojis)

2019-10-26 Thread Tedd Sterr
I presume that the majority of implementations will do the UTF-8 decoding before/during XML parsing, so with offsets specified as bytes they will likely awkwardly re-encode the string again to be able to cross-reference these byte offsets with the codepoint* offsets they need. For those which

Re: [Standards] Support for stickers (custom emojis)

2019-10-25 Thread Sam Whited
On Fri, Oct 25, 2019, at 14:25, Marvin W wrote: > Yes and no. multi-codepoint emojis are still valid characters when > split, whereas multi-byte codepoints cannot be split. There is > nothing wrong with displaying the flag  as ​ *, so your > implementation is always capable in strictly

Re: [Standards] Support for stickers (custom emojis)

2019-10-25 Thread Marvin W
On 10/25/19 3:15 PM, Sam Whited wrote: On Thu, Oct 24, 2019, at 18:32, Marvin W wrote: XMPP uses UTF-8, and there's almost no reason to use anything but UTF-8. I do agree that this is true inside XMPP, but the data being transported inside XMPP might be transcoded to non-xmpp transport

Re: [Standards] Support for stickers (custom emojis)

2019-10-25 Thread Sam Whited
On Thu, Oct 24, 2019, at 18:32, Marvin W wrote: > 1) At some point we might want to allow the usage of UTF-16 or any >other encoding. Byte counts would have to be translated when re- >encoding which a server is probably unable to do generically. XMPP uses UTF-8, and there's almost no

Re: [Standards] Support for stickers (custom emojis)

2019-10-24 Thread Marvin W
On 10/24/19 9:40 PM, Kim Alvefur wrote: We should refrain from using things like grapheme clusters in wire formats, as those are subject to changes in upcoming Unicode versions and thus the wire format would be understood differently depending on the Unicode version implemented by the client.

Re: [Standards] Support for stickers (custom emojis)

2019-10-24 Thread Kim Alvefur
On Thu, Oct 24, 2019 at 08:32:04PM +0200, Marvin W wrote: > Thus, I would vote for using codepoints. I agree. > The rule should just be that clients should not do that on outgoing > data. I agree with this as well. > We should refrain from using things like grapheme clusters in wire formats, >

Re: [Standards] Support for stickers (custom emojis)

2019-10-24 Thread Marvin W
On 10/21/19 4:06 PM, Jonathan Lennox wrote: The right concept here is probably "grapheme clusters", as defined in Unicode Standard Annex 29. ICU has support for this. We should refrain from using things like grapheme clusters in wire formats, as those are subject to changes in upcoming

Re: [Standards] Support for stickers (custom emojis)

2019-10-24 Thread Andrey Gagarin
пн, 21 окт. 2019 г. в 19:08, Jonathan Lennox : > The right concept here is probably "grapheme clusters", as defined in > Unicode Standard Annex 29. ICU has support for this. > We have succeded implementing reference processing on three clients and on the server side. And not one of the

Re: [Standards] Support for stickers (custom emojis)

2019-10-21 Thread Sam Whited
On Sun, Oct 20, 2019, at 18:39, JC Brand wrote: > You don't need tons of ways, you can just follow the instructions. If > the sending client is buggy, then this will become clear over time. "Following the instructions" may mean different things to different clients in this case. One might treat

Re: [Standards] Support for stickers (custom emojis)

2019-10-21 Thread Sam Whited
On Mon, Oct 21, 2019, at 14:06, Jonathan Lennox wrote: > The right concept here is probably "grapheme clusters", as defined in > Unicode Standard Annex 29. ICU has support for this. This was also my suggestion at a summit a few years ago. However, the downside here is that it significantly

Re: [Standards] Support for stickers (custom emojis)

2019-10-21 Thread Jonathan Lennox
On Saturday, October 19 2019, "Sam Whited" wrote to "standards@xmpp.org" saying: > On Sat, Oct 19, 2019, at 04:57, JC Brand wrote: > > You might still have an offset in between two codepoints that should > > ideally be shown together like "EU" making the EU flag, but this seems > > less of an

Re: [Standards] Support for stickers (custom emojis)

2019-10-20 Thread JC Brand
On Sat, Oct 19, 2019 at 04:41:19PM +, Sam Whited wrote: > On Sat, Oct 19, 2019, at 04:57, JC Brand wrote: > > You might still have an offset in between two codepoints that should > > ideally be shown together like "EU" making the EU flag, but this seems > > less of an issue to me. > > I don't

Re: [Standards] Support for stickers (custom emojis)

2019-10-18 Thread JC Brand
On Thu, Oct 17, 2019 at 01:46:26PM +, Sam Whited wrote: > TL;DR — we should avoid using XEP-0372 until "TODO: define character > appropriately" is removed and resolved. XEP-0394 (Message Markup) works similarly to XEP-0372 and defines the "start" and "end" values in "units of unicode code

Re: [Standards] Support for stickers (custom emojis)

2019-10-17 Thread Kim Alvefur
On Thu, Oct 17, 2019 at 04:17:23PM +0100, Matthew Wild wrote: > On Thu, 17 Oct 2019 at 13:34, JC Brand wrote: > > "some other entity" isn't terribly well defined. How do I (or the > > recipient of my stickers) know what other entity to ask? > > It's part of the identifier, e.g. >

Re: [Standards] Support for stickers (custom emojis)

2019-10-17 Thread Matthew Wild
On Thu, 17 Oct 2019 at 16:17, Matthew Wild wrote: > > On Thu, 17 Oct 2019 at 13:34, JC Brand wrote: > > > > On Thu, Oct 17, 2019 at 01:23:18PM +0200, Marvin W wrote: > > > > > > - I already dislike the fact that we do HTTP requests to arbitrary servers > > > for file transfers, as we might be

Re: [Standards] Support for stickers (custom emojis)

2019-10-17 Thread Matthew Wild
On Thu, 17 Oct 2019 at 13:34, JC Brand wrote: > > On Thu, Oct 17, 2019 at 01:23:18PM +0200, Marvin W wrote: > > > - I already dislike the fact that we do HTTP requests to arbitrary servers > > for file transfers, as we might be leaking IP addresses in such cases. > > The file servers are usually

Re: [Standards] Support for stickers (custom emojis)

2019-10-17 Thread Sam Whited
TL;DR — we should avoid using XEP-0372 until "TODO: define character appropriately" is removed and resolved. On Thu, Oct 17, 2019, at 10:07, JC Brand wrote: > Instead, I propose that we use XEP-0372 references to indicate that > a particular shortname (e.g. :dancingpanda:) should be replaced with

Re: [Standards] Support for stickers (custom emojis)

2019-10-17 Thread Sergey Ilinykh
> I'm not sure what you mean. The "begin" and "end" attributes on the > element can encapsulate the shortname of the sticker. I mean where the encapsulated text has to be somehow presented to the user or fully replaced with the sticker as if it never was there. I believe in some cases (not

Re: [Standards] Support for stickers (custom emojis)

2019-10-17 Thread JC Brand
On Thu, Oct 17, 2019 at 03:05:32PM +0300, Sergey Ilinykh wrote: >Doesn't SIMS ([1]https://xmpp.org/extensions/xep-0385.html) resolve all >the concerns yet? Yes, SIMS is more comprehensive than my example. It's still uses a XEP-0372 reference, so it's conceptually similar. >We can

Re: [Standards] Support for stickers (custom emojis)

2019-10-17 Thread JC Brand
On Thu, Oct 17, 2019 at 01:23:18PM +0200, Marvin W wrote: > Hi, > Regarding your proposal: > - You should still add a hash in the reference somehow so that clients *can* > cache entries (even if you won't do it in Converse) Sure. > - I already dislike the fact that we do HTTP requests to

Re: [Standards] Support for stickers (custom emojis)

2019-10-17 Thread Sergey Ilinykh
Doesn't SIMS (https://xmpp.org/extensions/xep-0385.html) resolve all the concerns yet? We can have a with cid: uri too. The only thing is missed as for me is an attribute where the referenced text has to be removed or not. Best Regards, Sergey чт, 17 окт. 2019 г. в 14:24, Marvin W : > Hi, >

Re: [Standards] Support for stickers (custom emojis)

2019-10-17 Thread Marvin W
Hi, Regarding your proposal: - You should still add a hash in the reference somehow so that clients *can* cache entries (even if you won't do it in Converse) - I already dislike the fact that we do HTTP requests to arbitrary servers for file transfers, as we might be leaking IP addresses in

Re: [Standards] Support for stickers (custom emojis)

2019-10-17 Thread Philipp Hörist
Just make a HTTP HEAD request, look how big the content is then start to download. This is not a problem specific of stickers, you have that problem always when you want to download data, potentially the data can always be big. or you add an attribute size, and trust the sending client. regards

[Standards] Support for stickers (custom emojis)

2019-10-17 Thread JC Brand
Hello I'm currently working on adding support for non-unicode emojis to Converse.js. Currently, users can't upload their own images to be used for custom emojis, mostly because Converse is a thin client with no backend support for it. So to add custom emojis, the web host needs to edit a