Re: [Standards] xep-0313 missing features

[Lazar Otasevic]

if you set both it goes forwards (from old to new)? how to set direction
explicitly in such case?

On Tue, Mar 6, 2018 at 2:19 PM, Kevin Smith  wrote:

> On 6 Mar 2018, at 13:14, Jonas Wielicki  wrote:
> > I think it would be great to have a way to limit the MAM query to an
> end-ID
> > indeed. Matt, Kevin, any chance we get that in?
>
> Set both the before and after IDs. Done. Just need to make sure 313’s
> clear that this is supported, and that if done you page backwards.
>
> /K
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Philipp Hörist]

2018-03-06 14:19 GMT+01:00 Kevin Smith :

> On 6 Mar 2018, at 13:14, Jonas Wielicki  wrote:
> > I think it would be great to have a way to limit the MAM query to an
> end-ID
> > indeed. Matt, Kevin, any chance we get that in?
>
> Set both the before and after IDs. Done. Just need to make sure 313’s
> clear that this is supported, and that if done you page backwards.
>
> /K
>
>
That makes no sense at all.
If you want to filter a MAM Query then you add additional filter options.
Leave RSM alone.
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Jonas Wielicki]

On Dienstag, 6. März 2018 14:19:41 CET Kevin Smith wrote:
> On 6 Mar 2018, at 13:14, Jonas Wielicki  wrote:
> > I think it would be great to have a way to limit the MAM query to an
> > end-ID
> > indeed. Matt, Kevin, any chance we get that in?
> 
> Set both the before and after IDs. Done. Just need to make sure 313’s clear
> that this is supported, and that if done you page backwards.

Would you be updating RSM in the process or would this be a MAM-specific 
addition to RSM? The latter would be unfortunate both for client 
implementations (separation of concern) and for possible future development of 
RSM. The former would be unfortunate because reverse paging may not always be 
desirable for all RSM users when both  and  are given.

Am I missing something?

kind regards,
Jonas

signature.asc
Description: This is a digitally signed message part.
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Kevin Smith]

On 6 Mar 2018, at 13:14, Jonas Wielicki  wrote:
> I think it would be great to have a way to limit the MAM query to an end-ID 
> indeed. Matt, Kevin, any chance we get that in?

Set both the before and after IDs. Done. Just need to make sure 313’s clear 
that this is supported, and that if done you page backwards.

/K
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Jonas Wielicki]

On Dienstag, 6. März 2018 13:23:21 CET Lazar Otasevic wrote:
> Conclusion:
> 
> 1. it is not possible to iterate efficiently backwards by including both
> & in the query because once  is included in the query
> then it iterates forwards. that means when iterating backwards client has
> to omit  and fetch entire pages and then the last page will mostly
> overlap with some of the local messages, which is a waste.
> 2. it is not possible to determine "holes" in the archive reliably, because
> client can not know what is the last message archive-id, because our own
> sent messages have no feedback from the server once the message is archived
> what is its archive-id ... that means that client has to fetch ALL messages
> from MAM once again just to be sure that holes are filled, even though many
> message-bodies are already received/sent during live communication.
> 
> Basically In the current state all our own messages are "holes" in the
> local archive, not to mention all kind of "bad network" scenarios,
> multi-device and longer offline periods.
> 
> Making separate requests, one for archive ids and one for content would
> make:
> - much less waste in the sync because only ids would be wasted, and not the
> content
> - possible to fill multiple holes in one request by fetching content that
> is really needed
> - make possible for push payloads to contain only message ids (when clients
> want to handle encrypted messages locally by fetching them and only them)
> currently it is doable by giving to the push one id before the wanted
> message

So I might be wrong, but for the sake of it, I think there is a sane and not 
too complex way to do archive sync with MAM:

1. On startup, you put a "hole marker" at the end of your local archive. A 
hole marker is essentially just the stanza-id of the last message at the time 
the marker is created.
2. Iterate over all hole markers, from newest to oldest. Download everything 
between the last message *before* and the first message *behind* the hole 
marker. During download, move the hole marker accordingly to deal wtih 
disconnects while downloading. When finished, remove the hole marker; move on 
to the next hole marker if there’s any.

This should work and should also work with current semantics. I appreciate 
that there *might* be some overlap between the last page of a hole and already 
received messages. This is unfortunate, but can trivially be solved by 
comparing stanza-id (if available locally) or origin-id or message id in that 
order. (N.B.: once we get self-carbons, we’ll always have the stanza-id)

I also wrote that down in more detail here: 
https://github.com/jabbercat/jabbercat/issues/26#issuecomment-370333729


I think it would be great to have a way to limit the MAM query to an end-ID 
indeed. Matt, Kevin, any chance we get that in?

kind regards,
Jonas

signature.asc
Description: This is a digitally signed message part.
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Jonas Wielicki]

On Dienstag, 6. März 2018 14:02:49 CET Philipp Hörist wrote:
> You dont need to determine holes, maybe you should first come up with a
> scenario where this "hole" of yours can even happen.
> 
> 1. if you receive a message live you get the archive-id, then you save it.
> 2. You go offline
> 3. Once you come later you request after = archive-id
> 4. repeat

Disconnect *while* (3) is in progress and after you have received at least one 
live message.

kind regards,
Jonas

signature.asc
Description: This is a digitally signed message part.
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Philipp Hörist]

You dont need to determine holes, maybe you should first come up with a
scenario where this "hole" of yours can even happen.

1. if you receive a message live you get the archive-id, then you save it.
2. You go offline
3. Once you come later you request after = archive-id
4. repeat

Please show me what can go wrong with this simple approach

So the only thing that would possibly overlap here is a message that you
sent after the last message you received, so in most cases that is 1-2
messages.
Yeah technically a waste but there is no efficient way to get the
archive-id of a sent message, its always going to need another stanza from
the server to tell you.



2018-03-06 13:23 GMT+01:00 Lazar Otasevic :

> Conclusion:
>
> 1. it is not possible to iterate efficiently backwards by including both
> & in the query because once  is included in the query
> then it iterates forwards. that means when iterating backwards client has
> to omit  and fetch entire pages and then the last page will mostly
> overlap with some of the local messages, which is a waste.
> 2. it is not possible to determine "holes" in the archive reliably,
> because client can not know what is the last message archive-id, because
> our own sent messages have no feedback from the server once the message is
> archived what is its archive-id ... that means that client has to fetch ALL
> messages from MAM once again just to be sure that holes are filled, even
> though many message-bodies are already received/sent during live
> communication.
>
> Basically In the current state all our own messages are "holes" in the
> local archive, not to mention all kind of "bad network" scenarios,
> multi-device and longer offline periods.
>
> Making separate requests, one for archive ids and one for content would
> make:
> - much less waste in the sync because only ids would be wasted, and not
> the content
> - possible to fill multiple holes in one request by fetching content that
> is really needed
> - make possible for push payloads to contain only message ids (when
> clients want to handle encrypted messages locally by fetching them and only
> them) currently it is doable by giving to the push one id before the wanted
> message
>
> Thanks.
>
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Lazar Otasevic]

Conclusion:

1. it is not possible to iterate efficiently backwards by including both
& in the query because once  is included in the query
then it iterates forwards. that means when iterating backwards client has
to omit  and fetch entire pages and then the last page will mostly
overlap with some of the local messages, which is a waste.
2. it is not possible to determine "holes" in the archive reliably, because
client can not know what is the last message archive-id, because our own
sent messages have no feedback from the server once the message is archived
what is its archive-id ... that means that client has to fetch ALL messages
from MAM once again just to be sure that holes are filled, even though many
message-bodies are already received/sent during live communication.

Basically In the current state all our own messages are "holes" in the
local archive, not to mention all kind of "bad network" scenarios,
multi-device and longer offline periods.

Making separate requests, one for archive ids and one for content would
make:
- much less waste in the sync because only ids would be wasted, and not the
content
- possible to fill multiple holes in one request by fetching content that
is really needed
- make possible for push payloads to contain only message ids (when clients
want to handle encrypted messages locally by fetching them and only them)
currently it is doable by giving to the push one id before the wanted
message

Thanks.
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Lazar Otasevic]

yes thats what i thought - carbon would be good

On Mon, Mar 5, 2018 at 6:14 PM, Kevin Smith  wrote:

> On 5 Mar 2018, at 16:42, Lazar Otasevic  wrote:
> > @Philipp BTW, how do you handle own messages? how does client discover
> archived id for newly sent message?
>
> It’s a thing we still have to sort out (probably by having Carbons send
> your own messages back including the archive id). We know that one’s a
> problem.
>
> /K
>
> >
> >
> > On Mon, Mar 5, 2018 at 4:48 PM, Lazar Otasevic 
> wrote:
> > Kevin suggested to use both (before..after) and i see no way to query
> first 10 or last 10, its a must have.
> > I know the count may be omitted and that makes it more difficult because
> if i query always 10 - before then i may fetch more than i need - its a
> waste
> >
> > On Mon, Mar 5, 2018 at 2:12 PM, Philipp Hörist 
> wrote:
> > 
> >   
> > 
> >   
> > urn:xmpp:mam:2
> >   
> > 
> >
> >   10
> >   
> > 
> >   
> > 
> >
> > This gives you the last 10 messages of the whole Archive, you can add
> time / jid filtering to get the last 10 from a JID or the last 10 inside a
> timeframe
> >
> > 2018-03-05 14:06 GMT+01:00 Lazar Otasevic :
> > because, why not. clients should be able to choose direction i guess.
> not every needs to work the same?
> >
> > On Mon, Mar 5, 2018 at 1:56 PM, Philipp Hörist 
> wrote:
> >
> > 2018-03-05 13:42 GMT+01:00 Lazar :
> > yes we need explicit controll to request last 10 or first 10 results
> then, i dont see it possible now
> >
> >
> >
> > why do you need that? You can definitly do a query like that, but i dont
> see why you ever want to do that.
> >
> > ___
> > Standards mailing list
> > Info: https://mail.jabber.org/mailman/listinfo/standards
> > Unsubscribe: standards-unsubscr...@xmpp.org
> > ___
> >
> >
> >
> > ___
> > Standards mailing list
> > Info: https://mail.jabber.org/mailman/listinfo/standards
> > Unsubscribe: standards-unsubscr...@xmpp.org
> > ___
> >
> >
> >
> > ___
> > Standards mailing list
> > Info: https://mail.jabber.org/mailman/listinfo/standards
> > Unsubscribe: standards-unsubscr...@xmpp.org
> > ___
> >
> >
> >
> > ___
> > Standards mailing list
> > Info: https://mail.jabber.org/mailman/listinfo/standards
> > Unsubscribe: standards-unsubscr...@xmpp.org
> > ___
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Kevin Smith]

On 5 Mar 2018, at 16:42, Lazar Otasevic  wrote:
> @Philipp BTW, how do you handle own messages? how does client discover 
> archived id for newly sent message?

It’s a thing we still have to sort out (probably by having Carbons send your 
own messages back including the archive id). We know that one’s a problem.

/K

> 
> 
> On Mon, Mar 5, 2018 at 4:48 PM, Lazar Otasevic  wrote:
> Kevin suggested to use both (before..after) and i see no way to query first 
> 10 or last 10, its a must have.
> I know the count may be omitted and that makes it more difficult because if i 
> query always 10 - before then i may fetch more than i need - its a waste
> 
> On Mon, Mar 5, 2018 at 2:12 PM, Philipp Hörist  wrote:
> 
>   
> 
>   
> urn:xmpp:mam:2
>   
> 
>
>   10
>   
> 
>   
> 
> 
> This gives you the last 10 messages of the whole Archive, you can add time / 
> jid filtering to get the last 10 from a JID or the last 10 inside a timeframe
> 
> 2018-03-05 14:06 GMT+01:00 Lazar Otasevic :
> because, why not. clients should be able to choose direction i guess. not 
> every needs to work the same?
> 
> On Mon, Mar 5, 2018 at 1:56 PM, Philipp Hörist  wrote:
> 
> 2018-03-05 13:42 GMT+01:00 Lazar :
> yes we need explicit controll to request last 10 or first 10 results then, i 
> dont see it possible now
> 
> 
> 
> why do you need that? You can definitly do a query like that, but i dont see 
> why you ever want to do that.
> 
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
> 
> 
> 
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
> 
> 
> 
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
> 
> 
> 
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___

___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Lazar Otasevic]

Yes i thought so... then my own messages(content) will come trough MAM
again, even though I already have them, and its a waste of traffic :(
Huh, i guess i have to live with it, but i would really appreciate if
something more efficient exists in XMPP.
In this case even my initial proposal with fetching ids only wouldn't help.
We would need pairs of (origin-id archived-id) but its too much.

Thaniks

On Mon, Mar 5, 2018 at 6:03 PM, Philipp Hörist  wrote:

> you cant discover your archive id on outgoing messages
>
> you set a 'origin-id' on the outgoing message https://xmpp.org/
> extensions/xep-0359.html
>
> if you request later the archive and discover a origin-id you can check
> your db if it is already present
>
>
>
> 2018-03-05 17:42 GMT+01:00 Lazar Otasevic :
>
>> @Philipp BTW, how do you handle own messages? how does client discover
>> archived id for newly sent message?
>>
>>
>> On Mon, Mar 5, 2018 at 4:48 PM, Lazar Otasevic 
>> wrote:
>>
>>> Kevin suggested to use both (before..after) and i see no way to query
>>> first 10 or last 10, its a must have.
>>> I know the count may be omitted and that makes it more difficult because
>>> if i query always 10 - before then i may fetch more than i need - its a
>>> waste
>>>
>>> On Mon, Mar 5, 2018 at 2:12 PM, Philipp Hörist 
>>> wrote:
>>>
 
   
 
   
 urn:xmpp:mam:2
   
 

   10
   
 
   
 

 This gives you the last 10 messages of the whole Archive, you can add
 time / jid filtering to get the last 10 from a JID or the last 10 inside a
 timeframe

 2018-03-05 14:06 GMT+01:00 Lazar Otasevic :

> because, why not. clients should be able to choose direction i guess.
> not every needs to work the same?
>
> On Mon, Mar 5, 2018 at 1:56 PM, Philipp Hörist 
> wrote:
>
>>
>> 2018-03-05 13:42 GMT+01:00 Lazar :
>>
>>> yes we need explicit controll to request last 10 or first 10 results
>>> then, i dont see it possible now
>>>
>>>
>>>
>> why do you need that? You can definitly do a query like that, but i
>> dont see why you ever want to do that.
>>
>> ___
>> Standards mailing list
>> Info: https://mail.jabber.org/mailman/listinfo/standards
>> Unsubscribe: standards-unsubscr...@xmpp.org
>> ___
>>
>>
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
>

 ___
 Standards mailing list
 Info: https://mail.jabber.org/mailman/listinfo/standards
 Unsubscribe: standards-unsubscr...@xmpp.org
 ___


>>>
>>
>> ___
>> Standards mailing list
>> Info: https://mail.jabber.org/mailman/listinfo/standards
>> Unsubscribe: standards-unsubscr...@xmpp.org
>> ___
>>
>>
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Philipp Hörist]

you cant discover your archive id on outgoing messages

you set a 'origin-id' on the outgoing message
https://xmpp.org/extensions/xep-0359.html

if you request later the archive and discover a origin-id you can check
your db if it is already present



2018-03-05 17:42 GMT+01:00 Lazar Otasevic :

> @Philipp BTW, how do you handle own messages? how does client discover
> archived id for newly sent message?
>
>
> On Mon, Mar 5, 2018 at 4:48 PM, Lazar Otasevic 
> wrote:
>
>> Kevin suggested to use both (before..after) and i see no way to query
>> first 10 or last 10, its a must have.
>> I know the count may be omitted and that makes it more difficult because
>> if i query always 10 - before then i may fetch more than i need - its a
>> waste
>>
>> On Mon, Mar 5, 2018 at 2:12 PM, Philipp Hörist 
>> wrote:
>>
>>> 
>>>   
>>> 
>>>   
>>> urn:xmpp:mam:2
>>>   
>>> 
>>>
>>>   10
>>>   
>>> 
>>>   
>>> 
>>>
>>> This gives you the last 10 messages of the whole Archive, you can add
>>> time / jid filtering to get the last 10 from a JID or the last 10 inside a
>>> timeframe
>>>
>>> 2018-03-05 14:06 GMT+01:00 Lazar Otasevic :
>>>
 because, why not. clients should be able to choose direction i guess.
 not every needs to work the same?

 On Mon, Mar 5, 2018 at 1:56 PM, Philipp Hörist 
 wrote:

>
> 2018-03-05 13:42 GMT+01:00 Lazar :
>
>> yes we need explicit controll to request last 10 or first 10 results
>> then, i dont see it possible now
>>
>>
>>
> why do you need that? You can definitly do a query like that, but i
> dont see why you ever want to do that.
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
>

 ___
 Standards mailing list
 Info: https://mail.jabber.org/mailman/listinfo/standards
 Unsubscribe: standards-unsubscr...@xmpp.org
 ___


>>>
>>> ___
>>> Standards mailing list
>>> Info: https://mail.jabber.org/mailman/listinfo/standards
>>> Unsubscribe: standards-unsubscr...@xmpp.org
>>> ___
>>>
>>>
>>
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Lazar Otasevic]

@Philipp BTW, how do you handle own messages? how does client discover
archived id for newly sent message?


On Mon, Mar 5, 2018 at 4:48 PM, Lazar Otasevic  wrote:

> Kevin suggested to use both (before..after) and i see no way to query
> first 10 or last 10, its a must have.
> I know the count may be omitted and that makes it more difficult because
> if i query always 10 - before then i may fetch more than i need - its a
> waste
>
> On Mon, Mar 5, 2018 at 2:12 PM, Philipp Hörist 
> wrote:
>
>> 
>>   
>> 
>>   
>> urn:xmpp:mam:2
>>   
>> 
>>
>>   10
>>   
>> 
>>   
>> 
>>
>> This gives you the last 10 messages of the whole Archive, you can add
>> time / jid filtering to get the last 10 from a JID or the last 10 inside a
>> timeframe
>>
>> 2018-03-05 14:06 GMT+01:00 Lazar Otasevic :
>>
>>> because, why not. clients should be able to choose direction i guess.
>>> not every needs to work the same?
>>>
>>> On Mon, Mar 5, 2018 at 1:56 PM, Philipp Hörist 
>>> wrote:
>>>

 2018-03-05 13:42 GMT+01:00 Lazar :

> yes we need explicit controll to request last 10 or first 10 results
> then, i dont see it possible now
>
>
>
 why do you need that? You can definitly do a query like that, but i
 dont see why you ever want to do that.

 ___
 Standards mailing list
 Info: https://mail.jabber.org/mailman/listinfo/standards
 Unsubscribe: standards-unsubscr...@xmpp.org
 ___


>>>
>>> ___
>>> Standards mailing list
>>> Info: https://mail.jabber.org/mailman/listinfo/standards
>>> Unsubscribe: standards-unsubscr...@xmpp.org
>>> ___
>>>
>>>
>>
>> ___
>> Standards mailing list
>> Info: https://mail.jabber.org/mailman/listinfo/standards
>> Unsubscribe: standards-unsubscr...@xmpp.org
>> ___
>>
>>
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Lazar Otasevic]

Kevin suggested to use both (before..after) and i see no way to query first
10 or last 10, its a must have.
I know the count may be omitted and that makes it more difficult because if
i query always 10 - before then i may fetch more than i need - its a waste

On Mon, Mar 5, 2018 at 2:12 PM, Philipp Hörist  wrote:

> 
>   
> 
>   
> urn:xmpp:mam:2
>   
> 
>
>   10
>   
> 
>   
> 
>
> This gives you the last 10 messages of the whole Archive, you can add time
> / jid filtering to get the last 10 from a JID or the last 10 inside a
> timeframe
>
> 2018-03-05 14:06 GMT+01:00 Lazar Otasevic :
>
>> because, why not. clients should be able to choose direction i guess. not
>> every needs to work the same?
>>
>> On Mon, Mar 5, 2018 at 1:56 PM, Philipp Hörist 
>> wrote:
>>
>>>
>>> 2018-03-05 13:42 GMT+01:00 Lazar :
>>>
 yes we need explicit controll to request last 10 or first 10 results
 then, i dont see it possible now



>>> why do you need that? You can definitly do a query like that, but i dont
>>> see why you ever want to do that.
>>>
>>> ___
>>> Standards mailing list
>>> Info: https://mail.jabber.org/mailman/listinfo/standards
>>> Unsubscribe: standards-unsubscr...@xmpp.org
>>> ___
>>>
>>>
>>
>> ___
>> Standards mailing list
>> Info: https://mail.jabber.org/mailman/listinfo/standards
>> Unsubscribe: standards-unsubscr...@xmpp.org
>> ___
>>
>>
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Philipp Hörist]

I just want to state again, using MAM like that you will run into problems.

It feels like you want to bend MAM to something you are fixed about, but
MAM works only good if you use it like it was intended.
And no, you cant satisfy all possible use styles of syncing things in one
XEP.

2018-03-05 14:20 GMT+01:00 Philipp Hörist :

> No.
>
> XEP
> Note: The  element MAY be omitted
>
> 2018-03-05 14:12 GMT+01:00 Lazar Otasevic :
>
>> does response has to return count?
>>
>> On Mon, Mar 5, 2018 at 2:06 PM, Lazar Otasevic 
>> wrote:
>>
>>> because, why not. clients should be able to choose direction i guess.
>>> not every needs to work the same?
>>>
>>> On Mon, Mar 5, 2018 at 1:56 PM, Philipp Hörist 
>>> wrote:
>>>

 2018-03-05 13:42 GMT+01:00 Lazar :

> yes we need explicit controll to request last 10 or first 10 results
> then, i dont see it possible now
>
>
>
 why do you need that? You can definitly do a query like that, but i
 dont see why you ever want to do that.

 ___
 Standards mailing list
 Info: https://mail.jabber.org/mailman/listinfo/standards
 Unsubscribe: standards-unsubscr...@xmpp.org
 ___


>>>
>>
>> ___
>> Standards mailing list
>> Info: https://mail.jabber.org/mailman/listinfo/standards
>> Unsubscribe: standards-unsubscr...@xmpp.org
>> ___
>>
>>
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Philipp Hörist]

No.

XEP
Note: The  element MAY be omitted

2018-03-05 14:12 GMT+01:00 Lazar Otasevic :

> does response has to return count?
>
> On Mon, Mar 5, 2018 at 2:06 PM, Lazar Otasevic 
> wrote:
>
>> because, why not. clients should be able to choose direction i guess. not
>> every needs to work the same?
>>
>> On Mon, Mar 5, 2018 at 1:56 PM, Philipp Hörist 
>> wrote:
>>
>>>
>>> 2018-03-05 13:42 GMT+01:00 Lazar :
>>>
 yes we need explicit controll to request last 10 or first 10 results
 then, i dont see it possible now



>>> why do you need that? You can definitly do a query like that, but i dont
>>> see why you ever want to do that.
>>>
>>> ___
>>> Standards mailing list
>>> Info: https://mail.jabber.org/mailman/listinfo/standards
>>> Unsubscribe: standards-unsubscr...@xmpp.org
>>> ___
>>>
>>>
>>
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Lazar Otasevic]

does response has to return count?

On Mon, Mar 5, 2018 at 2:06 PM, Lazar Otasevic  wrote:

> because, why not. clients should be able to choose direction i guess. not
> every needs to work the same?
>
> On Mon, Mar 5, 2018 at 1:56 PM, Philipp Hörist 
> wrote:
>
>>
>> 2018-03-05 13:42 GMT+01:00 Lazar :
>>
>>> yes we need explicit controll to request last 10 or first 10 results
>>> then, i dont see it possible now
>>>
>>>
>>>
>> why do you need that? You can definitly do a query like that, but i dont
>> see why you ever want to do that.
>>
>> ___
>> Standards mailing list
>> Info: https://mail.jabber.org/mailman/listinfo/standards
>> Unsubscribe: standards-unsubscr...@xmpp.org
>> ___
>>
>>
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Philipp Hörist]

 
  

  
urn:xmpp:mam:2
  

   
  10
  

  


This gives you the last 10 messages of the whole Archive, you can add time
/ jid filtering to get the last 10 from a JID or the last 10 inside a
timeframe

2018-03-05 14:06 GMT+01:00 Lazar Otasevic :

> because, why not. clients should be able to choose direction i guess. not
> every needs to work the same?
>
> On Mon, Mar 5, 2018 at 1:56 PM, Philipp Hörist 
> wrote:
>
>>
>> 2018-03-05 13:42 GMT+01:00 Lazar :
>>
>>> yes we need explicit controll to request last 10 or first 10 results
>>> then, i dont see it possible now
>>>
>>>
>>>
>> why do you need that? You can definitly do a query like that, but i dont
>> see why you ever want to do that.
>>
>> ___
>> Standards mailing list
>> Info: https://mail.jabber.org/mailman/listinfo/standards
>> Unsubscribe: standards-unsubscr...@xmpp.org
>> ___
>>
>>
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Lazar Otasevic]

because, why not. clients should be able to choose direction i guess. not
every needs to work the same?

On Mon, Mar 5, 2018 at 1:56 PM, Philipp Hörist  wrote:

>
> 2018-03-05 13:42 GMT+01:00 Lazar :
>
>> yes we need explicit controll to request last 10 or first 10 results
>> then, i dont see it possible now
>>
>>
>>
> why do you need that? You can definitly do a query like that, but i dont
> see why you ever want to do that.
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Philipp Hörist]

2018-03-05 13:42 GMT+01:00 Lazar :

> yes we need explicit controll to request last 10 or first 10 results then,
> i dont see it possible now
>
>
>
why do you need that? You can definitly do a query like that, but i dont
see why you ever want to do that.
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Philipp Hörist]

@Lazar

Maybe it helps to write down how most xmpp devs use MAM

Say your user starts with 0 messages locally

your first query to MAM will only use timestamp say 1 day back from the
current time


  

  
urn:xmpp:mam:2
  
  
2018-03-04T00:00:00Z
  

  


you record now the mam id of the last message you received.

and then whenever user loses connection or you feel you want to sync the
archive you execute


  

  
urn:xmpp:mam:2
  

   
  100
  SAVED-MAM-ID

  


thats it, you will request all messages since the last known mam id, there
cant be any holes.

if you want to request into the other direction you have to record the
first timestamp of the first mam message, and request from there back with
timestamps (from 01.03.2018 - 03.04.2018 for example)

if you fetch the messages from your db, you should order first for
timestamp and in a second step for row-id

regards

2018-03-05 13:28 GMT+01:00 Philipp Hörist :

> RSM turns your result set into smaller subsets specified with the page
> size
>
> 10
>
> then you can switch one page forward with  or one page backwards
> with  thats the idea behind RSM
>
> there is no RSM inside a RSM.
>
> 2018-03-05 12:29 GMT+01:00 Edwin Mons :
>
>> Why would it not be allowed?  The only confusing thing there would be if
>> your (possibly server-mandated) page size is smaller than the number of
>> items between before and after, as before and after have conflicting
>> indicators of which part of the items you'd want to receive.
>>
>> Edwin
>>
>>
>>
>>
>> On 05/03/2018 12:11, Lazar Otasevic wrote:
>>
>> Kevin says we can :) I asked him to clarify... we'll see.
>>
>> On Mon, Mar 5, 2018 at 11:45 AM, Philipp Hörist 
>> wrote:
>>
>>>

 You can already fill holes with the current spec, by specifying both
  and  on the RSM query can’t you? If that doesn’t
 obviously follow from the text, that’s something we should clarify.



>>> No i dont think you can do that. You can either specify  OR
>>> .
>>> RSM is for going one page further or backwards with  and 
>>> inside a already filtered result, you cannot filter the result again with
>>> upper and lower limits.
>>>
>>>
>>> ___
>>> Standards mailing list
>>> Info: https://mail.jabber.org/mailman/listinfo/standards
>>> Unsubscribe: standards-unsubscr...@xmpp.org
>>> ___
>>>
>>>
>>
>>
>> ___
>> Standards mailing list
>> Info: https://mail.jabber.org/mailman/listinfo/standards
>> Unsubscribe: standards-unsubscr...@xmpp.org
>> ___
>>
>>
>>
>> ___
>> Standards mailing list
>> Info: https://mail.jabber.org/mailman/listinfo/standards
>> Unsubscribe: standards-unsubscr...@xmpp.org
>> ___
>>
>>
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Lazar]

yes we need explicit controll to request last 10 or first 10 results then, i 
dont see it possible now

Sent from my iPhone

> On 5. Mar 2018, at 13:28, Philipp Hörist  wrote:
> 
> RSM turns your result set into smaller subsets specified with the page size 
> 
> 10
> 
> then you can switch one page forward with  or one page backwards with 
>  thats the idea behind RSM
> 
> there is no RSM inside a RSM.
> 
> 2018-03-05 12:29 GMT+01:00 Edwin Mons :
>> Why would it not be allowed?  The only confusing thing there would be if 
>> your (possibly server-mandated) page size is smaller than the number of 
>> items between before and after, as before and after have conflicting 
>> indicators of which part of the items you'd want to receive.
>> 
>> Edwin
>> 
>> 
>> 
>> 
>>> On 05/03/2018 12:11, Lazar Otasevic wrote:
>>> Kevin says we can :) I asked him to clarify... we'll see.
>>> 
 On Mon, Mar 5, 2018 at 11:45 AM, Philipp Hörist  
 wrote:
> 
> 
> You can already fill holes with the current spec, by specifying both 
>  and  on the RSM query can’t you? If that doesn’t 
> obviously follow from the text, that’s something we should clarify.
> 
> 
 
 No i dont think you can do that. You can either specify  OR 
 .
 RSM is for going one page further or backwards with 
  and  inside a already filtered result, you cannot filter 
 the result again with upper and lower limits.
 
 
 ___
 Standards mailing list
 Info: https://mail.jabber.org/mailman/listinfo/standards
 Unsubscribe: standards-unsubscr...@xmpp.org
 ___
 
>>> 
>>> 
>>> 
>>> ___
>>> Standards mailing list
>>> Info: https://mail.jabber.org/mailman/listinfo/standards
>>> Unsubscribe: standards-unsubscr...@xmpp.org
>>> ___
>> 
>> 
>> ___
>> Standards mailing list
>> Info: https://mail.jabber.org/mailman/listinfo/standards
>> Unsubscribe: standards-unsubscr...@xmpp.org
>> ___
>> 
> 
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Philipp Hörist]

RSM turns your result set into smaller subsets specified with the page size

10

then you can switch one page forward with  or one page backwards
with  thats the idea behind RSM

there is no RSM inside a RSM.

2018-03-05 12:29 GMT+01:00 Edwin Mons :

> Why would it not be allowed?  The only confusing thing there would be if
> your (possibly server-mandated) page size is smaller than the number of
> items between before and after, as before and after have conflicting
> indicators of which part of the items you'd want to receive.
>
> Edwin
>
>
>
>
> On 05/03/2018 12:11, Lazar Otasevic wrote:
>
> Kevin says we can :) I asked him to clarify... we'll see.
>
> On Mon, Mar 5, 2018 at 11:45 AM, Philipp Hörist 
> wrote:
>
>>
>>>
>>> You can already fill holes with the current spec, by specifying both
>>>  and  on the RSM query can’t you? If that doesn’t
>>> obviously follow from the text, that’s something we should clarify.
>>>
>>>
>>>
>> No i dont think you can do that. You can either specify  OR
>> .
>> RSM is for going one page further or backwards with  and 
>> inside a already filtered result, you cannot filter the result again with
>> upper and lower limits.
>>
>>
>> ___
>> Standards mailing list
>> Info: https://mail.jabber.org/mailman/listinfo/standards
>> Unsubscribe: standards-unsubscr...@xmpp.org
>> ___
>>
>>
>
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
>
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Edwin Mons]

Why would it not be allowed?  The only confusing thing there would be if
your (possibly server-mandated) page size is smaller than the number of
items between before and after, as before and after have conflicting
indicators of which part of the items you'd want to receive.

Edwin



On 05/03/2018 12:11, Lazar Otasevic wrote:
> Kevin says we can :) I asked him to clarify... we'll see.
>
> On Mon, Mar 5, 2018 at 11:45 AM, Philipp Hörist  > wrote:
>
>
>
> You can already fill holes with the current spec, by
> specifying both  and  on the RSM query can’t
> you? If that doesn’t obviously follow from the text, that’s
> something we should clarify.
>
>
>
> No i dont think you can do that. You can either specify 
> OR .
> RSM is for going one page further or backwards with  and
>  inside a already filtered result, you cannot filter the
> result again with upper and lower limits.
>
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> 
> Unsubscribe: standards-unsubscr...@xmpp.org
> 
> ___
>
>
>
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___


___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Lazar Otasevic]

Kevin says we can :) I asked him to clarify... we'll see.

On Mon, Mar 5, 2018 at 11:45 AM, Philipp Hörist  wrote:

>
>>
>> You can already fill holes with the current spec, by specifying both
>>  and  on the RSM query can’t you? If that doesn’t
>> obviously follow from the text, that’s something we should clarify.
>>
>>
>>
> No i dont think you can do that. You can either specify  OR
> .
> RSM is for going one page further or backwards with  and 
> inside a already filtered result, you cannot filter the result again with
> upper and lower limits.
>
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Lazar Otasevic]

Hi Kevin,

I guess all of your questions are self explanatory, no need for answer,
because client can fit to MAM logic, but I would ask you to clarify just
this one:

> You can already fill holes with the current spec, by specifying both
 and  on the RSM query can’t you? If that doesn’t
obviously follow from the text, that’s something we should clarify.

I didn't yet get it clear how to specify direction when requesting with
both  and ?
I guess its doable, but one small detail is how can client be sure of the
correct MAM order in certain race conditions (latest sent message VS latest
received message) without using dates?

BTW Making things layered (id skeleton + content) is not the same as making
them "more complicated" its just making them more flexible, but ok,
irrelevant. I repeat MAM removed something that we already had before. Just
for my push example, when client receives only the ids in the payload -
then I have to provide entire message inside the payload? Because client
can not get a specific message(s) on demand by using id(s)... I guess
clients have to live with that.

Thanks for support.



On Mon, Mar 5, 2018 at 10:45 AM, Kevin Smith  wrote:

> On 4 Mar 2018, at 22:01, Lazar Otasevic  wrote:
> >> I really don't understand this argument. Doing a MAM query is exactly
> >> like fetching a list of ids, except that you also get the content as
> >> well. As I mentioned, I see no case in which you would want the ids
> >> and not the content. Even if you get the ids, you will fetch the
> >> content later.
> >
> > yes, MAM query is exactly "like" fetching a list of ids combined with
> content, but thats the problem i have from the start :)
> >
> > getting the ids+content in the same query is a waste in some (many)
> cases because local archive can have holes in it when network is bad and
> lots of disconnects happen (for example live messages come before sync is
> completed and then disconnect happens = next time we receive immediately
> some live messages and we have holes)
>
> Is there ever a situation where you want the id but don’t want the
> message? I can’t see why that would be the case - you don’t for hole
> filling, or recent sync. If you always want the body of all the messages
> you’re querying then doing multiple queries is unnecessary effort for both
> the client and server.
>
> > even when we dont have a hole, we may trigger a page fetch of 10
> messages where infect we need only 2-3, and again we have a waste
>
> In the case that there’s no hole, why would you ask for more messages than
> you need?
>
> > in my proposal waste would be only some of the ids discarder, so very
> little.
>
> Yet with MAM you already can do it with no waste, so I’m not sure what
> this is buying.
>
> > good thing is that ids can come immediately in iq response and client
> don't need to collect it out of the stream like in MAM now. and more good
> thing see below…
>
> They still need to do the latter for the subsequent content query, so I
> don’t see why this is an improvement.
>
> >> If you already have the content, why would the client ever perform a
> >> query that overlaps with ranges of messages that it already has? Just
> >> none of this makes any sense to me
> >
> > see previous - holes
>
> You can already fill holes with the current spec, by specifying both
>  and  on the RSM query can’t you? If that doesn’t
> obviously follow from the text, that’s something we should clarify.
>
> >> > I thought the goal of XMPP is to make clients less complex and more
> >> > reliable? Why is adding two (basic) features counted as increasing
> >> > complexity? I say basic because those features were kind of existent
> back in
> >> > 2005 in xep-0013. Now we removed those features, why?
> >>
> >> Can you explain what your ideal client logic would be? And how it is
> >> more simple than "the latest message I have has ID X, request all
> >> messages since X"?
> >
> > yes, i can give you not only ideal logic but also ideal SW design of a
> client...
> >
> > lets say I make client from 3 components: sender, receiver and sync.
> > goal is to have them decoupled, and the only shared part will be the
> local message archive
> >
> > here is how the archive is filled:
> > 1. sender - fills archive with message content when outgoing message is
> confirmed to be sent
> > 2. receiver - fills archive with message content when it receives a live
> message
> > 3. sync is the only one that takes care of the order, and also filling
> the holes. order is implicitly given by list of ids received from the
> server, and holes are easily detected by simple query to archive and then
> potential holes filled by an extra request (with collection of ids)
> >
> > note how everything can work decoupled, no special handling for sync on
> connect/disconnect/send/receive, no id tracking, etc... simple and no
> waste (except a bit of ids), every component can work with/without every
> other.
>
> 

Re: [Standards] xep-0313 missing features

[Philipp Hörist]

>
>
>
> You can already fill holes with the current spec, by specifying both
>  and  on the RSM query can’t you? If that doesn’t
> obviously follow from the text, that’s something we should clarify.
>
>
>
No i dont think you can do that. You can either specify  OR .
RSM is for going one page further or backwards with  and 
inside a already filtered result, you cannot filter the result again with
upper and lower limits.
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Kevin Smith]

On 4 Mar 2018, at 22:01, Lazar Otasevic  wrote:
>> I really don't understand this argument. Doing a MAM query is exactly
>> like fetching a list of ids, except that you also get the content as
>> well. As I mentioned, I see no case in which you would want the ids
>> and not the content. Even if you get the ids, you will fetch the
>> content later.
>  
> yes, MAM query is exactly "like" fetching a list of ids combined with 
> content, but thats the problem i have from the start :)
> 
> getting the ids+content in the same query is a waste in some (many) cases 
> because local archive can have holes in it when network is bad and lots of 
> disconnects happen (for example live messages come before sync is completed 
> and then disconnect happens = next time we receive immediately some live 
> messages and we have holes)

Is there ever a situation where you want the id but don’t want the message? I 
can’t see why that would be the case - you don’t for hole filling, or recent 
sync. If you always want the body of all the messages you’re querying then 
doing multiple queries is unnecessary effort for both the client and server.

> even when we dont have a hole, we may trigger a page fetch of 10 messages 
> where infect we need only 2-3, and again we have a waste

In the case that there’s no hole, why would you ask for more messages than you 
need?

> in my proposal waste would be only some of the ids discarder, so very little.

Yet with MAM you already can do it with no waste, so I’m not sure what this is 
buying.

> good thing is that ids can come immediately in iq response and client don't 
> need to collect it out of the stream like in MAM now. and more good thing see 
> below…

They still need to do the latter for the subsequent content query, so I don’t 
see why this is an improvement.

>> If you already have the content, why would the client ever perform a
>> query that overlaps with ranges of messages that it already has? Just
>> none of this makes any sense to me 
> 
> see previous - holes

You can already fill holes with the current spec, by specifying both  
and  on the RSM query can’t you? If that doesn’t obviously follow from 
the text, that’s something we should clarify.

>> > I thought the goal of XMPP is to make clients less complex and more
>> > reliable? Why is adding two (basic) features counted as increasing
>> > complexity? I say basic because those features were kind of existent back 
>> > in
>> > 2005 in xep-0013. Now we removed those features, why?
>> 
>> Can you explain what your ideal client logic would be? And how it is
>> more simple than "the latest message I have has ID X, request all
>> messages since X"?
> 
> yes, i can give you not only ideal logic but also ideal SW design of a 
> client...
> 
> lets say I make client from 3 components: sender, receiver and sync.
> goal is to have them decoupled, and the only shared part will be the local 
> message archive
> 
> here is how the archive is filled:
> 1. sender - fills archive with message content when outgoing message is 
> confirmed to be sent 
> 2. receiver - fills archive with message content when it receives a live 
> message
> 3. sync is the only one that takes care of the order, and also filling the 
> holes. order is implicitly given by list of ids received from the server, and 
> holes are easily detected by simple query to archive and then potential holes 
> filled by an extra request (with collection of ids)
> 
> note how everything can work decoupled, no special handling for sync on 
> connect/disconnect/send/receive, no id tracking, etc... simple and no waste 
> (except a bit of ids), every component can work with/without every other.

You’re removing a small bit of work for the client (keep track of holes when 
they happen, which is straightforward enough) and replacing it with a 
complicated sync protocol that isn’t needed. You keep alluding to the ability 
to just ‘get all the IDs’ and then request the messages you don’t have, but 
that’s obviously not possible in one pass, and you’re going to end up needing 
to do something (merkle tree?) that’s involving several round trips and a more 
complicated implementation for both client and server.

/K
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Lazar Otasevic]

See my answer to Mathew, maybe i clarified better. cu

On Sun, Mar 4, 2018 at 9:08 PM, Philipp Hörist  wrote:

>
> 2018-03-04 18:37 GMT+01:00 Lazar Otasevic :
>
>> Phillip,
>>
>> I feel like we dont entirely understand each other. Are you sure you did
>> read and understood my post?
>>
>> Getting array of ids gives you the order because ids should come in the
>> order respective messages are archived, and getting content is an extra
>> step above, unrelated to order. Content should be fetched by giving ANY set
>> of ids, like SQL SELECT...IN
>>
>> Where in my statements you see that I don't treat ids as opaque?
>>
>> Regards
>>
>>
>>
> You wrote that you react to a push notification, meaning you request one
> specific message without first getting a list of ids and checking what you
> are missing and the order.
>
> But i think i get what you want, just like email, request the headers but
> not the content, because you think its inefficient to download messages
> that you potentially already have.
>
>
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Lazar Otasevic]

>
>
> > Matthew,
>
> I really don't understand this argument. Doing a MAM query is exactly
> like fetching a list of ids, except that you also get the content as
> well. As I mentioned, I see no case in which you would want the ids
> and not the content. Even if you get the ids, you will fetch the
> content later.
>

yes, MAM query is exactly "like" fetching a list of ids combined with
content, but thats the problem i have from the start :)

getting the ids+content in the same query is a waste in some (many) cases
because local archive can have holes in it when network is bad and lots of
disconnects happen (for example live messages come before sync is completed
and then disconnect happens = next time we receive immediately some live
messages and we have holes)

even when we dont have a hole, we may trigger a page fetch of 10 messages
where infect we need only 2-3, and again we have a waste

in my proposal waste would be only some of the ids discarder, so very
little. good thing is that ids can come immediately in iq response and
client don't need to collect it out of the stream like in MAM now. and more
good thing see below...


If you already have the content, why would the client ever perform a
> query that overlaps with ranges of messages that it already has? Just
> none of this makes any sense to me


see previous - holes


> > I agree "back filling" feels wrong for someone and feels right for
> someone
> > else so its not an argument for technical discussion. Anyhow, direction
> of
> > iteration should be a client choice (server needs to provide both
> directions
> > possible).
>
> By back-filling I was less concerned about the direction and more
> concerned with showing the user a new message in the conversation
> without also showing them the messages before it. That is, showing a
> conversation with "holes" in.
>

any user friendly client should not prevent user to do live communication
while the sync is in progress. and there you gave a hole already.
conclusion: clients should not focus too much on solving something that is
out of their control, and that is holes. BTW viber works from newest
message to oldest ones, i have a video of viber on a bad network to prove
it.


> I totally *do* understand that some clients would like to fill the
> message log backwards dynamically as a user scrolls. Querying
> backwards from a known id is already possible.
>
> yes we elaborated that


> > I thought the goal of XMPP is to make clients less complex and more
> > reliable? Why is adding two (basic) features counted as increasing
> > complexity? I say basic because those features were kind of existent
> back in
> > 2005 in xep-0013. Now we removed those features, why?
>
> Can you explain what your ideal client logic would be? And how it is
> more simple than "the latest message I have has ID X, request all
> messages since X"?
>

yes, i can give you not only ideal logic but also ideal SW design of a
client...

lets say I make client from 3 components: sender, receiver and sync.
goal is to have them decoupled, and the only shared part will be the local
message archive

here is how the archive is filled:
1. sender - fills archive with message content when outgoing message is
confirmed to be sent
2. receiver - fills archive with message content when it receives a live
message
3. sync is the only one that takes care of the order, and also filling the
holes. order is implicitly given by list of ids received from the server,
and holes are easily detected by simple query to archive and then potential
holes filled by an extra request (with collection of ids)

note how everything can work decoupled, no special handling for sync on
connect/disconnect/send/receive, no id tracking, etc... simple and no waste
(except a bit of ids), every component can work with/without every other.

BTW using request with ids for push purposes is just a special case of that
request that we miss.

@Philip maybe you find some answers here?

...
PS one more thing why MAM direction is important to clients is - receipts,
because if client fetches first the message and then the receipt (old to
new) then message is in the wrong state (hint: multidevice or reinstall +
own messages/receipts)



>
> Regards,
> Matthew
>
> Regards, Lazar
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Matthew Wild]

On 4 March 2018 at 17:30, Lazar Otasevic  wrote:
> Matthew,
>
> I may be repeating: every efficient sync algorithm needs iterating trough
> ids and content separately. Of course we may make sync work without
> iterating trough ids, but implementations like that are just terrible, too
> complex, too coupled with the rest of design and not optimal.

I really don't understand this argument. Doing a MAM query is exactly
like fetching a list of ids, except that you also get the content as
well. As I mentioned, I see no case in which you would want the ids
and not the content. Even if you get the ids, you will fetch the
content later.

If you already have the content, why would the client ever perform a
query that overlaps with ranges of messages that it already has? Just
none of this makes any sense to me.

> I agree "back filling" feels wrong for someone and feels right for someone
> else so its not an argument for technical discussion. Anyhow, direction of
> iteration should be a client choice (server needs to provide both directions
> possible).

By back-filling I was less concerned about the direction and more
concerned with showing the user a new message in the conversation
without also showing them the messages before it. That is, showing a
conversation with "holes" in.

I totally *do* understand that some clients would like to fill the
message log backwards dynamically as a user scrolls. Querying
backwards from a known id is already possible.

> I thought the goal of XMPP is to make clients less complex and more
> reliable? Why is adding two (basic) features counted as increasing
> complexity? I say basic because those features were kind of existent back in
> 2005 in xep-0013. Now we removed those features, why?

Can you explain what your ideal client logic would be? And how it is
more simple than "the latest message I have has ID X, request all
messages since X"?

Regards,
Matthew
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Lazar Otasevic]

Phillip,

I feel like we dont entirely understand each other. Are you sure you did
read and understood my post?

Getting array of ids gives you the order because ids should come in the
order respective messages are archived, and getting content is an extra
step above, unrelated to order. Content should be fetched by giving ANY set
of ids, like SQL SELECT...IN

Where in my statements you see that I don't treat ids as opaque?

Regards

On Sun, Mar 4, 2018 at 9:57 AM, Philipp Hörist  wrote:

> Hi,
>
> If you use MAM like that you will run into problems.
> if you request a specific ID you lose the Order information inside the
> Archive.
> your only option is to use the timestamp for ordering, but many messages
> can have the same timestamp.
>
> XEP:
> These IDs are strings that servers may construct in any manner, and
> clients must treat as opaque strings (e.g. there is no requirement for them
> to be numeric, sequenced or GUIDs)
>
>
> 2018-03-03 23:53 GMT+01:00 Lazar Otasevic :
>
>> Hi, I think I miss some features here:
>>
>> 1. fetching messages by giving a set of ids, also similar like xep-013
>>
>> Fetching message by id(s) is needed  for example when i have a custom
>> push notification with a given message id(s) and client needs to get that
>> one message asap and show it in the chat.
>>
>> 2. fetching message ids instead of entire messages, similar like "message
>> headers" in xep-013
>>
>> Fetching just message ids is harder to explain why its needed, but I
>> think its a must have if one wants to make an efficient and reliable local
>> message archive synced with server archive, and to make it as separate
>> module independent from the rest of the client. Basically by fetching
>> message ids we try to detect "holes" in our local archive and then we fill
>> that holes by doing step 1. I think this is a standard way of doing the
>> sync algorithm, and its mind boggling why its not here in MAM already.
>>
>> Is it possible to make it into reality and what would be the next steps
>> from my side?
>>
>> Thanks.
>>
>>
>>
>> ___
>> Standards mailing list
>> Info: https://mail.jabber.org/mailman/listinfo/standards
>> Unsubscribe: standards-unsubscr...@xmpp.org
>> ___
>>
>>
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Jonas Wielicki]

On Samstag, 3. März 2018 23:53:21 CET Lazar Otasevic wrote:
> Hi, I think I miss some features here:
> 
> 1. fetching messages by giving a set of ids, also similar like xep-013
> 
> Fetching message by id(s) is needed  for example when i have a custom push
> notification with a given message id(s) and client needs to get that one
> message asap and show it in the chat.

From my limited understanding of MAM, it should be possible if your custom 
push thing gives you the stanza-id instead of the message-id. Query by stanza-
id is kinda what MAM is really good at.


kind regards,
Jonas

signature.asc
Description: This is a digitally signed message part.
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Lazar Otasevic]

Matthew,

I may be repeating: every efficient sync algorithm needs iterating trough
ids and content separately. Of course we may make sync work without
iterating trough ids, but implementations like that are just terrible, too
complex, too coupled with the rest of design and not optimal.

I agree "back filling" feels wrong for someone and feels right for someone
else so its not an argument for technical discussion. Anyhow, direction of
iteration should be a client choice (server needs to provide both
directions possible).

Please keep in mind multi device scenarios. If one device misses a lot of
messages then back filling "feels" proper way for me - I want new messages
first (live messages also)

I thought the goal of XMPP is to make clients less complex and more
reliable? Why is adding two (basic) features counted as increasing
complexity? I say basic because those features were kind of existent back
in 2005 in xep-0013. Now we removed those features, why?


Thanks


On Sun, Mar 4, 2018 at 4:36 PM, Matthew Wild  wrote:

> Hi Lazar,
>
> On 3 Mar 2018 22:54, "Lazar Otasevic"  wrote:
> >
> > Hi, I think I miss some features here:
> >
> > 1. fetching messages by giving a set of ids, also similar like xep-013
> >
> > Fetching message by id(s) is needed  for example when i have a custom
> push notification with a given message id(s) and client needs to get that
> one message asap and show it in the chat.
>
> Why not just do a normal query? It feels wrong of a chat UI to show a
> message in a chat and later 'back-fill' the conversation. A message may
> easily be taken out of context this way.
>
> > 2. fetching message ids instead of entire messages, similar like
> "message headers" in xep-013
>
> I don't see why this is helpful. Why would a client want the stanza
> metadata but not the stanza?
>
> > Fetching just message ids is harder to explain why its needed, but I
> think its a must have if one wants to make an efficient and reliable local
> message archive synced with server archive, and to make it as separate
> module independent from the rest of the client. Basically by fetching
> message ids we try to detect "holes" in our local archive and then we fill
> that holes by doing step 1. I think this is a standard way of doing the
> sync algorithm, and its mind boggling why its not here in MAM already.
>
> Similarly I don't see what advantage this brings. Hole detection is
> already possible today with the current protocol, with no overhead of
> additionally fetching IDs and doing an inefficient list comparison.
>
> I really want to keep the XEP as simple as we can, and right now I don't
> see enough justification for adding these features.
>
> Regards,
> Matthew
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Matthew Wild]

Hi Lazar,

On 3 Mar 2018 22:54, "Lazar Otasevic"  wrote:
>
> Hi, I think I miss some features here:
>
> 1. fetching messages by giving a set of ids, also similar like xep-013
>
> Fetching message by id(s) is needed  for example when i have a custom
push notification with a given message id(s) and client needs to get that
one message asap and show it in the chat.

Why not just do a normal query? It feels wrong of a chat UI to show a
message in a chat and later 'back-fill' the conversation. A message may
easily be taken out of context this way.

> 2. fetching message ids instead of entire messages, similar like "message
headers" in xep-013

I don't see why this is helpful. Why would a client want the stanza
metadata but not the stanza?

> Fetching just message ids is harder to explain why its needed, but I
think its a must have if one wants to make an efficient and reliable local
message archive synced with server archive, and to make it as separate
module independent from the rest of the client. Basically by fetching
message ids we try to detect "holes" in our local archive and then we fill
that holes by doing step 1. I think this is a standard way of doing the
sync algorithm, and its mind boggling why its not here in MAM already.

Similarly I don't see what advantage this brings. Hole detection is already
possible today with the current protocol, with no overhead of additionally
fetching IDs and doing an inefficient list comparison.

I really want to keep the XEP as simple as we can, and right now I don't
see enough justification for adding these features.

Regards,
Matthew
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___

Re: [Standards] xep-0313 missing features

[Philipp Hörist]

Hi,

If you use MAM like that you will run into problems.
if you request a specific ID you lose the Order information inside the
Archive.
your only option is to use the timestamp for ordering, but many messages
can have the same timestamp.

XEP:
These IDs are strings that servers may construct in any manner, and clients
must treat as opaque strings (e.g. there is no requirement for them to be
numeric, sequenced or GUIDs)


2018-03-03 23:53 GMT+01:00 Lazar Otasevic :

> Hi, I think I miss some features here:
>
> 1. fetching messages by giving a set of ids, also similar like xep-013
>
> Fetching message by id(s) is needed  for example when i have a custom push
> notification with a given message id(s) and client needs to get that one
> message asap and show it in the chat.
>
> 2. fetching message ids instead of entire messages, similar like "message
> headers" in xep-013
>
> Fetching just message ids is harder to explain why its needed, but I think
> its a must have if one wants to make an efficient and reliable local
> message archive synced with server archive, and to make it as separate
> module independent from the rest of the client. Basically by fetching
> message ids we try to detect "holes" in our local archive and then we fill
> that holes by doing step 1. I think this is a standard way of doing the
> sync algorithm, and its mind boggling why its not here in MAM already.
>
> Is it possible to make it into reality and what would be the next steps
> from my side?
>
> Thanks.
>
>
>
> ___
> Standards mailing list
> Info: https://mail.jabber.org/mailman/listinfo/standards
> Unsubscribe: standards-unsubscr...@xmpp.org
> ___
>
>
___
Standards mailing list
Info: https://mail.jabber.org/mailman/listinfo/standards
Unsubscribe: standards-unsubscr...@xmpp.org
___