Thanks for the proposal Joseph! I think I'm also leaning toward the
circular buffer approach. The one real concern there seems to be the
potential for "DDoS"-type scenarios when users hit the subscriber limit
using clients which have retry logic. Providing a metric for the number of
currently connected subscribers will hopefully help operators avoid this.
The default value for a new flag limiting subscriber count should be very
high (MAX_INT??) to maintain current behavior.
What do other folks think about this approach? Joseph's draft review is
here: https://reviews.apache.org/r/69307/
Greg
On Wed, Nov 14, 2018 at 6:35 PM Joseph Wu wrote:
> Heartbeats are currently the least-liked solution, for precisely the
> reason BenM stated. Clients of the API, such as the maintainers of the
> DC/OS UI, would also like to avoid making more connections than necessary
> and/or keeping additional state between connections.
>
>
> Currently, I am leaning towards keeping subscribers in a circular buffer.
> This solution is minimal in the code footprint and requires no client-side
> changes besides heavily incentivizing retry logic (which we already expect
> in most cases).
> One potential downside is having more subscribers than the (master flag)
> configured maximum. In this case, each client would kick out the first
> few; which would then retry and kick out the next few, etc. Each retry is
> equivalent to a GET /master/state, and the extra calls would basically
> erase the performance gains we have from streaming the events.
>
> Nevertheless, I think a reasonably high default would have minimal impact
> on both master performance and client connectivity. The code for this
> proposal can be found here:
> https://reviews.apache.org/r/69307/ (Just one review)
>
> On Sun, Nov 11, 2018 at 9:22 AM Benjamin Mahler
> wrote:
>
>> >- We can add heartbeats to the SUBSCRIBE call.
>> > This would need to be
>> > part of a separate operator Call, because one platform (browsers) that
>> > might subscribe to the master does not support two-way streaming.
>>
>> This doesn't make sense to me, the heartbeats should still be part of the
>> same connection (request and response are infinite and heartbeating) by
>> default. Splitting into a separate call is messy and shouldn't be what we
>> force everyone to do, it should only be done in cases that it's impossible
>> to use a single connection (e.g. browsers).
>>
>> On Sat, Nov 10, 2018 at 12:03 AM Joseph Wu wrote:
>>
>>> Hi all,
>>>
>>> During some internal scale testing, we noticed that, when Mesos streaming
>>> endpoints are accessed via certain proxies (or load balancers), the
>>> proxies
>>> might not close connections after they are complete. For the Mesos
>>> master,
>>> which only has the /api/v1 SUBSCRIBE streaming endpoint, this can
>>> generate
>>> unnecessary authorization requests and affects performance.
>>>
>>> We are considering a few potential solutions:
>>>
>>>- We can add heartbeats to the SUBSCRIBE call. This would need to be
>>>part of a separate operator Call, because one platform (browsers) that
>>>might subscribe to the master does not support two-way streaming.
>>>- We can add (optional) arguments to the SUBSCRIBE call, which tells
>>> the
>>>master to disconnect it after a while. And the client would have to
>>> remake
>>>the connection every so often.
>>>- We can change the master to hold subscribers in a circular buffer,
>>> and
>>>disconnect the oldest ones if there are too many connections.
>>>
>>> We're tracking progress on this issue here:
>>> https://issues.apache.org/jira/browse/MESOS-9258
>>> Some prototypes of the code changes involved are also linked in the JIRA.
>>>
>>> Please chime in if you have any suggestions or if any of these options
>>> would be undesirable/bad,
>>> ~Joseph
>>>
>>
