Re: Maximum AllWatcher frame size?

[John A Meinel]

Wouldn't that feel like a breach of contract where the first response is
supposed to get you in sync with the system?
I could definitely see doing a different code path for the first response
because you know there aren't any objects you have to compare against, etc.
Or because you can reasonably expect that the first response is going to be
an entirely different order of magnitude to everything else. And at some
point you do have to load in the whole model to get started.
Especially bad we ended up with something like the 5s watcher delay between
updates and the initial model load of a large model takes 100s because we
broke it up.

I'm generally in favor of avoiding any one thing growing without bound. I
do want us to consider the tradeoffs. (limiting the size of the response
then let's the number of responses to get up to date grow without bound)

John
=:->


On Jun 16, 2017 04:20, "Menno Smits"  wrote:



On 15 June 2017 at 20:10, roger peppe  wrote:

>
> Looking at the multiwatcher code, I don't think it would be too
> difficult to make
> it return only a limited number of delta elements in any single response.
> That would not necessarily be a hard limit (theoretically an individual
> delta could be very large - I don't think anyone imposes limits on
> name sizes, for example) but would probably be good enough
> in practice.
>

​+1

This seems like the best approach to me.

- Menno

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/
mailman/listinfo/juju-dev
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Maximum AllWatcher frame size?

[roger peppe]

On 14 June 2017 at 17:14, Cory Johns  wrote:
> So, I'm not sure if this resolves the issue.  The python websockets library
> supports fragmented messages but I think that is separate from the maximum
> message size that it's enforcing.  I think that maximum is for the final
> payload, which would be the entire AllWatcher JSON response.  If the
> controller doesn't break that in to multiple AllWatcher updates, then I
> think we'll still hit the maximum message size.
>
> I can increase the max size pretty easily, or even remove it entirely, as
> mentioned in the issue, but I'm a little bit leery of having no upper bound
> at all.  Do we know if there's a reasonable upper bound that even large
> models would fall within for the initial AllWatcher response?

Unfortunately there is no reasonable upper bound because models can
be arbitrarily large. I'd suggest setting it to a value that would be a
serious memory problem if it got that big.

Looking at the multiwatcher code, I don't think it would be too
difficult to make
it return only a limited number of delta elements in any single response.
That would not necessarily be a hard limit (theoretically an individual
delta could be very large - I don't think anyone imposes limits on
name sizes, for example) but would probably be good enough
in practice.

  cheers,
rog.

>
> On Wed, Jun 14, 2017 at 12:08 PM, Tim Penhey 
> wrote:
>>
>> It is configured in juju
>>
>>
>> // Use a 64k frame size for the websockets while we need to deal
>> // with x/net/websocket connections that don't deal with recieving
>> // fragmented messages.
>> const websocketFrameSize = 65536
>>
>>
>> On 14/06/17 12:00, Cory Johns wrote:
>> > Do we know what size the gorilla/websocket library uses for automatic
>> > chunking?
>> >
>> > On Wed, Jun 14, 2017 at 11:35 AM, John Meinel > > > wrote:
>> >
>> > In 2.1 we did not chunk, in 2.2 we switch to gorilla/websocket which
>> > does support chunking into frames. I don't think we do any internal
>> > "well that would be too much information so we wont send it all".
>> >
>> > John
>> > =:->
>> >
>> >
>> > On Wed, Jun 14, 2017 at 7:11 PM, Cory Johns
>> > > wrote:
>> >
>> > https://github.com/juju/python-libjuju/issues/136
>> >  raises the
>> > issue that, for large models, the initial AllWatcher response
>> > frame can be large enough that it overruns the default maximum
>> > frame size of the websocket library (1MB).  We can increase this
>> > limit fairly easily, but I wanted to find out whether there is
>> > any maximum size from the Juju size and if large enough frames
>> > could get chunked into multiple frames?
>> >
>> > --
>> > Juju-dev mailing list
>> > Juju-dev@lists.ubuntu.com 
>> > Modify settings or unsubscribe at:
>> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
>> > 
>> >
>> >
>> >
>> >
>> >
>
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Maximum AllWatcher frame size?

[Cory Johns]

So, I'm not sure if this resolves the issue.  The python websockets library
supports fragmented messages but I think that is separate from the maximum
message size that it's enforcing.  I think that maximum is for the final
payload, which would be the entire AllWatcher JSON response.  If the
controller doesn't break that in to multiple AllWatcher updates, then I
think we'll still hit the maximum message size.

I can increase the max size pretty easily, or even remove it entirely, as
mentioned in the issue, but I'm a little bit leery of having no upper bound
at all.  Do we know if there's a reasonable upper bound that even large
models would fall within for the initial AllWatcher response?

On Wed, Jun 14, 2017 at 12:08 PM, Tim Penhey 
wrote:

> It is configured in juju
>
>
> // Use a 64k frame size for the websockets while we need to deal
> // with x/net/websocket connections that don't deal with recieving
> // fragmented messages.
> const websocketFrameSize = 65536
>
>
> On 14/06/17 12:00, Cory Johns wrote:
> > Do we know what size the gorilla/websocket library uses for automatic
> > chunking?
> >
> > On Wed, Jun 14, 2017 at 11:35 AM, John Meinel  > > wrote:
> >
> > In 2.1 we did not chunk, in 2.2 we switch to gorilla/websocket which
> > does support chunking into frames. I don't think we do any internal
> > "well that would be too much information so we wont send it all".
> >
> > John
> > =:->
> >
> >
> > On Wed, Jun 14, 2017 at 7:11 PM, Cory Johns
> > > wrote:
> >
> > https://github.com/juju/python-libjuju/issues/136
> >  raises the
> > issue that, for large models, the initial AllWatcher response
> > frame can be large enough that it overruns the default maximum
> > frame size of the websocket library (1MB).  We can increase this
> > limit fairly easily, but I wanted to find out whether there is
> > any maximum size from the Juju size and if large enough frames
> > could get chunked into multiple frames?
> >
> > --
> > Juju-dev mailing list
> > Juju-dev@lists.ubuntu.com 
> > Modify settings or unsubscribe at:
> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
> > 
> >
> >
> >
> >
> >
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Maximum AllWatcher frame size?

[Cory Johns]

Do we know what size the gorilla/websocket library uses for automatic
chunking?

On Wed, Jun 14, 2017 at 11:35 AM, John Meinel 
wrote:

> In 2.1 we did not chunk, in 2.2 we switch to gorilla/websocket which does
> support chunking into frames. I don't think we do any internal "well that
> would be too much information so we wont send it all".
>
> John
> =:->
>
>
> On Wed, Jun 14, 2017 at 7:11 PM, Cory Johns 
> wrote:
>
>> https://github.com/juju/python-libjuju/issues/136 raises the issue that,
>> for large models, the initial AllWatcher response frame can be large enough
>> that it overruns the default maximum frame size of the websocket library
>> (1MB).  We can increase this limit fairly easily, but I wanted to find out
>> whether there is any maximum size from the Juju size and if large enough
>> frames could get chunked into multiple frames?
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailm
>> an/listinfo/juju-dev
>>
>>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Maximum AllWatcher frame size?

[John Meinel]

In 2.1 we did not chunk, in 2.2 we switch to gorilla/websocket which does
support chunking into frames. I don't think we do any internal "well that
would be too much information so we wont send it all".

John
=:->


On Wed, Jun 14, 2017 at 7:11 PM, Cory Johns 
wrote:

> https://github.com/juju/python-libjuju/issues/136 raises the issue that,
> for large models, the initial AllWatcher response frame can be large enough
> that it overruns the default maximum frame size of the websocket library
> (1MB).  We can increase this limit fairly easily, but I wanted to find out
> whether there is any maximum size from the Juju size and if large enough
> frames could get chunked into multiple frames?
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/juju-dev
>
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev