Re: backend/director admin states / std.resolve_backend? / PRIV_REQ?

[Nils Goroll]

On 09/05/18 11:34, Nils Goroll wrote:
> ban() cli 

VCL that is
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: backend/director admin states / std.resolve_backend? / PRIV_REQ?

[Nils Goroll]

I guess this thread still lacks my response to phks reply:

On 03/05/18 11:20, Poul-Henning Kamp wrote:
> Admin health is owned by the CLI and director implementations do not
> get to meddle with it.  (But they do get to see it)

I understand the point, but allowing vmods to change the admin health would make
sense in the same way as the ban() cli command makes sense: in some cases,
controlling such functionality in-band avoids additional complications.

>   Admin=sick means sick, no matter what the director implementation thinks.
> 
>   Admin=healthy means healthy, --//--
> 
>   Admin=??? means "Implementation decides if sick or healthy."
> 
> Best proposal for "???" seems to be "auto", even though it is not
> entirely on point.

Ack. But I'd still like to see "auto" be a write-only health state, resulting in
healthy or whatever the implementation returns for ->list().

> In the new API, there is a director method called ->list() which
> gets called during CLI command 'backend.list' so that UX can reflect
> what the implementation actually know as best as possible.

It seems to make a lot of sense to replace "probe" and "dependent" from my
proposal with an implementation specific value. So ->list() would fill out both
the fields replacing the current "Admin" and "Probe" fields.

>2) Headers in backend.list

Admin and Check ?

>3) What does H1 backend without probe do for ->list()

return Admin=Healthy Check=-

>4) What does H1 backend with probe do for ->list()

return Admin=Healthy Check=

>5) What do vmod_directors do for ->list()

sensible default:

return Admin=dependent Check=n/m number of healthy/total backends in the 
director


Also I'd like to push again for the "disabled" admin state: Yes, this could be
implemented with another director layer, but I think the additional flag belongs
to the fundamental director because the proposed functionality is fundamental
and can be found in all the loadbalancer appliances I've encountered so far.

> I'm pondering a "VCL_BACKEND std.resolve_backend(VCL_BACKEND)" to
> avoid what seems to me to be a lot of complexity in shard to do
> the same?

The complexity in shard is not for resolve=NOW, but for resolve=LAZY: We need to
keep track of all the load balancing parameters without actually making the
decision. At this point, they are stored in a PRIV_TASK which cannot cross the
client/backend boundary, so we need to special case based on which side we're
running on.

I think that replacing resolve=NOW with std.resolve_backend() would make the
situation even more confusing for users as resolve=LAZY has limitations and
resolve=NOW doesn't.

If anything, with the current code, std.resolve_backend() would make sense for
the the directors EXCEPT for shard.

If we wanted to generalize more, I think we'd need something like a PRIV_REQ:

- can be created on the client side

- ownership transferred to the backend side or fini'd if no backend
  request

Nils
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: backend/director admin states

[Poul-Henning Kamp]

In message 
, Dridi Boukelmoune writes:

>All we need is TheNewGuy(tm) to dump this on their desk :p

Indeed, they grow up so fast, don't they ? :-)

Anyway, I would like to get back to the actual underlying
state-machinery for directors...

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: backend/director admin states

[Dridi Boukelmoune]

On Fri, May 4, 2018 at 9:42 AM, Poul-Henning Kamp  wrote:
> 

> The reason we have include/tbl/cli_cmds.h in the first place was
> that I wanted some kind of "schema-light" for UX use, but that
> didn't happen.

Yes, that experiment failed, but I somehow succeeded in doing
something similar for varnishtest: a schema for commands and
a generic parser dispatches TheRightThing(tm) to callbacks. It
worked with nested commands too although I only implemented
a dozen in total. The generic parser squared some of the
concurrency issues too by keeping track of all parties.

I don't have time these days for such experiments :(


> I don't see a need for varnishd to ship code to varnishadm though,
> so I'd be happy with a design where varnishd CLI always spits out
> JSON, and varnishadm is responsible for formatting it, if required.

The reason I wanted to centralize the json2txt code is to also ensure
that varnishd -d and varnishtest logs stay readable. If we want
varnishadm to take advantage of its interactive mode to align columns
and whatnots, we can always have a json2matrix step and then render
it.

> As long as somebody who is not me writes the code for it...
>
> PS: We have a solid JSON parser in the tree already.

All we need is TheNewGuy(tm) to dump this on their desk :p

Cheers
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: backend/director admin states

[Poul-Henning Kamp]

In message 
, Dridi Boukelmoune writes:

>I experimented with this a while back and tried this:
>
>[...]
>
>I never completed the last experiment because the time I allocated
>eventually ran out and I lost motivation to nail down the schema part.

I appreciate the effort, having personally been down that road a
number of times in my career, all in vain.

I think my conclusion at this point is that I have not seen an
actual schema work anywhere - ever.  That is not to say that schemas
cannot ever work, they proabably can in certain limited circumstances.

The reason we have include/tbl/cli_cmds.h in the first place was
that I wanted some kind of "schema-light" for UX use, but that
didn't happen.

Instead varnishadm issues a "help" command, to get the _actual_
commands available.

Varnishadm was a convenience, and the intent was that people could
connect to the CLI with any random script they cared to write.

Once we did the PSK-authentication, telnet'ing became a lot less
relevant, and varnishadm became the defacto CLI access tool.

Therefore it is relevant to consider a design-change so varnishd
only responds with JSON (but it can and should be nicely formatted
JSON), and varnishadm gets to render that nicely for the user.

Amongst many other UX advantages, varnishadm can know the width of
terminal windows, and the availability of colors and exploit that.

But if you try to impose a schema between varnishd and varnishadm,
you either loose access to all the really nifty things varnishadm
could do, or you spend the rest of your life trying to define a
schema which can express them.

I would argue that almost everybody have given up on schemas by
now, and switched to shipping code:

Your average website sends a shitload of JS code and datastructures
for it to act on to your browser.

I don't see a need for varnishd to ship code to varnishadm though,
so I'd be happy with a design where varnishd CLI always spits out
JSON, and varnishadm is responsible for formatting it, if required.

As long as somebody who is not me writes the code for it...

PS: We have a solid JSON parser in the tree already.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: backend/director admin states

[Dridi Boukelmoune]

>>Maybe we should do that the other way around? Have all CLI commands
>>reply with a JSON data structure, and CLI consumers like varnishadm,
>>varnishtest or varnishd -d be able to output it in plain text (with
>>this code centralized in libvarnish and exposed in libvarnishapi).
>
> I thought about that, but didn't move on it, as I didn't want to
> write all the code myself ;-)

I experimented with this a while back and tried this:

- have a CLI commands reply with a JSON payload
  - same as today: fixed-length size, newline, payload
  - includes metadata such as the response status
- JSON payload includes the "plain text" renderer's name
  - example: "backend.list", "backend.list -p" etc
  - one per command plus variants that change the output format
  - didn't go as far as plain text rendering
- replace the response JSON payload with custom byte code
  - the render chooses between JSON or text rendering
  - byte code is JSON-like, but easier to parse
  - one generic JSON renderer for all commands
  - started experimenting with plain text conversion
- code generation from a schema for each output type
  - example: "backend.list", "backend.list -p" etc (same as before)
  - defensive assertions that let us know when we break the output
  - the idea was to start actually versioning the CLI from 2.0 on
  - and at least bump y in 2.y when we add stuff to the CLI

I never completed the last experiment because the time I allocated
eventually ran out and I lost motivation to nail down the schema part.
This was in the varnishreload [1] time frame, when I realized that
changes to the CLI broke the reload scripts we ship in the [4.1,5.x]
series and since we were maintaining them out of tree we had no easy
way to keep track of that. Which is why I also wanted to import
varnishreload upstream and have proper coverage.

I have thrown the code away, but the idea stuck. And it was a fun
experiment :)

Dridi

[1] https://github.com/varnishcache/pkg-varnish-cache/pull/70
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: backend/director admin states

[Poul-Henning Kamp]

In message 

Re: backend/director admin states

[Dridi Boukelmoune]

On Thu, May 3, 2018 at 1:04 PM, Poul-Henning Kamp  wrote:
> 
> In message 
> 
> , Guillaume Quintard writes:
>
>>All commands should have a JSON option, I reckon.
>
> Somebody should write more code for it then, I reckon :-)

Maybe we should do that the other way around? Have all CLI commands
reply with a JSON data structure, and CLI consumers like varnishadm,
varnishtest or varnishd -d be able to output it in plain text (with
this code centralized in libvarnish and exposed in libvarnishapi).

Dridi
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: backend/director admin states

[Guillaume Quintard]

Yp, that's on my plate. Which is rather large...

-- 
Guillaume Quintard

On Thu, May 3, 2018, 13:04 Poul-Henning Kamp  wrote:

> 
> In message  c5q-zxfqjcu0k...@mail.gmail.com>
> , Guillaume Quintard writes:
>
> >All commands should have a JSON option, I reckon.
>
> Somebody should write more code for it then, I reckon :-)
>
> --
> Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
> p...@freebsd.org | TCP/IP since RFC 956
> FreeBSD committer   | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: backend/director admin states

[Poul-Henning Kamp]

In message 
, Guillaume Quintard writes:

>All commands should have a JSON option, I reckon.

Somebody should write more code for it then, I reckon :-)

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: backend/director admin states

[Guillaume Quintard]

All commands should have a JSON option, I reckon.

-- 
Guillaume Quintard

On Thu, May 3, 2018, 12:52 Poul-Henning Kamp  wrote:

> 
> In message <1401d370-56dd-6006-2b3d-21771650f...@uplex.de>, Geoff Simmons
> write
> s:
>
> >>> Illustrated example: varnishadm backend.list output
>
> We should also think about varnishadm backend.list -j ?
>
> --
> Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
> p...@freebsd.org | TCP/IP since RFC 956
> FreeBSD committer   | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
> ___
> varnish-dev mailing list
> varnish-dev@varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: backend/director admin states

[Poul-Henning Kamp]

In message <1401d370-56dd-6006-2b3d-21771650f...@uplex.de>, Geoff Simmons write
s:

>>> Illustrated example: varnishadm backend.list output

We should also think about varnishadm backend.list -j ?

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: backend/director admin states

[Poul-Henning Kamp]

In message 

Re: backend/director admin states

[Dridi Boukelmoune]

> Best proposal for "???" seems to be "auto", even though it is not
> entirely on point.
>
> The implementation can control its view of the health two ways
>
>   A) Call VRT_SetHealth() and set it on the director.
>  (useful for probes as we know them)
>
>   B) Provide a ->healthy() method which will be asked whenever
>  anybody wants to know.
>  (useful for selecting directors like rr/fb/rnd/hash)

"auto" is what we have today in the CLI, and I don't remember ever
being confused by that when I was only a user. And it seems to
still match your A) and B) cases.

Dridi
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: backend/director admin states

[Poul-Henning Kamp]

OK, let me try to explain my mental model of this:

Admin health is owned by the CLI and director implementations do not
get to meddle with it.  (But they do get to see it)

  Admin=sick means sick, no matter what the director implementation thinks.

  Admin=healthy means healthy, --//--

  Admin=??? means "Implementation decides if sick or healthy."

Best proposal for "???" seems to be "auto", even though it is not
entirely on point.

The implementation can control its view of the health two ways

  A) Call VRT_SetHealth() and set it on the director.
 (useful for probes as we know them)

  B) Provide a ->healthy() method which will be asked whenever
 anybody wants to know.
 (useful for selecting directors like rr/fb/rnd/hash)

In the new API, there is a director method called ->list() which
gets called during CLI command 'backend.list' so that UX can reflect
what the implementation actually know as best as possible.

So as I see it:  We need to decide:

   1) Best word for "???" above.

   2) Headers in backend.list

   3) What does H1 backend without probe do for ->list()

   4) What does H1 backend with probe do for ->list()

   5) What do vmod_directors do for ->list()

Poul-Henning


PS: Related stuff:

I'm pondering a "VCL_BACKEND std.resolve_backend(VCL_BACKEND)" to
avoid what seems to me to be a lot of complexity in shard to do
the same?

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: backend/director admin states

[Geoff Simmons]

On 05/03/2018 10:49 AM, Dridi Boukelmoune wrote:
>> Illustrated example: varnishadm backend.list output
>>
>> Backend name  Admin Probe  Last change
>> boot.b1   probe 0/8 badWed, 02 May 2018 13:22:10 
>> GMT
>> boot.b2   probe 8/8 good   Wed, 02 May 2018 13:22:10 
>> GMT
>> boot.b3   healthy   1/8 badWed, 02 May 2018 13:22:10 
>> GMT
>> boot.b4   healthy   -  Wed, 02 May 2018 13:22:10 
>> GMT
>> boot.b5   sick  7/8 good   Wed, 02 May 2018 13:22:10 
>> GMT
>> boot.b6   sick  -  Wed, 02 May 2018 13:22:10 
>> GMT
>> boot.b7   disabled  7/8 good   Wed, 02 May 2018 13:22:10 
>> GMT
>> boot.b8   deleted   -  Wed, 02 May 2018 13:22:10 
>> GMT
>> boot.d1   dependent -  Wed, 02 May 2018 13:22:10 
>> GMT
>> boot.d2   probe 3/4 good   Wed, 02 May 2018 13:22:10 
>> GMT
> 
> We should make this output a bit more functional if we're going to
> break the output.

[...]

> Example:
> 
>> NameTypeHealthProbeStateCommentLast change
>> boot.www1stdautononegood-Wed, 02 May 2018 
>> 13:22:10 GMT
>> boot.www2stdsicknonebad-Wed, 02 May 2018 
>> 13:22:10 GMT
>> boot.wwwfallbackautoclustergood-Wed, 02 May 2018 
>> 13:22:10 GMT
> 
> The idea is to avoid blanks inside columns (except "last change/updated")
> to make it more friendly to a ny column oriented toollike /bin/sh's `read` or
> awk's fields.

I would also suggest ISO 8601 format for the 'Last change' column, to
make it more easily parseable:

2018-05-02T13:22:10Z

Then every whitespace-separated field corresponds to a data column.


Best,
Geoff
-- 
** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstraße 32
22301 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de



signature.asc
Description: OpenPGP digital signature
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: backend/director admin states

[Nils Goroll]

Hi,

On 02/05/18 17:07, Guillaume Quintard wrote:
> I think I could live with "auto" meaning "dependent/probe" depending on the 
> type
> of object.

Me too, but I also think that for varnishadm backend.list output, displaying the
actual facts is significantly more helpful.

> Not sure the disabled case is super useful. If I get what you say, the only
> thing differing from setting it to sick is that we keep using the open
> connections, instead of finishing in-flight requests. Is it worth it?

No, the suggestion is regarding a different level: Consider some application
server backends with local session store: Using shard / hash / explicit vcl, you
have implemented some kind of persistence such that requests with the same
session id always get sent to the same backend.

Now you plan to restart that backend and want to still send requests with
existing persisted session ids to it, but no requests without a session cookie.

One current way to do this is to essentially define every backend twice and use
once director for the "existing session" and one for the "no session" case.
Other options probably involve std.healthy(), the backend probe and retries.

I think we can do much better than this.

> I'd argue "deleted" should be "dead", just to complete the medically 
> depressing
> vocabulary list you started :-)

wfm.

> (and we'll need to bring back "saint" from the dead)

Greetings from the guy who currently tries to keep the saint's mode working 
with. ;)

Nils
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: backend/director admin states

[Guillaume Quintard]

I think I could live with "auto" meaning "dependent/probe" depending on the
type of object.

Not sure the disabled case is super useful. If I get what you say, the only
thing differing from setting it to sick is that we keep using the open
connections, instead of finishing in-flight requests. Is it worth it?

I'd argue "deleted" should be "dead", just to complete the medically
depressing vocabulary list you started :-)

(and we'll need to bring back "saint" from the dead)

-- 
Guillaume Quintard

On Wed, May 2, 2018 at 4:50 PM, Nils Goroll  wrote:

> this is a follow up to a discussion between phk and myself on
> #varnish-hacking
> today:
>
> phk is in the process of restructuring director/backend APIs. One of the
> areas
> currently reconsidered is the admin health status. As of now
> (68a78f94b3a4e702edf73dea4e11e626c4c20301), the admin health has become
> private
> to varnishd (off limits to vmods) and semantics have changed.
>
> We discussed which admin health states should exist and their semantics.
> Notice
> that for the "probe" admin health, the actual health state is determined
> by the
> probe.
>
> Here's my proposal:
>
> * auto
>
>   this state is the default and only exists as an argument to varnishadm
>   backend.set_health. It resolves to either
>
>   - "probe" if the backend/director has a probe
>   - "dependent" if the director has no probe, but a healthy callback
>   - "healthy"   otherwise (no probe and no healthy callback)
>
> * healthy
>
>   The backend/director is considered unconditionally healthy. Connections
> will
>   be attempted.
>
>   This state is used for backends without a probe and can be manually set
>   using varnishadm backend.set_health and from vmods.
>
> * sick
>
>   The backend/director is considered unconditionally sick. Connections
> will not
>   be attempted.
>
>   Can be manually set using varnishadm backend.set_health and from vmods.
>
> * disabled
>
>   The health state is determined as for "probe" or "healthy" but a newly
> added
>   VRT / vmod_std function (e.g. backend_disabled()) returns true.
> Directors may
>   offer the choice to use disabled backends or not.
>
>   Can be manually set using varnishadm backend.set_health and from vmods.
>
>   The use case is to flag a backend for plannend maintenance such that it
> is
>   used for existing sessions, but not for new ones.
>
> * probe
>
>   The health state is determined by a probe, yet connections can still be
>   attempted even if the probe is unsuccessful and directors may offer the
> choice
>   to still hand out unhealthy backends.
>
>   This state can not be set explicitly, but is selected through
>   varnishadm backend.set_health auto or from vmods.
>
> * dependent
>
>   The health state of the director is determined by the backends it refers
> to.
>
>   This state can not be set explicitly, but is selected through
>   varnishadm backend.set_health auto or from vmods.
>
> * deleted
>
>   The backend has been deleted (via a vmod) will be removed once all
> references
>   have been cleared.
>
>
>
> Illustrated example: varnishadm backend.list output
>
> Backend name  Admin Probe  Last change
> boot.b1   probe 0/8 badWed, 02 May 2018
> 13:22:10 GMT
> boot.b2   probe 8/8 good   Wed, 02 May 2018
> 13:22:10 GMT
> boot.b3   healthy   1/8 badWed, 02 May 2018
> 13:22:10 GMT
> boot.b4   healthy   -  Wed, 02 May 2018
> 13:22:10 GMT
> boot.b5   sick  7/8 good   Wed, 02 May 2018
> 13:22:10 GMT
> boot.b6   sick  -  Wed, 02 May 2018
> 13:22:10 GMT
> boot.b7   disabled  7/8 good   Wed, 02 May 2018
> 13:22:10 GMT
> boot.b8   deleted   -  Wed, 02 May 2018
> 13:22:10 GMT
> boot.d1   dependent -  Wed, 02 May 2018
> 13:22:10 GMT
> boot.d2   probe 3/4 good   Wed, 02 May 2018
> 13:22:10 GMT
>
> in detail:
>
> * boot.b1   probe 0/8 bad
>
> backend with a probe for which all recent health checks have failed
>
> result from set_health=auto or default
>
> * boot.b2   probe 8/8 good
>
> backend with a probe for which all recent health checks have succeeded
>
> result from set_health=auto or default
>
> * boot.b3   healthy   1/8 bad
>
> backend with a probe for which most recent health checks have failed
>
> result from set_health=healthy
>
> * boot.b4   healthy   -
>
> backend with no probe
>
> result from set_health=healthy, set_health=auto or default
>
> * boot.b5   sick  7/8 good
>
> backend with a probe for which most recent health checks have succeeded
>
> result from set_health=sick
>
> * boot.b6   sick  -
>
> backend with no probe
>
> result from set_health=sick
>
> * boot.b7