Re: SHA256 from file descriptor

[Poul-Henning Kamp]

Dridi Boukelmoune writes:

> > Yes, but what do we need that for ?
>
> We don't, I'm working in a VMOD that computes hashes of regular files
> among other things. This looked like generic code not specific to that
> VMOD.

Is it a VMOD you expect to be adopted into the tree, and if so, could
you say a bit more about it ?

If it is an out-of-tree VMOD, I'm not very keen on adding library
functions just for one VMOD (but can probably be persuaded if the
functions are tied to Varnish through data structures etc.)

This specific suggestion does not meet that threshold, and we would
also have to add an explicit test-case for gcov, so I really dont
see much benefit ?

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: SHA256 from file descriptor

[Poul-Henning Kamp]

Dridi Boukelmoune writes:

> My bad, in my mind replying about mmap implied that this was indeed
> about having a convenience functtion to compute the hash of a regular
> file.

Yes, but what do we need that for ?

> Would the attached patch be welcome?

If we have a need, that's pretty much how I would do it.

> Slightly off topic, why have both definitions?
>
> #define VSHA256_LEN 32
> #define VSHA256_DIGEST_LENGTH   32

I think I kept it to not break any VMODs which relied on it.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: SHA256 from file descriptor

[Poul-Henning Kamp]

Dridi Boukelmoune writes:
> On Tue, May 21, 2024 at 5:38=E2=80=AFPM Poul-Henning Kamp  dk> wrote:
> >
> > Dridi Boukelmoune writes:
> >
> > > Are we interested in that kind of change (see attached diff) or do we
> > > strictly stick to the original source that was bundled in libvarnish?
> >
> > What's the usecase ?
> >
> > If regular file:  Why not mmap and save the kernel/userland copies ?
>
> Because the code from which I extracted this function was doing a copy
> while computing the checksum on the fly, so I didn't consider mmap
> when I extracted the read part of the code.

But what is the use-case ?

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: SHA256 from file descriptor

[Poul-Henning Kamp]

Dridi Boukelmoune writes:

> Are we interested in that kind of change (see attached diff) or do we
> strictly stick to the original source that was bundled in libvarnish?

What's the usecase ?

If regular file:  Why not mmap and save the kernel/userland copies ?

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Coverity Scan: Analysis completed for varnish

[Poul-Henning Kamp]

> I think Coverity Scan learned new tricks because nothing happened
> since the last run:

Indeed and I am going through them carefully as we speak.

They are suprisingly tricky to decide...

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: gcc -fanalyze fyi

[Poul-Henning Kamp]

Nils Goroll writes:

> *) https://social.treehouse.systems/@thesamesam/110697332121438807

Yes, that is precisely the top priority item for GCC I have pointed
out for ages...

Or maybe not...

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Varnishtest client user agents

[Poul-Henning Kamp]

Dridi Boukelmoune writes:

> > I would prefer you do not overload any Well Known Headers.
> >
> > X-VTC: c1
> >
> > maybe ?
>
> In the spirit of rfc6648 i would rather use a "VTC-" prefix, so
> something more like
>
> VTC-Client: c1
>
> But also, since a VTC client mocks a user agent, I would rather use
> the standard header for this. We can also add a -noua option similar
> to -nolen, -nohost and -nodate and imply -noua when an explicit `-hdr
> "User-Agent: other"` is present.

So what about the servers ?

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Varnishtest client user agents

[Poul-Henning Kamp]

Dridi Boukelmoune writes:
> On Wed, Jul 12, 2023 at 5:21=E2=80=AFAM Poul-Henning Kamp  dk> wrote:
> >
> > 
> > Dridi Boukelmoune writes:
> >
> > > Would it be OK to have `client cNAME` send a `User-Agent: vtest
> > > (cNAME)` or similar to reduce some of it?
> >
> > I generally just use /c1 /c2 etc in the URL ?
>
> That's part of the frustration, whenever I'm troubleshooting a test
> case I have to jump through more hoops to get something useful. What I
> would like is something out of the box, such as clients advertising
> themselves in a User-Agent header.

I would prefer you do not overload any Well Known Headers.

X-VTC: c1

maybe ?

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Varnishtest client user agents

[Poul-Henning Kamp]

Dridi Boukelmoune writes:

> Would it be OK to have `client cNAME` send a `User-Agent: vtest
> (cNAME)` or similar to reduce some of it?

I generally just use /c1 /c2 etc in the URL ?


-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Idea for multi-level CLI access control

[Poul-Henning Kamp]

Dridi Boukelmoune writes:
> On Mon, Jun 26, 2023 at 6:39=E2=80=AFPM Poul-Henning Kamp  dk> wrote:
> >

> Regarding the specific suggestion above, I don't think we would be
> satisfied with this model. In the security barriers diagram [1] we
> identified the following roles:
>
> - ADMIN
> - OPER
> - BACKEND
> - ANON

My idea was not meant as a replacement for any of those roles,
it just an idea for how to implement CLI connections with
different access levels - if we want to have that.

> For CLI access, we would probably want a new role TENANT, managed by
> ADMIN. Ideally, everything in the cache (VCL, contents, operations
> like ban) would be virtually partitioned such that a tenant could not
> directly affect the resources of other tenants.

I think that is a bit beyond the scope of the current discussion, but
it is certainly relevant to keep it in mind.

> > * Varnishd should identify itself (-i/-n) in the 107 message so that the
> >   client can pick which secret file to use if it has access to multiple.
>
> If each "account" (admin or tenant) has one dedicated secret,
> this is probably not needed.

I dont see the admin/tenant split eliminating the potential benefit
of being able to hand out restricted CLI access secrets.

As for CLI plain-text:  I would really love to find a good and mostly
seamless way to use SSH for CLI access.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Idea for multi-level CLI access control

[Poul-Henning Kamp]

We talked about the overall security model during bugwash today and
while trimming the hedges I had the following idea:

Today the fundamental authentication to open a CLI port is that
that you have access to the exact and entire contents of the "secret"
file and can generate a proof of this.

We keep that, but...

1.  We allow varnishd to have multiple secret files.
When a CLI connection attempts to authenticate, varnishd tries
them all.

2.  Secret files can be "old style" or "new style", in both
cases the "proof" uses the entire content of the secret file,
byte for byte.

3.  "New style" secret files have the following syntax:

Lines which start with '#' are comments and are ignored.

First line:

"secret: "  NL

Then any number of rules:

("permit: " | "deny: ")  NL

varnishd always appends a "deny: ." rule at the end of the
list of rules.

All submitted CLI commands are tested against these rules in
the order they appear in the secret file, and the search
terminates when one of them matches.

A trivial example of a secret file could be:

secret: swordfish
deny: vcl
deny: stop
# Note: Do not name a backend "kanban"
deny: ban

Random notes:

* Ideally the help command output is also filtered through the rules.

* Varnishd should identify itself (-i/-n) in the 107 message so that the
  client can pick which secret file to use if it has access to multiple.

* Varnishadm could look for secret files in ~/.varnish/${-i/-n arg}

Comments ?

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

a bit out of action....

[Poul-Henning Kamp]

I'm a bit out of action due to bacterial pneumonia, but I'll be keeping
eye on things to the best of my ability.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: *.body VMOD arguments

[Poul-Henning Kamp]

Nils Goroll writes:

> But I can not remember if we ever talked about *.body as VMOD function/method
> arguments? If we did, please remind me.
>
> If not, I would suggest that we add those as a pointer to the respective
> objcore. Because VCL_BODY currently is typedef'd as const void *, we would
> need to change that and rewrite the existing VCC code for "VCL_BODY += ..." 
> and
> "unset VCL_BODY", but I see no fundamental problem with that.

We already have a BODY type ?

const struct type BODY[1] = {{
.magic =TYPE_MAGIC,
.name = "BODY",
    .noindent = 1,
}};


-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Bugwash today at 15:00 EU time...

[Poul-Henning Kamp]

Can we please try to get the bugwash back on track ?

If the monday 15:00 has become inconvenient for people we should try
to find another and better timeslot.

Poul-Henning

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Death on idea: filters on synth bodies

[Poul-Henning Kamp]

For the longest times I have wanted to run synth bodies through
VFP, to get rid of the temp VSB and to enable ESI processing
on error messages.

It is not going to happen.

It would become quite a mess in `vcl_synth{}`.

We do not run `vcl_deliver{}` after `vcl_synth{}, so there
both the VFP and VDP filter lists would have to be configured
at the same time, which in addition to requiring two filter
lists variables, makes it near impossible to produce the automatic
proposal for VDP filters based on the object.

But we would also still have to buffer the body in a VSB, because we
"suck" data through VFP from the core code, we dont push it in from
the fetch code.

It would be more feasible to do it in the case of `vcl_backend_error{}`
because we only need to deal with the VFP list there, but the temporary
VSB would still be required there too.

So, yeah:  Not going to happen.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

a possible micro-optimization

[Poul-Henning Kamp]

This is a crudely hacked together proof-of-concept where a "perfect hash"
is used to find out if a given HTTP header has filtering flags.

I expect, but am far from sure, that this will perform better than the
current sequential code.

If anybody has a chance to beat this up in some kind of a performance
test, I would be very interested to hear the results.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c
index 09fce7def..ef09a5f8f 100644
--- a/bin/varnishd/cache/cache_http.c
+++ b/bin/varnishd/cache/cache_http.c
@@ -57,6 +57,124 @@ const char H__Status[]	= "\010:status:";
 const char H__Proto[]	= "\007:proto:";
 const char H__Reason[]	= "\010:reason:";
 
+
+/*
+ * Generated with gperf --ignore-case over tbl/http_headers.h
+ * where third arg non-zero.
+ */
+
+static const unsigned char asso_values[] = {
+	39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39,
+	39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39,
+	39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39,
+	39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39,
+	39, 39, 39, 39, 39, 25, 39,  0, 20,  5, 39, 39, 39, 15,  0, 39,
+	10, 39,  0, 39, 15, 10, 39, 39,  0, 39, 39, 39, 39, 39, 39, 39,
+	39, 39, 39, 39, 39, 25, 39,  0, 20,  5, 39, 39, 39, 15,  0, 39,
+	10, 39,  0, 39, 15, 10, 39, 39,  0, 39, 39, 39, 39, 39, 39, 39,
+	39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39,
+	39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39,
+	39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39,
+	39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39,
+	39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39,
+	39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39,
+	39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39,
+	39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39, 39
+};
+
+static struct foobar {
+	char		*hdr;
+	unsigned	flag;
+} foobar[] = {
+	{ NULL },
+	{ NULL },
+	{ H_TE },
+	{ H_Age },
+	{ NULL },
+	{ H_Range },
+	{ NULL },
+	{ H_Upgrade },
+	{ H_If_Range },
+	{ NULL },
+	{ H_Connection },
+	{ NULL },
+	{ H_Trailer },
+	{ H_If_None_Match },
+	{ NULL },
+	{ NULL },
+	{ NULL },
+	{ H_Transfer_Encoding },
+	{ H_Proxy_Authenticate },
+	{ H_Proxy_Authorization },
+	{ H_Keep_Alive },
+	{ NULL },
+	{ NULL },
+	{ H_If_Match },
+	{ H_HTTP2_Settings },
+	{ NULL },
+	{ NULL },
+	{ NULL },
+	{ H_Content_Range },
+	{ H_If_Unmodified_Since },
+	{ NULL },
+	{ NULL },
+	{ H_If_Modified_Since },
+	{ H_Cache_Control },
+	{ NULL },
+	{ NULL },
+	{ NULL },
+	{ NULL },
+	{ H_Accept_Ranges }
+};
+
+static struct foobar *
+hdr_hash(const char *b, const char *e)
+{
+	unsigned u;
+	struct foobar *retval;
+
+	if (e == NULL)
+		return(NULL);
+	assert(e > b);
+	u = (unsigned)(e - b);
+	if (u < 1 || u > 19)
+		return(NULL);
+	if (u > 3)
+		u += asso_values[((const uint8_t*)b)[3]];
+	if (u > 38)
+		return(NULL);
+	retval = [u];
+	if (retval->hdr == NULL)
+		return(NULL);
+	if (strncasecmp(retval->hdr + 1, b, e - b))
+		return(NULL);
+	return(retval);
+}
+
+static void
+hdr_init(char *hdr, int flg)
+{
+	struct foobar *f;
+
+	hdr[0] = strlen(hdr + 1);
+	f = hdr_hash(hdr + 1, hdr + hdr[0]);
+	if (!flg)
+		return;
+	AN(f);
+	assert(f->hdr == hdr);
+	f->flag = flg;
+}
+
+/**/
+
+void
+HTTP_Init(void)
+{
+
+#define HTTPH(a, b, c) hdr_init(b, c);
+#include "tbl/http_headers.h"
+}
+
 /*
  * These two functions are in an incestuous relationship with the
  * order of macros in include/tbl/vsl_tags_http.h
@@ -735,6 +853,7 @@ http_DoConnection(struct http *hp, enum sess_close sc_close)
 	const char *h, *b, *e;
 	enum sess_close retval;
 	unsigned u, v;
+	struct foobar *f;
 
 	CHECK_OBJ_NOTNULL(hp, HTTP_MAGIC);
 	assert(sc_close == SC_REQ_CLOSE || sc_close == SC_RESP_CLOSE);
@@ -756,13 +875,9 @@ http_DoConnection(struct http *hp, enum sess_close sc_close)
 			retval = SC_NULL;
 
 		/* Refuse removal of well-known-headers if they would pass. */
-/*lint -save -e506 [constant value boolean] */
-#define HTTPH(a, x, c)		\
-		if (!((c) & HTTPH_R_PASS) &&			\
-		strlen(a) == u && !strncasecmp(a, b, u))	\
+		f = hdr_hash(b, e);
+		if (f != NULL && !(f->flag & HTTPH_R_PASS))
 			return (SC_RX_BAD);
-#include "tbl/http_headers.h"
-/*lint -restore */
 
 		for (v = HTTP_HDR_FIRST; v < hp->nhd; v++) {
 			Tcheck(hp->hd[v]);
@@ -898,13 +1013,16 @@ http_PutResponse(struct http *to, const ch

Working Group Last Call: HTTP Core Documents (fwd)

[Poul-Henning Kamp]

Time to read!

--- Forwarded Message

From: Tommy Pauly 
Message-id: <85b2c1d6-28d6-4988-9b91-068d3039b...@apple.com>
Date: Thu, 14 Jan 2021 09:49:22 -0800
To: HTTP Working Group 
Subject: Working Group Last Call: HTTP Core Documents

Hello HTTP WG,

The time has come to start our Working Group Last Call for our current 
'core' documents! The issues list for these documents 
has been whittled down and we're ready to have the working group 
review them in depth in preparation to progress them towards 
publication.

These are substantial drafts, so your time in reviewing is very much 
appreciated. We'll plan to have a three-and-a-half-week last call, so 
that we will get feedback prior to our planned interim dates. The last 
call will end on Monday, February 8, 2021.

The three drafts to review are:

HTTP Semantics, draft-ietf-httpbis-semantics
https://www.ietf.org/archive/id/draft-ietf-httpbis-semantics-14.html 
<https://www.ietf.org/archive/id/draft-ietf-httpbis-semantics-14.html>

HTTP/1.1, draft-ietf-httpbis-messaging
https://www.ietf.org/archive/id/draft-ietf-httpbis-messaging-14.html 
<https://www.ietf.org/archive/id/draft-ietf-httpbis-messaging-14.html>

HTTP Caching, draft-ietf-httpbis-cache
https://www.ietf.org/archive/id/draft-ietf-httpbis-cache-14.html 
<https://www.ietf.org/archive/id/draft-ietf-httpbis-cache-14.html>

Please send email to the working group list with comments and feedback, 
and state if you think these documents are ready to progress. You can 
also file issues here: https://github.com/httpwg/http-core/issues 
<https://github.com/httpwg/http-core/issues>.

Many thanks to Julian, Mark, and Roy for their hard work on editing 
these documents!

Best,
Tommy

--- End Forwarded Message

--
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.

___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: allocating backend attributes to layers

[Poul-Henning Kamp]

Nils Goroll writes:

> - For use with routing directors, we could make good use of some kind of
> "reservation ticket" such that a routing director can make it more likely
>  to get a connection once the fetch starts.

And what happens when VCL takes an entirely different path, how does the
reservation gets released ?  vmod_priv ?

> - When we added the connection stats, we put them at the backend level as, at
> the time, we did not want to introduce another user-visible layer. Should we
> change that, we should also move the connection stats to the endpoint layer.

'move' or 'also implement' ?

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

allocating backend attributes to layers

[Poul-Henning Kamp]

I'm working on the "backend-clone" thing for dynamic directors and
I have run into something I could surprisningly not answer:  Which
backend attributes lives at which levels ?

So roughly speaking our backend stack looks like this:

Director
|
+-> [CGI]
|
v
Backend probe, hosthdr
|
+-> [Http2, QUIC]
|
v
Http1
|
v
|
Endpoint
|
+-+
| |
|  proxyhdr prefix, proxy_header
v |
+<+
|
+--> TCPipv4, ipv6, port
|
+--> UDSpath
|
+--> [TLS]  ipv4, ipv6, port, [clientcert, 
pubkey]

I have added the obvious extension-points in [] 

I have allocated the some of the backend attributes in the right column.

But now I need help:

proxy_header


Can one send a PROXY header over a TLS connection ?

If not, it needs to go to the TCP and UDS levels


{connect|first_byte|between_bytes}_timeout
--

I dont think these would apply to [CGI], they would have
their own separate timeouts.

I lean that they belong at the Backend level, but does that make sense ?

What would first_byte and between_bytes refer to in H2 or QUIC
context ?  Payload bytes or overhead bytes ?

Same question for TLS ?

max_connections
---

Does it go to the Backend level or the Endpoint level ?

Endpoints pool and serve multiple backends, so if we
put it in at the Backend level, you have to summ all the
backends limits to get the limit at the Endpoint level.

Maybe we need to be able to set limits both at the
Backend and Endpoint levels ?


-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

VIP30 at todays bugwash

[Poul-Henning Kamp]

Please see:


https://github.com/varnishcache/varnish-cache/wiki/VIP30%3A-Plumbing%3A-vcl_raw%28%29-and-vcl_pipe%28%29

Discussion in #3471

I'd like to get a preliminary go/no-go on the conceptual idea at bugwash today.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

V-C.O upgrade footnotes

[Poul-Henning Kamp]

I have upgraded the V-C.O server to FreeBSD 12.2-RELEASE and
it had some minor fallout:

The 5.1, 5.2, 6.0 and 6.1 builds all failed in the automatic run,
due to a new check in CLANG which spotted that a double cannot
represent a LONG_MAX precisely.

For 5.1 and 5.2, I patched a checked- out tree to build docs and
freeze them as tar files, as we have done with the 2.x, 3.x and
4.x docs which go on the webpage.

I did not commit the fixes to the 5.x branches, in order to not
give anybody false hope that we still maintain them.

For 6.0 and 6.1 I committed the fix to the branches so and
the docs continues to be built when commits to those branches
happen.

Poul-Henning
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

2020-09-15 code freeze is lifted

[Poul-Henning Kamp]

Thanks to everybody, in particular Martin, for getting the 2020-09-15
release out the door.

Code freeze is lifted, but please start gently.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Making libunwind the default

[Poul-Henning Kamp]

Dridi Boukelmoune writes:

> After a brief discussion on IRC I agreed to provide comparative stack
> traces, I'll find some time to make a couple up.

Status in FreeBSD-land:

The libunwind backtrace does not look obviously superior, or for
that matter significantly different on FreeBSD.

Libunwind does not seem to support 32-bit Arm well, at least on FreeBSD.

Would it make sense to make this a packaging decision ?

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Decide Name/number of 2020-09-15 release

[Poul-Henning Kamp]

We're going to nail the name/number of the 2020-09-15 release at the
bugwash today.  Please come prepared.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Starting to implement Cache-Status (fwd)

[Poul-Henning Kamp]

Anybody got time for some VCL hacking ?

--- Forwarded Message
From: Mark Nottingham 
Message-Id: <97385692-bff5-4fd2-a2eb-ec499182d...@mnot.net>
Date: Thu, 6 Aug 2020 19:01:54 +1000
To: HTTP Working Group 
Subject: Starting to implement Cache-Status

Just as a heads-up, we're starting to get implementation of =
Cache-Status,[1], and I suspect that we're not too far from Last =
Call.[2]

So far I'm aware of:

* My implementation for Fastly VCL - =

* Amos' patch for Squid - =


Are there others? I'd like to get more feedback soon, before we ship it, =
so consider this a call for that.

Cheers,

1. =
https://httpwg.org/http-extensions/draft-ietf-httpbis-cache-header.html
2. Speaking as editor of that draft, not as a chair.

- --
Mark Nottingham   https://www.mnot.net/



--- End of Forwarded Message
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Compared performance of Varnish Cache on x86_64 and aarch64

[Poul-Henning Kamp]

Martin Grigorov writes:

> Any feedback and ideas how to tweak it (VCL or even patches) are very
> welcome!

First you need to tweak your benchmark setup.

   aarch64

  Thread Stats   Avg  Stdev Max   +/- Stdev
Latency   655.40us  798.70us  28.43ms   90.52%

Strictly speaking, you cannot rule out that the ARM machine
sends responses before it receives the request, because your
standard deviation is larger than your average.

In other words:  Those numbers tell us nothing.

If you want to do this comparison, and I would love for you to do so,
you really need to take the time it takes, and get your "noise" down.

Here is how you should do it:

for machine in ARM, INTEL
Reboot machine
For i in (at least) 1-5:
Run test for 5 minutes

If the results from the first run on each machine is very different
from the other four runs, you can disrecard it, as a startup/bootup
artifact.

Report the numbers for all the runs for both machines.

Make a plot of all those numbers, where you plot the reported
average +/- stddev as a line, and the max value as a dot/cross/box.

If you want to get fancy, you can do a Student's T test to tell
you if there is any real difference.  There's a program called
"ministat" which will do this for you.

Also:  I can highly recommend this book:

   http://www.larrygonick.com/titles/science/the-cartoon-guide-to-statistics/

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

VSB_quote has gotten messy

[Poul-Henning Kamp]

I finally got around to look at VSB_QUOTE_GLOB feature Guillaume committed
by accident some time ago, and it doesn't work correctly as far as I
can tell, for instance, the difference between inputs:
[]
and
["]
makes no sense to me.

However, I can hardly blame Guillaume, because it is not very
consistent or clear how VSB_QUOTE is supposed to work in the first
place, I just spent 4 hours trying to find out, because we sort of
made it up as we went.

I propose that b0d1a40f326f... gets backed out before it has any
use in the tree, and put an errata on the 6.4 release page to the
effect of "do not use VSB_QUOTE_GLOB".

I also propose that we should deprecate VSB_quote*() in its current
form, ie: leave around for the benefit of VMODers for 7.x, remove
in 8.x.

Finally, I propose a new and more well thought, and better documented
replacement, VSB_encode(), to be added shortly.

Comments ?


-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Summertime moves the bugwash!

[Poul-Henning Kamp]

For those participating in bugwash from outside the EU-hegemony:

Please note that we europeans have moved to summertime now, so the bugwash
at 1500 EU-time is now at 1300 UTC.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Support for AARCH64

[Poul-Henning Kamp]

In message 
, Guillaume Quintard writes:

>Offering arm64 packages requires a few things:

Don't we have packages for a bunch of non-x86 architectures on Redhat ?

I seem to recall Ingvar popping up with issues on s390 and other archs
every so often ?

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Support for AARCH64

[Poul-Henning Kamp]

In message 
, Emilio Fernandes writes:

>I'd like to ask whether AARCH64 architecture is officially supported one.

Hi Emilio,

In the sense that we meticulously makes sure that Varnish works on
all archtectures we can lay our hands on, including arm64:  Yes it
is supported.

You can see here which arch/os/compiler combos our daily testing
involves:

http://varnish-cache.org/vtest/

I'm pretty sure FreeBSD has arm64 varnish packages, but I'll let
our package-meisters answer with respect to package building on
Linux.

>Thank you for Varnish Cache!

You're welcome!


-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Code coverage by individual VTCs

[Poul-Henning Kamp]

I have run GCOV on the individual VTC tests and chewed through
the results and produced two listings:

http://phk.freebsd.dk/misc/vtcgcov_by_source_line.txt

This listing shows source lines in varnish-cache, how many
times they were executed running all the tests, and how
many tests they were executed in.  If the last number is
low, the tests are listed.

http://phk.freebsd.dk/misc/vtcgcov_by_vtc.txt

This listing shows the lowest number of VTCS which tested the
source lines tested by each VTC.

In other words, a value of one means that there are source
lines in varnish-cache only tested by that VTC and a value
of 258 probably means the VTC is not doing squat for us.

Please note that the tests were all run as root on FreeBSD 13-CURRENT,
so the results are not as comprehensive as they might look.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: changelog automation?

[Poul-Henning Kamp]

In message , Nils Goroll 
writes:
>does anyone have experience with something like
>
>https://github.com/hawkowl/towncrier
>
>at first sight, would this help with our changelog maintenance?

The changelogs I know which are built by semi-automated means are useless
commitlogs with the useful stuff omitted.

The value of a changelog is *precisely* the editorial work which condenses
the commitlog to a concise and useful document.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

http-core status for IETF 106 (Singapore) (fwd)

[Poul-Henning Kamp]

Time to start reading...

--- Forwarded Message
From: "Roy T. Fielding" 
Date: Wed, 13 Nov 2019 12:03:27 -0800
To: "ietf-http...@w3.org Group" 
Subject: http-core status for IETF 106 (Singapore)


> On Oct 25, 2019, at 3:00 PM, Tommy Pauly  wrote:
> 
> Our draft agenda for IETF 106 is available here:
> 
> https://github.com/httpwg/wg-materials/blob/gh-pages/ietf106/agenda.md

I assume everyone noticed that the -06 drafts for http-core were 
published last week:

  https://tools.ietf.org/html/draft-ietf-httpbis-semantics-06
  https://tools.ietf.org/html/draft-ietf-httpbis-messaging-06
  https://tools.ietf.org/html/draft-ietf-httpbis-cache-06

We will be discussing those at the Monday afternoon meeting in 
Singapore.

  https://datatracker.ietf.org/meeting/106/agenda#2019-11-18-08

Diffs since 05 can be seen at

  https://httpwg.org/http-core/diffs/diff_semantics_05_to_06.html
  https://httpwg.org/http-core/diffs/diff_messaging_05_to_06.html
  https://httpwg.org/http-core/diffs/diff_cache_05_to_06.html

and a frankenRFC diff of all changes since the last consensus RFCs, 
rearranged
according to the current draft structure to show just the word changes:

  https://httpwg.org/http-core/diffs/diff_semantics_frfc_to_06.html
  https://httpwg.org/http-core/diffs/diff_messaging_frfc_to_06.html
  https://httpwg.org/http-core/diffs/diff_cache_frfc_to_06.html

As always, the best way to track our work is on github at

  https://github.com/httpwg/http-core

and especially the list of commits and open issues:

  https://github.com/httpwg/http-core/commits/master
  https://github.com/httpwg/http-core/issues

For this meeting, we will probably focus on the issues marked with the 
label "discuss"

  https://github.com/httpwg/http-core/labels/discuss

(which are still being added to, so speak up if you want to discuss 
something).

The following 28 issues have been closed since the last meeting in 
Montreal:

  
https://github.com/httpwg/http-core/issues?utf8=%E2%9C%93=is%3Aissue
+is%3Aclosed+closed%3A%3E2019-07-19+sort%3Acreated-asc

with the bulk of text changes being about

 - refactoring the byte ranges grammar to be extensible (#196, #212
 - moving trailer fields to Semantics (#16, #117)
 - moving payload body requirements to Semantics (#159, #202)
 - moving retries to idempotent methods (#27, 
 - adding a port registration (#36)
 - adding a min supported URI length (#169)
 - incorporating the remaining RFC2818 (HTTP over TLS) text (#236)
 - replacing "cacheable by default" with heuristically cacheable (#54, 
#242)
 - defining requirements on caching incomplete responses (#25, #221)

See the "Changes since ..." sections at the end of each draft for a 
brief
summary of what has been changed.

We currently have 60 open issues remaining on the list, though 15 are
editorial and many others have only small bits left to do before 
closing.

Cheers,

Roy T. Fielding
Senior Principal Scientist, Adobe
--- End of Forwarded Message
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

apology...

[Poul-Henning Kamp]

I want to apologize for not being at the bugwash on monday, I had fully
planned to attend but something amazing happened and stole my time.

Long story short:  The Vice-President of University of Haute-Alasace
drowe 1000km to Denmark to help us get our Rational R1000/400
computer working and when I should have been bug-washing, the machine
was booting up.

I promise to not make it a habit.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: #3093 proposed patch

[Poul-Henning Kamp]

In message 
, Martin Blix Grydeland writes:

>But I do not think that this patch alone will be enough to fix #3093.

My patch was only meant as a proof of concept for the "DTRT" part
of the topic/discussion.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

#3093 proposed patch

[Poul-Henning Kamp]

I'm on the train with shitty connectivity, so I will not attempt
a pull request, but this is a proposed DTRT patch for 3093

(Note that the vtc in 3093 fails, because this DTRT)

Comments & Tests welcome.

Spotted along the way: Should we alloc std.cache_req_body() also
in vcl_backend_fetch{} ?


-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 778db6861..c27745d71 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -407,6 +407,7 @@ struct busyobj {
 	struct http		*bereq0;
 	struct http		*bereq;
 	struct http		*beresp;
+	struct objcore		*bereq_body;
 	struct objcore		*stale_oc;
 	struct objcore		*fetch_objcore;
 
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 38a25d6dd..a98ffce1e 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -241,6 +241,12 @@ vbf_stp_mkbereq(struct worker *wrk, struct busyobj *bo)
 	if (bo->req->req_body_status == REQ_BODY_NONE) {
 		bo->req = NULL;
 		ObjSetState(bo->wrk, bo->fetch_objcore, BOS_REQ_DONE);
+	} else if (bo->req->req_body_status == REQ_BODY_CACHED) {
+		AN(bo->req->body_oc);
+		bo->bereq_body = bo->req->body_oc;
+		HSH_Ref(bo->bereq_body);
+		bo->req = NULL;
+		ObjSetState(bo->wrk, bo->fetch_objcore, BOS_REQ_DONE);
 	}
 	return (F_STP_STARTFETCH);
 }
@@ -924,6 +930,8 @@ vbf_fetch_thread(struct worker *wrk, void *priv)
 	VCL_TaskLeave(bo->vcl, bo->privs);
 	http_Teardown(bo->bereq);
 	http_Teardown(bo->beresp);
+	if (bo->bereq_body != NULL)
+		HSH_DerefObjCore(bo->wrk, >bereq_body, 0);
 
 	if (bo->fetch_objcore->boc->state == BOS_FINISHED) {
 		AZ(bo->fetch_objcore->flags & OC_F_FAILED);
diff --git a/bin/varnishd/cache/cache_req_body.c b/bin/varnishd/cache/cache_req_body.c
index 13c4ed07f..4cdaff865 100644
--- a/bin/varnishd/cache/cache_req_body.c
+++ b/bin/varnishd/cache/cache_req_body.c
@@ -233,7 +233,7 @@ VRB_Iterate(struct req *req, objiterate_f *func, void *priv)
  */
 
 static int v_matchproto_(objiterate_f)
-httpq_req_body_discard(void *priv, unsigned flush, const void *ptr, ssize_t len)
+httpq_re_body_discard(void *priv, unsigned flush, const void *ptr, ssize_t len)
 {
 
 	(void)priv;
diff --git a/bin/varnishd/http1/cache_http1_fetch.c b/bin/varnishd/http1/cache_http1_fetch.c
index dc1515ef1..b8dd44c7c 100644
--- a/bin/varnishd/http1/cache_http1_fetch.c
+++ b/bin/varnishd/http1/cache_http1_fetch.c
@@ -104,7 +104,14 @@ V1F_SendReq(struct worker *wrk, struct busyobj *bo, uint64_t *ctr_hdrbytes,
 	/* Deal with any message-body the request might (still) have */
 	i = 0;
 
-	if (bo->req != NULL &&
+	if (bo->bereq_body != NULL) {
+		if (do_chunked)
+			V1L_Chunked(wrk);
+		(void)ObjIterate(bo->wrk, bo->bereq_body,
+		bo, vbf_iter_req_body, 0);
+		if (do_chunked)
+			V1L_EndChunk(wrk);
+	} else if (bo->req != NULL &&
 	(bo->req->req_body_status == REQ_BODY_CACHED || !onlycached)) {
 		if (do_chunked)
 			V1L_Chunked(wrk);
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: TLS sandboxing

[Poul-Henning Kamp]

In message 
, Dridi Boukelmoune writes:

>I have given more thought to this and I'm torn between "this is 2019
>and we still don't support HTTPS" and the very good reasons why we
>don't. However I still think we should support it and I gave a tiny
>bit more thoughts to this.

Absolutely, all such decisions needs to be revisited as circumstances
change.

>First, in-kernel TLS is coming,

Yes, and about f**king I might add...

(See also: https://people.freebsd.org/~gallatin/talks/euro2019.pdf)

Unfortunately, the kernel people only seem to want to do the easy
part :-(

https://reviews.freebsd.org/D21277

>But once we have that we circle back to handshakes. And I very much
>agree that a keyless approach would be best. However this is still
>2019 so I think we should own it, and not rely on a third party.

I'm not in any hurry to add crypto code to our plate in any way
shape and form.

The primary reason is that as far as I know there are *at best* 1½
persons in the project with skills which *approximate* what would
be required.

The secondary reason is that neither of them are particularly eager
to add another job to their workload.

I can *possibly* be persuaded that we can handle session keys, and
I will be perfectly happy to point at a 3rd party keyserver packages
and say "that's where you get your keys from".

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: [varnishcache/varnish-cache] explicit_bzero() causing havoc (#3051)

[Poul-Henning Kamp]

In message , Geoff Simmons write
s:

>But it didn't make it into the release? Then there's a somewhat buggy
>varnish.m4 in 6.3.0. The workaround is to just ignore the error
>messages, but they're going to be annoying to look at from now on.

It's not a LTS release so we'll survive...

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: [varnishcache/varnish-cache] explicit_bzero() causing havoc (#3051)

[Poul-Henning Kamp]

In message <7b7463af-bd3f-8cfe-acb9-d231cfaa3...@uplex.de>, Geoff Simmons write
s:

>The EXPLICIT_BZERO check is still AC_REQUIRE'd in varnish.m4, from
>VARNISH_PREREQ and _VARNISH_CHECK_DEVEL, which are used in VMOD
>development. Since it's not defined now, this leads to a cascade of
>error messages when autogen.sh is called for a VMOD.

Ticket please, that is out of my comfort area.

>It's worth pointing out, however, that Colin Percival concluded in his
>blog that not even this solution guarantees that the memset call won't
>be optimized out:
>
>https://www.daemonology.net/blog/2014-09-05-erratum.html

I took that as more of a judgement of the sanity of the ISO-C
committee and compiler writers in general, as a problem we need
to deal with.

>But this gets us much closer to something that will work on most
>platforms. It's OpenSSL's solution for wiping keys in memory, so one
>hopes that it works most of the time.

Ohh God!

Now you just inspired all "cyberforces" to start implementing
compiler optimizations...  :-)

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

VDD19Q3 wiki page

[Poul-Henning Kamp]

Please contribute:

https://github.com/varnishcache/varnish-cache/wiki/VDD19Q3

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Release 6.3 on next monday

[Poul-Henning Kamp]

We decided today that the 2019-09-15 release will happen next monday
and be called 6.3.

It will *not* be a long-term-support ("LTS") release.

Effectively immediately, we are in code freeze as far as source
code, but welcome any contribution pertaining to documentation and
release engineering.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: TLS sandboxing

[Poul-Henning Kamp]

In message , Nils Goroll writ
es:

>Yet with the H3/QUIC madness on the horizon, 

No, they actually dealt with this in the design, so that "keyless"
operation is more or less the natural architecture for QUIC.

If we want to do TCP/TLS, we should also aim firmly for the "keyless" model.

I'm hoping we can to raid the hitch source code to build the "keymaster" 
process.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

VTLAM: VTEST -> VCI

[Poul-Henning Kamp]

During discussions with Nils yesterday it became clear that the
overloading of "VTEST" for both the non-native version of varnishtest
and for our CI setup is silly.

To that end, I plan to rename the VTEST stuff to VCI as soon as a
round tuit can be spared.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: STRING_LIST deorbit burn completed

[Poul-Henning Kamp]

In message 
, Dridi Boukelmoune writes:

>I found an old ASAT in my garage and STRING_LIST usage is now
>gone from trunk.

Thanks!

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

a bit out of circulation

[Poul-Henning Kamp]

I managed to acquire an infected insect bite last week.
Its nothing serious (the penicilin-stomach is worse),
but it has reduced my bandwidth somewhat.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: STRING_LIST deorbit burn completed

[Poul-Henning Kamp]

In message 
, Dridi Boukelmoune writes:
>On Fri, Jun 21, 2019 at 1:01 PM Poul-Henning Kamp  wrote:


>My point was rather that "all uses of STRING_LIST in the tree" aren't
>removed yet:
>
>$ git grep STRING_LIST -- lib/libvmod_*
>lib/libvmod_directors/vmod.vcc:$Method BACKEND .backend(STRING_LIST)
>lib/libvmod_vtc/vmod.vcc:$Function VOID panic(STRING_LIST)

Hmm, I guess I should actually grep for STRING_LIST rather than
the functions operating on it :-)

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: STRING_LIST deorbit burn completed

[Poul-Henning Kamp]

In message 
, Dridi Boukelmoune writes:
>On Thu, Jun 20, 2019 at 9:29 AM Poul-Henning Kamp  wrote:
>>
>> I have removed the remaining uses of STRING_LIST in the tree, and
>> made vmodtool.py be slightly annoying if it is used.
>
>Not quite:

Right now I'm trying to get the message out, the annoyance will be
adjusted continuously.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

STRING_LIST deorbit burn completed

[Poul-Henning Kamp]

I have removed the remaining uses of STRING_LIST in the tree, and
made vmodtool.py be slightly annoying if it is used.

vmodtool.py will get more and more annoying about this over summer
and around august 1st, it will refuse to compile it.

The replacement is of course STRANDS.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

VCC now has type-driven properties

[Poul-Henning Kamp]

As you may have spotted in recent commits and notably 2962375b4ed80e,
VCC now has type driven properties.

This means that subexpressions of a given type can express that
property, no matter how we got to it in the first place.

If we defined a type driven property from DURATION called
"foobar" it would be possible to write:

set beresp.http.qqq = (beresp.ttl + 4m).foobar;

Right now, this has not changed the syntax of VCL, we're still 4.1
because the only use is for the storage.*.* variables for which
the syntax remains the exact same.

If we want to, some of the stuff in vmod_std can now be made
slightly more intutive syntax-wise:

BACKEND.healthy -> BOOL

IP.port -> INT

Note that these are *properties* `EXPR.name` they do not take any
VCL arguments beside the expression they work on.  (The C-implementation
can of course take other args, like VRT_CTX)

It would not be too hard to also do *methods* `EXPR.func(args)`,
so that one could also do things like:

set bereq.http.qqq = beres.ttl.format("%.3f")

But I am not going to implement that until we have more convincing
use cases than that.

A bit more work would be to allow vmods to implement type driven
properties and methods, but again:  Convincing use cases ?.

Proposals and ideas are welcome.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

varnish gcov

[Poul-Henning Kamp]

After a lot of comparisons I have changed the GCOV to use LLVM-8.0
because it is about twice as fast and (IMO) more honest than the
GCC6 we used.

This means that the overall score dropped a little bit.

I also rescaled the "sparklines" to make incremental changes in
the range we care about stand out more.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: VIP23 (VSL refactoring) design sketch

[Poul-Henning Kamp]

In message , Geoff Simmons write
s:

Now that I found your mock-up in my spam-folder (no idea why) and
have a semi-working laptop again:

>In the attachment you'll find:
>
>- A first implementation of a function VSLwv() to write binary fields
>into a log record buffer, using a printf-like format string and a va_list

For strict alignment platformns I think you have to explicitly use
memcpy(3) in all cases, and leave it to the compiler to decide if
it really needs to call it.

>I grepped through current VSL*() calls, and these are more or less all
>of the format characters we currently use: %s, %.*s, %.ns (for a length
>n, as in %.20s), %d, %u, %x, %f, %j* and %z* (for (u)intmax_t and size_t
>types).

Yes, the vocabulary isn't that big to begin with.

>As @Nils suspected, the weak spot is branch prediction. My gcc compiled
>the switch statement as a jump table (jmpq *rax), and kcachegrind
>reports 86% of indirect branch mispredicts there, far more than anything
>else -- glibc vfprintf comes in next with 13%.

A switch may not be the optimal performance choice. If we collect
stats on frequency of the various formats, a handsorted if-else-if
chain may be better.

>HTTP header records, with two variable-length strings, also won't be any
>shorter than the ASCII format. If they have two 2-byte length fields,
>they may come out as a couple of bytes longer (especially if they're
>also NUL-terminated).

We could ENUM the most common headers.  I've been considering that in
general, but the performance trade-off isn't obvious.

>   printf("%.*s: %f %f %f\n\n", *((unsigned short *)[24]),
>  [26], *((double *)[0]), *((double *)[8]),
>  *((double *)[16]));
>
>-.. I had pleasant thoughts like "phk is gonna kill me".

For a mock-ups to test a concept, I've done far worse myself :-)

>Something like the FLD()
>accessors will of course also be needed for VSLQ, varnishncsa etc.

All things considered, I suspect the generating VSL calls to be
the simpler part of the task and VSLQ to be the harder one, so
we should probably look carefully at that as well.

Poul-Henning

PS: Re: slinks idea with generating functions to emit the VSL
records, I will mention one evil trick I have seen, but say up front
that we are not going there:  If you machine-generate the producer
functions you can basically memcpy memcpy'ed the stack-frame into
a network packet (or VSL record).  This is horribly CPU-calling-convention
specific, and needs special casing for strings, but in olden days
before really smart compilers (ie: 68k) it was a pretty smart move.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: VIP23 (VSL refactoring) design sketch

[Poul-Henning Kamp]

In message 
, Dridi 
Boukelmoune writes:

>Interesting, for textual logs the effect of vsl_reclen is
>straightforward but how should we deal with binary records truncation?

Maybe redefine the limit to apply to individual strings (ie: header
values) rather than the full record ?

>> Some other lessons learned:
>>
>> The binary records are not necessarily shorter than the ASCII-formatted
>> records.

It is incredible how often people overlook this.  I've been ranting
about it for almost two decades now, in particular in context of the
userland/kernel API.

Imagine if in addition to errno there also were a const char *errstr
which the kernel could fill out, that would do *wonders*, in particular
for EINVAL.

My favourite example was a Nx64 card where the driver spent more
than 35kloc of source code on defining a ton of structs for ioctl(2)
usage but no matter what the kernel found deficient, the only thing
it told you was return(EINVAL).

I threw out the entire thing and added a single ioctl which passed a
struct with an input string, containing a command like
"set chan 4 ts 4-12" 
and another string out with an error message like:
"channel 11 already allocated to chan 5"
didn't even spend 1kloc on it...

>> comes to 48 bytes on my machine, about twice as long as a string like
>> "110 0 110 210 224 434". I don't think we can do much about that.

It depends what we know about the numbers.

For the 68K they had done a lot of statistics, and therefore their
relative jump instructions worked like this:

[opcode] 0x00 [16 bit signed displacement]
[opcode] 0xff [32 bit signed displacement]
[opcode] [8 bit signed displacement]

This gives good compression, without sinking in the
"variable length integers" like the crap in OSI/X509/H2/H3

Of course Motorola had the benefit that neither zero nor minus 1
gives any meaning as relative jump displacement.

We would probably want to sacrifice 0xfe rather than 0x00 (Benfords
Law and all that)

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: on the vsl format - VIP23 (VSL refactoring) design sketch

[Poul-Henning Kamp]

In message <6b8d3465-2245-ba3a-7c80-4b7620078...@uplex.de>, Nils Goroll writes:

>For the binary VSL records we already agreed that fixed length fields always
>come first. Because we have the length and know the size of the fixed part, we
>can could put in another STRING or header as follows:

Well, agreed and agreed...

If we go with the "printf-format" model, and we wait a bit before we
rearrange the order of fields, we can do A/B testing, and i think
that is a very good trade-off for maybe not gaining quite as much
performance up front.

Then once we have switched over and the dust have settled, we can
start to optimize field order.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: VIP23 (VSL refactoring) design sketch

[Poul-Henning Kamp]

In message <2291598a-aa17-ad9f-ed32-5b77b0ede...@uplex.de>, Geoff Simmons 
writes:

>I hope your data's OK.

No worries, I've been in this trade too long to trust any hardware :-)

Running FreeBSD-Current for 26 years makes you a little bit paranoid
about the software too :-)

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: VIP23 (VSL refactoring) design sketch

[Poul-Henning Kamp]

In message , Geoff Simmons 
writes:

>I'm not sure if we have a consensus, but I'd like to summarize what I've
>understood from the conversation so far. I at least would be willing to
>go ahead on this basis.

Sorry, I got distracted by a disk (actually NVME) failure on my laptop
yesterday, and have not tought more about it.  Can we conclude on
this monday morning instead ?


-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: VIP23 (VSL refactoring) design sketch

[Poul-Henning Kamp]

In message <93c9d1d0-9d1c-ba23-a7aa-cc5e597d1...@uplex.de>, Nils Goroll writes:

>But at least for the built-in tags, I find it more appealing to auto-generate
>logging functions for each SLT (not macros) with well-defined arguments, by
>example of
>
>void VSLb_ts(struct vsl_log *, const char *event, vtim_real first,
>vtim_real *pprev, vtim_real now);
>
>The generated code may or may not memcpy, but even if, why is this an issue?

Because of the cache footprint.

If each VSL record invokes its own C-function, we eat up a LOT of L1+L2
cache bandwidth. If they all call the same function, we do not.

(Timestamp was a badly chosen example, because we already wrap that
in a function call, but you get the idea...)


-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: unused mailing lists

[Poul-Henning Kamp]

In message <593b2696-c8bf-f50d-ddde-84f4219d8...@schokola.de>, Nils Goroll 
writes:

>I would have thought we already did exactly this.

Ok, I wasn't sure.

Never mind then.


-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: VIP23 (VSL refactoring) design sketch

[Poul-Henning Kamp]

In message <82919b48-25e0-d22f-1029-6d17db040...@uplex.de>, Geoff Simmons 
writes:

>Whether we get the support is yet to be finalized, but if so we'll give
>it high priority. phk asked me to post a design sketch in advance before
>we put in the major time and effort, so that others can comment, and
>warn us off we're going in the wrong direction.

Yes, I really hate it when people go off and do something in quiet
and come back with something which isn't quite right for some reason
they overlooked.

>- That means that varnishd writes binary fields into the log.

For all its problems, a single call to sprintf is actually pretty
efficient in terms of CPU cache etc, because we almost always have
it stuck there.

I'm not very keen to see a lot of macro-work here, for instance
wrapping three memcpy's for Timestamp etc.

In theory one could extend the printf syntax with %V and similar,
but that looses the compilers type-checking of arguments.

However, just because the "const char *fmt" and the varargs
are the *same* as printf, doesn't mean we have to *do* the same
thing as printf with them.

So as a total early morning strawman, before morning tea has
fully kicked in, but *after* I read your email:

VSLb(vsl, SLT_Timestamp, "%s: %.6f %.6f %.6f",
event, now, now - first, now - *pprev);

Becomes:

VSLb(vsl, SLT_Timestamp, "S%sT%fD%fD%f", 
event, now, now - first, now - *pprev);

And serialize into VSL as:

<8 byte double><8 byte double><8 byte double>

Meanwhile, the "S%sT%fD%fD%f" string (possibly transformed) tells
the client side that we have a string, a timestamp and two durations.

We could make the "veprintf" format string come out of the tbl/mumble.h
paired with the SLT_ define:

#include  magic produces:

#define SLT_Bin_Timestamp SLT_Timestamp | SLT_BINARY, "S%sT%fD%fD%f"

Then in source call becomes:

VSLb(vsl, SLT_Bin_Timestamp, event, now, now - first, now - *pprev);

That way nobody confuses it with a "regular" printf, and the
SLT_Timestamp bit alerts the VSL code to interpret the fmt string
as "veprintf" syntax.

This would be back-wards source compatible, making the the transition
easier, and could allow us to make A/B testing by substituting "real"
printf formats with a compiletime option.

The "veprintf" syntax needs to be carefully thought out, for instance
"S%f" should be a syntax error, and that needs to be caught somewhere,
preferably at compile time, but worst case we could walk all the
strings from  with a varnishd -x argument as part of
the build.

>- Outputs for structured, typed data, in particular JSON from varnishlog
>-j, fall out easily, since the data is already structured and typed
>(don't need to parse strings in the log).

So where does  come from ?

It probably makes a lot of sense to wrap all the information one single
place, possibly as .rst-oid format like ".vsc" files and have python
chew on that and output all of docs, JSON metadata and the macro
definitions for varnishd (which would not tbl/mumble.h then.)

Far out:

Given this level of abstraction we could reorder the arguments
between VSL and presentation, so for instance timestamp could be
serialized as:

<8 byte double><8 byte double><8 byte double>

So the VSLQ would always know where to find the doubles, but the
presentation code in libvarnishapi would (still) render the string
in front.

We chould also benchmark emitting a length before ASCIIZ to make
it faster to skip strings elements.

Either 'trick' might make VSLQ faster.


-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: unused mailing lists

[Poul-Henning Kamp]

In message <45384.1554798...@critter.freebsd.dk>, Poul-Henning Kamp writes:

Also varnish-dist it seems.

>It looks like varnish-test and varnish-bugs are not used anymore.
>
>Is it possible to configure so they reject any email sent, but the
>archives still remain visible in mailman ?

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

unused mailing lists

[Poul-Henning Kamp]

It looks like varnish-test and varnish-bugs are not used anymore.

Is it possible to configure so they reject any email sent, but the
archives still remain visible in mailman ?

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: bugwash 2019-04-01

[Poul-Henning Kamp]

In message <8d34478b-e99a-2fe2-de43-f9a49ea27...@schokola.de>, Nils Goroll 
writes:

>I will not be able to attend bugwash tomorrow

I'm not sure I can either, I will be at Hamburg Hbf. most of that hour.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: 2019-03-15 release

[Poul-Henning Kamp]

In message 
, Dridi Boukelmoune writes:
>On Thu, Mar 14, 2019 at 8:43 PM Poul-Henning Kamp  wrote:
>>
>> Do not cut the release until I give the OK.
>>
>> I/We have a couple of i's to dots and t's to dash first.
>
>And coverity reported a bunch of things too!

Those are part of it.

(Many thanks to Federico for getting a Coverity Run through!)

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

2019-03-15 release

[Poul-Henning Kamp]

Do not cut the release until I give the OK.

I/We have a couple of i's to dots and t's to dash first.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Doc updates on v-c.o

[Poul-Henning Kamp]

In message , Geoffrey Simmons wr
ites:

Seems like python3  fallout. should be fixed now

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

code-freeze-ish

[Poul-Henning Kamp]

Trivial doc-commits and similar polishing allowed.

Anything bigger than that needs an OK (PR/IRC) until the 15th.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: tabular CLI output proof of concept hack

[Poul-Henning Kamp]

In message 
, Dridi 
Boukelmoune writes:
>> But I am also seriously wondering if we should go even further and
>> have varnishd *only* produce JSON output, and leave the render-for-humans
>> aspect to varnishapi or even varnishadm.
>
>Ideally implement it in libvarnish.a and expose it in libvarnishapi,
>with a way for varnishadm to specify a window size in interactive
>mode. This way varnishd -d will be able to produce the plain text
>output.

I think you need to unpack that a bit before I follow ?

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

tabular CLI output proof of concept hack

[Poul-Henning Kamp]

While looking at #2896 it bothered me how much code we spend on
text-processing the CLI output.

Since we now have -j for machine-readable output, it would make
sense to make the non-j output more human-friendly, and avoid some
of all that text-processing at the same time.

I threw together the attached proof-of-concept hack, which adds
a central function to figure out how wide the columns of tabular
output should be, and to reformat it thusly.

The idea is that the CLI code, in this case vcl.list sticks
some ASCII control-chars in the vsb to mark the columns, which then
autosize once all the output is ready, rather than have all the
"lets set this column width to 28 ? 45 ? 51?" guesswork.

In the hack I have also added column headers for nicer output
so that it looks like this:

Status Readiness Ref Name
== = ===  
active auto/warm 1   vcl1  

(Ignore the amount of space/padding etc, that's for later)

I kind of like it.

But I am also seriously wondering if we should go even further and
have varnishd *only* produce JSON output, and leave the render-for-humans
aspect to varnishapi or even varnishadm.

Comments welcome...

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
diff --git a/bin/varnishd/cache/cache_vcl.c b/bin/varnishd/cache/cache_vcl.c
index a524a7981..72c5da737 100644
--- a/bin/varnishd/cache/cache_vcl.c
+++ b/bin/varnishd/cache/cache_vcl.c
@@ -657,6 +657,8 @@ vcl_cli_list(struct cli *cli, const char * const *av, void *priv)
 	(void)av;
 	(void)priv;
 	ASSERT_CLI();
+	
+	VCLI_Out(cli, "\x0eStatus\x02Readiness\x02Ref\x02Name\n");
 	VTAILQ_FOREACH(vcl, _head, list) {
 		if (vcl == vcl_active) {
 			flg = "active";
@@ -664,7 +666,7 @@ vcl_cli_list(struct cli *cli, const char * const *av, void *priv)
 			flg = "discarded";
 		} else
 			flg = "available";
-		VCLI_Out(cli, "%-10s %5s/%-8s %6u %s",
+		VCLI_Out(cli, "%s\x01%s/%s\x01%u\x01%s\x01",
 		flg, vcl->state, vcl->temp, vcl->busy, vcl->loaded_name);
 		if (vcl->label != NULL) {
 			VCLI_Out(cli, " -> %s", vcl->label->loaded_name);
diff --git a/bin/varnishd/mgt/mgt_vcl.c b/bin/varnishd/mgt/mgt_vcl.c
index 7d8acb572..32019cbf6 100644
--- a/bin/varnishd/mgt/mgt_vcl.c
+++ b/bin/varnishd/mgt/mgt_vcl.c
@@ -811,13 +811,14 @@ mcf_vcl_list(struct cli *cli, const char * const *av, void *priv)
 		}
 		free(p);
 	} else {
+		VCLI_Out(cli, "\x0eStatus\x02Readiness\x02Ref\x02Name\n");
 		VTAILQ_FOREACH(vp, , list) {
-			VCLI_Out(cli, "%-10s %5s",
+			VCLI_Out(cli, "%s\x01%s",
 			vp == active_vcl ? "active" : "available",
 			vp->state);
-			VCLI_Out(cli, "/%-8s", vp->warm ?
+			VCLI_Out(cli, "/%s\x01", vp->warm ?
 			VCL_STATE_WARM : VCL_STATE_COLD);
-			VCLI_Out(cli, " %6s %s", "-", vp->name);
+			VCLI_Out(cli, "-\x01%s\x01", vp->name);
 			if (mcf_is_label(vp)) {
 vd = VTAILQ_FIRST(>dfrom);
 AN(vd);
diff --git a/bin/varnishtest/tests/u00011.vtc b/bin/varnishtest/tests/u00011.vtc
index b53a9ba7b..6e7f8eb11 100644
--- a/bin/varnishtest/tests/u00011.vtc
+++ b/bin/varnishtest/tests/u00011.vtc
@@ -24,7 +24,8 @@ process p1 -expect-text 0 1 "PONG"
 
 process p1 -write "vcl.li\t\r"
 
-process p1 -expect-text 0 1 "active  auto/warm"
+process p1 -expect-text 0 0 "active"
+process p1 -expect-text 0 0 "auto/warm"
 
 process p1 -write "vcl.s\t\th\t vcl1\r"
 
diff --git a/lib/libvarnish/vcli_serve.c b/lib/libvarnish/vcli_serve.c
index e5cf0c411..3eeeb9431 100644
--- a/lib/libvarnish/vcli_serve.c
+++ b/lib/libvarnish/vcli_serve.c
@@ -247,6 +247,94 @@ cls_dispatch(struct cli *cli, const struct cli_proto *cp,
 		cp->func(cli, (const char * const *)av, cp->priv);
 }
 
+/*
+ * Reformat CLI output as table
+ */
+
+#define MAXCOL 10
+static char *
+Tabelize(const char *s)
+{
+	int colw[MAXCOL];
+	int ncol = 0;
+	int nln = 0;
+	int cc = 0;
+	int cw = 0;
+	const char *p;
+	char *q, *r;
+	size_t sz = 0;
+
+	assert(*s == '\x0e');
+	memset(colw, 0, sizeof colw);
+	for (p = ++s; *p ; p++) {
+		switch(*p) {
+		case '\x01':
+		case '\x02':
+			if (cw > colw[cc])
+colw[cc] = cw;
+			cc++;
+			cw = 0;
+			assert(cc < MAXCOL);
+			break;
+		case '\n':
+			if (cc > ncol)
+ncol = cc;
+			cc = 0;
+			cw = 0;
+			nln++;
+			break;
+		case '\t':
+			WRONG("TAB not allowed in tables");
+		default:
+			cw++;
+		}
+	}
+	for(cc = 0; cc < ncol; cc++)
+		sz += 1 + colw[cc];
+	sz *= nln + 1;
+	sz += 1;
+	q = malloc(sz);
+	AN(q);
+	r = q;

Re: VMOD havoc generating patch

[Poul-Henning Kamp]

In message 
, Dridi 
Boukelmoune writes:


>No, I mean when I look at the generated vcc_$VMOD_if.h, I see many
>occurrences of the $Prefix that are not guarded by VPFX() macros.

The VPFX/VARGS/VENUM macros are mostly intended as convenience for the
files the vmod-author writes.

I didn't VPFX the prototypes to not _force_ people to use the VPFX
in their code if they dont want to, but if the consensus is that
we should be pushing VPFX() usage, changing it is just stylistic.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: VMOD havoc generating patch

[Poul-Henning Kamp]

In message 
, Dridi 
Boukelmoune writes:
>On Mon, Feb 4, 2019 at 8:30 PM Poul-Henning Kamp  wrote:

>It seems incomplete though, currently the $Prefix is applied all over
>the place and the VPFX macro isn't used for function names. Is that
>still work in progress?

You mean the in-tree vmods ?

Yes, I only applied the necessary macros to make things work again,
in order to highlight that you dont *have* to use the macros to
write/port a VMOD to the new world order.

After the dust has settled and march release is out, I think we
should fully macro-ize the in-tree VMODS, but if you guys think
it is a better idea to go the full step now, I'm fine with that.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: VMOD havoc generating patch

[Poul-Henning Kamp]

In message 
, Dridi 
Boukelmoune writes:
>On Mon, Jan 28, 2019 at 1:55 PM Poul-Henning Kamp  wrote:

>I waited until my weekly CI against master would choke on this change,
>and it's very easy to support both before and after if your baseline
>is 6.0, which is more than fine since 4.1 only has a couple months to
>live:
>
>#include "vcc_$VMOD_if.h"
>
>/* Varnish < 6.2 compat */
>#ifndef VPFX
> #define VPFX(a) vmod_ ## a
> #define VARGS(a) vmod_ ## a ## _arg
> #define VENUM(a) vmod_enum_ ## a
>#endif
>
>I didn't try on a vmod with a $Prefix stanza, but I'm sure it can be
>helped if that snippet is not enough.

Yes, that would go a long way towards compatibility.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

VMOD havoc generating patch

[Poul-Henning Kamp]

Ticket 2810 is about the names generated by vmodtool and vcc, and
while there is a good intellectual argument for getting it right,
I am a little bit worried about how much havoc that causes.

This is a WIP patch headed in that direction, and I would like to
hear input from VMOD writers.

Ideally with this stuff finished, VMOD writers can version their
vmods using $Prefix and you will then be able to import multiple
different versions of the same VMOD in the same VCL.  Not sure that
is a good thing to do, but it proves that the name-space issue is
solved.

See the changes to the in-tree vmods for how this will look for you.



-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
diff --git a/doc/sphinx/reference/directors.rst b/doc/sphinx/reference/directors.rst
index b359133ae..8758b4535 100644
--- a/doc/sphinx/reference/directors.rst
+++ b/doc/sphinx/reference/directors.rst
@@ -155,7 +155,7 @@ Health Probes
 =
 
 It is possible in a VCL program to query the health of a director (see
-:ref:`func_healthy`). A director can report its health if it implements the
+:ref:`vmod_std.healthy`). A director can report its health if it implements the
 ``healthy`` function, it is otherwise always considered healthy.
 
 Unless you are making a dynamic backend, you need to take care of the
diff --git a/doc/sphinx/whats-new/upgrading-5.1.rst b/doc/sphinx/whats-new/upgrading-5.1.rst
index f350c58ff..62a1726c4 100644
--- a/doc/sphinx/whats-new/upgrading-5.1.rst
+++ b/doc/sphinx/whats-new/upgrading-5.1.rst
@@ -197,7 +197,7 @@ vcl_recv
 
 * Added ``req.storage``, which tells Varnish which storage backend to
   use if you choose to save the request body (see
-  :ref:`func_cache_req_body`).
+  :ref:`vmod_std.cache_req_body`).
 
 * ``return(vcl(LABEL))`` may not be called after a restart. It can
   only be called from the active VCL instance.
@@ -232,9 +232,9 @@ nuke limit is used in all cases.
 vmod_std
 
 
-* Added ``std.getenv()``, see :ref:`func_getenv`.
+* Added ``std.getenv()``, see :ref:`vmod_std.getenv`.
 
-* Added ``std.late_100_continue()``, see :ref:`func_late_100_continue`.
+* Added ``std.late_100_continue()``, see :ref:`vmod_std.late_100_continue`.
 
 Other changes
 =
diff --git a/doc/sphinx/whats-new/upgrading-5.2.rst b/doc/sphinx/whats-new/upgrading-5.2.rst
index fa137c35f..8edd71b6b 100644
--- a/doc/sphinx/whats-new/upgrading-5.2.rst
+++ b/doc/sphinx/whats-new/upgrading-5.2.rst
@@ -121,7 +121,7 @@ situation.
 vmod_std
 
 
-Added :ref:`func_file_exists`.
+Added :ref:`vmod_std.file_exists`.
 
 New VMODs in the standard distribution
 ~~
diff --git a/doc/sphinx/whats-new/upgrading-6.0.rst b/doc/sphinx/whats-new/upgrading-6.0.rst
index 510993143..7a0227995 100644
--- a/doc/sphinx/whats-new/upgrading-6.0.rst
+++ b/doc/sphinx/whats-new/upgrading-6.0.rst
@@ -461,9 +461,9 @@ backend, or set a value for the Host header in VCL.
 VMOD std
 
 
-:ref:`std.port(IP) ` always returns 0 when applied to a
+:ref:`std.port(IP) ` always returns 0 when applied to a
 ``*.ip`` variable whose value is set to ``0.0.0.0`` because the
-listener is UDS.  :ref:`std.set_ip_tos(INT) ` is
+listener is UDS.  :ref:`std.set_ip_tos(INT) ` is
 silently ignored when the listener is UDS.
 
 The ``shard`` director
@@ -519,7 +519,7 @@ except for ``req.restarts`` and ``req.xid``, which change by design.
 
 If you need to reset the client request headers to their original
 state (before changes in VCL), call
-:ref:`std.rollback(req) `.
+:ref:`std.rollback(req) `.
 
 ``return(restart)`` can now be called from ``vcl_recv{}``.
 
diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index a082b3cc5..7a5499af4 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -138,7 +138,7 @@ Other changes to VCL
 VMODs
 =
 
-Added the :ref:`func_fnmatch` function to :ref:`vmod_std(3)`, which
+Added the :ref:`vmod_std.fnmatch` function to :ref:`vmod_std(3)`, which
 you can use for shell-style wildcard matching. Wildcard patterns may
 be a good fit for matching URLs, to match against a pattern like
 ``/foo/*/bar/*``. The patterns can be built at runtime, if you need to
diff --git a/include/vrt.h b/include/vrt.h
index 35d9dcaa1..5863e2997 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -240,6 +240,7 @@ struct vmod_data {
 	const char			*file_id;
 
 	const char			*name;
+	const char			*func_name;
 	const void			*func;
 	intfunc_len;
 	const char			*proto;
diff --git a/lib/libvcc/vcc_vmod.c b/lib/libvcc/vcc_vmod.c
index 00f0cb0d0..678821429 100644
--- a/lib/libvcc/vcc_vmod.c
+++ b/lib/libvcc/vcc_vmod.c
@@ -307,8 +307,8 @@ vcc_ParseImport(struct vcc *tl)
 	VSB_printf(ifp->ini, "\tif (VRT_Vmod_Init(ctx,\n");
 	VSB_pr

Re: vtest status

[Poul-Henning Kamp]

In message , Frederic 
Lecaille writes:

>> Indeed, it's much easier now :-)  It's the first time I can build it.
>> I had to use this since there are probably still options that the user
>> has to tune :
>> 
>>make CC='gcc -DHAVE_SYS_VFS_H'
>
>with
>  make CC='gcc -O2 -s -D_GNU_SOURCE -DHAVE_SYS_VFS_H'
>
>we do not have anymore these warnings:

Better ?

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: vtest status

[Poul-Henning Kamp]

In message , Frederic 
Lecaille writes:
>On 1/9/19 11:09 PM, Poul-Henning Kamp wrote:
>> 
>> In message <1ca72b15-d581-4556-713a-a95106aa2...@haproxy.com>, Frederic 
>> Lecaille writes:
>> 
>> Try the stuff I just committed.
>> 
>> If you say "make vtest" you get no varnishsupport
>> 
>> If you say "make varnishtest" and point the variable at your
>> varnish source tree, you get varnish support.
>
>Everything is OK.
>Thank you a lot Poul-Henning.

Cool.

I'll update the README with some context...

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: vtest status

[Poul-Henning Kamp]

In message <1ca72b15-d581-4556-713a-a95106aa2...@haproxy.com>, Frederic 
Lecaille writes:

Try the stuff I just committed.

If you say "make vtest" you get no varnishsupport

If you say "make varnishtest" and point the variable at your
varnish source tree, you get varnish support.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: vtest status

[Poul-Henning Kamp]

In message <1ca72b15-d581-4556-713a-a95106aa2...@haproxy.com>, Frederic 
Lecaille writes:

>> My suggestion for now, is to let vtest live as a "source code
>> library" on github and not build and distribute it as stand-alone
>> packages.
>
>Currently, vtest convert.sh script make copies of lib/libvarnish/*.c
>which are already compiled when compiling varnish sources providing
>lib/libvarnish.a library.

The convert.sh script is really only meant to show me what the
difference is between the copy of vtest in the varnish cache
project and in the vtest project.

>Could not this vtest library be compiled against lib/libvarnish.a
>library?

The problem is that if you install a varnish package you do not
have libvarnish.a, it is an internal static library.

>As far as we do not have to make copies of file from varnish sources
>to compile vtest this would help to its maintenance.

Precisely.

Watch out for a couple of commits...

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: vtest status

[Poul-Henning Kamp]

In message <58d74416-8e0f-4a35-bb7c-67f02a680...@haproxy.com>, Frederic 
Lecaille writes:

>I hope everything is going well on your side.

Well, getting better:  One of my close friends died from stomach-cancer
right before X-mas and that threw all plans askew.

>So, if you need some help to make vtest project progress, please
>do not hesitate to ask.

I think what we need most right now, is to decide what shape vtest
should take?

I want to stress that I want this to be a collective decision,
so don't take anything I write here as edict or diktat.

Personally I am not very keen on turning vtest a "real" project
with releases, package-building and all that, because it would cause
me a lot of work which I don't think brings enough advantages.

The problem for me is that vtc_varnish has a very incestuos
relationship with varnish internals, internals which we do not want
to turn into documented or even open APIs, and that means that
vtc_varnish and varnishd must match versions.

A stand-alone vtest-package would either need to be compiled against
a specific version of varnish, or have some way to dynamically load
vtc_varnish from from the varnish source/package it is being used
against.

But that just moves the problem to the other side of the line, where
we need to open the internals of vtest up as a documented and
versioned API...

Some day (H3?) all that work may make sense, but I don't feel we
are there yet, at least not as far as cost/benefit is concerned.

My suggestion for now, is to let vtest live as a "source code
library" on github and not build and distribute it as stand-alone
packages.

Instead it will be up to the projects which use it to import
and build in their own projects.

That way HAproxy and leave vtc_varnish out of their compilation so
varnish does not become a build-dependency (or you can conditionally
compile vtc_varnish in, if it is already there.)

And we can compile it in Varnish including vtc_varnish, and include
vtc_haproxy in similar fashion. (actually, compilation of 
vtc_haproxy does not need haproxy to be installed, does it ?)

We should still provide a Makefile in the vtest github project which
compiles as much as possible (ie: vtc_varnish if it can find varnish
installed), that will make work on the shared sources easier for
all of us.

If we decide to do things that way, maybe you should call your
compiled version something like "hatest", while we stick with
"varnishtest", so we can reserve the "vtest" name for the future
runtime-extensible all-singing-and-dancing thing ?


Poul-Henning

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: will be away Dec 11 - Jan 11

[Poul-Henning Kamp]

In message <04313569-1497-6243-7008-04b70024e...@schokola.de>, Nils Goroll writ
es:
>before I forget again to mention this during bugwash:
>
>I will be away for a month to experience Christmas during summer.

.AU ?

.NZ ?

.ZA ?

.AR ?

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: [6.1] 2f2387038 Fix gensequences for BusyBox awk

[Poul-Henning Kamp]

Why is this fix not in -trunk ?

Poul-Henning

In message <20181107091318.f3356ac...@lists.varnish-cache.org>, Pål Hermunn 
Johanse
n writes:
>
>commit 2f2387038a7b9aca4c31c0f839cda3b7ab3391c0
>Author: Guillaume Quintard 
>Date:   Thu Apr 26 21:59:41 2018 +0200
>
>Fix gensequences for BusyBox awk
>
>I never thought that I'd have to fix a string concatenation problem in
>a BusyBox awk program to generate VT100 code in a container, but here we
>are:
>
>echo | awk 'END {print "foo" "" ++a, "foo" ++a}'
>
>should output "foo0 foo1", and for all the ?awk I tested, it does,
>except for BusyBox awk who thought funny to output "0 foo1", breaking the
>teken_state.h file.
>
>diff --git a/bin/varnishtest/gensequences b/bin/varnishtest/gensequences
>index 4337186b8..18a68b43b 100644
>--- a/bin/varnishtest/gensequences
>+++ b/bin/varnishtest/gensequences
>@@ -70,7 +70,7 @@ while (getline > 0) {
>   l_prefix_parent[n] = prefix;
>   l_prefix_suffix[n] = sequence[i];
>   if (!l_prefix_name[n])
>-  l_prefix_name[n] = "teken_state_" ++npr;
>+  l_prefix_name[n] = "teken_state_" "" ++npr;
>   prefix = n;
>   }
> 
>_______
>varnish-commit mailing list
>varnish-com...@varnish-cache.org
>https://www.varnish-cache.org/lists/mailman/listinfo/varnish-commit
>

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Fwd: Identifying our deliverables (naming QUIC's HTTP binding) (fwd)

[Poul-Henning Kamp]

As expected:  QUIC == HTTP/3

Start reading up...

Poul-Henning

--- Forwarded Message

From: Mark Nottingham 
Date: Tue, 30 Oct 2018 11:18:36 +1100
To: HTTP Working Group 
Message-Id: <67d04361-ce05-4dcc-aa99-3da259bbf...@mnot.net>

Everyone,

FYI, this might come to the WG in Bangkok.

Cheers,


> Begin forwarded message:
> 
> From: Mark Nottingham 
> Subject: Identifying our deliverables
> Date: 28 October 2018 at 11:30:55 am AEDT
> To: IETF QUIC WG 
> Archived-At: 

> 
> You might recall that we've previously talked about renaming our 
deliverables to clarify their relationship to the input documents 
(sometimes referred to as "gQUIC vs iQUIC").
> 
> After a fair amount of discussion with various folks, it seems like 
there's confidence that calling our core deliverable just "QUIC" will 
work, since Google's variant will fade out of use if we succeed, and 
there aren't any good alternatives; our best candidate was "QUIC/2", but 
it was pointed out that this additional level of versioning (since we 
already have a wire version) will cause yet more confusion.
> 
> However, in those discussions, a related concern was identified; 
confusion between QUIC-the-transport-protocol, and 
QUIC-the-HTTP-binding. I and others have seen a number of folks not 
closely involved in this work conflating the two, even though they're 
now separate things.
> 
> To address this, I'd like to suggest that -- after coordination with 
the HTTP WG -- we rename our the HTTP document to "HTTP/3", and using 
the final ALPN token "h3". Doing so clearly identifies it as another 
binding of HTTP semantics to the wire protocol -- just as HTTP/2 did -- 
so people understand its separation from QUIC.
> 
> As part of that discussion, I'd suggest that we formalise that its 
maintenance (as well as that of QPACK) pass off to the HTTP WG once 
we've published it.
> 
> I've talked about this with a number of folks. The only concerns I've 
heard so far are:
> 
> a) That QUIC isn't yet proven. That's true, but the name won't be 
formalised or used on the wire until the RFC is published, so we have a 
good amount of time to back away. Even then, if it fails in the market, 
we can always skip to HTTP/4 one day, if we need to. 
> 
> b) That discussing naming is a distraction from our technical work. I 
agree with the concern overall, but we have a responsibility to 
communicate clearly with external developers and users, so I'd like to 
have a *limited* discussion about this. Please, don't bike shed.
> 
> *Chair hat* 
> 
> We plan on reserving a very small period of time to discuss this in 
Bangkok; barring serious objections (of which "what about this name that 
I like instead?" is not one), we'll bring it up with the HTTP WG in 
Bangkok as well.
> 
> Cheers,
> 

- --
Mark Nottingham   https://www.mnot.net/
--- End of Forwarded Message
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: style(9), dir-locals.el and emacs-foo

[Poul-Henning Kamp]

In message <16b46047-10bf-af78-6cd6-a7bc75088...@uplex.de>, Geoff Simmons 
writes:

>I'm writing in the hope that someone will be better versed in emacs-lisp
>than I am.

My best suggestion is to look here:

https://www.freebsd.org/doc/en/books/developers-handbook/emacs.html

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: next VDD

[Poul-Henning Kamp]

In message 
, Federico 
Schwindt writes:

>It might be a bit earlier but personally knowing the where and when for the
>next VDD will help immensely to sort the time off at work and whatnot,
>especially the date.
>
>Do we have any ideas or suggestions?

I've been wanting to visit London before brexit, and a VDD could be an
excuse to do so.

Anywhere nort-europe-ish is fine with me.

Poul-Henning

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: mini benchmark vtim formatting

[Poul-Henning Kamp]

In message <6e3db4b7-5817-4c60-8c20-f4dce1ebf...@schokola.de>, Nils Goroll 
writes:

>>  printf("%ju.%06ju",
>>  (uint64_t)floor(foo),
>>  (uint64_t)floor((foo*1e6)) % 100UL);
>
>nice trick. On Linux, this gets us down to the same order of magnitude as a
>%lu.%06lu with uint64_t

The important thing here is the FP rendering, not the FP math, which is
instantaneous for all practical purposes.

It's kind of surprising to me that "%.f" hasn't gotten specific
optimization, it is a lot simpler thing than the general FP formatting
problem, but if there is any code I'm not keen on wading into, it is
FP formatting code...

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: mini benchmark vtim formatting

[Poul-Henning Kamp]

In message <853f9d84-9737-8a0c-e718-e73031302...@schokola.de>, Nils Goroll writ
es:

>* when representing time as a double, formatting a timestamp is now much more
>  expensive (7x - 20x) than retrieving the time

So the interesting question for me now, is if we can shave if we
stick with double and simply by tweaking the formatting.

Something like:

printf("%ju.%06ju",
(uint64_t)floor(foo),
(uint64_t)floor((foo*1e6)) % 100UL);

(Obviously to be hidden with PF-like macro)

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Divorcing varnishtest from varnish-cache

[Poul-Henning Kamp]

I have made a first export/import of VTest now:

https://github.com/vtest/VTest

The Makefile is very rudimentary and mostly so that I could see that
the convert.sh script does something sensible.

Let the experiments start...

Poul-Henning

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Divorcing varnishtest from varnish-cache

[Poul-Henning Kamp]

In message <20180924140433.68447...@barghest.suse.de>, Marcus 
=?UTF-8?B?UsO8Y2tlcnQ=?= writes:
>On Sat, 22 Sep 2018 17:34:17 +0000
>"Poul-Henning Kamp"  wrote:
>> 4. No binary packages/releases will be built at this time
>> 
>>HAproxy and Varnish-Cache can/will import by whatever
>>means from the new common 'vtest' project, and build
>>vtest the way which works best for their project.
>>(V-C obviously have some backwards compat issues)
>
>From a distribution standpoint ... why not make it into a library from
>the beginning to avoid code duplication?

Because I want to see where this takes us as we gain some hands-on
experience with sharing code, rather than try to plot out a course
which may not be relevant to our needs.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Divorcing varnishtest from varnish-cache

[Poul-Henning Kamp]

I guess my age is showing, as I almost forgot the most important thing:

   0. We need a cute 'vtest' logo

:-)

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: Divorcing varnishtest from varnish-cache

[Poul-Henning Kamp]

In message <71064.1537637...@critter.freebsd.dk>, "Poul-Henning Kamp" writes:

>1. Try to gain control of the empty 'vtest' account on github
>   I've sent them a message.  (Do we know any insiders ?)
>   Alternative name suggestions if this fails are most welcome.

OK, color me amazed...  I now have the @vtest account on github.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Divorcing varnishtest from varnish-cache

[Poul-Henning Kamp]

We discussed this at our developer summit in Hamburg yesterday
and I had made some experiments on the train down there.

The suggested plan right now is:

1. Try to gain control of the empty 'vtest' account on github
   I've sent them a message.  (Do we know any insiders ?)
   Alternative name suggestions if this fails are most welcome.

2. Create a vtest repository which contains:

   ./README
   ./Makefile
   ./lib/
(slimmed down) files from lib/libvarnish
   ./src/
bin/varnishtest/*.[ch]
   ./tests/
a?.vtc tests, purged from varnishd usage

   My in-train test got it down to 29kLOC.

3. Give developer access to interested & trustworthy parties

   We should probably define an informal review/design process, so
   that nobody drags the carpet under somebody else by accident.

4. No binary packages/releases will be built at this time

   HAproxy and Varnish-Cache can/will import by whatever
   means from the new common 'vtest' project, and build
   vtest the way which works best for their project.
   (V-C obviously have some backwards compat issues)

5. See where that takes us.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: RFC: unify directors and backends vcl interface

[Poul-Henning Kamp]

In message , Nils Goroll 
writes:
>On 19/09/2018 22:20, Poul-Henning Kamp wrote:
>> One thing I *really* miss right now is some serious high-end
>> benchmarking, to inform decisions about how much CPU we can afford
>> to "waste" on stuff like this.
>
>Apologies for playing this back to you, but you seem to imply that we would 
>need
>some kind of run-time method resolution and this is in no way what I had been
>thinking of when I proposed to dedicate an "object accessor" method.

I was talking in context of the broader backend-issues, refcounting
and all that.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: RFC: unify directors and backends vcl interface

[Poul-Henning Kamp]

In message 
, Dridi 
Boukelmoune writes:

>I think the confusion is mostly that we make an amalgam of the VMOD
>object and the underlying director. [...]

Well, there is a lot of ways to get confused, and I have yet to find
any obvious ways to make all the confusion go away.

I already thought about the hidden "are you a backend?" method Dridi
proposes, but it opens a lot of corner cases and becomes quite
complicated.

Most of the ideas I have toyed with, which supressed a *lot* of
confusion is stuff which forces a major version release, and I'm
not quite sure I'm ready for that yet.

One thing I *really* miss right now is some serious high-end
benchmarking, to inform decisions about how much CPU we can afford
to "waste" on stuff like this.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Absent from bugwash today...

[Poul-Henning Kamp]

I got out of bed at 03:45 and will not be present at bugwash today.

Sorry.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Back in the saddle: Bugwash today at 13!

[Poul-Henning Kamp]

I'm back in the saddle and will attend bugwash today at 13 as usual.

We have a load of tickets with bugwash label for today, so it
would speed things up if people think about them before the
bugwash.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Bugwash today: AWOL

[Poul-Henning Kamp]

I will be away from my keyboard at bugwash today.

Regular attendance from next monday

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

AWOL at bugwash today

[Poul-Henning Kamp]

I will not be present for bugwash today.

Feel free to deal with any available laundry.

Poul-Henning

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: CLI command vmod.tell

[Poul-Henning Kamp]

In message <CABoVN9CmtyVAo_CS9QK+m1Z7FY295=wanluvca5kdjqsfpu...@mail.gmail.com>
, Dridi Boukelmoune writes:

>> vmod.tell [VCL_PATTERN:]VMODNAME arguments [...]

>Can we also pass a nonce to the VMOD? So that "global" actions can be
>skipped once done when a VMOD like xkey is targeted with a VCL pattern
>matching more than once.

You are entirely in control of what "arguments" are and how
they are interpreted.


-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

CLI command vmod.tell

[Poul-Henning Kamp]

In message <e8975e99-3370-641f-362f-5841d38b9...@uplex.de>, Geoff Simmons write
s:

>>  vmod.tell  somevcl::somevmod "bla bla bla"

I looked a bit at this.

There is a hurt of locking issues for VMOD writers trying to do
anything smart, but trivial stuff like setting global options for
the VMOD etc, shouldn't cause trouble.

VCC wise I think this will be a "$CLI" stanza, and the function
will get called with a VRT_CTX and a char **, and return an enum
VCLI_status_e.

I think I prefer this naming:

vmod.tell [VCL_PATTERN:]VMODNAME arguments [...]

The default VCL_PATTERN will be the active VCL.

In other words:

vmod.tell std bugger off

only hits the std vmod in the active vcl, whereas

vmod.tell *:std bugger off

will hit the std vmod in all vcls which has imported it.

When used without a pattern:


If the active VCL has not imported a vmod of that name
status is CLIS_CANT.

If that VMOD does not implement $CLI it will be CLIS_UNIMPL.

Otherwise the VMODs $CLI determines the status.

When used with a pattern:
-

If no VCLs are matched by pattern:  CLIS_CANT

If the VMOD is not loaded in any of the matched VCLs:  CLIS_CANT

If no hit VMOD implements $CLI: CLIS_UNIMPL.

If any hit VMOD's $CLI returns non-200:  new status CLIS_KABOOM (=202)

Otherwise CLIS_OK

Still waffling on:
--

Should names of VCLs be put into cli output as subheadings when
pattern is used so VMODs won't have to do that, or should they
only be added in front of VMODs which produce cli output.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: VDD summary

[Poul-Henning Kamp]

In message <cabovn9c1cddewjp+2bepwqnoy-yatqxseoymurynk_7t7jq...@mail.gmail.com>
, Dridi Boukelmoune writes:

>And you may have several competing vmods loaded with the same name
>thanks to vmod caching. 

Well, due to having multiple loaded VCLs really.

It would have to be possible to say

vmod.tell  somevcl::somevmod "bla bla bla"

To target a single one, but it is probably ill-advised to insist
on always only targeting a single one.

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: VDD summary

[Poul-Henning Kamp]

In message 

VDD summary

[Poul-Henning Kamp]

Thanks to everybody at the VDD in Oslo yesterday, I think we got a
lot done and we certainly ended up with one of the most actionable
summaries we have ever had.

See you in Hamburg next time:

27.9. optional VDD if we want
28.9. VDD
29.9. optional fun day including TMBG concert *) or techno party 
depending on taste

*) need to book yourself if you want to go


H2::Send_Frame move to ->error
#1799

Pluggable ops for TCP-Pools and TCP operations 
=> Proper way to do this is to generalize and come up with multi-protocol 
backend support
=> Unknown/large scope: unlikely for a Sept release

#2573 @^#@&^#@-Headers (vmod_header[s])
=> New vmod_header. @Steven writes/proposes .vcc file
=> New parameter: header-charset ?  return 400 if not acceptable.  Default=what 
VCL can handle.

SELinux: maintain a policy upstream
=> we ship a policy if we can test it (dridi to figure this out) (That really 
goes for *anything* in our tree)

CLI:JSON:
=> First step is implement -j in all commands

VFP/VDP in vmods
=> If filter-list is set, setting do_{gzip|gunzip|esi} gives Failure

libvgz compilation warnings

inflate.c:1232:25: error: this statement may fall through 
[-Werror=implicit-fallthrough=]

 state->mode = LENGTH;

 ^~~~

either: -Wno-error=implicit-fallthrough= if the code supports it, or just 
-Wno-error

(new zlib code is still affected by this)


Reported upstream:

https://github.com/madler/zlib/issues/316


=> separate "varnish" CFLAGS (warnings) from the rest

=> don't pass varnish CFLAGS to foreign code

=> don't treat sanitizer flags separately? 



Features for Sept 2018 release?

VCL variables? (see https://etherpad.wikimedia.org/p/vdd17q3 lines 103-114)

release proc/docs etc.

* docs:

changes.rst / vs. sphinx/whats-new/changes-X.X.rst

new definition: doc/changes.rst only as a hint to release managers


#1799
=> Pål wrote this: 
https://github.com/varnishcache/varnish-cache/compare/master...hermunn:master-cache-behavior-vmods
 and Martin has added some comments. Also, Nils had some comments in an email.
=> Implement func pointer zegerman_catflap() veto function from HSH_Lookup{} 
slink to write PR
=> Reimplement req.grace, vcl_hit{} becomes "return(deliver)". (Pål does a PR 
for this)

Director/Refcount/Probe
=> phk suggests BACKEND std.resolve(BACKEND) (slink says it doesn't work for 
shard directors, which phk thinks is shard's problem, not his.)
   -> explained and would be solved by cookie idea below
   -> a general mechanism for passing private data from the client to backend 
side may be interesting --> describe some use cases to phk to persuade him of 
the usefulness

   -> this may be where introducing variables becomes interesting, e.g. 
declare INT req.var.foo, which then becomes available as an INT as bereq.var.foo
XXX: why wouldn't it simply by var.foo both places ?

=> new CLI command: backend.tell  args
=> ... if glob matches multiple backends, error-returns are ignored
=> ... Should "?" args be magic ?
=> "lightweight" directors for req.backend_hint, storing "cookie" on req.ws 
which is copied to bereq.ws and reinstated. (?)
=> What happens if backend_hint is shard with shards under it ?  How do 
2nd-level shards get cookies ?
  -> slink volunteers to write PR if phk wants him to

VDP cleanup and API changes (#2673)
=> Sounds fine, consider slink's comments, reiterate, then PHK to review

VRT_CTX, priv->free(), calls outside VCL methods (VDI) (and object destructors?)
=> This one sounds sensible, but requires some homework wrt. breaking APIs and 
if priv's need broken more (ie: dynamic objects)

(again) native VCC type for regexen
=> VCL_REGEXP, compiler turns CONST-STRING into vre

Next VDD on Norway-Denmark boat ?
No: TMBG concert in Hamburg


-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev

Re: backend/director admin states

[Poul-Henning Kamp]

In message <CABoVN9Bu--1K9veWNx_YY0rMF+3vTg=5mckegrjx0fjrvtt...@mail.gmail.com>
, Dridi Boukelmoune writes:

>All we need is TheNewGuy(tm) to dump this on their desk :p

Indeed, they grow up so fast, don't they ? :-)

Anyway, I would like to get back to the actual underlying
state-machinery for directors...

-- 
Poul-Henning Kamp   | UNIX since Zilog Zeus 3.20
p...@freebsd.org | TCP/IP since RFC 956
FreeBSD committer   | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.
___
varnish-dev mailing list
varnish-dev@varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev