Re: How to "unit" test HAProxy configurations (and HTTP servers in general)

[Ciprian Dorin Craciun]

On Thu, Dec 19, 2019 at 7:08 AM Илья Шипицин  wrote:
>> Really I would have expected to find a lightweight and generic Python
>> or Ruby "framework" that allows one to write such tests in a more
>> "programmatic" manner...  :(
>
> python mechanize ?


Looking at the documentation, https://mechanize.readthedocs.io/, it
seems targeted towards HTML testing and automation, say something more
similar to Selenium.

But the direction is quite good, i.e. it includes programatic
definition, however I'm serching something targeting mainly the
HTTP-layer, and as added-value if it allows me to write "application"
tests the better.  :)


Aside from `mechanize`, there seems to be two categories of tools, at
opposite ends, without any overlapping:
* we have the web-application testers, like Postman, `mechanize` (from
what I've seen), a few other (especially NodeJS-based)
implementations, that focus exclusively on the application layer;
(i.e. HTML, JSON, forms, etc.)
* we have the low-level HTTP-layer load generating tools like `wrk` or
`ab`, that have almost no capability of checking the "application"
layer, and most of the time neither the HTTP layer;
* and we have `vtest` / `varnishtest` (and perhaps a few others that I
haven't found yet) that also focus on the HTTP-layer, but mainly only
target the proxy / cache aspect of it, and lack basic functionality
for any application testing;

I'm searching for something "in-between".  :)

Thanks,
Ciprian.

Re: How to "unit" test HAProxy configurations (and HTTP servers in general)

[Ciprian Dorin Craciun]

On Thu, Dec 19, 2019 at 9:10 AM Jarno Huuskonen  wrote:
> > So my question to the HAProxy community is if anyone knows / uses a
> > generic HTTP unit testing framework.
>
> Have you looked into varnishtest(vtest) ? There're examples in haproxy source
> reg-tests directory.


Interesting;  for reference I'll put the relevant links bellow:

  https://github.com/vtest/VTest -- the actual tool used by HAProxy
  https://github.com/haproxy/haproxy/tree/master/reg-tests
  https://varnish-cache.org/docs/trunk/reference/varnishtest.html
  https://varnish-cache.org/docs/trunk/reference/vtc.html#haproxy

I'll have a more closer look, however at a first glance the following
sticks out to me:
* (as a plus)  it seems to be quite competent and allows many
use-cases;  (even checking very low-level details such as TCP
window-size;)
* (as a minus)  the syntax is quite awkward;  therefore it will be
very hard for most to write checks;
* (as a minus)  it can't be extended into more "application-side"
testing;  (i.e. what Postman does best;)
* (as a minus)  I think it's targeted more at testing small,
individual, features, as for example I don't think it's able to
generate the test cases (including URLs, headers, etc.)
programatically, for example taking them from a file, etc.;

Thanks for the pointer, I'll take a closer look and perhaps I'm
mistaken about many of the above,
Ciprian.

Re: How to "unit" test HAProxy configurations (and HTTP servers in general)

[Jarno Huuskonen]

Hi,

On Wed, Dec 18, Ciprian Dorin Craciun wrote:
> Hello all!
> 
> [First of all this question isn't only HAProxy specific, but can be
> extended to any HTTP server (including Apache, Nginx, and any other
> web application out there);  however I think it is especially
> important for HAProxy given how many HTTP-routing / mangling
> capabilities it has.]
> 
> I have a quite complex HAProxy configuration and I want to make sure
> that while changing some ACL's and rules I don't break something else.
> 
> Therefore I set out to find a tool that allows me to "unit test" an
> HTTP server (thus including HAProxy).  And to my surprise I didn't
> find one...  Obviously there are many "framework" unit test platforms
> out-there, each tailored for the underlying framework, such as Django,
> Flask, Rails, Spring, Go-lang, etc.;  however nothing "generic" that
> can test a web server by plain HTTP requests.
> 
> 
> So my question to the HAProxy community is if anyone knows / uses a
> generic HTTP unit testing framework.
> 
> (I have already written a few custom Bash scripts that given an URL
> create a file on disk, and given it resides in a Git repository, I can
> easily `git diff ./tests/responses` to see if anything changed, but
> this is too "barbaric"...)  :)

Have you looked into varnishtest(vtest) ? There're examples in haproxy source
reg-tests directory.

-Jarno

-- 
Jarno Huuskonen

Re: How to "unit" test HAProxy configurations (and HTTP servers in general)

[Илья Шипицин]

чт, 19 дек. 2019 г. в 01:07, Ciprian Dorin Craciun <
ciprian.crac...@gmail.com>:

> On Wed, Dec 18, 2019 at 8:23 PM Илья Шипицин  wrote:
> >> redirects are easy to test. using any framework.
> >
> > for example, jmeter (or any other http query tool)
> >
> >> * apply authentication for `admin`;
> >> * force some caching headers for `web`, `static` and `media`;
> >
> > same here. assert in jmeter would serve caching headers check
> >
> >> * apply some "sanity" checks to requests / responses (i.e. except
> >> admin, the rest should only serve `GET` requests);
> >> * deny any request that doesn't match a set of domains
> >
> > same here. just fire random query using jmeter
>
>
> JMeter, like Postman, don't seem very adequate tools for this purpose;
>  for starters they are too centered around a UI (especially for
> "editing" the tests).  (Although they have CLI runners, and perhaps
> their "tests collections" can be serialized in a file and put in a Git
> repository, I don't think one can easily use it with e.g. a custom
> editor, etc., like any "normal" configuration.)
>
> For example:
> * https://jmeter.apache.org/demos/ -- an XML file for JMeter JMX files
> used as "test plans";
> *
> https://github.com/Blazemeter/taurus/blob/master/examples/functional/postman-sample-collection.json
> -- a JSON file (random example) for Postman;
>
>
> The closest I could find something that resembles "usability" is
> Tavern, which at least has the advantage of YAML as opposed to "raw"
> XML or JSON:
> * https://github.com/taverntesting/tavern
> *
> https://github.com/taverntesting/tavern/blob/master/example/simple/test_server.tavern.yaml
>
>
> Really I would have expected to find a lightweight and generic Python
> or Ruby "framework" that allows one to write such tests in a more
> "programmatic" manner...  :(
>

python mechanize ?


>
> Ciprian.
>

Re: How to "unit" test HAProxy configurations (and HTTP servers in general)

[Ciprian Dorin Craciun]

On Wed, Dec 18, 2019 at 8:23 PM Илья Шипицин  wrote:
>> redirects are easy to test. using any framework.
>
> for example, jmeter (or any other http query tool)
>
>> * apply authentication for `admin`;
>> * force some caching headers for `web`, `static` and `media`;
>
> same here. assert in jmeter would serve caching headers check
>
>> * apply some "sanity" checks to requests / responses (i.e. except
>> admin, the rest should only serve `GET` requests);
>> * deny any request that doesn't match a set of domains
>
> same here. just fire random query using jmeter


JMeter, like Postman, don't seem very adequate tools for this purpose;
 for starters they are too centered around a UI (especially for
"editing" the tests).  (Although they have CLI runners, and perhaps
their "tests collections" can be serialized in a file and put in a Git
repository, I don't think one can easily use it with e.g. a custom
editor, etc., like any "normal" configuration.)

For example:
* https://jmeter.apache.org/demos/ -- an XML file for JMeter JMX files
used as "test plans";
* 
https://github.com/Blazemeter/taurus/blob/master/examples/functional/postman-sample-collection.json
-- a JSON file (random example) for Postman;


The closest I could find something that resembles "usability" is
Tavern, which at least has the advantage of YAML as opposed to "raw"
XML or JSON:
* https://github.com/taverntesting/tavern
* 
https://github.com/taverntesting/tavern/blob/master/example/simple/test_server.tavern.yaml


Really I would have expected to find a lightweight and generic Python
or Ruby "framework" that allows one to write such tests in a more
"programmatic" manner...  :(

Ciprian.

Re: How to "unit" test HAProxy configurations (and HTTP servers in general)

[Илья Шипицин]

ср, 18 дек. 2019 г. в 22:34, Ciprian Dorin Craciun <
ciprian.crac...@gmail.com>:

> On Wed, Dec 18, 2019 at 6:47 PM Илья Шипицин  wrote:
> > you are talking about testing ACL. can you provide some example ?
>
>
> So let's assume I have a given HAProxy configuration, full of ACL's
> and rules, that apply certain "firewalling", authentication /
> authorization, "mangling" operations to the HTTP request (e.g. drop
> "bad-bots", update headers, redirects, routing to various backends,
> etc.).
>
> Now how can I test that the HAProxy configuration actually
> "implements" what it's proposes to?  I.e. how can I be sure that the
> rules are in the proper order, that no ACL's are missing, etc.
>
> My answer would be:  fire an HTTP request and see if it "does" what it
> should.  (Perhaps expose as HTTP headers some "state" values to help
> in checking things.)
>
>
>
>
> My concrete example would be this:  I find HAProxy wonderful for any
> non trivial HTTP deployment (and in fact anything "touched" by the
> Internet);  unfortunately the configuration language (with it's flat
> ACL's and request / response rules) is like "assembler" (as opposed to
> say Python).  Therefore I've written myself a HAProxy "configurator"
> in Python that based on simple Python code generates the full HAProxy
> configuration.
>
> For example:
>
>
> https://github.com/cipriancraciun/haproxy-configurator/blob/master/examples/example-01.py
>
> https://github.com/cipriancraciun/haproxy-configurator/blob/master/examples/_configs/example-01.cfg
>
> , the Python script is (hopefully) readable and clearly shows the
> intent of the resulting configuration:
> * redirect everything via HTTPS;
> * redirect `example.com` to `www.example.com`;
> * redirect `/admin/*` to `admin.example.com/admin/*`
> , same for `/blog/*`;
>

redirects are easy to test. using any framework.

for example, jmeter (or any other http query tool)


> * apply authentication for `admin`;
> * force some caching headers for `web`, `static` and `media`;
>

same here. assert in jmeter would serve caching headers check


> * apply some "sanity" checks to requests / responses (i.e. except
> admin, the rest should only serve `GET` requests);
> * deny any request that doesn't match a set of domains
>

same here. just fire random query using jmeter


>
> The resulting file is around 639 lines, and (given how I've chosen to
> identify ACL's) is quite hard to "follow by hand".
>
> So my question now is how do I test it...  Fire HTTP requests at it!  :)
>
>
> I hope this gives everyone a glimpse into my use-case,
> Ciprian.
>

Re: How to "unit" test HAProxy configurations (and HTTP servers in general)

[Ciprian Dorin Craciun]

On Wed, Dec 18, 2019 at 6:47 PM Илья Шипицин  wrote:
> you are talking about testing ACL. can you provide some example ?


So let's assume I have a given HAProxy configuration, full of ACL's
and rules, that apply certain "firewalling", authentication /
authorization, "mangling" operations to the HTTP request (e.g. drop
"bad-bots", update headers, redirects, routing to various backends,
etc.).

Now how can I test that the HAProxy configuration actually
"implements" what it's proposes to?  I.e. how can I be sure that the
rules are in the proper order, that no ACL's are missing, etc.

My answer would be:  fire an HTTP request and see if it "does" what it
should.  (Perhaps expose as HTTP headers some "state" values to help
in checking things.)




My concrete example would be this:  I find HAProxy wonderful for any
non trivial HTTP deployment (and in fact anything "touched" by the
Internet);  unfortunately the configuration language (with it's flat
ACL's and request / response rules) is like "assembler" (as opposed to
say Python).  Therefore I've written myself a HAProxy "configurator"
in Python that based on simple Python code generates the full HAProxy
configuration.

For example:

  
https://github.com/cipriancraciun/haproxy-configurator/blob/master/examples/example-01.py
  
https://github.com/cipriancraciun/haproxy-configurator/blob/master/examples/_configs/example-01.cfg

, the Python script is (hopefully) readable and clearly shows the
intent of the resulting configuration:
* redirect everything via HTTPS;
* redirect `example.com` to `www.example.com`;
* redirect `/admin/*` to `admin.example.com/admin/*`, same for `/blog/*`;
* apply authentication for `admin`;
* force some caching headers for `web`, `static` and `media`;
* apply some "sanity" checks to requests / responses (i.e. except
admin, the rest should only serve `GET` requests);
* deny any request that doesn't match a set of domains

The resulting file is around 639 lines, and (given how I've chosen to
identify ACL's) is quite hard to "follow by hand".

So my question now is how do I test it...  Fire HTTP requests at it!  :)


I hope this gives everyone a glimpse into my use-case,
Ciprian.

Re: How to "unit" test HAProxy configurations (and HTTP servers in general)

[Илья Шипицин]

ср, 18 дек. 2019 г. в 21:16, Ciprian Dorin Craciun <
ciprian.crac...@gmail.com>:

> Hello all!
>
> [First of all this question isn't only HAProxy specific, but can be
> extended to any HTTP server (including Apache, Nginx, and any other
> web application out there);  however I think it is especially
> important for HAProxy given how many HTTP-routing / mangling
> capabilities it has.]
>
> I have a quite complex HAProxy configuration and I want to make sure
> that while changing some ACL's and rules I don't break something else.
>
> Therefore I set out to find a tool that allows me to "unit test" an
> HTTP server (thus including HAProxy).  And to my surprise I didn't
> find one...  Obviously there are many "framework" unit test platforms
> out-there, each tailored for the underlying framework, such as Django,
> Flask, Rails, Spring, Go-lang, etc.;  however nothing "generic" that
> can test a web server by plain HTTP requests.
>
>
> So my question to the HAProxy community is if anyone knows / uses a
> generic HTTP unit testing framework.
>


I use openresty test framework (it works with nginx):
https://openresty.gitbooks.io/programming-openresty/content/testing/index.html
good question how to implement things using haproxy

you are talking about testing ACL. can you provide some example ?


>
> (I have already written a few custom Bash scripts that given an URL
> create a file on disk, and given it resides in a Git repository, I can
> easily `git diff ./tests/responses` to see if anything changed, but
> this is too "barbaric"...)  :)
>
> Thanks,
> Ciprian.
>
>

How to "unit" test HAProxy configurations (and HTTP servers in general)

[Ciprian Dorin Craciun]

Hello all!

[First of all this question isn't only HAProxy specific, but can be
extended to any HTTP server (including Apache, Nginx, and any other
web application out there);  however I think it is especially
important for HAProxy given how many HTTP-routing / mangling
capabilities it has.]

I have a quite complex HAProxy configuration and I want to make sure
that while changing some ACL's and rules I don't break something else.

Therefore I set out to find a tool that allows me to "unit test" an
HTTP server (thus including HAProxy).  And to my surprise I didn't
find one...  Obviously there are many "framework" unit test platforms
out-there, each tailored for the underlying framework, such as Django,
Flask, Rails, Spring, Go-lang, etc.;  however nothing "generic" that
can test a web server by plain HTTP requests.


So my question to the HAProxy community is if anyone knows / uses a
generic HTTP unit testing framework.

(I have already written a few custom Bash scripts that given an URL
create a file on disk, and given it resides in a Git repository, I can
easily `git diff ./tests/responses` to see if anything changed, but
this is too "barbaric"...)  :)

Thanks,
Ciprian.