Re: [PATCH] MAJOR: contrib: porting spoa_server to support python3
Hi, Thank you for the fast merge Willy and Thierry. I will keep this process in mind for any future contribution. Cheers, Gilchrist On Mon, May 11, 2020, 11:01 Willy Tarreau wrote: > On Mon, May 11, 2020 at 10:32:27AM +0200, Thierry Fournier wrote: > > Hi, thanks Willy, > > > > This a good idea to support python3. > > > > In facts, python2 is no longer supported, but it is not ready to > disappear > > because a lot of scripts in many enterprises are written in python2 > > (exemple: Centos7/RHEL7 the package manager yum is written in python2). > > > > In other way the spoe-server is newer and I guess it is not so used, so > > I should be a good idea to move to python3. > > > > So, its ok for me. > > Thanks for the quick response. So that's now merged :-) > > Gilchrist, thank you for this really clean work and the extensive > tests you've run on it. > > If we notice there's more contribution to this code, maybe we'll have to > push one step further like we did to support many openssl versions, which > consists in always sticking to the most recent API and emulate it for the > old ones. In your case that would for example mean that instead of having > this : > > - value = PyString_FromStringAndSize(args[i].name.str, > args[i].name.len); > + value = PY_STRING_FROM_STRING_AND_SIZE(args[i].name.str, > args[i].name.len); > > You would switch it to "PyUnicode_FromStringAndSize" (which is the new one > from python3) and add this in your compatibility layer to keep suppor for > 2.7: > > #define PyUnicode_FromStringAndSize PyString_FromStringAndSize > > That's something that can almost never be done in a single step, as it's > the best way to mix everything and cause breakage, but with your work > this seems to become within reach. But given that I don't know if we > should expect many contributions there, I don't know if it's worth > investing more time on it! > > Cheers, > Willy >
Re: [RFC PATCH 1/1] MEDIUM: ssl: add alternative way to load certificates with io_uring
On Mon, May 11, 2020 at 5:32 PM William Lallemand wrote: > By removing, it just means: > - In the case of a directory there is basicaly nothing to do but remove > the bundle code, everything will be loaded as usual, it will just be > done in separate SSL_CTX. > - In the configuration, if you specify a bundle with "example.pem" to > load the .rsa and .ecdsa, I'll probably add a shim to emulate the > previous behavior, but they will be loaded in separate SSL_CTX. understood, this will indeed simplify a lot of things. Thanks for the clarification! -- William
Re: [RFC PATCH 1/1] MEDIUM: ssl: add alternative way to load certificates with io_uring
On Mon, May 11, 2020 at 04:34:07PM +0200, William Dauchy wrote: > > I'm going to remove the support for multi-cert bundle once 2.2 is > > released, so it will simplify a lot of things. I encourage people > > writing new features to not support multi-cert bundles, more > > particularly on the CLI. > > So you mean that > example.pem.rsa > example.pem.ecdsa > will be loaded separately as all the other certificates? > I'm not 100% sure what you meant behind removing support of "multi-cert > bundle". What we call "multi-cert bundle" is the ability to load a certificate for each key type (RSA,DSA,ECDSA) in the same SSL_CTX. It was a hack implemented this way for OpenSSL 1.0.2 because there wasn't any callback available to chose a certificate type in this version. It wasn't implemented by all other SSL libraries. With OpenSSL 1.1.1, a client_hello callback was implemented, letting us chose the certificate type ourselves instead of letting OpenSSL do it. In my opinion this API is better. The multi-cert bundles are tricky and need specific conditions everywhere in the loading code. It complexify a lot of things. For example the loading code needs to verify if all SNIs exists in all certificate types, and generates the right SSL_CTX combinations, leading to multiple SSL_CTX created if the SNI weren't matching. By removing, it just means: - In the case of a directory there is basicaly nothing to do but remove the bundle code, everything will be loaded as usual, it will just be done in separate SSL_CTX. - In the configuration, if you specify a bundle with "example.pem" to load the .rsa and .ecdsa, I'll probably add a shim to emulate the previous behavior, but they will be loaded in separate SSL_CTX. Regarding the steps during the connection, during the SNI lookup, instead of having one sni_ctx node containing a SSL_CTX with all certificates, you will have one sni_ctx for each certificate type and the right one will be selected during lookup. Note that you can configure more precisely the SSL, for each certificate file, by loading separately the certs in a crt-list, instead of configuring all certificates on the bind line. -- William Lallemand
Re: [RFC PATCH 1/1] MEDIUM: ssl: add alternative way to load certificates with io_uring
Hello William L. Thank you for your answer. On Mon, May 11, 2020 at 1:01 PM William Lallemand wrote: > I'm wondering if we coud have the timing directly in the logs or on the > CLI, because I'm actually also doing this kind of things, and people > could probably optimize the loading if they could see the loading time. I agree it is something which was missing while doing my tests. Remove/re-add my timer to do my tests each time I was doing some modifications; to a point where I noted to ask you if you had some tooling I missed. So that's totally something I could do in a separate patch! What is your preferred timing function? > Your patch is really interesting in my opinion, a lot of people could > benefits of something like that. Regarding the reload part, I have some > ideas to pass the ckch structures to the new processes without reloading > the files one the filesystem, but I didn't implement anything yet. ok; interesting. > I'm currently reworking the ssl_sock.c and .h files, and splitting them > in several files, so you will probably need to rebase your patch on top > of this. At the moment I separated this in 3 parts: 1 part for the > crt-list/directories, 1 part for the ckch, and 1 part for the > SSL_CTX/bind. I'm not happy yet with what I have, but I'll do my best to > push that as soon as possible in the git repository, I don't want the > 2.2 to be released in the current state of the files. good to know, and that's mostly why I avoided too much changes on the file to avoid having too much rebase work while evolving my functionality. > I don't really know the io_uring API in details so I trust you on the > implementation. Yes, I've been using it for a few months now, so I'm starting to better understand how to properly use it. There are indeed a few different options where you could easily get trapped, and making the code more complicated. I however was planning to ask Jens for a quick review when I will have something more feature complete. > Regarding the name of the file, ssl_load.c, it is really generic, I > don't mind having a more specific file only for loading the SSL > certificates with io_uring. Yes I totally agree; as it was more of an experimentation I will do a second path on the naming and improve that. As I also started some poll changes, it also helps me to better name things to make elements more reusable. > I'm wondering if we couldn't push directly the certificate in an X509 > struture instead of storing it in a intermediate buffer, and use the > ckch instead of a specific cert_iobuf. That just some thoughts, I don't > know io_uring enough. I will look into that and it was also on my todo but for now I considered it ok for a first implementation. It mostly depends if we are able to properly allocate memory at the right time, while keeping the code clear enough. For now I considered it as a possible incremental improvement. Thanks for the input. > The crtlist files could also benefits from this, since they can contain > a lot of certificates too. Yes, indeed, for now I limited myself to the cases I was concerned about to avoid too much testing. I will improve that. I should have warned about it in my commit message. > I'm seeing that you are using in ssl_load_preparetree() the > SSL_SOCK_KEYTYPES_NAME array, which is only used for merging a > certificate bundle. I know that this can be confusing in the code > because the bundle implementation is crappy. But in a directory we > don't use any filename extension to load a PEM, we load everything as a > PEM file with the exception of .key, .ocsp, .sctl and .issuer, which are > loaded separatly with their own functions. (The cert_exts[] array is > used for that in ssl_sock.c) ok, good to recall all files should be taken into account; for now I blindly implemented my only known case. This will be part of the above point to take into account all possible options. > I'm going to remove the support for multi-cert bundle once 2.2 is > released, so it will simplify a lot of things. I encourage people > writing new features to not support multi-cert bundles, more > particularly on the CLI. So you mean that example.pem.rsa example.pem.ecdsa will be loaded separately as all the other certificates? I'm not 100% sure what you meant behind removing support of "multi-cert bundle". > Unfortunately your patch is too late for 2.2, but I think it could be > great for the next release! Yes, and it was clear for me since the beginning, I would not have the time to finish things early enough. Thanks a lot for your feedbacks. I will try to combine the one from Willy and yours for a future submission. I also hope we could find a good middle point between something perfect and something we could improve along the time. -- William
Re: [RFC PATCH 0/1] using io_uring to load certificates
Hello Willy, Thank you for your feedbacks. On Sat, May 9, 2020 at 6:39 AM Willy Tarreau wrote: > Your work could be extended to other files, like errorfiles, maps, ACLs, > etc. However with the config parser being very linear, it's extremely > unlikely that we could actually benefit from this in places other than > certs. What makes certs specific is that a single config line can induce > hundreds of thousands of file accesses and that these ones do benefit in > being performed in parallel. Yes, I take your comment as also something to improve in my commit message explanation. > Despite this I'm wondering if we should think about a generic approach > like an async file open with a callback on what to call once the file > is opened, or optionally an automatic async read and a callback set up > for received data. This would bring the benefit of allowing to perform > runtime FS accesses without destroying performance: right now, if you > try to access the file system from Lua or maybe from within one of the > device detection libs, or if you'd like to reload some files from the > CLI, the whole thread totally freezes until the I/O is completed, which > is why such accesses are absolutely forbidden. But done asynchronously > it could be different. We could imagine an async read API to be used > from Lua. We could even imagine being able to trigger reloading certain > files (ACL patterns or error files maybe) from the CLI, or even to serve > static files. This will of course not solve the security issues that > come with doing this, but at least it could solve the performance issue. wow I did not thought that far, but we could make things more generic indeed. I will try to keep that in mind and improve things along the way. Regarding the CLI, I'm however a bit worried about the `chroot` support though, which is important from my point of view. > Probably that the work should be split into a functional and an > implementation layer. The functional layer is the ability to perform > async file operations. The implementation is done using uring. But for > other OSes we could imagine using an emulation layer based on AIO or > even a bunch of threads, which could still possibly provide some > benefits during startup when loading tons of certificates, especially > when that's from spinning disks (not sure if those are still in use > though, last time I saw one was when I tried to repair my old Ultra5 > a few months ago). ok will try to keep that in mind. not sure I will be able to come up with a 100% nice interface but I will at least try to have something we can iterate on. > From day one I've been totally convinced it's possible to create a new > poller entirely based on io-uring. We could avoid a number of the ugly > tricks we use in epoll to avoid useless epoll_ctl() calls. What's nice > about our pollers is that they're entirely modular so it is perfectly > possible to create your own and experiment with it. yes totally. For now I've duplicated to a new poller, but depending on the result and diff, I might simply propose a similar #ifdef IO_URING in ev_poll.c to activate it or not. I will see the final result. Regarding ev_epoll, there is also an hybrid implementation to use `epoll_wait` but uring for all `epoll_ctl` calls. Thanks a lot for your inputs, -- William
[PATCH 3/3] CLEANUP: select: enhance readability in init
while reading the code, I thought it was clearer to put one instruction
per line as it is mostly done elsewhere
Signed-off-by: William Dauchy
---
src/ev_select.c | 12
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/src/ev_select.c b/src/ev_select.c
index 167bb7eb7..d15f3d828 100644
--- a/src/ev_select.c
+++ b/src/ev_select.c
@@ -225,9 +225,11 @@ static int init_select_per_thread()
int fd_set_bytes;
fd_set_bytes = sizeof(fd_set) * (global.maxsock + FD_SETSIZE - 1) /
FD_SETSIZE;
- if ((tmp_evts[DIR_RD] = (fd_set *)calloc(1, fd_set_bytes)) == NULL)
+ tmp_evts[DIR_RD] = (fd_set *)calloc(1, fd_set_bytes);
+ if (tmp_evts[DIR_RD] == NULL)
goto fail_rd;
- if ((tmp_evts[DIR_WR] = (fd_set *)calloc(1, fd_set_bytes)) == NULL)
+ tmp_evts[DIR_WR] = (fd_set *)calloc(1, fd_set_bytes);
+ if (tmp_evts[DIR_WR] == NULL)
goto fail_wr;
return 1;
fail_wr:
@@ -238,8 +240,10 @@ static int init_select_per_thread()
static void deinit_select_per_thread()
{
- free(tmp_evts[DIR_WR]); tmp_evts[DIR_WR] = NULL;
- free(tmp_evts[DIR_RD]); tmp_evts[DIR_RD] = NULL;
+ free(tmp_evts[DIR_WR]);
+ tmp_evts[DIR_WR] = NULL;
+ free(tmp_evts[DIR_RD]);
+ tmp_evts[DIR_RD] = NULL;
}
/*
--
2.26.2
[PATCH 1/3] BUG/MINOR: pollers: remove uneeded free in global init
Since commit d4604adeaa8c ("MAJOR: threads/fd: Make fd stuffs
thread-safe"), we init pollers per thread using a helper. It was still
correct for mono-thread mode until commit cd7879adc2c4 ("BUG/MEDIUM:
threads: Run the poll loop on the main thread too"). We now use a deinit
helper for all threads, making those free uneeded.
Only poll and select are affected by this very minor issue.
it could be backported from v1.8 to v2.1.
Fixes: cd7879adc2c4 ("BUG/MEDIUM: threads: Run the poll loop on the main
thread too")
Signed-off-by: William Dauchy
---
src/ev_poll.c | 1 -
src/ev_select.c | 5 +
2 files changed, 1 insertion(+), 5 deletions(-)
diff --git a/src/ev_poll.c b/src/ev_poll.c
index 538f07057..8df244566 100644
--- a/src/ev_poll.c
+++ b/src/ev_poll.c
@@ -286,7 +286,6 @@ static int _do_init(struct poller *p)
fail_swevt:
free(fd_evts[DIR_RD]);
fail_srevt:
- free(poll_events);
p->pref = 0;
return 0;
}
diff --git a/src/ev_select.c b/src/ev_select.c
index ab021b97f..acfdbb94a 100644
--- a/src/ev_select.c
+++ b/src/ev_select.c
@@ -255,7 +255,7 @@ static int _do_init(struct poller *p)
p->private = NULL;
if (global.maxsock > FD_SETSIZE)
- goto fail_revt;
+ goto fail_srevt;
fd_set_bytes = sizeof(fd_set) * (global.maxsock + FD_SETSIZE - 1) /
FD_SETSIZE;
@@ -272,9 +272,6 @@ static int _do_init(struct poller *p)
fail_swevt:
free(fd_evts[DIR_RD]);
fail_srevt:
- free(tmp_evts[DIR_WR]);
- free(tmp_evts[DIR_RD]);
- fail_revt:
p->pref = 0;
return 0;
}
--
2.26.2
[PATCH 2/3] BUG/MINOR: select: remove uneeded free in thread init
we wrongly free `tmp_evts[DIR_RD]` in error case.
it could be backported from v1.8 to v2.1.
Fixes: d4604adeaa8c ("MAJOR: threads/fd: Make fd stuffs thread-safe")
Signed-off-by: William Dauchy
---
src/ev_select.c | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/src/ev_select.c b/src/ev_select.c
index acfdbb94a..167bb7eb7 100644
--- a/src/ev_select.c
+++ b/src/ev_select.c
@@ -226,13 +226,13 @@ static int init_select_per_thread()
fd_set_bytes = sizeof(fd_set) * (global.maxsock + FD_SETSIZE - 1) /
FD_SETSIZE;
if ((tmp_evts[DIR_RD] = (fd_set *)calloc(1, fd_set_bytes)) == NULL)
- goto fail;
+ goto fail_rd;
if ((tmp_evts[DIR_WR] = (fd_set *)calloc(1, fd_set_bytes)) == NULL)
- goto fail;
+ goto fail_wr;
return 1;
- fail:
+ fail_wr:
free(tmp_evts[DIR_RD]);
- free(tmp_evts[DIR_WR]);
+ fail_rd:
return 0;
}
--
2.26.2
[PATCH 0/3] minor fixes for pollers
Hello, Here are a few minor fixes on pollers as I'm currently spending some time in this area. William Dauchy (3): BUG/MINOR: pollers: remove uneeded free in global init BUG/MINOR: select: remove uneeded free in thread init CLEANUP: select: enhance readability in init src/ev_poll.c | 1 - src/ev_select.c | 25 + 2 files changed, 13 insertions(+), 13 deletions(-) -- 2.26.2
Re: [RFC PATCH 1/1] MEDIUM: ssl: add alternative way to load certificates with io_uring
On Fri, May 08, 2020 at 04:34:27PM +0200, William Dauchy wrote: > experimental patch to make use of IO_URING to batch load certificates; > this drastically reduces the number of syscall and might benefit to > setup with large number of certificates. > it uses liburing in order to batch operation as follow: > for each certificate directory, we apply the same operations on each > file: > - statx > - openat > - read > - close > the results are stored in an ebtree. Then when we need to load them > with the SSL lib, instead of providing a path, we provide a buffer to > be consumed. The tree is freed after each directory. > > for now it requires a quite large limit of file descriptors, as all > operations types are done one after another; so the limit of fd should > be set higher than the number of certificates you have to load. This > part is probably going to evolve very soon as IO_URING plans are to be > able to chain operations with a given pre-defined file descriptor. > > on a setup with 25k certificates I was able to measure a minimum of 20% > gain on the init time when the filesystem cache is empty. My testing is > as follow; for each directory: > - put a timer before `ssl_sock_load_cert_list_file`, including > `ssl_load_certiodir` in the case of io_uring case. > - measure the time at the end of those operations, just after > `ssl_free_certiodir` in the case of io_uring. I'm wondering if we coud have the timing directly in the logs or on the CLI, because I'm actually also doing this kind of things, and people could probably optimize the loading if they could see the loading time. Your patch is really interesting in my opinion, a lot of people could benefits of something like that. Regarding the reload part, I have some ideas to pass the ckch structures to the new processes without reloading the files one the filesystem, but I didn't implement anything yet. I'm currently reworking the ssl_sock.c and .h files, and splitting them in several files, so you will probably need to rebase your patch on top of this. At the moment I separated this in 3 parts: 1 part for the crt-list/directories, 1 part for the ckch, and 1 part for the SSL_CTX/bind. I'm not happy yet with what I have, but I'll do my best to push that as soon as possible in the git repository, I don't want the 2.2 to be released in the current state of the files. I don't really know the io_uring API in details so I trust you on the implementation. Regarding the name of the file, ssl_load.c, it is really generic, I don't mind having a more specific file only for loading the SSL certificates with io_uring. I'm wondering if we couldn't push directly the certificate in an X509 struture instead of storing it in a intermediate buffer, and use the ckch instead of a specific cert_iobuf. That just some thoughts, I don't know io_uring enough. The crtlist files could also benefits from this, since they can contain a lot of certificates too. I'm seeing that you are using in ssl_load_preparetree() the SSL_SOCK_KEYTYPES_NAME array, which is only used for merging a certificate bundle. I know that this can be confusing in the code because the bundle implementation is crappy. But in a directory we don't use any filename extension to load a PEM, we load everything as a PEM file with the exception of .key, .ocsp, .sctl and .issuer, which are loaded separatly with their own functions. (The cert_exts[] array is used for that in ssl_sock.c) I'm going to remove the support for multi-cert bundle once 2.2 is released, so it will simplify a lot of things. I encourage people writing new features to not support multi-cert bundles, more particularly on the CLI. Unfortunately your patch is too late for 2.2, but I think it could be great for the next release! Regards, -- William Lallemand
Re: [PATCH] enable arm64 builds in travis-ci
сб, 9 мая 2020 г. в 11:45, Willy Tarreau : > On Sat, May 09, 2020 at 08:11:27AM +0200, Vincent Bernat wrote: > > ? 8 mai 2020 14:25 +02, Willy Tarreau: > > > > >> > Let's increase the timeout to see if it has a chance to finish, no ? > > >> > > > >> > > >> yes > > > > > > OK now pushed. It's really annoying to work blindly like this. The > > > build model Travis uses is broken by design. Requiring to commit > > > something for testing is utterly wrong. And doing so within the > > > project that's supposed to being test is further wrong. We already > > > have 44 patches only on .travis.yml! If this continues like this, > > > I predict that a "pre-CI" solution will appear to test if your > > > change is likely to trigger a travis error before it gets merged... > > > > You can push changes to a (throwable) branch instead. > > Good point, that can also be a solution. But it remains completely > hackish. It's basically abusing a versioning system to use it as a > messaging system to indicate "please build with this". > > Willy > I created several topics (no answer yet). as for travis-ci rights, it's totally undocumented. but I suspect travis grants rights based on github rights. i.e. github admin becomes travis admin as well. https://travis-ci.community/t/arm64-fails-with-non-clear-reason/8529 https://travis-ci.community/t/undocumented-require-admin-permissions/8530 https://travis-ci.community/t/undocumented-operation-requires-create-request-access-to-repository/8528
Re: [PATCH] MAJOR: contrib: porting spoa_server to support python3
On Mon, May 11, 2020 at 10:32:27AM +0200, Thierry Fournier wrote: > Hi, thanks Willy, > > This a good idea to support python3. > > In facts, python2 is no longer supported, but it is not ready to disappear > because a lot of scripts in many enterprises are written in python2 > (exemple: Centos7/RHEL7 the package manager yum is written in python2). > > In other way the spoe-server is newer and I guess it is not so used, so > I should be a good idea to move to python3. > > So, its ok for me. Thanks for the quick response. So that's now merged :-) Gilchrist, thank you for this really clean work and the extensive tests you've run on it. If we notice there's more contribution to this code, maybe we'll have to push one step further like we did to support many openssl versions, which consists in always sticking to the most recent API and emulate it for the old ones. In your case that would for example mean that instead of having this : - value = PyString_FromStringAndSize(args[i].name.str, args[i].name.len); + value = PY_STRING_FROM_STRING_AND_SIZE(args[i].name.str, args[i].name.len); You would switch it to "PyUnicode_FromStringAndSize" (which is the new one from python3) and add this in your compatibility layer to keep suppor for 2.7: #define PyUnicode_FromStringAndSize PyString_FromStringAndSize That's something that can almost never be done in a single step, as it's the best way to mix everything and cause breakage, but with your work this seems to become within reach. But given that I don't know if we should expect many contributions there, I don't know if it's worth investing more time on it! Cheers, Willy
Re: [*EXT*] Re: 404 + VN when enabling h2 in front of keycloak
On Sun, Apr 26, 2020 at 02:20:50PM +0200, Ionel GARDAIS wrote: > I give a try to other browsers. > Chrome and Brave both fails, even in private browsing. > > Firefox however succeeded in private browsing but failed in classic browsing, > even after clearing all caches. > > I gave a try to FF75.0 in Windows : it fails both in classic and private > browsing. Ionel, you should really take a capture on the server side. I'm seeing two possibilities. One is that the SSO doesn't support load balancing well, and that in H2 you get multiple requests in parallel going to different servers while in H1 they are very sequential. Another possibility is regarding to the header fields syntax as suggested by Jarno, but in 2.1 they're always lower case for both H1 and H2, so that cannot be the only difference. In both cases I encourage you to take a capture between haproxy and the servers to compare between a working client and a failing one. I'm pretty sure you'll spot a difference (missing cookie on a request, two requests going to different servers, etc). You can also post them here but be careful not to leave confidential information in them :-) Cheers, Willy
Re: [PATCH] MINOR: http_fetch: capture.req.ver not compatible with H2
Hi Baptiste, On Fri, Apr 24, 2020 at 09:23:28AM +0200, Baptiste wrote: > The function smp_fetch_capture_req_ver called when using the fetch > capture.req.ver don't return the right protocol version when H2 is in > use. It returns only "HTTP/1.1". > > This patch fixes this behavior and now the expected string is returned, > whatever protocol is used. Unfortunately no, it will even break the sample fetch. Look what the doc says: capture.req.ver : string This extracts the request's HTTP version and returns either "HTTP/1.0" or "HTTP/1.1". Unlike "req.ver", it can be used in both request, response, and logs because it relies on a persistent flag. Now you turn it into "req.ver" again, which will break processing on the response path. Also it leaves res.ver inconsistent, and is also inconsistent with the "either/or" promise in the doc. And for HTTP/0.9 it would fail to extract anything instead of returning HTTP/1.0 corresponding to the upgraded version. Maybe what could be done instead is to detect the front mux and return "HTTP/2" if connected to the h2 mux otherwise fall back to the other ones. What would really be cleaner would be to use two bits in the HTTP message to store the version instead, and already reserve values 2 & 3 for HTTP 2 and 3 (and update the doc and the response path as well). I've checked and HTTP_MSGF_VER_11 is very seldom used, most of the times a simple bit check on both bits at once to verify that 1 or above is supported will be enough. Willy
Re: [PATCH] MAJOR: contrib: porting spoa_server to support python3
Hi, thanks Willy,
This a good idea to support python3.
In facts, python2 is no longer supported, but it is not ready to disappear
because a lot of scripts in many enterprises are written in python2
(exemple: Centos7/RHEL7 the package manager yum is written in python2).
In other way the spoe-server is newer and I guess it is not so used, so
I should be a good idea to move to python3.
So, its ok for me.
Thierry
> On 11 May 2020, at 10:19, Willy Tarreau wrote:
>
> CCing Thierry as I suspect he didn't notice it :-)
>
> Thierry, I'm just seeking a simple instruction such as "let's merge it",
> "please give me more time" or "don't do it, it will break X or Y". I'd
> rather not miss it before 2.2 without a good reason.
>
> Thanks,
> Willy
>
> On Wed, May 06, 2020 at 12:25:31PM +, Gilchrist Dadaglo wrote:
>>
>>Background:
>>Python 2 is no longer supported since January, 1st 2020 as per
>> https://antiphishing.ozon.io/4/0@MA85Se5hVy0@Ftkk8IJL5AfiE-5CIh6PYjO-yQQskH-5ghbuD3ZQFFEUV7nusU8YNJqjq5OWm_raT4Eb1h8La3tdyS-n5cB6l_rrp3Ffo4Ol_v5uzolbbzLQtISQhFt93yhkg6noOlI1@87af154d3ed6896fbeb877900d8310458c756a70
>>The purpose of this change is to make the spoa_server contrib library
>>compatible with Python 3 to allow transition to Python 3.
>>
>>Test Settings:
>>ps_python.py:
>>...
>>spoa.set_var_null("null", spoa.scope_txn)
>>spoa.set_var_boolean("boolean", spoa.scope_txn, True)
>>spoa.set_var_int32("int32", spoa.scope_txn, 1234)
>>spoa.set_var_uint32("uint32", spoa.scope_txn, 1234)
>>spoa.set_var_int64("int64", spoa.scope_txn, 1234)
>>spoa.set_var_uint64("uint64", spoa.scope_txn, 1234)
>>spoa.set_var_ipv4("ipv4", spoa.scope_txn,
>> ipaddress.IPv4Address(u"127.0.0.1"))
>>spoa.set_var_ipv6("ipv6", spoa.scope_txn,
>> ipaddress.IPv6Address(u"1::f"))
>>spoa.set_var_str("str", spoa.scope_txn, "1::f")
>>spoa.set_var_bin("bin", spoa.scope_txn, "1:\x01:\x42f\x63\x63")
>>spoa.set_var_str("python_version", spoa.scope_sess,
>> str(sys.version_info))
>>...
>>haproxy.cfg:
>>...
>>http-request capture var(txn.verb.null),debug len 1
>>http-request capture var(txn.verb.boolean),debug len 1
>>http-request capture var(txn.verb.int32),debug len 4
>>http-request capture var(txn.verb.uint32),debug len 4
>>http-request capture var(txn.verb.int64),debug len 4
>>http-request capture var(txn.verb.uint64),debug len 4
>>http-request capture var(txn.verb.ipv4),debug len 16
>>http-request capture var(txn.verb.ipv6),debug len 45
>>http-request capture var(txn.verb.str),debug len 32
>>http-request capture var(txn.verb.bin),debug len 32
>>http-request capture var(sess.verb.python_version),debug len 100
>>...
>>
>>Test result:
>>Python 3.8:
>>ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR--
>> 1/1/0/0/0 0/0
>> {|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=3,
>> minor=8, micro=1, releaselevel='final', serial=0)} "POST / HTTP/1.1"
>>Python 3.7:
>>ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR--
>> 1/1/0/0/0 0/0
>> {|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=3,
>> minor=7, micro=6, releaselevel='final', serial=0)} "POST / HTTP/1.1"
>>Python 3.6:
>>ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR--
>> 1/1/0/0/0 0/0
>> {|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=3,
>> minor=6, micro, releaselevel='final', serial=0)} "POST / HTTP/1.1"
>>Python 2.7:
>>ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR--
>> 1/1/0/0/0 0/0
>> {|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=2,
>> minor=7, micro, releaselevel='final', serial=0)} "POST / HTTP/1.1"
>>
>>Not tested:
>>Python <2.7
>> ---
>> haproxy/contrib/spoa_server/Makefile| 37 +
>> haproxy/contrib/spoa_server/README | 10 +-
>> haproxy/contrib/spoa_server/ps_python.c | 179 +++-
>> haproxy/contrib/spoa_server/ps_python.h | 52 +++
>> 4 files changed, 241 insertions(+), 37 deletions(-)
>> create mode 100644 haproxy/contrib/spoa_server/ps_python.h
>>
>
>> diff --git a/haproxy/contrib/spoa_server/Makefile
>> b/haproxy/contrib/spoa_server/Makefile
>> index f075282..e7b20db 100644
>> --- a/haproxy/contrib/spoa_server/Makefile
>> +++ b/haproxy/contrib/spoa_server/Makefile
>> @@ -23,10 +23,47 @@ endif
>>
>> ifneq ($(USE_PYTHON),)
>> OBJS + ps_python.o
>> +
>> +# "--embed" flag is supported (and required) only from python 3.8+
>> +check_python_config : $(shell if python3-config --embed; then echo
>> "python3.8+"; \
>> +elif hash python3-config; then echo "python3"; \
>> +elif hash python-config; then echo "python2"; fi)
>> +
>> +ifeq ($(check_python_config), python3.8+)
>>
Re: 2.0.14 PCRE2 JIT compilation failed
Hi Veiko, On Fri, Apr 24, 2020 at 01:08:40PM +, Veiko Kukk wrote: > It has happened to many of us that after asking for help, a good idea to > test/debug comes. Oh yes, it's quite common that putting down a problem helps seeing it better. This is also why when you can explain a problem to someone in person, you often don't need to finish your sentence and immediately find the solution :-) > It turned out to be selinx issue. > > #= haproxy_t == > > # This avc can be allowed using the boolean 'cluster_use_execmem' > allow haproxy_t self:process execmem; That's very interesting, thanks for sharing! > I wonder if somewhere in HAproxy documentation about pcre jit, it is > mentioned that in case of selinux, selinux rules must be changed for the jit > to work. If not, would be nice to add it. I'm sure it's not mentioned and that probably that nobody thought about this before. Could you please propose a patch against INSTALL to mention this ? There's also section 11 of management.txt which could get a few lines about it I guess, reusing your discoveries. Thanks, Willy
Re: [PATCH] MAJOR: contrib: porting spoa_server to support python3
CCing Thierry as I suspect he didn't notice it :-)
Thierry, I'm just seeking a simple instruction such as "let's merge it",
"please give me more time" or "don't do it, it will break X or Y". I'd
rather not miss it before 2.2 without a good reason.
Thanks,
Willy
On Wed, May 06, 2020 at 12:25:31PM +, Gilchrist Dadaglo wrote:
>
> Background:
> Python 2 is no longer supported since January, 1st 2020 as per
> https://www.python.org/doc/sunset-python-2/
> The purpose of this change is to make the spoa_server contrib library
> compatible with Python 3 to allow transition to Python 3.
>
> Test Settings:
> ps_python.py:
> ...
> spoa.set_var_null("null", spoa.scope_txn)
> spoa.set_var_boolean("boolean", spoa.scope_txn, True)
> spoa.set_var_int32("int32", spoa.scope_txn, 1234)
> spoa.set_var_uint32("uint32", spoa.scope_txn, 1234)
> spoa.set_var_int64("int64", spoa.scope_txn, 1234)
> spoa.set_var_uint64("uint64", spoa.scope_txn, 1234)
> spoa.set_var_ipv4("ipv4", spoa.scope_txn,
> ipaddress.IPv4Address(u"127.0.0.1"))
> spoa.set_var_ipv6("ipv6", spoa.scope_txn,
> ipaddress.IPv6Address(u"1::f"))
> spoa.set_var_str("str", spoa.scope_txn, "1::f")
> spoa.set_var_bin("bin", spoa.scope_txn, "1:\x01:\x42f\x63\x63")
> spoa.set_var_str("python_version", spoa.scope_sess,
> str(sys.version_info))
> ...
> haproxy.cfg:
> ...
> http-request capture var(txn.verb.null),debug len 1
> http-request capture var(txn.verb.boolean),debug len 1
> http-request capture var(txn.verb.int32),debug len 4
> http-request capture var(txn.verb.uint32),debug len 4
> http-request capture var(txn.verb.int64),debug len 4
> http-request capture var(txn.verb.uint64),debug len 4
> http-request capture var(txn.verb.ipv4),debug len 16
> http-request capture var(txn.verb.ipv6),debug len 45
> http-request capture var(txn.verb.str),debug len 32
> http-request capture var(txn.verb.bin),debug len 32
> http-request capture var(sess.verb.python_version),debug len 100
> ...
>
> Test result:
> Python 3.8:
> ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR--
> 1/1/0/0/0 0/0
> {|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=3,
> minor=8, micro=1, releaselevel='final', serial=0)} "POST / HTTP/1.1"
> Python 3.7:
> ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR--
> 1/1/0/0/0 0/0
> {|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=3,
> minor=7, micro=6, releaselevel='final', serial=0)} "POST / HTTP/1.1"
> Python 3.6:
> ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR--
> 1/1/0/0/0 0/0
> {|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=3,
> minor=6, micro=10, releaselevel='final', serial=0)} "POST / HTTP/1.1"
> Python 2.7:
> ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR--
> 1/1/0/0/0 0/0
> {|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=2,
> minor=7, micro=17, releaselevel='final', serial=0)} "POST / HTTP/1.1"
>
> Not tested:
> Python <2.7
> ---
> haproxy/contrib/spoa_server/Makefile| 37 +
> haproxy/contrib/spoa_server/README | 10 +-
> haproxy/contrib/spoa_server/ps_python.c | 179 +++-
> haproxy/contrib/spoa_server/ps_python.h | 52 +++
> 4 files changed, 241 insertions(+), 37 deletions(-)
> create mode 100644 haproxy/contrib/spoa_server/ps_python.h
>
> diff --git a/haproxy/contrib/spoa_server/Makefile
> b/haproxy/contrib/spoa_server/Makefile
> index f075282..e7b20db 100644
> --- a/haproxy/contrib/spoa_server/Makefile
> +++ b/haproxy/contrib/spoa_server/Makefile
> @@ -23,10 +23,47 @@ endif
>
> ifneq ($(USE_PYTHON),)
> OBJS += ps_python.o
> +
> +# "--embed" flag is supported (and required) only from python 3.8+
> +check_python_config := $(shell if python3-config --embed; then echo
> "python3.8+"; \
> +elif hash python3-config; then echo "python3"; \
> +elif hash python-config; then echo "python2"; fi)
> +
> +ifeq ($(check_python_config), python3.8+)
> +PYTHON_DEFAULT_INC := $(shell python3-config --includes)
> +PYTHON_DEFAULT_LIB := $(shell python3-config --libs --embed)
> +else ifeq ($(check_python_config), python3)
> +PYTHON_DEFAULT_INC := $(shell python3-config --includes)
> +PYTHON_DEFAULT_LIB := $(shell python3-config --libs)
> +else ifeq ($(check_python_config), python2)
> +PYTHON_DEFAULT_INC := $(shell python-config --includes)
> +PYTHON_DEFAULT_LIB := $(shell python-config --libs)
> +endif
> +
> +
> +# Add default path
> +ifneq ($(PYTHON_DEFAULT_INC),)
> +CFLAGS += $(PYTHON_DEFAULT_INC)
> +else
> CFLAGS += -I/usr/include/python2.7
> +endif
> +ifneq ($(PYTHON_DEFAULT_LIB),)
> +LDLIBS += $(PYTHON_DEFAULT_LIB)
> +else
> LDLIBS +=
Re: [PATCH] spellcheck patches
On Wed, May 06, 2020 at 12:55:14AM +0500, ??? wrote: > Hello, > > another round of typo fixes. All three applied, thanks Ilya. Willy
Re: [PATCH] travis-ci: upgrade LibreSSL versions (LibreSSL-3.1.1 released)
Hi Ilya, On Sat, May 09, 2020 at 09:44:07PM +0500, ??? wrote: > Hello, > > patch attached. Applied, thanks! Willy
AW: Nieuws voor de Domeinnaam mydns.nl
Geachte heer of mevrouw, Via deze mail willen wij u op de hoogte stellen dat de domein mydns.nl aangeboden wordt. Heeft U interesse? Alvast een prettig begin van de week gewenst! Lotta Vink
