Re: 1.8.7 http-tunnel doesn't seem to work? (but default http-keep-alive does)

[Willy Tarreau]

On Tue, Apr 17, 2018 at 11:47:18PM +0200, PiBa-NL wrote:
> Op 17-4-2018 om 17:46 schreef Willy Tarreau:
> > On Tue, Apr 17, 2018 at 04:33:07PM +0200, Olivier Houchard wrote:
> > > After talking with Willy, here is an updated patch that does that.
> > > That way, the day we'll want to use EV_ONESHOT, we'll be ready, and won't
> > > miss any event.
> > Now merged, thanks guys!
> > Willy
> 
> Thanks!
> Works as expected with current master.
> 
> Do i dare ask a estimate when 1.8.8 might be released? (oh just did. ;) )

Probably tomorrow evening, with another fix we're working on.

> O well, i guess ill run production with nokqueue for a little while, that
> works alright for me sofar.

Or keep running on 1.8.7 with this patch ;-)

Willy

Re: 1.8.7 http-tunnel doesn't seem to work? (but default http-keep-alive does)

[PiBa-NL]

Op 17-4-2018 om 17:46 schreef Willy Tarreau:

On Tue, Apr 17, 2018 at 04:33:07PM +0200, Olivier Houchard wrote:

After talking with Willy, here is an updated patch that does that.
That way, the day we'll want to use EV_ONESHOT, we'll be ready, and won't
miss any event.

Now merged, thanks guys!
Willy


Thanks!
Works as expected with current master.

Do i dare ask a estimate when 1.8.8 might be released? (oh just did. ;) )
O well, i guess ill run production with nokqueue for a little while, 
that works alright for me sofar.


Regards,
PiBa-NL (Pieter)

Re: 1.8.7 http-tunnel doesn't seem to work? (but default http-keep-alive does)

[Willy Tarreau]

On Tue, Apr 17, 2018 at 04:33:07PM +0200, Olivier Houchard wrote:
> After talking with Willy, here is an updated patch that does that.
> That way, the day we'll want to use EV_ONESHOT, we'll be ready, and won't
> miss any event.

Now merged, thanks guys!
Willy

Re: 1.8.7 http-tunnel doesn't seem to work? (but default http-keep-alive does)

[Olivier Houchard]

Hi again,

On Tue, Apr 17, 2018 at 01:07:49PM +0200, Olivier Houchard wrote:
[...]
> We only need one to prevent kevent() from trying to scanning the kqueue, so
> only setting kev[0] should be enough. It's inside an #ifdef because
> EV_RECEIPT was only implemented recently in OpenBSD, so all users may not have
> it.
> My hope is, if EV_RECEIPT is not set, we will indeed read even needlessly,
> but we will read them again at the next kevent() call. If it does not, and
> events are lost, then we'll have to add an extra entry that generates an 
> error.

After talking with Willy, here is an updated patch that does that.
That way, the day we'll want to use EV_ONESHOT, we'll be ready, and won't
miss any event.

Regards,

Olivier

>From 56701f85b3311657ea6915df94ec75ac7a83eb54 Mon Sep 17 00:00:00 2001
From: Olivier Houchard 
Date: Mon, 16 Apr 2018 13:24:48 +0200
Subject: [PATCH] BUG/MEDIUM: kqueue: When adding new events, provide an output
 to get errors.

When adding new events using kevent(), if there's an error, because we're
trying to delete an event that wasn't there, or because the fd has already
been closed, kevent() will either add an event in the eventlist array if
there's enough room for it, and keep on handling other events, or stop and
return -1.
We want it to process all the events, so give it a large-enough array to
store any error.

This should be backported to 1.8.
---
 src/ev_kqueue.c | 43 +--
 1 file changed, 37 insertions(+), 6 deletions(-)

diff --git a/src/ev_kqueue.c b/src/ev_kqueue.c
index a103ece9d..ebfd5d210 100644
--- a/src/ev_kqueue.c
+++ b/src/ev_kqueue.c
@@ -31,6 +31,7 @@
 /* private data */
 static int kqueue_fd[MAX_THREADS]; // per-thread kqueue_fd
 static THREAD_LOCAL struct kevent *kev = NULL;
+static struct kevent *kev_out = NULL; // Trash buffer for kevent() to write 
the eventlist in
 
 /*
  * kqueue() poller
@@ -43,6 +44,8 @@ REGPRM2 static void _do_poll(struct poller *p, int exp)
int updt_idx, en;
int changes = 0;
 
+   timeout.tv_sec  = 0;
+   timeout.tv_nsec = 0;
/* first, scan the update list to find changes */
for (updt_idx = 0; updt_idx < fd_nbupdt; updt_idx++) {
fd = fd_updt[updt_idx];
@@ -81,13 +84,21 @@ REGPRM2 static void _do_poll(struct poller *p, int exp)
HA_ATOMIC_OR([fd].polled_mask, tid_bit);
}
}
-   if (changes)
-   kevent(kqueue_fd[tid], kev, changes, NULL, 0, NULL);
+   if (changes) {
+#ifdef EV_RECEIPT
+   kev[0].flags |= EV_RECEIPT;
+#else
+   /* If EV_RECEIPT isn't defined, just add an invalid entry,
+* so that we get an error and kevent() stops before scanning
+* the kqueue.
+*/
+   EV_SET([changes++], -1, EVFILT_WRITE, EV_DELETE, 0, 0, 
NULL);
+#endif
+   kevent(kqueue_fd[tid], kev, changes, kev_out, changes, 
);
+   }
fd_nbupdt = 0;
 
delta_ms= 0;
-   timeout.tv_sec  = 0;
-   timeout.tv_nsec = 0;
 
if (!exp) {
delta_ms= MAX_DELAY_MS;
@@ -150,8 +161,12 @@ static int init_kqueue_per_thread()
 {
int fd;
 
-   /* we can have up to two events per fd (*/
-   kev = calloc(1, sizeof(struct kevent) * 2 * global.maxsock);
+   /* we can have up to two events per fd, so allocate enough to store
+* 2*fd event, and an extra one, in case EV_RECEIPT isn't defined,
+* so that we can add an invalid entry and get an error, to avoid
+* scanning the kqueue uselessly.
+*/
+   kev = calloc(1, sizeof(struct kevent) * (2 * global.maxsock + 1));
if (kev == NULL)
goto fail_alloc;
 
@@ -194,6 +209,15 @@ REGPRM1 static int _do_init(struct poller *p)
 {
p->private = NULL;
 
+   /* we can have up to two events per fd, so allocate enough to store
+* 2*fd event, and an extra one, in case EV_RECEIPT isn't defined,
+* so that we can add an invalid entry and get an error, to avoid
+* scanning the kqueue uselessly.
+*/
+   kev_out = calloc(1, sizeof(struct kevent) * (2 * global.maxsock + 1));
+   if (!kev_out)
+   goto fail_alloc;
+
kqueue_fd[tid] = kqueue();
if (kqueue_fd[tid] < 0)
goto fail_fd;
@@ -203,6 +227,9 @@ REGPRM1 static int _do_init(struct poller *p)
return 1;
 
  fail_fd:
+   free(kev_out);
+   kev_out = NULL;
+fail_alloc:
p->pref = 0;
return 0;
 }
@@ -220,6 +247,10 @@ REGPRM1 static void _do_term(struct poller *p)
 
p->private = NULL;
p->pref = 0;
+   if (kev_out) {
+   free(kev_out);
+   kev_out = NULL;
+   }
 }
 
 /*
-- 
2.14.3

Re: 1.8.7 http-tunnel doesn't seem to work? (but default http-keep-alive does)

[Olivier Houchard]

Hi Pieter,

On Mon, Apr 16, 2018 at 10:41:48PM +0200, PiBa-NL wrote:
> Hi Olivier,
> 
> Op 16-4-2018 om 17:09 schreef Olivier Houchard:
> > After some discussion with Willy, we came with a solution that may fix your
> > problem with kqueue.
> > Can you test the attached patch and let me know if it fixes it for you ?
> > 
> > Minor variation of the patch, that uses EV_RECEIPT if available, to avoid
> > scanning needlessly the kqueue.
> > 
> > Regards,
> > 
> > Olivier
> 
> Thanks the patch solves the issue i experienced at least for the testcase
> that i had. (And doesn't seem to cause obvious new issues that i could
> quickly spot..) Both with and without EV_RECEIPT on kev[0] it seems to work
> the same for my testcase..
> 
> Just a few thoughts though:
> Now only the first kev[0] gets the EV_RECEIPT flag, shouldn't it be added to
> all items in the array? Now sometimes 3 changes are send and only 2
> 'results' are reported back. If i read right the EV_RECEIPT should 'force' a
> result for each change send. Also is there a reason you put it inside a
> '#ifdef' ? It seems to me a hard requirement to not read any possible
> pending events when sending the list of updated filters at that moment.?. Or
> perhaps its possible to call kevent only once? Both sending changes, and
> receiving new events in 1 big go and without the RECEIPT flag?
> 

We only need one to prevent kevent() from trying to scanning the kqueue, so
only setting kev[0] should be enough. It's inside an #ifdef because
EV_RECEIPT was only implemented recently in OpenBSD, so all users may not have
it.
My hope is, if EV_RECEIPT is not set, we will indeed read even needlessly,
but we will read them again at the next kevent() call. If it does not, and
events are lost, then we'll have to add an extra entry that generates an error.
We considered using only one kevent() call for both adding events, and reading
them, but it makes it difficult to respect maxpollevents, because we'd need
enough room for all errors and reading the new events. 

> There are now more 'changes' send than required that need to be disregarded
> with a 'error flag' by kqueue
> Doesn't that (slightly) affect performance? Or would checking a bitmask
> beforehand not be cheaper than what kevent itself needs to do to ignore an
> item and 'error report' some of the changes.?. I've not tried to measure
> this, but technically i think there will be a few more cpu operations needed
> overall this way.?.

It should be fine. What kevent() does is :
foreach (change) {
ret = process_change();
if (ret != 0 || flags & EV_RECEIPT)
nerrors++;
}
if (nerrors > 0)
return nerrors;
return scan_kqueue();

So it should virtually have no impact on performances.

Regards,

Olivier

Re: 1.8.7 http-tunnel doesn't seem to work? (but default http-keep-alive does)

[PiBa-NL]

Hi Olivier,

Op 16-4-2018 om 17:09 schreef Olivier Houchard:

After some discussion with Willy, we came with a solution that may fix your
problem with kqueue.
Can you test the attached patch and let me know if it fixes it for you ?

Minor variation of the patch, that uses EV_RECEIPT if available, to avoid
scanning needlessly the kqueue.

Regards,

Olivier


Thanks the patch solves the issue i experienced at least for the 
testcase that i had. (And doesn't seem to cause obvious new issues that 
i could quickly spot..) Both with and without EV_RECEIPT on kev[0] it 
seems to work the same for my testcase..


Just a few thoughts though:
Now only the first kev[0] gets the EV_RECEIPT flag, shouldn't it be 
added to all items in the array? Now sometimes 3 changes are send and 
only 2 'results' are reported back. If i read right the EV_RECEIPT 
should 'force' a result for each change send. Also is there a reason you 
put it inside a '#ifdef' ? It seems to me a hard requirement to not read 
any possible pending events when sending the list of updated filters at 
that moment.?. Or perhaps its possible to call kevent only once? Both 
sending changes, and receiving new events in 1 big go and without the 
RECEIPT flag?


There are now more 'changes' send than required that need to be 
disregarded with a 'error flag' by kqueue
Doesn't that (slightly) affect performance? Or would checking a bitmask 
beforehand not be cheaper than what kevent itself needs to do to ignore 
an item and 'error report' some of the changes.?. I've not tried to 
measure this, but technically i think there will be a few more cpu 
operations needed overall this way.?.


Regards,

PiBa-NL (Pieter)

Re: 1.8.7 http-tunnel doesn't seem to work? (but default http-keep-alive does)

[Olivier Houchard]

Hi,

On Mon, Apr 16, 2018 at 03:37:34PM +0200, Olivier Houchard wrote:
> Hi Pieter,
> 
> On Fri, Apr 13, 2018 at 06:50:50AM +, Pi Ba wrote:
> > Using poll (startup with -dk) the request works properly.
> 
> After some discussion with Willy, we came with a solution that may fix your
> problem with kqueue.
> Can you test the attached patch and let me know if it fixes it for you ?
> 
> Thanks !
> 
> Olivier


Minor variation of the patch, that uses EV_RECEIPT if available, to avoid
scanning needlessly the kqueue.

Regards,

Olivier
>From 2229159329ec539c7875943af08c539064dcd76b Mon Sep 17 00:00:00 2001
From: Olivier Houchard 
Date: Mon, 16 Apr 2018 13:24:48 +0200
Subject: [PATCH] BUG/MEDIUM: When adding new events, provide an output to get
 errors.

When adding new events using kevent(), if there's an error, because we're
trying to delete an event that wasn't there, or because the fd has already
been closed, kevent() will either add an event in the eventlist array if
there's enough room for it, and keep on handling other events, or stop and
return -1.
We want it to process all the events, so give it a large-enough array to
store any error.

This should be backported to 1.8.
---
 src/ev_kqueue.c | 25 +
 1 file changed, 21 insertions(+), 4 deletions(-)

diff --git a/src/ev_kqueue.c b/src/ev_kqueue.c
index a103ece9d..4306c4372 100644
--- a/src/ev_kqueue.c
+++ b/src/ev_kqueue.c
@@ -31,6 +31,7 @@
 /* private data */
 static int kqueue_fd[MAX_THREADS]; // per-thread kqueue_fd
 static THREAD_LOCAL struct kevent *kev = NULL;
+static struct kevent *kev_out = NULL; // Trash buffer for kevent() to write 
the eventlist in
 
 /*
  * kqueue() poller
@@ -43,6 +44,8 @@ REGPRM2 static void _do_poll(struct poller *p, int exp)
int updt_idx, en;
int changes = 0;
 
+   timeout.tv_sec  = 0;
+   timeout.tv_nsec = 0;
/* first, scan the update list to find changes */
for (updt_idx = 0; updt_idx < fd_nbupdt; updt_idx++) {
fd = fd_updt[updt_idx];
@@ -81,13 +84,15 @@ REGPRM2 static void _do_poll(struct poller *p, int exp)
HA_ATOMIC_OR([fd].polled_mask, tid_bit);
}
}
-   if (changes)
-   kevent(kqueue_fd[tid], kev, changes, NULL, 0, NULL);
+   if (changes) {
+#ifdef EV_RECEIPT
+   kev[0].flags |= EV_RECEIPT;
+#endif
+   kevent(kqueue_fd[tid], kev, changes, kev_out, changes, 
);
+   }
fd_nbupdt = 0;
 
delta_ms= 0;
-   timeout.tv_sec  = 0;
-   timeout.tv_nsec = 0;
 
if (!exp) {
delta_ms= MAX_DELAY_MS;
@@ -194,6 +199,10 @@ REGPRM1 static int _do_init(struct poller *p)
 {
p->private = NULL;
 
+   kev_out = calloc(1, sizeof(struct kevent) * 2 * global.maxsock);
+   if (!kev_out)
+   goto fail_alloc;
+
kqueue_fd[tid] = kqueue();
if (kqueue_fd[tid] < 0)
goto fail_fd;
@@ -203,6 +212,9 @@ REGPRM1 static int _do_init(struct poller *p)
return 1;
 
  fail_fd:
+   free(kev_out);
+   kev_out = NULL;
+fail_alloc:
p->pref = 0;
return 0;
 }
@@ -220,6 +232,10 @@ REGPRM1 static void _do_term(struct poller *p)
 
p->private = NULL;
p->pref = 0;
+   if (kev_out) {
+   free(kev_out);
+   kev_out = NULL;
+   }
 }
 
 /*
@@ -250,6 +266,7 @@ REGPRM1 static int _do_fork(struct poller *p)
return 1;
 }
 
+
 /*
  * It is a constructor, which means that it will automatically be called before
  * main(). This is GCC-specific but it works at least since 2.95.
-- 
2.14.3

Re: 1.8.7 http-tunnel doesn't seem to work? (but default http-keep-alive does)

[Olivier Houchard]

Hi Pieter,

On Fri, Apr 13, 2018 at 06:50:50AM +, Pi Ba wrote:
> Using poll (startup with -dk) the request works properly.

After some discussion with Willy, we came with a solution that may fix your
problem with kqueue.
Can you test the attached patch and let me know if it fixes it for you ?

Thanks !

Olivier
>From 3c0a505e5f163989239ffb5267ddf7c1ed549fb9 Mon Sep 17 00:00:00 2001
From: Olivier Houchard 
Date: Mon, 16 Apr 2018 13:24:48 +0200
Subject: [PATCH] BUG/MEDIUM: When adding new events, provide an output to get
 errors.

When adding new events using kevent(), if there's an error, because we're
trying to delete an event that wasn't there, or because the fd has already
been closed, kevent() will either add an event in the eventlist array if
there's enough room for it, and keep on handling other events, or stop and
return -1.
We want it to process all the events, so give it a large-enough array to
store any error.

This should be backported to 1.8.
---
 src/ev_kqueue.c | 18 +++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/src/ev_kqueue.c b/src/ev_kqueue.c
index a103ece9d..faf3ba2c2 100644
--- a/src/ev_kqueue.c
+++ b/src/ev_kqueue.c
@@ -31,6 +31,7 @@
 /* private data */
 static int kqueue_fd[MAX_THREADS]; // per-thread kqueue_fd
 static THREAD_LOCAL struct kevent *kev = NULL;
+static struct kevent *kev_out = NULL; // Trash buffer for kevent() to write 
the eventlist in
 
 /*
  * kqueue() poller
@@ -43,6 +44,8 @@ REGPRM2 static void _do_poll(struct poller *p, int exp)
int updt_idx, en;
int changes = 0;
 
+   timeout.tv_sec  = 0;
+   timeout.tv_nsec = 0;
/* first, scan the update list to find changes */
for (updt_idx = 0; updt_idx < fd_nbupdt; updt_idx++) {
fd = fd_updt[updt_idx];
@@ -82,12 +85,10 @@ REGPRM2 static void _do_poll(struct poller *p, int exp)
}
}
if (changes)
-   kevent(kqueue_fd[tid], kev, changes, NULL, 0, NULL);
+   kevent(kqueue_fd[tid], kev, changes, kev_out, changes, 
);
fd_nbupdt = 0;
 
delta_ms= 0;
-   timeout.tv_sec  = 0;
-   timeout.tv_nsec = 0;
 
if (!exp) {
delta_ms= MAX_DELAY_MS;
@@ -194,6 +195,10 @@ REGPRM1 static int _do_init(struct poller *p)
 {
p->private = NULL;
 
+   kev_out = calloc(1, sizeof(struct kevent) * 2 * global.maxsock);
+   if (!kev_out)
+   goto fail_alloc;
+
kqueue_fd[tid] = kqueue();
if (kqueue_fd[tid] < 0)
goto fail_fd;
@@ -203,6 +208,9 @@ REGPRM1 static int _do_init(struct poller *p)
return 1;
 
  fail_fd:
+   free(kev_out);
+   kev_out = NULL;
+fail_alloc:
p->pref = 0;
return 0;
 }
@@ -220,6 +228,10 @@ REGPRM1 static void _do_term(struct poller *p)
 
p->private = NULL;
p->pref = 0;
+   if (kev_out) {
+   free(kev_out);
+   kev_out = NULL;
+   }
 }
 
 /*
-- 
2.16.3

Re: 1.8.7 http-tunnel doesn't seem to work? (but default http-keep-alive does)

[Willy Tarreau]

On Fri, Apr 13, 2018 at 06:50:50AM +, Pi Ba wrote:
> Using poll (startup with -dk) the request works properly.

Thank you Pieter!

Willy

Re: 1.8.7 http-tunnel doesn't seem to work? (but default http-keep-alive does)

[Pi Ba]

Using poll (startup with -dk) the request works properly.

Op vr 13 apr. 2018 08:29 schreef Willy Tarreau :

> Hi Pieter,
>
> On Fri, Apr 13, 2018 at 01:12:41AM +0200, PiBa-NL wrote:
> > To clarify the issue is not that haproxy uses cpu by looping, the issue
> is
> > that haproxy prevents the page from loading in the browser. The 'fix' on
> the
> > old version after the commit introducing the issue was to call the EV_SET
> > write delete *less* often. Or maybe my understanding of what is does is
> just
> > wrong :).
>
> Ah, sorry, maybe I'm mixing multiple problems, I understood it was looping.
> So if it's indeed missing events, it's different. Maybe we have an issue
> with a sequence of enable/disable on the polling, causing it to be disabled
> and not re-enabled sometimes. If you can test with "-dk" to disable kqueue
> and switch to poll, that will be great because I'll be able to test under
> the same conditions. And if it doesn't fail it will tell us it's really
> related to the kqueue poller.
>
> Willy
>

Re: 1.8.7 http-tunnel doesn't seem to work? (but default http-keep-alive does)

[Willy Tarreau]

Hi Pieter,

On Fri, Apr 13, 2018 at 01:12:41AM +0200, PiBa-NL wrote:
> To clarify the issue is not that haproxy uses cpu by looping, the issue is
> that haproxy prevents the page from loading in the browser. The 'fix' on the
> old version after the commit introducing the issue was to call the EV_SET
> write delete *less* often. Or maybe my understanding of what is does is just
> wrong :).

Ah, sorry, maybe I'm mixing multiple problems, I understood it was looping.
So if it's indeed missing events, it's different. Maybe we have an issue
with a sequence of enable/disable on the polling, causing it to be disabled
and not re-enabled sometimes. If you can test with "-dk" to disable kqueue
and switch to poll, that will be great because I'll be able to test under
the same conditions. And if it doesn't fail it will tell us it's really
related to the kqueue poller.

Willy

Re: 1.8.7 http-tunnel doesn't seem to work? (but default http-keep-alive does)

[PiBa-NL]

Hi Willy,

And a second mail as i just thought of one extra thing you wrote that 
maybe i misunderstand or perhaps confused you with a small remark about 
cpu usage in my earlier mail (that was a side effect of my other earlier 
but totally wrong code change..).

I'm suspecting we could have something wrong with the polled_mask, maybe
sometimes it's removed too early somewhere, preventing the delete(write)
from being performed, which would explain why it loops.

To clarify the issue is not that haproxy uses cpu by looping, the issue 
is that haproxy prevents the page from loading in the browser. The 'fix' 
on the old version after the commit introducing the issue was to call 
the EV_SET write delete *less* often. Or maybe my understanding of what 
is does is just wrong :).


Op 13-4-2018 om 0:57 schreef PiBa-NL:

Hi Willy,

Op 13-4-2018 om 0:22 schreef Willy Tarreau:

I'm suspecting we could have something wrong with the polled_mask, maybe
sometimes it's removed too early somewhere, preventing the delete(write)
from being performed, which would explain why it loops.

By the way you must really not try to debug an
old version but stick to the latest fixes.
Okay testing from now on with current master, just thought it would be 
easier to backtrack if i knew what particular new/missing event would 
possibly cause it. And it could have been simpler to find a fix just 
after the problem was introduced, but it seems it ain't that simple :).


I'm seeing two things that could be of interest to test :
   - remove the two "if (fdtab[fd].polled_mask & tid_bit)" conditions
 to delete the events. It will slightly inflate the list of events
 but not that much. If it fixes the problem it means that the
 polled_mask is sometimes wrong. Please do that with the updated
 master.
Removing the 'if polled_mask' does not fix the issue, in fact that 
makes it worse. the "srvrep[0007:0008]: HTTP/1.1 401 Unauthorized" is 
also not shown anymore without those checks..


   - switch to poll() just to see if you have the same so that we can
 figure if only the kqueue code triggers the issue. poll() doesn't
 rely on polled_mask at all.

Using poll (startup with -dk) the request works properly.


Many thanks for your tests.
Willy


Regards,

PiBa-NL (Pieter)



Regards,

PiBa-NL (Pieter)

Re: 1.8.7 http-tunnel doesn't seem to work? (but default http-keep-alive does)

[PiBa-NL]

Hi Willy,

Op 13-4-2018 om 0:22 schreef Willy Tarreau:

By the way you must really not try to debug an
old version but stick to the latest fixes.
Okay testing from now on with current master, just thought it would be 
easier to backtrack if i knew what particular new/missing event would 
possibly cause it. And it could have been simpler to find a fix just 
after the problem was introduced, but it seems it ain't that simple :).


I'm seeing two things that could be of interest to test :
   - remove the two "if (fdtab[fd].polled_mask & tid_bit)" conditions
 to delete the events. It will slightly inflate the list of events
 but not that much. If it fixes the problem it means that the
 polled_mask is sometimes wrong. Please do that with the updated
 master.
Removing the 'if polled_mask' does not fix the issue, in fact that makes 
it worse. the "srvrep[0007:0008]: HTTP/1.1 401 Unauthorized" is also not 
shown anymore without those checks..


   - switch to poll() just to see if you have the same so that we can
 figure if only the kqueue code triggers the issue. poll() doesn't
 rely on polled_mask at all.

Using poll (startup with -dk) the request works properly.


Many thanks for your tests.
Willy


Regards,

PiBa-NL (Pieter)

Re: 1.8.7 http-tunnel doesn't seem to work? (but default http-keep-alive does)

[PiBa-NL]

Hi Willy,

Op 12-4-2018 om 1:19 schreef Willy Tarreau:

Thank you very much for pointing the exact line that causes you trouble.
Well exact line.. probably not the right one. And yes just removing that 
line indeed breaks something else. (as expected..)

Would you have the ability to try the latest 1.9-dev just by chance ?
Yes when i started compiling from source, thats where i started, the 
current master branch. Though (sadly) it has the same broken effect.


Ill try and investigate a bit more.. Though if you think of a possible 
patch, i'm happy to try anything out ;)


Regards,
PiBa-NL (Pieter)

Re: 1.8.7 http-tunnel doesn't seem to work? (but default http-keep-alive does)

[Willy Tarreau]

Hi Pieter,

On Thu, Apr 12, 2018 at 12:16:21AM +0200, PiBa-NL wrote:
> Hi List / Willy,
> 
> Removing the line below 'fixes' my issue with kqueue poller and NTLM
> authentication with option http-tunnel..
> Though i'm sure something is then broken then horribly also (CPU go's
> 100%..). And i'm not sure what the proper fix would be. (ive got to little
> knowledge of what the various flags do and C++ aint a language i normally
> ever look at.. )

Don't worry for this, it's already great that you went that far!

> The 'breaking' commit was this one: 
> http://git.haproxy.org/?p=haproxy-1.8.git;a=commit;h=f839593dd26ec210ba66d74b2a4c2040dd1ed806
> 
> Can you take a new look at that piece of code? (as the commit was yours ;) )
> Thanks in advance :).

Thank you very much for pointing the exact line that causes you trouble.
I'm pretty sure that your fix breaks something else and causes some events
to possibly be missed in some cases, but at the moment I'm having a hard
time figuring the details. It could be that an event (stop reading) is
improperly reported through the mux in tunnel mode.

Would you have the ability to try the latest 1.9-dev just by chance ? I'm
interested, since the FDs work a bit differently there. If it happens not
to malfunction it could help us compare the behaviours between the two to
more easily spot the culprit.

Thanks!
Willy

Re: 1.8.7 http-tunnel doesn't seem to work? (but default http-keep-alive does)

[PiBa-NL]

Hi List / Willy,

Removing the line below 'fixes' my issue with kqueue poller and NTLM 
authentication with option http-tunnel..
Though i'm sure something is then broken then horribly also (CPU go's 
100%..). And i'm not sure what the proper fix would be. (ive got to 
little knowledge of what the various flags do and C++ aint a language i 
normally ever look at.. )
The 'breaking' commit was this one: 
http://git.haproxy.org/?p=haproxy-1.8.git;a=commit;h=f839593dd26ec210ba66d74b2a4c2040dd1ed806


Can you take a new look at that piece of code? (as the commit was yours ;) )
Thanks in advance :).

Regards,
PiBa-NL (Pieter)

 src/ev_kqueue.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/ev_kqueue.c b/src/ev_kqueue.c
index a103ece..49e7302 100644
--- a/src/ev_kqueue.c
+++ b/src/ev_kqueue.c
@@ -78,7 +78,7 @@ REGPRM2 static void _do_poll(struct poller *p, int exp)
         else if (fdtab[fd].polled_mask & tid_bit)
             EV_SET([changes++], fd, EVFILT_WRITE, EV_DELETE, 
0, 0, NULL);


-            HA_ATOMIC_OR([fd].polled_mask, tid_bit);
+//            HA_ATOMIC_OR([fd].polled_mask, tid_bit);
     }
 }
 if (changes)



Op 10-4-2018 om 23:11 schreef PiBa-NL:

Hi Haproxy List,

I upgraded to 1.8.7 (coming from 1.8.3) and found i could no-longer 
use one of our IIS websites. The login procedure thats using windows 
authentication / ntlm seems to fail..
Removing option http-tunnel seems to fix this though. Afaik 
http-tunnel 'should' switch to tunnelmode after the first request and 
as such should have no issue sending the credentials the the server.?.


Below are:  config / haproxy -vv / tcpdump / sess all

Is it a known issue? Is there anything else i can provide?

Regards,

PiBa-NL (Pieter)

-
# Automaticaly generated, dont edit manually.
# Generated on: 2018-04-10 21:00
global
    maxconn            1000
    log            192.168.8.10    local1    info
    stats socket /tmp/haproxy.socket level admin
    gid            80
    nbproc            1
    nbthread            1
    hard-stop-after        15m
    chroot                /tmp/haproxy_chroot
    daemon
    tune.ssl.default-dh-param    2048
    defaults
    option log-health-checks


frontend site.domain.nl2
    bind            192.168.8.5:443 name 192.168.8.5:443  ssl  crt 
/var/etc/haproxy/site.domain.nl2.pem crt-list 
/var/etc/haproxy/site.domain.nl2.crt_list

    mode            http
    log            global
    option            httplog
    option            http-tunnel
    maxconn            100
    timeout client        1h
    option tcplog
    default_backend website-intern_http_ipvANY

backend site-intern_http_ipvANY
    mode            http
    log            global
    option            http-tunnel
    timeout connect        10s
    timeout server        1h
    retries            3
    server            site 192.168.13.44:443 ssl  weight 1.1 verify none

-
[2.4.3-RELEASE][root@pfsense_5.local]/root: haproxy -vv
HA-Proxy version 1.8.7 2018/04/07
Copyright 2000-2018 Willy Tarreau 

Build options :
  TARGET  = freebsd
  CPU = generic
  CC  = cc
  CFLAGS  = -O2 -pipe -fstack-protector -fno-strict-aliasing 
-fno-strict-aliasing -Wdeclaration-after-statement -fwrapv 
-fno-strict-overflow -Wno-address-of-packed-member 
-Wno-null-dereference -Wno-unused-label -DFREEBSD_PORTS
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 
USE_ACCEPT4=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1 
USE_PCRE_JIT=1


Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), 
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")

Built with PCRE version : 8.40 2017-01-11
Running on PCRE version : 8.40 2017-01-11
PCRE library supports JIT : yes
Built with multi-threading support.
Encrypted password support via crypt(3): yes
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with Lua version : Lua 5.3.4
Built with OpenSSL version : OpenSSL 1.0.2m-freebsd  2 Nov 2017
Running on OpenSSL version : OpenSSL 1.0.2m-freebsd  2 Nov 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2

Available polling systems :
 kqueue : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use kqueue.

Available filters :
    [TRACE] trace
    [COMP] compression
    [SPOE] spoe
-
tcpdump of : Client 8.32>Haproxy 8.5:

21:09:13.452118 IP 192.168.8.32.51658 > 192.168.8.5.443: Flags [S], 
seq 1417754656, win 8192, options [mss 1260,nop,wscale 
8,nop,nop,sackOK], length 0
21:09:13.452312 IP 192.168.8.5.443 > 

1.8.7 http-tunnel doesn't seem to work? (but default http-keep-alive does)

[PiBa-NL]

Hi Haproxy List,

I upgraded to 1.8.7 (coming from 1.8.3) and found i could no-longer use 
one of our IIS websites. The login procedure thats using windows 
authentication / ntlm seems to fail..
Removing option http-tunnel seems to fix this though. Afaik http-tunnel 
'should' switch to tunnelmode after the first request and as such should 
have no issue sending the credentials the the server.?.


Below are:  config / haproxy -vv / tcpdump / sess all

Is it a known issue? Is there anything else i can provide?

Regards,

PiBa-NL (Pieter)

-
# Automaticaly generated, dont edit manually.
# Generated on: 2018-04-10 21:00
global
    maxconn            1000
    log            192.168.8.10    local1    info
    stats socket /tmp/haproxy.socket level admin
    gid            80
    nbproc            1
    nbthread            1
    hard-stop-after        15m
    chroot                /tmp/haproxy_chroot
    daemon
    tune.ssl.default-dh-param    2048
    defaults
    option log-health-checks


frontend site.domain.nl2
    bind            192.168.8.5:443 name 192.168.8.5:443  ssl  crt 
/var/etc/haproxy/site.domain.nl2.pem crt-list 
/var/etc/haproxy/site.domain.nl2.crt_list

    mode            http
    log            global
    option            httplog
    option            http-tunnel
    maxconn            100
    timeout client        1h
    option tcplog
    default_backend website-intern_http_ipvANY

backend site-intern_http_ipvANY
    mode            http
    log            global
    option            http-tunnel
    timeout connect        10s
    timeout server        1h
    retries            3
    server            site 192.168.13.44:443 ssl  weight 1.1 verify none

-
[2.4.3-RELEASE][root@pfsense_5.local]/root: haproxy -vv
HA-Proxy version 1.8.7 2018/04/07
Copyright 2000-2018 Willy Tarreau 

Build options :
  TARGET  = freebsd
  CPU = generic
  CC  = cc
  CFLAGS  = -O2 -pipe -fstack-protector -fno-strict-aliasing 
-fno-strict-aliasing -Wdeclaration-after-statement -fwrapv 
-fno-strict-overflow -Wno-address-of-packed-member -Wno-null-dereference 
-Wno-unused-label -DFREEBSD_PORTS
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 
USE_ACCEPT4=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1 
USE_PCRE_JIT=1


Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), 
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")

Built with PCRE version : 8.40 2017-01-11
Running on PCRE version : 8.40 2017-01-11
PCRE library supports JIT : yes
Built with multi-threading support.
Encrypted password support via crypt(3): yes
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with Lua version : Lua 5.3.4
Built with OpenSSL version : OpenSSL 1.0.2m-freebsd  2 Nov 2017
Running on OpenSSL version : OpenSSL 1.0.2m-freebsd  2 Nov 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2

Available polling systems :
 kqueue : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use kqueue.

Available filters :
    [TRACE] trace
    [COMP] compression
    [SPOE] spoe
-
tcpdump of : Client 8.32>Haproxy 8.5:

21:09:13.452118 IP 192.168.8.32.51658 > 192.168.8.5.443: Flags [S], seq 
1417754656, win 8192, options [mss 1260,nop,wscale 8,nop,nop,sackOK], 
length 0
21:09:13.452312 IP 192.168.8.5.443 > 192.168.8.32.51658: Flags [S.], seq 
1950703403, ack 1417754657, win 65228, options [mss 1260,nop,wscale 
7,sackOK,eol], length 0
21:09:13.453030 IP 192.168.8.32.51658 > 192.168.8.5.443: Flags [.], ack 
1, win 260, length 0
21:09:13.457740 IP 192.168.8.32.51658 > 192.168.8.5.443: Flags [P.], seq 
1:190, ack 1, win 260, length 189
21:09:13.457762 IP 192.168.8.5.443 > 192.168.8.32.51658: Flags [.], ack 
190, win 510, length 0
21:09:13.459503 IP 192.168.8.5.443 > 192.168.8.32.51658: Flags [.], seq 
1:1261, ack 190, win 511, length 1260
21:09:13.459516 IP 192.168.8.5.443 > 192.168.8.32.51658: Flags [.], seq 
1261:2521, ack 190, win 511, length 1260
21:09:13.459527 IP 192.168.8.5.443 > 192.168.8.32.51658: Flags [P.], seq 
2521:2686, ack 190, win 511, length 165
21:09:13.460342 IP 192.168.8.32.51658 > 192.168.8.5.443: Flags [.], ack 
2686, win 260, length 0
21:09:13.478984 IP 192.168.8.32.51658 > 192.168.8.5.443: Flags [P.], seq 
190:316, ack 2686, win 260, length 126
21:09:13.479038 IP 192.168.8.5.443 > 192.168.8.32.51658: Flags [.], ack 
316, win 510, length 0
21:09:13.480105 IP 192.168.8.5.443 > 192.168.8.32.51658: Flags [P.], seq 
2686:2737, ack 316, win 511, length 51
21:09:13.490136 IP 192.168.8.32.51658 >