Re: Intermittent bug with asyncio and MS Edge

[Barry Scott]

> On 25 Mar 2020, at 06:12, Frank Millman  wrote:
> 
> On 2020-03-24 8:39 PM, Barry Scott wrote:
>>> On 24 Mar 2020, at 11:54, Frank Millman  wrote:
>>> 
>>> 
>>> I decided to concentrate on using Wireshark to detect the difference 
>>> between a Python3.7 session and a Python3.8 session. Already I can see some 
>>> differences.
>>> 
>>> There is only one version of my program. I am simply running it with either 
>>> 'py -3.7 ' or 'py -3.8'. And I am ignoring Chrome at this stage, as it is 
>>> only that Edge shows the problem.
>>> 
>>> First point - Python3.7 also shows a lot of [RST, ACK] lines. My guess is 
>>> that this is caused by my 'protocol violation' of sending a 'Keep-Alive' 
>>> header and then closing the connection. Python3.7 does not suffer from 
>>> dropping files, so I now think this is a sidetrack. I will fix my program 
>>> when this is all over, but for now I don't want to touch it.
>> Yes your protocol violation is why you see [RST, ACK].
>> I'm confused you know that the code has a critical bug in it and you have 
>> not fixed it?
>> Just send "Connection: close" and I'd assume all will work.
> 
> Well, the reason is simply that I wanted to understand why my code that 
> worked all the way from 3.4 through 3.7 stopped working in 3.8. I realise 
> that my code is faulty, but I still wanted to know what the trigger was that 
> caused the bug to appear.

Got it, I'd not picked up on you wishes to find the details of why.

> 
> From my testing with Wireshark, I can see that both Edge and Chrome create 20 
> connections to GET 20 files. The difference seems to be that Chrome does not 
> attempt to re-use a connection, even though both client and server have sent 
> Keep-Alive headers. Edge does attempt to re-use the connection.

Chrome will reuse the connections if you load enough files from the same server.
All browser do this to get a page displayed as fast as possible.
Because HTTP is half-duplex its the round trip time rather then bandwidth that 
controls
the time taken get all the asserts needed to render a page most of the time.

> 
> The difference between 3.7 and 3.8 is that 3.7 sends the data in separate 
> packets for the status, each header, and then each chunk, whereas 3.8 sends 
> the whole lot in a single packet.

That will change the timing enough to expose the problem.

I often find with network code bugs that its changes that makes the
timing different that expos bugs in supposed tested and working code.

In this case you could track down what was the cause, but it's often the case 
that it's not practical to do that.
You may not know what changed between when the code worked and when it broke in 
many cases.

> My guess is that 3.7 is slower to send the files, so Edge starts up all 20 
> connections before it has finished receiving the first one, whereas with 3.8, 
> by the time it has opened a few connections the first file has been received, 
> so it tries to re-use the same connection to receive the next one. By then I 
> have closed the connection. If I am right, it is surprising that my program 
> worked *some* of the time.

I have lost count of the number of times when we have found a bug in some code 
that we say "how did it ever work?".

> 
> The same reasoning would explain why it worked when connecting from a remote 
> host. There would be enough delay to force it into the same behaviour as 3.7.

Yep and if there are any middle boxes, routers, man-in-middle-proxies, etc the 
timing/packets get pushed around.

> 
> It has been an interesting ride, and I have learned a lot. I will now look 
> into fixing my program. The easy fix is to just send 'Connection: Close', but 
> I will do it properly and implement 'Keep-Alive'.

Barry

> 
> Thanks all
> 
> Frank
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Chris Angelico]

On Wed, Mar 25, 2020 at 5:14 PM Frank Millman  wrote:
> My guess is that 3.7 is slower to send the files, so Edge starts up all
> 20 connections before it has finished receiving the first one, whereas
> with 3.8, by the time it has opened a few connections the first file has
> been received, so it tries to re-use the same connection to receive the
> next one. By then I have closed the connection. If I am right, it is
> surprising that my program worked *some* of the time.

That is definitely plausible. It would also mean that, depending on
exact browser settings, you might see the same problem crop up in
other situations. Definitely worth the time to investigate this one,
since you now know the ACTUAL reason and can solve the ACTUAL bug.

> It has been an interesting ride, and I have learned a lot. I will now
> look into fixing my program. The easy fix is to just send 'Connection:
> Close', but I will do it properly and implement 'Keep-Alive'.

Yep, and honestly, popping a quick "Connection: Close" isn't a
terrible solution. I have one app where I had a similar weird problem
with keep-alive, and made the opposite decision to you: rather than
sink in the hours to figure out the actual cause, I just slapped in a
"Connection: Close" and moved on. Part of the problem is that my issue
only shows up on HTTPS connections and not on plain HTTP ones, which
makes it extremely fiddly to wireshark (since all the traffic I care
about is encrypted, unless through some miracle the bug is visible
when it has no symptoms). But hey, that's what TODO files are for

Thank you for including us on your debugging journey. To anyone else
out there with a bizarre problem, I recommend reading over this thread
to get an idea of how to get help - Frank demonstrated a willingness
to sink his own time into this, a comprehensive writeup of all useful
symptoms, and a truly interesting problem report.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Frank Millman]

On 2020-03-24 8:39 PM, Barry Scott wrote:




On 24 Mar 2020, at 11:54, Frank Millman  wrote:


I decided to concentrate on using Wireshark to detect the difference between a 
Python3.7 session and a Python3.8 session. Already I can see some differences.

There is only one version of my program. I am simply running it with either 'py 
-3.7 ' or 'py -3.8'. And I am ignoring Chrome at this stage, as it is only that 
Edge shows the problem.

First point - Python3.7 also shows a lot of [RST, ACK] lines. My guess is that 
this is caused by my 'protocol violation' of sending a 'Keep-Alive' header and 
then closing the connection. Python3.7 does not suffer from dropping files, so 
I now think this is a sidetrack. I will fix my program when this is all over, 
but for now I don't want to touch it.


Yes your protocol violation is why you see [RST, ACK].

I'm confused you know that the code has a critical bug in it and you have not 
fixed it?
Just send "Connection: close" and I'd assume all will work.



Well, the reason is simply that I wanted to understand why my code that 
worked all the way from 3.4 through 3.7 stopped working in 3.8. I 
realise that my code is faulty, but I still wanted to know what the 
trigger was that caused the bug to appear.


From my testing with Wireshark, I can see that both Edge and Chrome 
create 20 connections to GET 20 files. The difference seems to be that 
Chrome does not attempt to re-use a connection, even though both client 
and server have sent Keep-Alive headers. Edge does attempt to re-use the 
connection.


The difference between 3.7 and 3.8 is that 3.7 sends the data in 
separate packets for the status, each header, and then each chunk, 
whereas 3.8 sends the whole lot in a single packet.


My guess is that 3.7 is slower to send the files, so Edge starts up all 
20 connections before it has finished receiving the first one, whereas 
with 3.8, by the time it has opened a few connections the first file has 
been received, so it tries to re-use the same connection to receive the 
next one. By then I have closed the connection. If I am right, it is 
surprising that my program worked *some* of the time.


The same reasoning would explain why it worked when connecting from a 
remote host. There would be enough delay to force it into the same 
behaviour as 3.7.


It has been an interesting ride, and I have learned a lot. I will now 
look into fixing my program. The easy fix is to just send 'Connection: 
Close', but I will do it properly and implement 'Keep-Alive'.


Thanks all

Frank
--
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Barry Scott]

> On 24 Mar 2020, at 11:54, Frank Millman  wrote:
> 
> On 2020-03-23 1:56 PM, Frank Millman wrote:
>> On 2020-03-23 12:57 PM, Chris Angelico wrote:
>>> On Mon, Mar 23, 2020 at 8:03 PM Frank Millman  wrote:
 
 On 2020-03-22 12:11 PM, Chris Angelico wrote:
> On Sun, Mar 22, 2020 at 8:30 PM Frank Millman  wrote:
>> 
>> On 2020-03-22 10:45 AM, Chris Angelico wrote:
> 
> If you can recreate the problem with a single socket and multiple
> requests, that would be extremely helpful. I also think it's highly
> likely that this is the case.
> 
 
 I am working on a stripped-down version, but I realise there are a few
 things I have not grasped.
 
 Hope you don't mind, but can you scan through what follows and tell me
 if I am on the right lines?
>>> 
>>> No probs!
>>> 
>> [...]
>> Really appreciate the one-on-one tuition. I am learning a lot!
>>> 
 If this all makes sense, I should write two versions of the client
 program, one using a single connection, and one using a pool of 
 connections.
 
>>> 
>>> Possibly! I think you'll most likely see that one of those behaves
>>> perfectly normally, and you only trigger the issue in the other. So
>>> you could move forward with just one test program.
>>> 
>> Well, I have got the first one working - single connection - and so far it 
>> has not gone wrong.
>> However, it is difficult to be sure that I am comparing apples with apples. 
>> I have written my test server to handle 'Keep-Alive' correctly, but as I 
>> mentioned earlier, my live program closes the connection after each 
>> transfer. So now I have to make my test server do the same, and change my 
>> test client to react to that and re-open the connection each time. I will 
>> make the changes and see how that behaves.
>> Of course now I am in the murky waters of trying to second-guess how Edge 
>> reacts to that. Presumably that is where Wireshark will be useful. I will 
>> keep you posted.
> 
> Here is a progress report.
> 
> I decided to concentrate on using Wireshark to detect the difference between 
> a Python3.7 session and a Python3.8 session. Already I can see some 
> differences.
> 
> There is only one version of my program. I am simply running it with either 
> 'py -3.7 ' or 'py -3.8'. And I am ignoring Chrome at this stage, as it is 
> only that Edge shows the problem.
> 
> First point - Python3.7 also shows a lot of [RST, ACK] lines. My guess is 
> that this is caused by my 'protocol violation' of sending a 'Keep-Alive' 
> header and then closing the connection. Python3.7 does not suffer from 
> dropping files, so I now think this is a sidetrack. I will fix my program 
> when this is all over, but for now I don't want to touch it.

Yes your protocol violation is why you see [RST, ACK].

I'm confused you know that the code has a critical bug in it and you have not 
fixed it?
Just send "Connection: close" and I'd assume all will work.

If my last message was not clear let me know what is unclear.

It not interesting that you get away with it in 3.7 and not 3.8 because the 
code is wrong.

> When I send the response to the initial connection, I write a status line, 
> then multiple header lines, then an html file. I then close the connection. 
> Python3.7 sends a packet with the status, then a separate packet for each 
> header, then a packet with the file. Python3.8 sends a packet with the 
> status, then merges everything else into a single packet and sends it in one 
> go. I just mention this as an indication that quite a lot has changed between 
> my two versions of Python.
> 
> I have one frustration with Wireshark. I will mention it in case anyone has a 
> solution.
> 
> I can see that Edge opens multiple connections. I am trying to track the 
> activity on each connection separately. I can export the data to csv, which 
> makes it easier to work on. But while the TCP lines include the source and 
> destination ports, the HTTP lines do not, so I don't know which connection 
> they belong to. If I view the data in Wireshark's gui it does show the ports, 
> so the data is there somewhere. Does anyone know how to include it in the csv 
> output?

There is a option to follow a single TCP or HTTP connection in Wireshark.
Right click on one pf the packets then choose Follow/HTTP stream.

The way to share the data is as a PCAP file that allows someone else to look at 
the capture
with tools like wireshark and others.

> 
> That's all for now. I will keep you posted.

Barry

> 
> Frank
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list 
> 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Frank Millman]

On 2020-03-24 1:54 PM, Frank Millman wrote:

On 2020-03-23 1:56 PM, Frank Millman wrote:

I have one frustration with Wireshark. I will mention it in case anyone 
has a solution.


I can see that Edge opens multiple connections. I am trying to track the 
activity on each connection separately. I can export the data to csv, 
which makes it easier to work on. But while the TCP lines include the 
source and destination ports, the HTTP lines do not, so I don't know 
which connection they belong to. If I view the data in Wireshark's gui 
it does show the ports, so the data is there somewhere. Does anyone know 
how to include it in the csv output?




I solved my Wireshark problem by exporting the data as text. A much more 
wordy format, and fiddly to parse, but the info I am looking for is there.


I am getting some interesting results now, such as a second GET 
interrupting an existing session. However, my brain is now frazzled, so 
I will continue tomorrow.


Frank
--
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Frank Millman]

On 2020-03-23 1:56 PM, Frank Millman wrote:

On 2020-03-23 12:57 PM, Chris Angelico wrote:

On Mon, Mar 23, 2020 at 8:03 PM Frank Millman  wrote:


On 2020-03-22 12:11 PM, Chris Angelico wrote:
On Sun, Mar 22, 2020 at 8:30 PM Frank Millman  
wrote:


On 2020-03-22 10:45 AM, Chris Angelico wrote:


If you can recreate the problem with a single socket and multiple
requests, that would be extremely helpful. I also think it's highly
likely that this is the case.



I am working on a stripped-down version, but I realise there are a few
things I have not grasped.

Hope you don't mind, but can you scan through what follows and tell me
if I am on the right lines?


No probs!



[...]

Really appreciate the one-on-one tuition. I am learning a lot!




If this all makes sense, I should write two versions of the client
program, one using a single connection, and one using a pool of 
connections.




Possibly! I think you'll most likely see that one of those behaves
perfectly normally, and you only trigger the issue in the other. So
you could move forward with just one test program.



Well, I have got the first one working - single connection - and so far 
it has not gone wrong.


However, it is difficult to be sure that I am comparing apples with 
apples. I have written my test server to handle 'Keep-Alive' correctly, 
but as I mentioned earlier, my live program closes the connection after 
each transfer. So now I have to make my test server do the same, and 
change my test client to react to that and re-open the connection each 
time. I will make the changes and see how that behaves.


Of course now I am in the murky waters of trying to second-guess how 
Edge reacts to that. Presumably that is where Wireshark will be useful. 
I will keep you posted.


Here is a progress report.

I decided to concentrate on using Wireshark to detect the difference 
between a Python3.7 session and a Python3.8 session. Already I can see 
some differences.


There is only one version of my program. I am simply running it with 
either 'py -3.7 ' or 'py -3.8'. And I am ignoring Chrome at this stage, 
as it is only that Edge shows the problem.


First point - Python3.7 also shows a lot of [RST, ACK] lines. My guess 
is that this is caused by my 'protocol violation' of sending a 
'Keep-Alive' header and then closing the connection. Python3.7 does not 
suffer from dropping files, so I now think this is a sidetrack. I will 
fix my program when this is all over, but for now I don't want to touch it.


When I send the response to the initial connection, I write a status 
line, then multiple header lines, then an html file. I then close the 
connection. Python3.7 sends a packet with the status, then a separate 
packet for each header, then a packet with the file. Python3.8 sends a 
packet with the status, then merges everything else into a single packet 
and sends it in one go. I just mention this as an indication that quite 
a lot has changed between my two versions of Python.


I have one frustration with Wireshark. I will mention it in case anyone 
has a solution.


I can see that Edge opens multiple connections. I am trying to track the 
activity on each connection separately. I can export the data to csv, 
which makes it easier to work on. But while the TCP lines include the 
source and destination ports, the HTTP lines do not, so I don't know 
which connection they belong to. If I view the data in Wireshark's gui 
it does show the ports, so the data is there somewhere. Does anyone know 
how to include it in the csv output?


That's all for now. I will keep you posted.

Frank

--
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Chris Angelico]

On Mon, Mar 23, 2020 at 10:58 PM Frank Millman  wrote:
>
> On 2020-03-23 12:57 PM, Chris Angelico wrote:
> > On Mon, Mar 23, 2020 at 8:03 PM Frank Millman  wrote:
> >>
> >> On 2020-03-22 12:11 PM, Chris Angelico wrote:
> >>> On Sun, Mar 22, 2020 at 8:30 PM Frank Millman  wrote:
> 
>  On 2020-03-22 10:45 AM, Chris Angelico wrote:
> >>>
> >>> If you can recreate the problem with a single socket and multiple
> >>> requests, that would be extremely helpful. I also think it's highly
> >>> likely that this is the case.
> >>>
> >>
> >> I am working on a stripped-down version, but I realise there are a few
> >> things I have not grasped.
> >>
> >> Hope you don't mind, but can you scan through what follows and tell me
> >> if I am on the right lines?
> >
> > No probs!
> >
>
> [...]
>
> Really appreciate the one-on-one tuition. I am learning a lot!

Heh, I'm always happy to help people who are willing to learn. Also,
you've brought an interesting problem to the table. :)

> Of course now I am in the murky waters of trying to second-guess how
> Edge reacts to that. Presumably that is where Wireshark will be useful.
> I will keep you posted.
>

Yep, it's going to be a matter of monitoring the exact headers sent
each direction.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Barry Scott]

> On 23 Mar 2020, at 09:02, Frank Millman  wrote:
> 
> On 2020-03-22 12:11 PM, Chris Angelico wrote:
>> On Sun, Mar 22, 2020 at 8:30 PM Frank Millman  wrote:
>>> 
>>> On 2020-03-22 10:45 AM, Chris Angelico wrote:
>> If you can recreate the problem with a single socket and multiple
>> requests, that would be extremely helpful. I also think it's highly
>> likely that this is the case.
> 
> I am working on a stripped-down version, but I realise there are a few things 
> I have not grasped.
> 
> Hope you don't mind, but can you scan through what follows and tell me if I 
> am on the right lines?
> 
> Both the client and the server can send a header with 'Keep-alive', but what 
> does it actually mean?
> 
> If the client sends it, does that mean that it wants the server to keep the 
> connection open, and only close it when the client closes it from the other 
> end?
> 
> Conversely, if the server sends it, does it mean that it wants the client to 
> keep the connection open? If so, under what condition would the server close 
> the connection (other than a timeout). Should the server only send 
> 'Keep-alive' if it receives 'Keep-alive'?

See https://tools.ietf.org/html/rfc2616  
secton 14.10 for the details.

When the browser sends "Connection: keep-alive" it mean that the browser wishes 
to have
a persistent connection. The server is free to choose.

If the server replies with "Connection: close" the connection is not persistent 
and will
be closed after the response.

If the server replies with "Connection: keep-aiive" then connect is persistent 
that means that
after this response the server will allow another transaction, GET, POST etc.

Note: You need to know if the client is using HTTP/1.0 or HTTP/1.1 protocol the 
rules are different in
each version.

> 
> In my program, when I send a file in response to a GET, I send a header with 
> 'Keep-alive', then I send the file, then I close the connection. This now 
> seems wrong. It could even be the cause of my bug, but then why has it only 
> appeared now? Both Edge and Chrome send 'Keep-alive' headers.

Its a protocol violation. In your case you must send "Connection: close". Fix 
that and I think your code is likely to work.

If you care about performance add the extra code to handle persistent 
connections. Its speeds up the browsers
and should reduce the impact on the server as well.


> If I am thinking along the right lines, then the exchange should go like this 
> -
> 
> Client sends request, with 'Keep-alive' header.
> 
> Server sends response, but does not close the connection. The server should 
> reply with a 'Keep-alive' header. If it does not, the client will close the 
> connection, in which case the server will also close.

You must include a "Connection:" head for HTTP/1.1 I believe.

> 
> Assuming that they both send 'Keep-alive', the onus is on the client to close 
> the connection when it has no more requests.

Yes.

> The server should have a timeout in case the client goes away.

Yes to stop resources running out.

> 
> Assuming that the above is correct, the client will rely on 'Content-length' 
> to determine when it has received the entire request. If the client has more 
> than one request, it will send the first, wait for the response, when fully 
> received as per the 'Content-length' it will send the next one, until all 
> requests have been sent and all responses received, at which point it will 
> close the connection.

You can use "Content-Length" header or chunked encoding.

> 
> All this assumes only one connection. Alternatively the client could open 
> multiple connections for the requests. In that case it would make sense to 
> use 'Connection: Close', so that the server can close the connection straight 
> away, making it available for reuse.

Modern browsers open many persistent connection. I recall its 8/site in the 
case of chrome.


> 
> If this all makes sense, I should write two versions of the client program, 
> one using a single connection, and one using a pool of connections.

Is this for a test program?

You can use curl for the single connection case.


> 
> All comments appreciated!
> 
> Frank

Barry

> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Frank Millman]

On 2020-03-23 12:57 PM, Chris Angelico wrote:

On Mon, Mar 23, 2020 at 8:03 PM Frank Millman  wrote:


On 2020-03-22 12:11 PM, Chris Angelico wrote:

On Sun, Mar 22, 2020 at 8:30 PM Frank Millman  wrote:


On 2020-03-22 10:45 AM, Chris Angelico wrote:


If you can recreate the problem with a single socket and multiple
requests, that would be extremely helpful. I also think it's highly
likely that this is the case.



I am working on a stripped-down version, but I realise there are a few
things I have not grasped.

Hope you don't mind, but can you scan through what follows and tell me
if I am on the right lines?


No probs!



[...]

Really appreciate the one-on-one tuition. I am learning a lot!




If this all makes sense, I should write two versions of the client
program, one using a single connection, and one using a pool of connections.



Possibly! I think you'll most likely see that one of those behaves
perfectly normally, and you only trigger the issue in the other. So
you could move forward with just one test program.



Well, I have got the first one working - single connection - and so far 
it has not gone wrong.


However, it is difficult to be sure that I am comparing apples with 
apples. I have written my test server to handle 'Keep-Alive' correctly, 
but as I mentioned earlier, my live program closes the connection after 
each transfer. So now I have to make my test server do the same, and 
change my test client to react to that and re-open the connection each 
time. I will make the changes and see how that behaves.


Of course now I am in the murky waters of trying to second-guess how 
Edge reacts to that. Presumably that is where Wireshark will be useful. 
I will keep you posted.


Frank

--
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Chris Angelico]

On Mon, Mar 23, 2020 at 8:03 PM Frank Millman  wrote:
>
> On 2020-03-22 12:11 PM, Chris Angelico wrote:
> > On Sun, Mar 22, 2020 at 8:30 PM Frank Millman  wrote:
> >>
> >> On 2020-03-22 10:45 AM, Chris Angelico wrote:
> >
> > If you can recreate the problem with a single socket and multiple
> > requests, that would be extremely helpful. I also think it's highly
> > likely that this is the case.
> >
>
> I am working on a stripped-down version, but I realise there are a few
> things I have not grasped.
>
> Hope you don't mind, but can you scan through what follows and tell me
> if I am on the right lines?

No probs!

> Both the client and the server can send a header with 'Keep-alive', but
> what does it actually mean?
>
> If the client sends it, does that mean that it wants the server to keep
> the connection open, and only close it when the client closes it from
> the other end?
>
> Conversely, if the server sends it, does it mean that it wants the
> client to keep the connection open? If so, under what condition would
> the server close the connection (other than a timeout). Should the
> server only send 'Keep-alive' if it receives 'Keep-alive'?

I'm pretty sure the server should never say "keep-alive" if the client
said "close". (There's a bit of an oddity in that the default changed
between, I think, HTTP 1.0 and 1.1. But that's a minor discrepancy.)
Generally, you'll see one of these options:

1) Client says "Connection: close". This means "I'm only gonna be
doing this one request, then we're done". Good for non-browser usage,
automated scripts etc.

2) Client says "Keep-Alive", server says "Close". This means the
client would be okay with connection reuse, but the server's saying no
(maybe it doesn't support it, or maybe it's in a soft shutdown state
where it wants to end connections, or whatever). After the current
request, the connection will be closed.

3) Both ends say "Connection: Keep-Alive". After the request, the
socket isn't closed, and can be reused for some other request.

I'm not sure how a client would (or should) handle it if the server
says "Connection: Keep-Alive" after the client said "Connection:
Close". Probably the client would just close its end of the socket and
that'd be that.

> In my program, when I send a file in response to a GET, I send a header
> with 'Keep-alive', then I send the file, then I close the connection.
> This now seems wrong. It could even be the cause of my bug, but then why
> has it only appeared now? Both Edge and Chrome send 'Keep-alive' headers.
>

It mightn't be fundamentally WRONG (there needs to be *some* way to
signal that you're now done), but it would probably be suboptimal. If
you're always going to close the connection after one request, just
send Connection: Close.

> If I am thinking along the right lines, then the exchange should go like
> this -
>
> Client sends request, with 'Keep-alive' header.
>
> Server sends response, but does not close the connection. The server
> should reply with a 'Keep-alive' header. If it does not, the client will
> close the connection, in which case the server will also close.
>
> Assuming that they both send 'Keep-alive', the onus is on the client to
> close the connection when it has no more requests. The server should
> have a timeout in case the client goes away.

I think either end is allowed to close the connection.

> Assuming that the above is correct, the client will rely on
> 'Content-length' to determine when it has received the entire request.
> If the client has more than one request, it will send the first, wait
> for the response, when fully received as per the 'Content-length' it
> will send the next one, until all requests have been sent and all
> responses received, at which point it will close the connection.

Yep. Which makes things very interesting (for serene values of
"interesting", such as "we're all gonna die") when you want to use
connection pooling with something that fundamentally doesn't have a
length, such as a chunked transfer. Which you are doing. That may or
may not actually be significant.

> All this assumes only one connection. Alternatively the client could
> open multiple connections for the requests. In that case it would make
> sense to use 'Connection: Close', so that the server can close the
> connection straight away, making it available for reuse.

What you'll often see is that the client has some sort of connection
limit, and reuses those same connections. For instance, it might have
a maximum of four connections per server (where "server" has to
include the name as well as its IP and port, due to TLS SNI), so it
requests an HTML page on one connection (with keep-alive), parses
that, and then fires off one request on the reused connection and
three others on new connections. As those requests complete, it sends
new requests on the same sockets.

> If this all makes sense, I should write two versions of the client
> program, one using a single connection, and one using 

Re: Intermittent bug with asyncio and MS Edge

[Frank Millman]

On 2020-03-22 12:11 PM, Chris Angelico wrote:

On Sun, Mar 22, 2020 at 8:30 PM Frank Millman  wrote:


On 2020-03-22 10:45 AM, Chris Angelico wrote:


If you can recreate the problem with a single socket and multiple
requests, that would be extremely helpful. I also think it's highly
likely that this is the case.



I am working on a stripped-down version, but I realise there are a few 
things I have not grasped.


Hope you don't mind, but can you scan through what follows and tell me 
if I am on the right lines?


Both the client and the server can send a header with 'Keep-alive', but 
what does it actually mean?


If the client sends it, does that mean that it wants the server to keep 
the connection open, and only close it when the client closes it from 
the other end?


Conversely, if the server sends it, does it mean that it wants the 
client to keep the connection open? If so, under what condition would 
the server close the connection (other than a timeout). Should the 
server only send 'Keep-alive' if it receives 'Keep-alive'?


In my program, when I send a file in response to a GET, I send a header 
with 'Keep-alive', then I send the file, then I close the connection. 
This now seems wrong. It could even be the cause of my bug, but then why 
has it only appeared now? Both Edge and Chrome send 'Keep-alive' headers.


If I am thinking along the right lines, then the exchange should go like 
this -


Client sends request, with 'Keep-alive' header.

Server sends response, but does not close the connection. The server 
should reply with a 'Keep-alive' header. If it does not, the client will 
close the connection, in which case the server will also close.


Assuming that they both send 'Keep-alive', the onus is on the client to 
close the connection when it has no more requests. The server should 
have a timeout in case the client goes away.


Assuming that the above is correct, the client will rely on 
'Content-length' to determine when it has received the entire request. 
If the client has more than one request, it will send the first, wait 
for the response, when fully received as per the 'Content-length' it 
will send the next one, until all requests have been sent and all 
responses received, at which point it will close the connection.


All this assumes only one connection. Alternatively the client could 
open multiple connections for the requests. In that case it would make 
sense to use 'Connection: Close', so that the server can close the 
connection straight away, making it available for reuse.


If this all makes sense, I should write two versions of the client 
program, one using a single connection, and one using a pool of connections.


All comments appreciated!

Frank

--
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Barry Scott]

> On 22 Mar 2020, at 09:41, Frank Millman  wrote:
> 
> On 2020-03-22 11:00 AM, Barry Scott wrote:
>>> On 22 Mar 2020, at 07:56, Frank Millman  wrote:
>>> 
>>> On 2020-03-21 8:04 PM, Barry Scott wrote:
 I'd look at the network traffic with wireshark to see if there is anything 
 different between edge and the other browsers.
>>> 
>>> You are leading me into deep waters here :-)  I have never used Wireshark 
>>> before. I have now downloaded it and am running it - it generates a *lot* 
>>> of data, most of which I do not understand yet!
>> You can tell wireshark to only capture on one interface and to only capture 
>> packets for port 80.
>> (Captureing HTTPS means you cannot decode the packets without going deeper I 
>> recall)
>> Then you can tell wireshark to decode the captured data for http to drop a 
>> lot of the lower level details.
> 
> Thanks. I am more or less doing that. Interestingly the [RST,ACK] messages 
> appear on the tcp packets, so if I filter on http I do not see them.
> 
>>> 
>>> One thing immediately stands out. When I run it with MS Edge and Python3.8, 
>>> it shows a lot of lines highlighted in red, with the symbols [RST,ACK]. 
>>> They do not appear when running Chrome, and they do not appear when running 
>>> Python3.7.
>> As Chris said that should not happen.
> 
> As I replied to Chris, they appear in packets sent *from* Python *to* Edge.
> 
>>> 
>>> I have another data point. I tried putting an asyncio.sleep() after sending 
>>> each file. A value of 0.01 made no difference, but a value of 0.1 makes the 
>>> problem go away.
>> What is the async wait to wait for the transmit buffers to drain?
> 
> Not sure what you are asking. I am just doing what it says in the docs -
> 
> =
> 
> write(data)
> The method attempts to write the data to the underlying socket immediately. 
> If that fails, the data is queued in an internal write buffer until it can be 
> sent.
> 
> The method should be used along with the drain() method:
> 
> stream.write(data)
> await stream.drain()
> 
> =
> 
> coroutine drain()
> Wait until it is appropriate to resume writing to the stream. Example:
> 
> writer.write(data)
> await writer.drain()
> This is a flow control method that interacts with the underlying IO write 
> buffer. When the size of the buffer reaches the high watermark, drain() 
> blocks until the size of the buffer is drained down to the low watermark and 
> writing can be resumed. When there is nothing to wait for, the drain() 
> returns immediately.
> 
> =

That you called drain() is what I was asking. I'm a Twisted user and have not 
used asyncio yet.

Barry

> 
> Frank
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Barry Scott]

> On 22 Mar 2020, at 11:59, Frank Millman  wrote:
> 
> On 2020-03-22 1:01 PM, Chris Angelico wrote:
>> On Sun, Mar 22, 2020 at 12:45 AM Frank Millman  wrote:
>>> 
>>> Hi all
>>> 
>>> I have a strange intermittent bug.
>>> 
>>> The role-players -
>>>  asyncio on Python 3.8 running on Windows 10
>>>  Microsoft Edge running as a browser on the same machine
>>> 
>>> The bug does not occur with Python 3.7.
>>> It does not occur with Chrome or Firefox.
>>> It does not occur when MS Edge connects to another host on the network,
>>> running the same Python program (Python 3.8 on Fedora 31).
>> What exact version of Python 3.7 did you test? I'm looking through the
>> changes to asyncio and came across this one, which may have some
>> impact.
> 
> Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 
> bit (AMD64)] on win32

Can you confirm that you have implemented Connection: keep-alive?
This means that the browser can send a 2nd GET on the same connection.

Barry


> 
> Frank
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Frank Millman]

On 2020-03-22 1:01 PM, Chris Angelico wrote:

On Sun, Mar 22, 2020 at 12:45 AM Frank Millman  wrote:


Hi all

I have a strange intermittent bug.

The role-players -
  asyncio on Python 3.8 running on Windows 10
  Microsoft Edge running as a browser on the same machine

The bug does not occur with Python 3.7.
It does not occur with Chrome or Firefox.
It does not occur when MS Edge connects to another host on the network,
running the same Python program (Python 3.8 on Fedora 31).


What exact version of Python 3.7 did you test? I'm looking through the
changes to asyncio and came across this one, which may have some
impact.


Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 
64 bit (AMD64)] on win32


Frank

--
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Kouli]

The RST from Python is probably caused here by HTTP 1.1 server closing TCP
connection without signalling "Connection: Close" in response headers: a
fast HTTP client will send another HTTP request before its TCP stack
detects the connection is being closed - and packets containing this new
requests will be replied with RST.

When you delay your response (as you mentioned), the Edge browser probably
opens more HTTP connections and will not send more HTTP requests in a
single connection as described above.

You should search for the cause Python closes the TCP connection (instead
of waiting for another HTTP request).

Kouli

On Sun, Mar 22, 2020 at 12:04 PM Chris Angelico  wrote:

> On Sun, Mar 22, 2020 at 12:45 AM Frank Millman  wrote:
> >
> > Hi all
> >
> > I have a strange intermittent bug.
> >
> > The role-players -
> >  asyncio on Python 3.8 running on Windows 10
> >  Microsoft Edge running as a browser on the same machine
> >
> > The bug does not occur with Python 3.7.
> > It does not occur with Chrome or Firefox.
> > It does not occur when MS Edge connects to another host on the network,
> > running the same Python program (Python 3.8 on Fedora 31).
>
> What exact version of Python 3.7 did you test? I'm looking through the
> changes to asyncio and came across this one, which may have some
> impact.
>
> https://bugs.python.org/issue36801
>
> Also this one made a change that introduced a regression that was
> subsequently fixed. Could be interesting.
>
> https://bugs.python.org/issue36802
>
> What happens if you try awaiting your writes? I think it probably
> won't make any difference, though - from my reading of the source, I
> believe that "await writer.write(...)" is the same as
> "writer.write(...); await writer.drain()", so it's going to be exactly
> the same as you're already doing.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Chris Angelico]

On Sun, Mar 22, 2020 at 12:45 AM Frank Millman  wrote:
>
> Hi all
>
> I have a strange intermittent bug.
>
> The role-players -
>  asyncio on Python 3.8 running on Windows 10
>  Microsoft Edge running as a browser on the same machine
>
> The bug does not occur with Python 3.7.
> It does not occur with Chrome or Firefox.
> It does not occur when MS Edge connects to another host on the network,
> running the same Python program (Python 3.8 on Fedora 31).

What exact version of Python 3.7 did you test? I'm looking through the
changes to asyncio and came across this one, which may have some
impact.

https://bugs.python.org/issue36801

Also this one made a change that introduced a regression that was
subsequently fixed. Could be interesting.

https://bugs.python.org/issue36802

What happens if you try awaiting your writes? I think it probably
won't make any difference, though - from my reading of the source, I
believe that "await writer.write(...)" is the same as
"writer.write(...); await writer.drain()", so it's going to be exactly
the same as you're already doing.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Chris Angelico]

On Sun, Mar 22, 2020 at 8:42 PM Frank Millman  wrote:
>
> On 2020-03-22 11:00 AM, Barry Scott wrote:
> >
> >
> >> On 22 Mar 2020, at 07:56, Frank Millman  wrote:
> >>
> >> On 2020-03-21 8:04 PM, Barry Scott wrote:
> >>> I'd look at the network traffic with wireshark to see if there is 
> >>> anything different between edge and the other browsers.
> >>
> >> You are leading me into deep waters here :-)  I have never used Wireshark 
> >> before. I have now downloaded it and am running it - it generates a *lot* 
> >> of data, most of which I do not understand yet!
> >
> > You can tell wireshark to only capture on one interface and to only capture 
> > packets for port 80.
> > (Captureing HTTPS means you cannot decode the packets without going deeper 
> > I recall)
> >
> > Then you can tell wireshark to decode the captured data for http to drop a 
> > lot of the lower level details.
> >
>
> Thanks. I am more or less doing that. Interestingly the [RST,ACK]
> messages appear on the tcp packets, so if I filter on http I do not see
> them.
>

I'm not 100% sure what "filter on HTTP" actually means, and it might
show only the data packets. Instead, filter on "from port 80 or to
port 80", which should show you the entire connection including the
SYN - SYN/ACK - ACK handshake, every data and acknowledgement packet,
and then whichever closing sequence gets used.

> >> I have another data point. I tried putting an asyncio.sleep() after 
> >> sending each file. A value of 0.01 made no difference, but a value of 0.1 
> >> makes the problem go away.
> >
> > What is the async wait to wait for the transmit buffers to drain?
> >
>
> Not sure what you are asking. I am just doing what it says in the docs -
>

I think his question is "what would asyncio be sleeping for, since you
already made it wait till it was drained". And my guess is that
there's a bug somewhere; the question is where.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Chris Angelico]

On Sun, Mar 22, 2020 at 8:30 PM Frank Millman  wrote:
>
> On 2020-03-22 10:45 AM, Chris Angelico wrote:
> > On Sun, Mar 22, 2020 at 6:58 PM Frank Millman  wrote:
> >>> I'd look at the network traffic with wireshark to see if there is 
> >>> anything different between edge and the other browsers.
> >>>
> >>
> >> You are leading me into deep waters here :-)  I have never used
> >> Wireshark before. I have now downloaded it and am running it - it
> >> generates a *lot* of data, most of which I do not understand yet!
> >>
> >> One thing immediately stands out. When I run it with MS Edge and
> >> Python3.8, it shows a lot of lines highlighted in red, with the symbols
> >> [RST,ACK]. They do not appear when running Chrome, and they do not
> >> appear when running Python3.7.
> >
> > Interesting. RST means "Reset" and is sent when the connection is
> > closed. Which direction were these packets sent (Edge to Python or
> > Python to Edge)? You can tell by the source and destination ports -
> > one of them is going to be the port Python is listening on (eg 80 or
> > 443), so if the destination port is 80, it's being sent *to* Python,
> > and if the source port is 80, it's being sent *from* Python.
> >
>
> They are all being sent *from* Python *to* Edge.

Very interesting indeed. What that *might* mean is that Python is
misinterpreting something and then believing that the connection has
been closed, so it responds "Okay, I'm closing the connection". Or
possibly it sees some sort of error condition.

> >> I have another data point. I tried putting an asyncio.sleep() after
> >> sending each file. A value of 0.01 made no difference, but a value of
> >> 0.1 makes the problem go away.
> >
> > Interesting also.
> >
> > Can you recreate the problem without Edge? It sounds like something's
> > going on with concurrent transfers, so it'd be awesome if you can
> > replace Edge with another Python program, and then post both programs.
> >
>
> Do you mean write a program that emulates a browser - make a connection,
> receive the HTML page, send a GET request for each file, and receive the
> results?
>
> I will give it a go!

Yes - although the HTML page is most likely irrelevant. You could just
make a connection and then spam requests. Or make multiple
connections.

Actually, that's another thing worth checking. Is Edge using a single
connection for all requests, or separate connections for each request,
or something in between (eg a pool of four connections and spreading
requests between them)? You'll be able to recognize different
connections by the port numbers Edge uses, which are guaranteed unique
among concurrent connections, and are most likely to not be reused for
a while even if the other is closed.

> > Also of interest: Does the problem go away if you change "Connection:
> > Keep-Alive" to "Connection: Close" in your headers?
> >
>
> Yes, the problem does go away.
>

This makes me think that the answer to the previous question is going
to involve some connection reuse.

If you can recreate the problem with a single socket and multiple
requests, that would be extremely helpful. I also think it's highly
likely that this is the case.

My theory: Your program is sending a large amount of data to the lower
level API functions, which attempt to send a large amount of data to
the socket. At some point, something gets told "sorry, can't handle
all that data, hold some of it back" at a time when it's not prepared
to do so, and it misinterprets it as an error. This error results in
the connection being closed.

But that's just a theory.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Frank Millman]

On 2020-03-22 11:00 AM, Barry Scott wrote:




On 22 Mar 2020, at 07:56, Frank Millman  wrote:

On 2020-03-21 8:04 PM, Barry Scott wrote:

I'd look at the network traffic with wireshark to see if there is anything 
different between edge and the other browsers.


You are leading me into deep waters here :-)  I have never used Wireshark 
before. I have now downloaded it and am running it - it generates a *lot* of 
data, most of which I do not understand yet!


You can tell wireshark to only capture on one interface and to only capture 
packets for port 80.
(Captureing HTTPS means you cannot decode the packets without going deeper I 
recall)

Then you can tell wireshark to decode the captured data for http to drop a lot 
of the lower level details.



Thanks. I am more or less doing that. Interestingly the [RST,ACK] 
messages appear on the tcp packets, so if I filter on http I do not see 
them.






One thing immediately stands out. When I run it with MS Edge and Python3.8, it 
shows a lot of lines highlighted in red, with the symbols [RST,ACK]. They do 
not appear when running Chrome, and they do not appear when running Python3.7.


As Chris said that should not happen.



As I replied to Chris, they appear in packets sent *from* Python *to* Edge.



I have another data point. I tried putting an asyncio.sleep() after sending 
each file. A value of 0.01 made no difference, but a value of 0.1 makes the 
problem go away.


What is the async wait to wait for the transmit buffers to drain?



Not sure what you are asking. I am just doing what it says in the docs -

=

write(data)
The method attempts to write the data to the underlying socket 
immediately. If that fails, the data is queued in an internal write 
buffer until it can be sent.


The method should be used along with the drain() method:

stream.write(data)
await stream.drain()

=

coroutine drain()
Wait until it is appropriate to resume writing to the stream. Example:

writer.write(data)
await writer.drain()
This is a flow control method that interacts with the underlying IO 
write buffer. When the size of the buffer reaches the high watermark, 
drain() blocks until the size of the buffer is drained down to the low 
watermark and writing can be resumed. When there is nothing to wait for, 
the drain() returns immediately.


=

Frank

--
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Frank Millman]

On 2020-03-22 10:45 AM, Chris Angelico wrote:

On Sun, Mar 22, 2020 at 6:58 PM Frank Millman  wrote:

I'd look at the network traffic with wireshark to see if there is anything 
different between edge and the other browsers.



You are leading me into deep waters here :-)  I have never used
Wireshark before. I have now downloaded it and am running it - it
generates a *lot* of data, most of which I do not understand yet!

One thing immediately stands out. When I run it with MS Edge and
Python3.8, it shows a lot of lines highlighted in red, with the symbols
[RST,ACK]. They do not appear when running Chrome, and they do not
appear when running Python3.7.


Interesting. RST means "Reset" and is sent when the connection is
closed. Which direction were these packets sent (Edge to Python or
Python to Edge)? You can tell by the source and destination ports -
one of them is going to be the port Python is listening on (eg 80 or
443), so if the destination port is 80, it's being sent *to* Python,
and if the source port is 80, it's being sent *from* Python.



They are all being sent *from* Python *to* Edge.


I have another data point. I tried putting an asyncio.sleep() after
sending each file. A value of 0.01 made no difference, but a value of
0.1 makes the problem go away.


Interesting also.

Can you recreate the problem without Edge? It sounds like something's
going on with concurrent transfers, so it'd be awesome if you can
replace Edge with another Python program, and then post both programs.



Do you mean write a program that emulates a browser - make a connection, 
receive the HTML page, send a GET request for each file, and receive the 
results?


I will give it a go!


Also of interest: Does the problem go away if you change "Connection:
Keep-Alive" to "Connection: Close" in your headers?



Yes, the problem does go away.

Frank

--
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Barry Scott]

> On 22 Mar 2020, at 07:56, Frank Millman  wrote:
> 
> On 2020-03-21 8:04 PM, Barry Scott wrote:
>>> On 21 Mar 2020, at 13:43, Frank Millman  wrote:
>>> 
>>> Hi all
>>> 
>>> I have a strange intermittent bug.
>>> 
>>> The role-players -
>>>asyncio on Python 3.8 running on Windows 10
>>>Microsoft Edge running as a browser on the same machine
>>> 
>>> The bug does not occur with Python 3.7.
>>> It does not occur with Chrome or Firefox.
>>> It does not occur when MS Edge connects to another host on the network, 
>>> running the same Python program (Python 3.8 on Fedora 31).
>>> 
>>> The symptoms -
>>>On receiving a connection, I send an HTML page to the browser,
>>>which has 20 lines like this -
>>> 
>>>
>>>
>>>...
>>> 
>>> Intermittently, one or other of the script files is not received by MS Edge.
>>> 
> [...]
>>> >> I don't know whether the problem lies with Python or MS Edge, but as 
> it does not happen with Python 3.7, I am suspecting that something changed in 
> 3.8 which does not match MS Edge's expectations.
>> I'd look at the network traffic with wireshark to see if there is anything 
>> different between edge and the other browsers.
> 
> You are leading me into deep waters here :-)  I have never used Wireshark 
> before. I have now downloaded it and am running it - it generates a *lot* of 
> data, most of which I do not understand yet!

You can tell wireshark to only capture on one interface and to only capture 
packets for port 80.
(Captureing HTTPS means you cannot decode the packets without going deeper I 
recall)

Then you can tell wireshark to decode the captured data for http to drop a lot 
of the lower level details.


> 
> One thing immediately stands out. When I run it with MS Edge and Python3.8, 
> it shows a lot of lines highlighted in red, with the symbols [RST,ACK]. They 
> do not appear when running Chrome, and they do not appear when running 
> Python3.7.

As Chris said that should not happen.

> 
> I have another data point. I tried putting an asyncio.sleep() after sending 
> each file. A value of 0.01 made no difference, but a value of 0.1 makes the 
> problem go away.

What is the async wait to wait for the transmit buffers to drain?

> 
> I will keep digging, but I thought I would post this information now in case 
> it helps with diagnosis.
> 
> Frank
> -- 
> https://mail.python.org/mailman/listinfo/python-list 
> 
Barry
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Chris Angelico]

On Sun, Mar 22, 2020 at 6:58 PM Frank Millman  wrote:
> > I'd look at the network traffic with wireshark to see if there is anything 
> > different between edge and the other browsers.
> >
>
> You are leading me into deep waters here :-)  I have never used
> Wireshark before. I have now downloaded it and am running it - it
> generates a *lot* of data, most of which I do not understand yet!
>
> One thing immediately stands out. When I run it with MS Edge and
> Python3.8, it shows a lot of lines highlighted in red, with the symbols
> [RST,ACK]. They do not appear when running Chrome, and they do not
> appear when running Python3.7.

Interesting. RST means "Reset" and is sent when the connection is
closed. Which direction were these packets sent (Edge to Python or
Python to Edge)? You can tell by the source and destination ports -
one of them is going to be the port Python is listening on (eg 80 or
443), so if the destination port is 80, it's being sent *to* Python,
and if the source port is 80, it's being sent *from* Python.

> I have another data point. I tried putting an asyncio.sleep() after
> sending each file. A value of 0.01 made no difference, but a value of
> 0.1 makes the problem go away.

Interesting also.

Can you recreate the problem without Edge? It sounds like something's
going on with concurrent transfers, so it'd be awesome if you can
replace Edge with another Python program, and then post both programs.

Also of interest: Does the problem go away if you change "Connection:
Keep-Alive" to "Connection: Close" in your headers?

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Frank Millman]

On 2020-03-21 8:04 PM, Barry Scott wrote:




On 21 Mar 2020, at 13:43, Frank Millman  wrote:

Hi all

I have a strange intermittent bug.

The role-players -
asyncio on Python 3.8 running on Windows 10
Microsoft Edge running as a browser on the same machine

The bug does not occur with Python 3.7.
It does not occur with Chrome or Firefox.
It does not occur when MS Edge connects to another host on the network, running 
the same Python program (Python 3.8 on Fedora 31).

The symptoms -
On receiving a connection, I send an HTML page to the browser,
which has 20 lines like this -



...

Intermittently, one or other of the script files is not received by MS Edge.


[...]
>> I don't know whether the problem lies with Python or MS Edge, but as 
it does not happen with Python 3.7, I am suspecting that something 
changed in 3.8 which does not match MS Edge's expectations.


I'd look at the network traffic with wireshark to see if there is anything 
different between edge and the other browsers.



You are leading me into deep waters here :-)  I have never used 
Wireshark before. I have now downloaded it and am running it - it 
generates a *lot* of data, most of which I do not understand yet!


One thing immediately stands out. When I run it with MS Edge and 
Python3.8, it shows a lot of lines highlighted in red, with the symbols 
[RST,ACK]. They do not appear when running Chrome, and they do not 
appear when running Python3.7.


I have another data point. I tried putting an asyncio.sleep() after 
sending each file. A value of 0.01 made no difference, but a value of 
0.1 makes the problem go away.


I will keep digging, but I thought I would post this information now in 
case it helps with diagnosis.


Frank

--
https://mail.python.org/mailman/listinfo/python-list

Re: Intermittent bug with asyncio and MS Edge

[Barry Scott]

> On 21 Mar 2020, at 13:43, Frank Millman  wrote:
> 
> Hi all
> 
> I have a strange intermittent bug.
> 
> The role-players -
>asyncio on Python 3.8 running on Windows 10
>Microsoft Edge running as a browser on the same machine
> 
> The bug does not occur with Python 3.7.
> It does not occur with Chrome or Firefox.
> It does not occur when MS Edge connects to another host on the network, 
> running the same Python program (Python 3.8 on Fedora 31).
> 
> The symptoms -
>On receiving a connection, I send an HTML page to the browser,
>which has 20 lines like this -
> 
>
>
>...
> 
> Intermittently, one or other of the script files is not received by MS Edge.
> 
> I have checked the Network tab in Developer Tools in MS Edge. It shows the 
> first few requests getting a Status 200 OK, then some are shown as 'Pending'. 
> This seems to be where the problem occurs.
> 
> I am hoping that someone can give me some hints about how to debug this.
> 
> My function to send the script file looks like this -
> 
>async def send_file(writer, fname):
> 
>status = 'HTTP/1.1 200 OK\r\n'
>writer.write(status.encode())
> 
>headers = []
>headers.append(('CONNECTION', 'keep-alive'))
>headers.append(('DATE',
>email.utils.formatdate(usegmt=True)))
>headers.append(('SERVER',
>   f'Python {sys.version.split()[0]} asyncio'))
>headers.append(('Content-type', 'text/javascript'))
>headers('Transfer-Encoding', 'chunked'))
>for key, val in headers:
>writer.write(f'{key}: {val}\r\n'.encode())
>writer.write('\r\n'.encode())
>await writer.drain()
> 
>with open(fname 'rb') as fd:
>chunk = fd.read(8192)
>while chunk:
>writer.write(hex(len(chunk))[2:].encode() + b'\r\n')
>writer.write(chunk + b'\r\n')
>await writer.drain()
>chunk = fd.read(8192)
>writer.write(b'0\r\n\r\n')
>await writer.drain()
> 
>writer.close()
>await writer.wait_closed()
> 
> I have asked the same question on StackOverflow, from an MS Edge perspective -
> https://stackoverflow.com/questions/60785767/ms-edge-randomly-does-not-load-script
> 
> I don't know whether the problem lies with Python or MS Edge, but as it does 
> not happen with Python 3.7, I am suspecting that something changed in 3.8 
> which does not match MS Edge's expectations.

I'd look at the network  traffic with wireshark to see if there is anything 
different between edge and the other browsers.

Aside: headers are case blind, but usually written capitalised. Some buggy 
software expects that its capitalised.

I wonder if it will start working if you do not do chunked (your chunked 
encoding looks correct).
You can use os.stat to find the size of the file for the Content-Length header.

Barry



> 
> Any hints much appreciated.
> 
> Frank Millman
> 
> -- 
> https://mail.python.org/mailman/listinfo/python-list
> 

-- 
https://mail.python.org/mailman/listinfo/python-list