D2392: debugcommands: add debugwireproto command
This revision was automatically updated to reflect the committed changes. Closed by commit rHG36f21b975efb: debugcommands: add debugwireproto command (authored by indygreg, committed by ). REPOSITORY rHG Mercurial CHANGES SINCE LAST UPDATE https://phab.mercurial-scm.org/D2392?vs=6250=6283 REVISION DETAIL https://phab.mercurial-scm.org/D2392 AFFECTED FILES mercurial/debugcommands.py tests/test-completion.t tests/test-help.t tests/test-ssh-proto.t CHANGE DETAILS diff --git a/tests/test-ssh-proto.t b/tests/test-ssh-proto.t --- a/tests/test-ssh-proto.t +++ b/tests/test-ssh-proto.t @@ -12,10 +12,29 @@ $ echo 0 > foo $ hg -q add foo $ hg commit -m initial - $ cd .. + +A no-op connection performs a handshake + + $ hg debugwireproto --localssh << EOF + > EOF + creating ssh peer from handshake results + +Raw peers don't perform any activity + + $ hg debugwireproto --localssh --peer raw << EOF + > EOF + using raw connection to peer + $ hg debugwireproto --localssh --peer ssh1 << EOF + > EOF + creating ssh peer for wire protocol version 1 + $ hg debugwireproto --localssh --peer ssh2 << EOF + > EOF + creating ssh peer for wire protocol version 2 Test a normal behaving server, for sanity + $ cd .. + $ hg --debug debugpeer ssh://user@dummy/server running * "*/tests/dummyssh" 'user@dummy' 'hg -R server serve --stdio' (glob) (no-windows !) running * "*\tests/dummyssh" "user@dummy" "hg -R server serve --stdio" (glob) (windows !) @@ -33,11 +52,19 @@ Server should answer the "hello" command in isolation - $ hg -R server serve --stdio << EOF - > hello + $ hg -R server debugwireproto --localssh --peer raw << EOF + > raw + > hello\n + > readline + > readline > EOF - 384 - capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN + using raw connection to peer + i> write(6) -> None: + i> hello\n + o> readline() -> 4: + o> 384\n + o> readline() -> 384: + o> capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN\n `hg debugserve --sshstdio` works @@ -80,16 +107,33 @@ Server should reply with capabilities and should send "1\n\n" as a successful reply with empty response to the "between". - $ hg -R server serve --stdio << EOF - > hello - > between - > pairs 81 - > - + $ hg -R server debugwireproto --localssh --peer raw << EOF + > raw + > hello\n + > readline + > readline + > raw + > between\n + > pairs 81\n + > - + > readline + > readline > EOF - 384 - capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN - 1 - + using raw connection to peer + i> write(6) -> None: + i> hello\n + o> readline() -> 4: + o> 384\n + o> readline() -> 384: + o> capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN\n + i> write(98) -> None: + i> between\n + i> pairs 81\n + i> - + o> readline() -> 2: + o> 1\n + o> readline() -> 1: + o> \n SSH banner is not printed by default, ignored by clients @@ -127,26 +171,63 @@ And test the banner with the raw protocol - $ SSHSERVERMODE=banner hg -R server serve --stdio << EOF - > hello - > between - > pairs 81 - > - + $ SSHSERVERMODE=banner hg -R server debugwireproto --localssh --peer raw << EOF + > raw + > hello\n + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > raw + > between\n + > pairs 81\n + > - + > readline + > readline > EOF - banner: line 0 - banner: line 1 - banner: line 2 - banner: line 3 - banner: line 4 - banner: line 5 - banner: line 6 - banner: line 7 - banner: line 8 - banner: line 9 - 384 - capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN - 1 - + using raw connection to peer + i> write(6) -> None: + i> hello\n + o> readline() -> 15: + o> banner: line 0\n + o> readline() -> 15:
D2392: debugcommands: add debugwireproto command
indygreg updated this revision to Diff 6250. REPOSITORY rHG Mercurial CHANGES SINCE LAST UPDATE https://phab.mercurial-scm.org/D2392?vs=6139=6250 REVISION DETAIL https://phab.mercurial-scm.org/D2392 AFFECTED FILES mercurial/debugcommands.py tests/test-completion.t tests/test-help.t tests/test-ssh-proto.t CHANGE DETAILS diff --git a/tests/test-ssh-proto.t b/tests/test-ssh-proto.t --- a/tests/test-ssh-proto.t +++ b/tests/test-ssh-proto.t @@ -12,10 +12,29 @@ $ echo 0 > foo $ hg -q add foo $ hg commit -m initial - $ cd .. + +A no-op connection performs a handshake + + $ hg debugwireproto --localssh << EOF + > EOF + creating ssh peer from handshake results + +Raw peers don't perform any activity + + $ hg debugwireproto --localssh --peer raw << EOF + > EOF + using raw connection to peer + $ hg debugwireproto --localssh --peer ssh1 << EOF + > EOF + creating ssh peer for wire protocol version 1 + $ hg debugwireproto --localssh --peer ssh2 << EOF + > EOF + creating ssh peer for wire protocol version 2 Test a normal behaving server, for sanity + $ cd .. + $ hg --debug debugpeer ssh://user@dummy/server running * "*/tests/dummyssh" 'user@dummy' 'hg -R server serve --stdio' (glob) (no-windows !) running * "*\tests/dummyssh" "user@dummy" "hg -R server serve --stdio" (glob) (windows !) @@ -33,11 +52,19 @@ Server should answer the "hello" command in isolation - $ hg -R server serve --stdio << EOF - > hello + $ hg -R server debugwireproto --localssh --peer raw << EOF + > raw + > hello\n + > readline + > readline > EOF - 384 - capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN + using raw connection to peer + i> write(6) -> None: + i> hello\n + o> readline() -> 4: + o> 384\n + o> readline() -> 384: + o> capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN\n `hg debugserve --sshstdio` works @@ -80,16 +107,33 @@ Server should reply with capabilities and should send "1\n\n" as a successful reply with empty response to the "between". - $ hg -R server serve --stdio << EOF - > hello - > between - > pairs 81 - > - + $ hg -R server debugwireproto --localssh --peer raw << EOF + > raw + > hello\n + > readline + > readline + > raw + > between\n + > pairs 81\n + > - + > readline + > readline > EOF - 384 - capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN - 1 - + using raw connection to peer + i> write(6) -> None: + i> hello\n + o> readline() -> 4: + o> 384\n + o> readline() -> 384: + o> capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN\n + i> write(98) -> None: + i> between\n + i> pairs 81\n + i> - + o> readline() -> 2: + o> 1\n + o> readline() -> 1: + o> \n SSH banner is not printed by default, ignored by clients @@ -127,26 +171,63 @@ And test the banner with the raw protocol - $ SSHSERVERMODE=banner hg -R server serve --stdio << EOF - > hello - > between - > pairs 81 - > - + $ SSHSERVERMODE=banner hg -R server debugwireproto --localssh --peer raw << EOF + > raw + > hello\n + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > raw + > between\n + > pairs 81\n + > - + > readline + > readline > EOF - banner: line 0 - banner: line 1 - banner: line 2 - banner: line 3 - banner: line 4 - banner: line 5 - banner: line 6 - banner: line 7 - banner: line 8 - banner: line 9 - 384 - capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN - 1 - + using raw connection to peer + i> write(6) -> None: + i> hello\n + o> readline() -> 15: + o> banner: line 0\n + o> readline() -> 15: + o> banner: line 1\n + o> readline() -> 15: + o> banner: line 2\n + o> readline() -> 15: + o> banner: line 3\n + o>
D2392: debugcommands: add debugwireproto command
indygreg added a comment. In https://phab.mercurial-scm.org/D2392#40790, @yuja wrote: > This is OT, but I couldn't find a way to phabread this long series. `:D2462` (the topmost patch > in yadda) appeared to lack many changes, and `:D2392` missed some dependency patches. > And I gave up. `hg phabsend` doesn't appear to update parent revisions when re-submitting??? Anyway, I updated parent revisions manually through the web UI and I think things are all cleaned up. REPOSITORY rHG Mercurial REVISION DETAIL https://phab.mercurial-scm.org/D2392 To: indygreg, #hg-reviewers, yuja Cc: yuja, sid0, mercurial-devel ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D2392: debugcommands: add debugwireproto command
yuja requested changes to this revision. yuja added a comment. This revision now requires changes to proceed. This is OT, but I couldn't find a way to phabread this long series. `:D2462` (the topmost patch in yadda) appeared to lack many changes, and `:D2392` missed some dependency patches. And I gave up. INLINE COMMENTS > debugcommands.py:2661 > +# shared state from interfering with server operation. > +args = [util.hgexecutable(), '-R', repo.root, 'debugserve', > +'--sshstdio'] Perhaps `util.hgcmd()` is more appropriate because we don't run the command by the system shell. > debugcommands.py:2707 > +data = ''.join(l.lstrip() for l in lines) > +data = ast.literal_eval(b'''b"%s"''' % data) > +stdin.write(data) util.unescapestr() can be used if we want to allow only string literals. REPOSITORY rHG Mercurial REVISION DETAIL https://phab.mercurial-scm.org/D2392 To: indygreg, #hg-reviewers, yuja Cc: yuja, sid0, mercurial-devel ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D2392: debugcommands: add debugwireproto command
indygreg updated this revision to Diff 6139. indygreg edited the summary of this revision. REPOSITORY rHG Mercurial CHANGES SINCE LAST UPDATE https://phab.mercurial-scm.org/D2392?vs=6005=6139 REVISION DETAIL https://phab.mercurial-scm.org/D2392 AFFECTED FILES mercurial/debugcommands.py tests/test-completion.t tests/test-help.t tests/test-ssh-proto.t CHANGE DETAILS diff --git a/tests/test-ssh-proto.t b/tests/test-ssh-proto.t --- a/tests/test-ssh-proto.t +++ b/tests/test-ssh-proto.t @@ -12,10 +12,29 @@ $ echo 0 > foo $ hg -q add foo $ hg commit -m initial - $ cd .. + +A no-op connection performs a handshake + + $ hg debugwireproto --localssh << EOF + > EOF + creating ssh peer from handshake results + +Raw peers don't perform any activity + + $ hg debugwireproto --localssh --peer raw << EOF + > EOF + using raw connection to peer + $ hg debugwireproto --localssh --peer ssh1 << EOF + > EOF + creating ssh peer for wire protocol version 1 + $ hg debugwireproto --localssh --peer ssh2 << EOF + > EOF + creating ssh peer for wire protocol version 2 Test a normal behaving server, for sanity + $ cd .. + $ hg --debug debugpeer ssh://user@dummy/server running * "*/tests/dummyssh" 'user@dummy' 'hg -R server serve --stdio' (glob) (no-windows !) running * "*\tests/dummyssh" "user@dummy" "hg -R server serve --stdio" (glob) (windows !) @@ -33,11 +52,19 @@ Server should answer the "hello" command in isolation - $ hg -R server serve --stdio << EOF - > hello + $ hg -R server debugwireproto --localssh --peer raw << EOF + > raw + > hello\n + > readline + > readline > EOF - 384 - capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN + using raw connection to peer + i> write(6) -> None: + i> hello\n + o> readline() -> 4: + o> 384\n + o> readline() -> 384: + o> capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN\n `hg debugserve --sshstdio` works @@ -80,16 +107,33 @@ Server should reply with capabilities and should send "1\n\n" as a successful reply with empty response to the "between". - $ hg -R server serve --stdio << EOF - > hello - > between - > pairs 81 - > - + $ hg -R server debugwireproto --localssh --peer raw << EOF + > raw + > hello\n + > readline + > readline + > raw + > between\n + > pairs 81\n + > - + > readline + > readline > EOF - 384 - capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN - 1 - + using raw connection to peer + i> write(6) -> None: + i> hello\n + o> readline() -> 4: + o> 384\n + o> readline() -> 384: + o> capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN\n + i> write(98) -> None: + i> between\n + i> pairs 81\n + i> - + o> readline() -> 2: + o> 1\n + o> readline() -> 1: + o> \n SSH banner is not printed by default, ignored by clients @@ -127,26 +171,63 @@ And test the banner with the raw protocol - $ SSHSERVERMODE=banner hg -R server serve --stdio << EOF - > hello - > between - > pairs 81 - > - + $ SSHSERVERMODE=banner hg -R server debugwireproto --localssh --peer raw << EOF + > raw + > hello\n + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > readline + > raw + > between\n + > pairs 81\n + > - + > readline + > readline > EOF - banner: line 0 - banner: line 1 - banner: line 2 - banner: line 3 - banner: line 4 - banner: line 5 - banner: line 6 - banner: line 7 - banner: line 8 - banner: line 9 - 384 - capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ unbundle=HG10GZ,HG10BZ,HG10UN - 1 - + using raw connection to peer + i> write(6) -> None: + i> hello\n + o> readline() -> 15: + o> banner: line 0\n + o> readline() -> 15: + o> banner: line 1\n + o> readline() -> 15: + o> banner: line 2\n + o> readline() ->
D2392: debugcommands: add debugwireproto command
indygreg planned changes to this revision. indygreg added inline comments. INLINE COMMENTS > debugcommands.py:2630 > +# separation. This prevents a whole class of potential bugs around > +# shared state from interfering with server operation. > + I'm having second thoughts about this. The reason is that from a testing perspective (which is the primary driver behind this work), `read()` isn't very reliable because of timing issues. Depending on operating system settings, system performance, etc, operations like `read(-1)` can return a variable number of bytes because they return only what's available on the wire. `write()`, however, is more reliable. When you `write()` to something in Python, Python makes as many system calls as necessary to ensure all bytes are delivered. So a `write()` at the Python level is mostly deterministic. I think the concerns around process separation here aren't that significant. So I think I'm going to rework this (yet again) to spawn the SSH server in process and to only monitor I/O operations that are deterministic. This may mean only monitoring `write()` calls on pipes and *possibly* monitoring `readline()` and `read(N)`. But if we monitor `write()` on both peers since they are both in-process, then `read()` monitoring is redundant. That could be useful to debug behavior. But for tests demonstrating the wire protocol exchange, it's less useful. REPOSITORY rHG Mercurial REVISION DETAIL https://phab.mercurial-scm.org/D2392 To: indygreg, #hg-reviewers Cc: sid0, mercurial-devel ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D2392: debugcommands: add debugwireproto command
indygreg added a subscriber: sid0. indygreg added a comment. @sid0: I reckon we could find a way to take what I did in this commit and turn it into a wire protocol conformance tester. i.e. if we could refactor the test so it instantiates the server via alternate mechanisms, we could point the test at a separate server implementation as a means to tease out implementation differences. Just thought you'd like to know in case Mononoke would be interested in this. REPOSITORY rHG Mercurial REVISION DETAIL https://phab.mercurial-scm.org/D2392 To: indygreg, #hg-reviewers Cc: sid0, mercurial-devel ___ Mercurial-devel mailing list Mercurial-devel@mercurial-scm.org https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
D2392: debugcommands: add debugwireproto command
indygreg created this revision. Herald added a subscriber: mercurial-devel. Herald added a reviewer: hg-reviewers. REVISION SUMMARY We currently don't have a low-level mechanism for sending arbitrary wire protocol commands. Having a generic and robust mechanism for sending wire protocol commands, examining wire data, etc would make it vastly easier to test the wire protocol and debug server operation. This is a problem I've wanted a solution for numerous times, especially recently as I've been hacking on a new version of the wire protocol. This commit establishes a `hg debugwireproto` command for sending data to a peer. The command invents a mini language for specifying actions to take. This will enable a lot of flexibility for issuing commands and testing variations for how commands are sent. Right now, we only support low-level raw sends and receives. These are probably the least valuable commands to intended users of this command. But they are the most useful commands to implement to bootstrap the feature (I've chosen to reimplement test-ssh-proto.t using this command to prove its usefulness). We invent a mechanism to wrap a file object so we can observe activity on it. We have similar functionality in badserverext.py, but that's a test-only extension and is pretty specific to HTTP server. The new functionality needed to live in core because it needs to be accessible to `hg debugwireproto`. My eventual goal of `hg debugwireproto` is to allow calling wire protocol commands with a human-friendly interface. Essentially, people can type in a command name and arguments and `hg debugwireproto` will figure out how to send that on the wire. I'd love to eventually be able to save the server's raw response to a file. This would allow us to e.g. call "getbundle" wire protocol commands easily. test-ssh-proto.t has been updated to use the new command in lieu of piping directly to a server process. As part of the transition, test behavior improved. Before, we piped all request data to the server at once. Now, we have explicit control over the ordering of operations. e.g. we can send one command, receive its response, then send another command. This will allow us to more robustly test race conditions, buffering behavior, etc. There were some subtle changes in test behavior. For example, previous behavior would often send trailing newlines to the server. The new mechanism doesn't treat literal newlines specially and requires newlines be escaped in the payload. Because the new logging code is very low level, it is easy to introduce race conditions in tests. For example, the number of bytes returned by a read() may vary depending on load. This is why tests make heavy use of "readline" for consuming data: the result of that operation should be deterministic and not subject to race conditions. There are still some uses of "readavailable." However, those are only for reading from stderr. I was able to reproduce timing issues with my system under load when using "readavailable" globally. But if I "readline" to grab stdout, "readavailable" appears to work deterministically for stderr. I think this is because the server writes to stderr first. As long as the OS delivers writes to pipes in the same order they were made, this should work. If there are timing issues, we can introduce a mechanism to readline from stderr. REPOSITORY rHG Mercurial REVISION DETAIL https://phab.mercurial-scm.org/D2392 AFFECTED FILES mercurial/debugcommands.py mercurial/util.py tests/test-completion.t tests/test-help.t tests/test-ssh-proto.t CHANGE DETAILS diff --git a/tests/test-ssh-proto.t b/tests/test-ssh-proto.t --- a/tests/test-ssh-proto.t +++ b/tests/test-ssh-proto.t @@ -12,10 +12,29 @@ $ echo 0 > foo $ hg -q add foo $ hg commit -m initial - $ cd .. + +A no-op connection performs a handshake + + $ hg debugwireproto --localssh << EOF + > EOF + creating ssh peer from handshake results + +Raw peers don't perform any activity + + $ hg debugwireproto --localssh --peer raw << EOF + > EOF + using raw connection to peer + $ hg debugwireproto --localssh --peer ssh1 << EOF + > EOF + creating ssh peer for wire protocol version 1 + $ hg debugwireproto --localssh --peer ssh2 << EOF + > EOF + creating ssh peer for wire protocol version 2 Test a normal behaving server, for sanity + $ cd .. + $ hg --debug debugpeer ssh://user@dummy/server running * "*/tests/dummyssh" 'user@dummy' 'hg -R server serve --stdio' (glob) (no-windows !) running * "*\tests/dummyssh" "user@dummy" "hg -R server serve --stdio" (glob) (windows !) @@ -33,26 +52,51 @@ Server should answer the "hello" command in isolation - $ hg -R server serve --stdio << EOF - > hello + $ hg -R server debugwireproto --localssh --peer raw << EOF + > raw + > hello\n + > readline + > readline > EOF - 384
