D2871: wireproto: service multiple command requests per HTTP request

2018-03-24 Thread yuja (Yuya Nishihara) 
yuja added inline comments.

INLINE COMMENTS

> wireprotoserver.py:557
>  elif action == 'noop':
>  pass
>  else:

Nit: `return False` instead of returning None?

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D2871

To: indygreg, #hg-reviewers, durin42
Cc: yuja, durin42, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D2871: wireproto: service multiple command requests per HTTP request

2018-03-21 Thread indygreg (Gregory Szorc) 
This revision was automatically updated to reflect the committed changes.
Closed by commit rHGbbea991635d0: wireproto: service multiple command requests 
per HTTP request (authored by indygreg, committed by ).

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D2871?vs=7147=7235

REVISION DETAIL
  https://phab.mercurial-scm.org/D2871

AFFECTED FILES
  mercurial/help/internals/wireprotocol.txt
  mercurial/wireprotoserver.py
  tests/test-http-api-httpv2.t

CHANGE DETAILS

diff --git a/tests/test-http-api-httpv2.t b/tests/test-http-api-httpv2.t
--- a/tests/test-http-api-httpv2.t
+++ b/tests/test-http-api-httpv2.t
@@ -412,4 +412,153 @@
   s> received: \n
   s> {"action": "noop"}
 
+Multiple requests to regular command URL are not allowed
+
+  $ send << EOF
+  > httprequest POST api/$HTTPV2/ro/customreadonly
+  > accept: $MEDIATYPE
+  > content-type: $MEDIATYPE
+  > user-agent: test
+  > frame 1 command-name eos customreadonly
+  > frame 3 command-name eos customreadonly
+  > EOF
+  using raw connection to peer
+  s> POST /api/exp-http-v2-0001/ro/customreadonly HTTP/1.1\r\n
+  s> Accept-Encoding: identity\r\n
+  s> accept: application/mercurial-exp-framing-0002\r\n
+  s> content-type: application/mercurial-exp-framing-0002\r\n
+  s> user-agent: test\r\n
+  s> content-length: 40\r\n
+  s> host: $LOCALIP:$HGPORT\r\n (glob)
+  s> \r\n
+  s> 
\x0e\x00\x00\x01\x00\x11customreadonly\x0e\x00\x00\x03\x00\x11customreadonly
+  s> makefile('rb', None)
+  s> HTTP/1.1 200 OK\r\n
+  s> Server: testing stub value\r\n
+  s> Date: $HTTP_DATE$\r\n
+  s> Content-Type: text/plain\r\n
+  s> Content-Length: 46\r\n
+  s> \r\n
+  s> multiple commands cannot be issued to this URL
+
+Multiple requests to "multirequest" URL are allowed
+
+  $ send << EOF
+  > httprequest POST api/$HTTPV2/ro/multirequest
+  > accept: $MEDIATYPE
+  > content-type: $MEDIATYPE
+  > user-agent: test
+  > frame 1 command-name eos customreadonly
+  > frame 3 command-name eos customreadonly
+  > EOF
+  using raw connection to peer
+  s> POST /api/exp-http-v2-0001/ro/multirequest HTTP/1.1\r\n
+  s> Accept-Encoding: identity\r\n
+  s> accept: application/mercurial-exp-framing-0002\r\n
+  s> content-type: application/mercurial-exp-framing-0002\r\n
+  s> user-agent: test\r\n
+  s> *\r\n (glob)
+  s> host: $LOCALIP:$HGPORT\r\n (glob)
+  s> \r\n
+  s> 
\x0e\x00\x00\x01\x00\x11customreadonly\x0e\x00\x00\x03\x00\x11customreadonly
+  s> makefile('rb', None)
+  s> HTTP/1.1 200 OK\r\n
+  s> Server: testing stub value\r\n
+  s> Date: $HTTP_DATE$\r\n
+  s> Content-Type: application/mercurial-exp-framing-0002\r\n
+  s> Transfer-Encoding: chunked\r\n
+  s> \r\n
+  s> *\r\n (glob)
+  s> \x1d\x00\x00\x01\x00Bcustomreadonly bytes response
+  s> \r\n
+  s> 23\r\n
+  s> \x1d\x00\x00\x03\x00Bcustomreadonly bytes response
+  s> \r\n
+  s> 0\r\n
+  s> \r\n
+
+Interleaved requests to "multirequest" are processed
+
+  $ send << EOF
+  > httprequest POST api/$HTTPV2/ro/multirequest
+  > accept: $MEDIATYPE
+  > content-type: $MEDIATYPE
+  > user-agent: test
+  > frame 1 command-name have-args listkeys
+  > frame 3 command-name have-args listkeys
+  > frame 3 command-argument eoa \x09\x00\x09\x00namespacebookmarks
+  > frame 1 command-argument eoa \x09\x00\x0a\x00namespacenamespaces
+  > EOF
+  using raw connection to peer
+  s> POST /api/exp-http-v2-0001/ro/multirequest HTTP/1.1\r\n
+  s> Accept-Encoding: identity\r\n
+  s> accept: application/mercurial-exp-framing-0002\r\n
+  s> content-type: application/mercurial-exp-framing-0002\r\n
+  s> user-agent: test\r\n
+  s> content-length: 85\r\n
+  s> host: $LOCALIP:$HGPORT\r\n (glob)
+  s> \r\n
+  s> 
\x08\x00\x00\x01\x00\x12listkeys\x08\x00\x00\x03\x00\x12listkeys\x16\x00\x00\x03\x00"
 \x00\x00namespacebookmarks\x17\x00\x00\x01\x00" \x00\n
+  s> \x00namespacenamespaces
+  s> makefile('rb', None)
+  s> HTTP/1.1 200 OK\r\n
+  s> Server: testing stub value\r\n
+  s> Date: $HTTP_DATE$\r\n
+  s> Content-Type: application/mercurial-exp-framing-0002\r\n
+  s> Transfer-Encoding: chunked\r\n
+  s> \r\n
+  s> 6\r\n
+  s> \x00\x00\x00\x03\x00B
+  s> \r\n
+  s> 24\r\n
+  s> \x1e\x00\x00\x01\x00Bbookmarks\n
+  s> namespaces\n
+  s> phases
+  s> \r\n
+  s> 0\r\n
+  s> \r\n
+
+Restart server to disable read-write access
+
+  $ killdaemons.py
+  $ cat > server/.hg/hgrc << EOF
+  > [experimental]
+  > web.apiserver = true
+  > web.api.debugreflect = true
+  > web.api.http-v2 = true
+  > [web]
+  > push_ssl = false
+  > EOF
+
+  $ hg -R server serve -p $HGPORT -d --pid-file hg.pid -E error.log
+  $ cat hg.pid > $DAEMON_PIDS
+
+Attempting to run a read-write command

D2871: wireproto: service multiple command requests per HTTP request

2018-03-21 Thread durin42 (Augie Fackler) 
durin42 added inline comments.

INLINE COMMENTS

> wireprotoserver.py:508
> +
> +assert authedperm in (b'ro', b'rw')
> +wirecommand = wireproto.commands[command['command']]

worth not using assert here? I don't think this is attacker-controlled?

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D2871

To: indygreg, #hg-reviewers
Cc: durin42, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D2871: wireproto: service multiple command requests per HTTP request

2018-03-19 Thread indygreg (Gregory Szorc) 
indygreg updated this revision to Diff 7147.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D2871?vs=7055=7147

REVISION DETAIL
  https://phab.mercurial-scm.org/D2871

AFFECTED FILES
  mercurial/help/internals/wireprotocol.txt
  mercurial/wireprotoserver.py
  tests/test-http-api-httpv2.t

CHANGE DETAILS

diff --git a/tests/test-http-api-httpv2.t b/tests/test-http-api-httpv2.t
--- a/tests/test-http-api-httpv2.t
+++ b/tests/test-http-api-httpv2.t
@@ -412,4 +412,153 @@
   s> received: \n
   s> {"action": "noop"}
 
+Multiple requests to regular command URL are not allowed
+
+  $ send << EOF
+  > httprequest POST api/$HTTPV2/ro/customreadonly
+  > accept: $MEDIATYPE
+  > content-type: $MEDIATYPE
+  > user-agent: test
+  > frame 1 command-name eos customreadonly
+  > frame 3 command-name eos customreadonly
+  > EOF
+  using raw connection to peer
+  s> POST /api/exp-http-v2-0001/ro/customreadonly HTTP/1.1\r\n
+  s> Accept-Encoding: identity\r\n
+  s> accept: application/mercurial-exp-framing-0002\r\n
+  s> content-type: application/mercurial-exp-framing-0002\r\n
+  s> user-agent: test\r\n
+  s> content-length: 40\r\n
+  s> host: $LOCALIP:$HGPORT\r\n (glob)
+  s> \r\n
+  s> 
\x0e\x00\x00\x01\x00\x11customreadonly\x0e\x00\x00\x03\x00\x11customreadonly
+  s> makefile('rb', None)
+  s> HTTP/1.1 200 OK\r\n
+  s> Server: testing stub value\r\n
+  s> Date: $HTTP_DATE$\r\n
+  s> Content-Type: text/plain\r\n
+  s> Content-Length: 46\r\n
+  s> \r\n
+  s> multiple commands cannot be issued to this URL
+
+Multiple requests to "multirequest" URL are allowed
+
+  $ send << EOF
+  > httprequest POST api/$HTTPV2/ro/multirequest
+  > accept: $MEDIATYPE
+  > content-type: $MEDIATYPE
+  > user-agent: test
+  > frame 1 command-name eos customreadonly
+  > frame 3 command-name eos customreadonly
+  > EOF
+  using raw connection to peer
+  s> POST /api/exp-http-v2-0001/ro/multirequest HTTP/1.1\r\n
+  s> Accept-Encoding: identity\r\n
+  s> accept: application/mercurial-exp-framing-0002\r\n
+  s> content-type: application/mercurial-exp-framing-0002\r\n
+  s> user-agent: test\r\n
+  s> *\r\n (glob)
+  s> host: $LOCALIP:$HGPORT\r\n (glob)
+  s> \r\n
+  s> 
\x0e\x00\x00\x01\x00\x11customreadonly\x0e\x00\x00\x03\x00\x11customreadonly
+  s> makefile('rb', None)
+  s> HTTP/1.1 200 OK\r\n
+  s> Server: testing stub value\r\n
+  s> Date: $HTTP_DATE$\r\n
+  s> Content-Type: application/mercurial-exp-framing-0002\r\n
+  s> Transfer-Encoding: chunked\r\n
+  s> \r\n
+  s> *\r\n (glob)
+  s> \x1d\x00\x00\x01\x00Bcustomreadonly bytes response
+  s> \r\n
+  s> 23\r\n
+  s> \x1d\x00\x00\x03\x00Bcustomreadonly bytes response
+  s> \r\n
+  s> 0\r\n
+  s> \r\n
+
+Interleaved requests to "multirequest" are processed
+
+  $ send << EOF
+  > httprequest POST api/$HTTPV2/ro/multirequest
+  > accept: $MEDIATYPE
+  > content-type: $MEDIATYPE
+  > user-agent: test
+  > frame 1 command-name have-args listkeys
+  > frame 3 command-name have-args listkeys
+  > frame 3 command-argument eoa \x09\x00\x09\x00namespacebookmarks
+  > frame 1 command-argument eoa \x09\x00\x0a\x00namespacenamespaces
+  > EOF
+  using raw connection to peer
+  s> POST /api/exp-http-v2-0001/ro/multirequest HTTP/1.1\r\n
+  s> Accept-Encoding: identity\r\n
+  s> accept: application/mercurial-exp-framing-0002\r\n
+  s> content-type: application/mercurial-exp-framing-0002\r\n
+  s> user-agent: test\r\n
+  s> content-length: 85\r\n
+  s> host: $LOCALIP:$HGPORT\r\n (glob)
+  s> \r\n
+  s> 
\x08\x00\x00\x01\x00\x12listkeys\x08\x00\x00\x03\x00\x12listkeys\x16\x00\x00\x03\x00"
 \x00\x00namespacebookmarks\x17\x00\x00\x01\x00" \x00\n
+  s> \x00namespacenamespaces
+  s> makefile('rb', None)
+  s> HTTP/1.1 200 OK\r\n
+  s> Server: testing stub value\r\n
+  s> Date: $HTTP_DATE$\r\n
+  s> Content-Type: application/mercurial-exp-framing-0002\r\n
+  s> Transfer-Encoding: chunked\r\n
+  s> \r\n
+  s> 6\r\n
+  s> \x00\x00\x00\x03\x00B
+  s> \r\n
+  s> 24\r\n
+  s> \x1e\x00\x00\x01\x00Bbookmarks\n
+  s> namespaces\n
+  s> phases
+  s> \r\n
+  s> 0\r\n
+  s> \r\n
+
+Restart server to disable read-write access
+
+  $ killdaemons.py
+  $ cat > server/.hg/hgrc << EOF
+  > [experimental]
+  > web.apiserver = true
+  > web.api.debugreflect = true
+  > web.api.http-v2 = true
+  > [web]
+  > push_ssl = false
+  > EOF
+
+  $ hg -R server serve -p $HGPORT -d --pid-file hg.pid -E error.log
+  $ cat hg.pid > $DAEMON_PIDS
+
+Attempting to run a read-write command via multirequest on read-only URL is 
not allowed
+
+  $ send << EOF
+  > httprequest POST api/$HTTPV2/ro/multirequest
+  > accept: $MEDIATYPE
+  >

D2871: wireproto: service multiple command requests per HTTP request

2018-03-14 Thread indygreg (Gregory Szorc) 
indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  Now that our new frame-based protocol server can understand how
  to ingest multiple, possibly interleaved, command requests, let's
  hook it up to the HTTP server.
  
  The code on the HTTP side of things is still a bit hacky. We need
  a bit of work around error handling, content types, etc. But it's
  a start.
  
  Among the added tests, we demonstrate that a client can send frames
  for multiple commands iterleaved with each other and that a later
  issued command can respond before the first one has finished
  sending. This makes our multi-request model technically superior
  to the previous "batch" command.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D2871

AFFECTED FILES
  mercurial/help/internals/wireprotocol.txt
  mercurial/wireprotoserver.py
  tests/test-http-api-httpv2.t

CHANGE DETAILS

diff --git a/tests/test-http-api-httpv2.t b/tests/test-http-api-httpv2.t
--- a/tests/test-http-api-httpv2.t
+++ b/tests/test-http-api-httpv2.t
@@ -397,4 +397,153 @@
   s> received: \n
   s> {"action": "noop"}
 
+Multiple requests to regular command URL are not allowed
+
+  $ send << EOF
+  > httprequest POST api/$HTTPV2/ro/capabilities
+  > accept: $MEDIATYPE
+  > content-type: $MEDIATYPE
+  > user-agent: test
+  > frame 1 command-name eos capabilities
+  > frame 3 command-name eos capabilities
+  > EOF
+  using raw connection to peer
+  s> POST /api/exp-http-v2-0001/ro/capabilities HTTP/1.1\r\n
+  s> Accept-Encoding: identity\r\n
+  s> accept: application/mercurial-exp-framing-0002\r\n
+  s> content-type: application/mercurial-exp-framing-0002\r\n
+  s> user-agent: test\r\n
+  s> content-length: 36\r\n
+  s> host: $LOCALIP:$HGPORT\r\n (glob)
+  s> \r\n
+  s> 
\x0c\x00\x00\x01\x00\x11capabilities\x0c\x00\x00\x03\x00\x11capabilities
+  s> makefile('rb', None)
+  s> HTTP/1.1 200 OK\r\n
+  s> Server: testing stub value\r\n
+  s> Date: $HTTP_DATE$\r\n
+  s> Content-Type: text/plain\r\n
+  s> Content-Length: 46\r\n
+  s> \r\n
+  s> multiple commands cannot be issued to this URL
+
+Multiple requests to "multirequest" URL are allowed
+
+  $ send << EOF
+  > httprequest POST api/$HTTPV2/ro/multirequest
+  > accept: $MEDIATYPE
+  > content-type: $MEDIATYPE
+  > user-agent: test
+  > frame 1 command-name eos capabilities
+  > frame 3 command-name eos capabilities
+  > EOF
+  using raw connection to peer
+  s> POST /api/exp-http-v2-0001/ro/multirequest HTTP/1.1\r\n
+  s> Accept-Encoding: identity\r\n
+  s> accept: application/mercurial-exp-framing-0002\r\n
+  s> content-type: application/mercurial-exp-framing-0002\r\n
+  s> user-agent: test\r\n
+  s> content-length: 36\r\n
+  s> host: $LOCALIP:$HGPORT\r\n (glob)
+  s> \r\n
+  s> 
\x0c\x00\x00\x01\x00\x11capabilities\x0c\x00\x00\x03\x00\x11capabilities
+  s> makefile('rb', None)
+  s> HTTP/1.1 200 OK\r\n
+  s> Server: testing stub value\r\n
+  s> Date: $HTTP_DATE$\r\n
+  s> Content-Type: application/mercurial-exp-framing-0002\r\n
+  s> Transfer-Encoding: chunked\r\n
+  s> \r\n
+  s> *\r\n (glob)
+  s> Y\x01\x00\x01\x00Blookup branchmap pushkey known getbundle 
unbundlehash streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ 
unbundle=HG10GZ,HG10BZ,HG10UN
+  s> \r\n
+  s> *\r\n (glob)
+  s> Y\x01\x00\x03\x00Blookup branchmap pushkey known getbundle 
unbundlehash streamreqs=generaldelta,revlogv1 $USUAL_BUNDLE2_CAPS_SERVER$ 
unbundle=HG10GZ,HG10BZ,HG10UN
+  s> \r\n
+  s> 0\r\n
+  s> \r\n
+
+Interleaved requests to "multirequest" are processed
+
+  $ send << EOF
+  > httprequest POST api/$HTTPV2/ro/multirequest
+  > accept: $MEDIATYPE
+  > content-type: $MEDIATYPE
+  > user-agent: test
+  > frame 1 command-name have-args listkeys
+  > frame 3 command-name have-args listkeys
+  > frame 3 command-argument eoa \x09\x00\x09\x00namespacebookmarks
+  > frame 1 command-argument eoa \x09\x00\x0a\x00namespacenamespaces
+  > EOF
+  using raw connection to peer
+  s> POST /api/exp-http-v2-0001/ro/multirequest HTTP/1.1\r\n
+  s> Accept-Encoding: identity\r\n
+  s> accept: application/mercurial-exp-framing-0002\r\n
+  s> content-type: application/mercurial-exp-framing-0002\r\n
+  s> user-agent: test\r\n
+  s> content-length: 85\r\n
+  s> host: $LOCALIP:$HGPORT\r\n (glob)
+  s> \r\n
+  s> 
\x08\x00\x00\x01\x00\x12listkeys\x08\x00\x00\x03\x00\x12listkeys\x16\x00\x00\x03\x00"
 \x00\x00namespacebookmarks\x17\x00\x00\x01\x00" \x00\n
+  s> \x00namespacenamespaces
+  s> makefile('rb', None)
+  s> HTTP/1.1 200 OK\r\n
+  s> Server: testing stub value\r\n
+  s> Date: $HTTP_DATE$\r\n
+  s> Content-Type: