Hi Remy, On Tue, Sep 15, 2020 at 6:16 PM Rémy Maucherat <r...@apache.org> wrote:
> On Tue, Sep 15, 2020 at 5:08 PM Martin Grigorov <mgrigo...@apache.org> > wrote: > > > Hi Mark, > > > > On Tue, Sep 15, 2020 at 3:34 PM Mark Thomas <ma...@apache.org> wrote: > > > > > On 15/09/2020 12:46, Martin Grigorov wrote: > > > > On Tue, Sep 15, 2020 at 2:37 PM Martin Grigorov < > mgrigo...@apache.org> > > > > wrote: > > > > > > > >> Hi, > > > >> > > > >> I am running some load tests on Tomcat and I've noticed that when > > HTTP2 > > > is > > > >> enabled the throughput drops considerably. > > > >> > > > >> Here are the steps to reproduce: > > > >> > > > >> 1) Enable HTTP2, e.g. by commenting out this connector: > > > >> > > > >> > > > > > > https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112 > > > >> > > > >> 2) Download Vegeta load tool from: > > > >> https://github.com/tsenart/vegeta/releases/ > > > >> > > > >> 3) Run the load tests: > > > >> > > > >> 3.1) HTTP/1.1 > > > >> echo -e '{"method": "GET", "url": "http://localhost:8080/examples/ > "}' > > | > > > >> vegeta attack -format=json -rate=0 -max-workers=1000 -duration=10s > | > > > >> vegeta encode > /tmp/http1.json; and vegeta report -type=json > > > >> /tmp/http1.json | jq . > > > >> > > > >> 3.2) HTTP2 > > > >> echo -e '{"method": "GET", "url": "https://localhost:8443/examples/ > > "}' > > > | > > > >> vegeta attack -format=json -http2 -rate=0 -max-workers=1000 > -insecure > > > >> -duration=10s | vegeta encode > /tmp/http2.json; and vegeta report > > > >> -type=json /tmp/http2.json | jq . > > > >> > > > >> As explained at https://github.com/tsenart/vegeta#-rate -rate=0 > means > > > >> that Vegeta will try to send as many requests as possible with the > > > >> configured number of workers. > > > >> I use '-insecure' because I use self-signed certificate. > > > >> > > > >> On my machine I get around 14-15K reqs/sec for HTTP1.1 with only > > > responses > > > >> with code=200 . > > > >> But for HTTP2 Tomcat starts returning such kind of errors: > > > >> > > > >> "errors": [ > > > >> "Get \"https://localhost:8443/examples/\": http2: server sent > > > GOAWAY > > > >> and closed the connection; LastStreamID=9259, > ErrCode=PROTOCOL_ERROR, > > > >> debug=\"Stream [9,151] has been closed for some time\"", > > > >> "http2: server sent GOAWAY and closed the connection; > > > >> LastStreamID=9259, ErrCode=PROTOCOL_ERROR, debug=\"Stream [9,151] > has > > > been > > > >> closed for some time\"", > > > >> "Get \"https://localhost:8443/examples/\": http2: server sent > > > GOAWAY > > > >> and closed the connection; LastStreamID=239, ErrCode=PROTOCOL_ERROR, > > > >> debug=\"Stream [49] has been closed for some time\"" > > > >> ] > > > >> > > > >> when I ask for more than 2000 reqs/sec, i.e. -rate=2000/1s > > > > > > That indicates that the client has sent a frame associated with a > stream > > > that the server closed previously and that that stream has been removed > > > from the Map of known streams to make room for new ones. See > > > Http2UpgardeHandler.pruneClosedStreams() > > > > > > It looks like the client is making assumptions about server behaviour > > > that go beyond the requirements of RFC 7540, section 5.3.4. > > > > > > > This is possible! > > I've just tested with two more HTTP2 impls: > > > > 1) Node.js > > > > http2-server.js > > =================================================== > > const http2 = require('http2'); > > const fs = require('fs'); > > > > const server = http2.createSecureServer({ > > key: fs.readFileSync('/path/to/server.key'), > > cert: fs.readFileSync('/path/to/server.crt') > > }); > > server.on('error', (err) => console.error(err)); > > > > server.on('stream', (stream, headers) => { > > // stream is a Duplex > > stream.respond({ > > 'content-type': 'text/plain; charset=utf-8', > > ':status': 200 > > }); > > stream.end('Hello world!'); > > }); > > > > server.listen(18080); > > =================================================== > > > > run with: node http2-server.js > > > > Runs fine with -rate=0 and gives around 8K reqs/sec > > > > 2) Rust > > > > Cargo.toml > > =================================================== > > [package] > > name = "my-http2-server" > > version = "0.0.1" > > publish = false > > authors = ["Martin Grigorov <mgrigo...@apache.org>"] > > license = "MIT/Apache-2.0" > > description = "Load test HTTP/2 " > > repository = "https://github.com/martin-g/http2-server-rust"; > > keywords = ["http2"] > > edition = "2018" > > > > [dependencies] > > actix-web = { version = "3", features = ["openssl"] } > > openssl = { version = "0.10", features = ["v110"] } > > =================================================== > > > > src/main.rs > > =================================================== > > use actix_web::{web, App, HttpRequest, HttpServer, Responder}; > > use openssl::ssl::{SslAcceptor, SslFiletype, SslMethod}; > > > > async fn index(_req: HttpRequest) -> impl Responder { > > "Hello world!" > > } > > > > #[actix_web::main] > > async fn main() -> std::io::Result<()> { > > let mut builder = > > SslAcceptor::mozilla_intermediate(SslMethod::tls()).unwrap(); > > builder > > .set_private_key_file("/path/to/server.key", SslFiletype::PEM) > > .unwrap(); > > builder.set_certificate_chain_file("/path/to/server.crt").unwrap(); > > > > HttpServer::new(|| App::new().route("/", web::get().to(index))) > > .bind_openssl("127.0.0.1:18080", builder)? > > .run() > > .await > > } > > =================================================== > > > > run with: cargo run > > Again no errors, throughput: 3K reqs/sec > > > > I will test with Netty tomorrow too, but so far it looks like only Tomcat > > fails under load. > > > > > > > > > >> All the access logs look like: > > > >> > > > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" > > 200 > > > >> 1126 > > > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" > > 200 > > > >> 1126 > > > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" > > 200 > > > >> 1126 > > > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" > > 200 > > > >> 1126 > > > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" > > 200 > > > >> 1126 > > > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" > > 200 > > > >> 1126 > > > >> > > > >> i.e. there are no error codes, just 200. > > > >> Vegeta reports the error with status code = 0. I think this just > means > > > >> that it didn't get a proper HTTP response but just TCP error. > > > >> There are no errors in catalina.out. > > > >> > > > >> Are there any settings I can tune to get better throughput with > HTTP2 > > ? > > > >> > > > >> Tomcat 10.0.0-M8. > > > > > > If you really want to maximise throughput then you need to reduce the > > > number of concurrent requests to (or a little above) the number of > cores > > > available on the server. Go higher and you'll start to see throughput > > > tail off due to context switching. > > > > > > If you want to demonstrate throughput with a large number of clients > > > you'll probably need to experiment with both maxThreads, > > > > > > > I've forgot to say that I use maxThreads=8. > > > > > > > maxConcurrentStreams and maxConcurrentStreamExecution. > > > > > > If I had to guess, I'd expect maxConcurrentStreams == > > > maxConcurrentStreamExecution and low numbers for all of them to give > the > > > best results. > > > > > > > I will check those tomorrow! > > > > I use h2load, and IMO I have very good performance with h2c. No errors. > However, same as your tool, it's concurrency flag only controls how many > streams it will use *inside a single connection* [I respect that, but I > I think this is not the case here. As far as I can see Vegeta does not use custom ClientConnPool and the default pool in net::http2 shares many connections per host:port key: https://github.com/golang/net/blob/62affa334b73ec65ed44a326519ac12c421905e3/http2/client_conn_pool.go > would have expected instead the tool would also use multiple connections in > an attempt to simulate multiple users, but nope]. This can only go so far > and the performance numbers caused me to add the flags to limit stream > concurrency. > > Rémy >
