On Tue, Sep 15, 2020 at 5:08 PM Martin Grigorov <mgrigo...@apache.org> wrote:
> Hi Mark, > > On Tue, Sep 15, 2020 at 3:34 PM Mark Thomas <ma...@apache.org> wrote: > > > On 15/09/2020 12:46, Martin Grigorov wrote: > > > On Tue, Sep 15, 2020 at 2:37 PM Martin Grigorov <mgrigo...@apache.org> > > > wrote: > > > > > >> Hi, > > >> > > >> I am running some load tests on Tomcat and I've noticed that when > HTTP2 > > is > > >> enabled the throughput drops considerably. > > >> > > >> Here are the steps to reproduce: > > >> > > >> 1) Enable HTTP2, e.g. by commenting out this connector: > > >> > > >> > > > https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112 > > >> > > >> 2) Download Vegeta load tool from: > > >> https://github.com/tsenart/vegeta/releases/ > > >> > > >> 3) Run the load tests: > > >> > > >> 3.1) HTTP/1.1 > > >> echo -e '{"method": "GET", "url": "http://localhost:8080/examples/"}' > | > > >> vegeta attack -format=json -rate=0 -max-workers=1000 -duration=10s | > > >> vegeta encode > /tmp/http1.json; and vegeta report -type=json > > >> /tmp/http1.json | jq . > > >> > > >> 3.2) HTTP2 > > >> echo -e '{"method": "GET", "url": "https://localhost:8443/examples/ > "}' > > | > > >> vegeta attack -format=json -http2 -rate=0 -max-workers=1000 -insecure > > >> -duration=10s | vegeta encode > /tmp/http2.json; and vegeta report > > >> -type=json /tmp/http2.json | jq . > > >> > > >> As explained at https://github.com/tsenart/vegeta#-rate -rate=0 means > > >> that Vegeta will try to send as many requests as possible with the > > >> configured number of workers. > > >> I use '-insecure' because I use self-signed certificate. > > >> > > >> On my machine I get around 14-15K reqs/sec for HTTP1.1 with only > > responses > > >> with code=200 . > > >> But for HTTP2 Tomcat starts returning such kind of errors: > > >> > > >> "errors": [ > > >> "Get \"https://localhost:8443/examples/\": http2: server sent > > GOAWAY > > >> and closed the connection; LastStreamID=9259, ErrCode=PROTOCOL_ERROR, > > >> debug=\"Stream [9,151] has been closed for some time\"", > > >> "http2: server sent GOAWAY and closed the connection; > > >> LastStreamID=9259, ErrCode=PROTOCOL_ERROR, debug=\"Stream [9,151] has > > been > > >> closed for some time\"", > > >> "Get \"https://localhost:8443/examples/\": http2: server sent > > GOAWAY > > >> and closed the connection; LastStreamID=239, ErrCode=PROTOCOL_ERROR, > > >> debug=\"Stream [49] has been closed for some time\"" > > >> ] > > >> > > >> when I ask for more than 2000 reqs/sec, i.e. -rate=2000/1s > > > > That indicates that the client has sent a frame associated with a stream > > that the server closed previously and that that stream has been removed > > from the Map of known streams to make room for new ones. See > > Http2UpgardeHandler.pruneClosedStreams() > > > > It looks like the client is making assumptions about server behaviour > > that go beyond the requirements of RFC 7540, section 5.3.4. > > > > This is possible! > I've just tested with two more HTTP2 impls: > > 1) Node.js > > http2-server.js > =================================================== > const http2 = require('http2'); > const fs = require('fs'); > > const server = http2.createSecureServer({ > key: fs.readFileSync('/path/to/server.key'), > cert: fs.readFileSync('/path/to/server.crt') > }); > server.on('error', (err) => console.error(err)); > > server.on('stream', (stream, headers) => { > // stream is a Duplex > stream.respond({ > 'content-type': 'text/plain; charset=utf-8', > ':status': 200 > }); > stream.end('Hello world!'); > }); > > server.listen(18080); > =================================================== > > run with: node http2-server.js > > Runs fine with -rate=0 and gives around 8K reqs/sec > > 2) Rust > > Cargo.toml > =================================================== > [package] > name = "my-http2-server" > version = "0.0.1" > publish = false > authors = ["Martin Grigorov <mgrigo...@apache.org>"] > license = "MIT/Apache-2.0" > description = "Load test HTTP/2 " > repository = "https://github.com/martin-g/http2-server-rust"; > keywords = ["http2"] > edition = "2018" > > [dependencies] > actix-web = { version = "3", features = ["openssl"] } > openssl = { version = "0.10", features = ["v110"] } > =================================================== > > src/main.rs > =================================================== > use actix_web::{web, App, HttpRequest, HttpServer, Responder}; > use openssl::ssl::{SslAcceptor, SslFiletype, SslMethod}; > > async fn index(_req: HttpRequest) -> impl Responder { > "Hello world!" > } > > #[actix_web::main] > async fn main() -> std::io::Result<()> { > let mut builder = > SslAcceptor::mozilla_intermediate(SslMethod::tls()).unwrap(); > builder > .set_private_key_file("/path/to/server.key", SslFiletype::PEM) > .unwrap(); > builder.set_certificate_chain_file("/path/to/server.crt").unwrap(); > > HttpServer::new(|| App::new().route("/", web::get().to(index))) > .bind_openssl("127.0.0.1:18080", builder)? > .run() > .await > } > =================================================== > > run with: cargo run > Again no errors, throughput: 3K reqs/sec > > I will test with Netty tomorrow too, but so far it looks like only Tomcat > fails under load. > > > > > >> All the access logs look like: > > >> > > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" > 200 > > >> 1126 > > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" > 200 > > >> 1126 > > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" > 200 > > >> 1126 > > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" > 200 > > >> 1126 > > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" > 200 > > >> 1126 > > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" > 200 > > >> 1126 > > >> > > >> i.e. there are no error codes, just 200. > > >> Vegeta reports the error with status code = 0. I think this just means > > >> that it didn't get a proper HTTP response but just TCP error. > > >> There are no errors in catalina.out. > > >> > > >> Are there any settings I can tune to get better throughput with HTTP2 > ? > > >> > > >> Tomcat 10.0.0-M8. > > > > If you really want to maximise throughput then you need to reduce the > > number of concurrent requests to (or a little above) the number of cores > > available on the server. Go higher and you'll start to see throughput > > tail off due to context switching. > > > > If you want to demonstrate throughput with a large number of clients > > you'll probably need to experiment with both maxThreads, > > > > I've forgot to say that I use maxThreads=8. > > > > maxConcurrentStreams and maxConcurrentStreamExecution. > > > > If I had to guess, I'd expect maxConcurrentStreams == > > maxConcurrentStreamExecution and low numbers for all of them to give the > > best results. > > > > I will check those tomorrow! > I use h2load, and IMO I have very good performance with h2c. No errors. However, same as your tool, it's concurrency flag only controls how many streams it will use *inside a single connection* [I respect that, but I would have expected instead the tool would also use multiple connections in an attempt to simulate multiple users, but nope]. This can only go so far and the performance numbers caused me to add the flags to limit stream concurrency. Rémy
