Hi,
Let me start by saying that gRPC is awesome :) Now to my issue:
I have implemented a gRPC server that has a single streaming RPC, in C++
(just like the getting started tutorials).
Then, I started multiple channels and connected to the server to see how
many connections it can handle, before even dealing with the streams, along
the lines of (one std::thread per connection):
std::shared_ptr<grpc::Channel>
> channel = grpc::CreateCustomChannel("localhost:8080", credentials,
> channel_arguments);
> while (true) {
> channel->GetState(true); // Tries to connect.
> absl::SleepFor(absl::Seconds(1));
> }
I started 1000 concurrent connections, and I can verify they are connected
with:
$ netstat -anp |grep 57751 |wc -l
> (Not all processes could be identified, non-owned process info
> will not be shown, you would have to be root to see it all.)
> 3964
The problem is, as I increase the number of concurrent connections, the
server crashes with:
terminate called after throwing an instance of 'std::system_error'
> what(): Resource temporarily unavailable
I imagine that is due the socket limits of my OS (Ubuntu 18.04), which is
set around 1024 (ulimit -n, unsure why it allowed 3964 connections but the
magnitude makes sense). Even if I raise that limit, it is too easy to just
increase the number of concurrent connections and make it crash again
(tried with 100k). My use case is that I want to bring up a service that
will have very few servers running, but, it seems too easy to bring it down
if I do the described above.
My questions are:
- Is there a way to handle the incoming connections at the gRPC library
level?
- Is there a way to make gRPC not crash and instead just not accept new
connections if the limit is reached?
- Would using iptables and limiting connections by IP be a temporary
solution?
- Related to the above, could I set some parameter that would disconnect
connection from channels at the server side that aren't activated (e.g.: by
calling a stream)? That way I could also use iptables and limit the max
number of connections from an IP to X per second (better than limiting the
number of connections by IP, I think).
- Are there other ways and gRPC best practices to prevent abuse or a
buggy client that could start too many connections? (The docs in grpc.io
don't cover that).
- Are there any open issues in GitHub about this problem?
Thanks in advance,
Bruno
--
You received this message because you are subscribed to the Google Groups
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/grpc-io/0076d023-331a-4413-9d97-e5677f458f72%40googlegroups.com.
