I have a couple more questions: - Did you get a jstack of the process? If so I assume you saw lots of Netty threads like "New I/O boss", "New I/O worker", etc. because of having many KuduClient instances. Is that right? - Just curious: are your edge node clients in the same data center as Kudu or are you going across the WAN with your client API writes? This should not affect client threads but has application architecture implications (i.e. are you buffering or dropping events at the edge node?) when the WAN link or the Kudu service is unavailable for some reason.
In general, we recommend sharing Kudu client instances to avoid too many threads. A single Kudu client and Netty setup should be able to handle all the threads in the process. An example of this is the static Kudu client cache we use for the Spark integration at https://github.com/apache/kudu/blob/master/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala#L445 Hope that helps, Mike On Thu, Jan 17, 2019 at 11:52 AM Alexey Serbin <[email protected]> wrote: > Hi Boris, > > Kudu servers have a setting for connection inactivity period: idle > connections to the servers will be automatically closed after the specified > time (--rpc_default_keepalive_time_ms is the flag). So, from that > perspective idle clients is not a big concern to the Kudu server side. > > As for your question, right now Kudu doesn't have a way to initiate a > shutdown of an idle client from the server side. > > BTW, I'm curious what it was in your case you reported: were there too > many idle Kudu client objects around created by the same application? Or > that was something else, like a single idle Kudu Java client that created > so many threads? > > > Thanks, > > Alexey > > On Wed, Jan 16, 2019 at 1:31 PM Boris Tyukin <[email protected]> > wrote: > >> sorry it is Java >> >> On Wed, Jan 16, 2019 at 3:32 PM Mike Percy <[email protected]> wrote: >> >>> Java or C++ / Python client? >>> >>> Mike >>> >>> Sent from my iPhone >>> >>> > On Jan 16, 2019, at 12:27 PM, Boris Tyukin <[email protected]> >>> wrote: >>> > >>> > Hi guys, >>> > >>> > is there a setting on Kudu server to close/clean-up inactive Kudu >>> clients? >>> > >>> > we just found some rogue code that did not close client on code >>> completion and wondering if we can prevent this in future on Kudu server >>> level rather than relying on good developers. >>> > >>> > That code caused 22,000 threads opened on our edge node over the last >>> few days. >>> > >>> > Boris >>> >>>
