> We have a single broker Kafka isn't really designed to run with a single broker setup. Whilst I doubt it impacts your problems here, you should immediately bump your broker count to 3 or more.
Our performance tests were run with the JVM producer and consumer, and we didn't notice any real slowdown on that side either when enabling SSL. Just to check the obvious: are your producers and consumers going over a fast connection to the broker? > This GC you notice, is it on the broker(s) or on the clients which hosts the producer/consumers? Either way, this is a good hint and something I hadn't considered in my setup. I'll see if I can get some GC logs/metrics on our setup for this. Thanks for the input. On the broker - we never looked at consumers or producers, but that might be a good place to look. On Tue, Sep 6, 2016 at 5:40 AM, Jaikiran Pai <jai.forums2...@gmail.com> wrote: > On Tuesday 06 September 2016 09:42 AM, Jaikiran Pai wrote: > >> Hi Todd, >> >> Note that this relies on the fix in KAFKA-4050 and switching the PRNG to >>> SHA1PRNG. >>> >> >> Thanks, I hadn't noticed that JIRA. I'll make sure we use it in our >> setup. However, right now, (intentionally) we have single broker within the >> setup and based on the JIRA details, I believe this one won't affect us for >> now. >> > > I read that JIRA again and saw the stacktrace. I might be wrong when I > said this probably doesn't affect us given that the code path appears to > the same for other SSL interactions. I'm going to introduce this patch (or > in fact the latest 0.10 branch) in our setup and see how it goes. > > -Jaikiran > > > > >> >> >>> Right now, we're specifically avoiding moving consume traffic to SSL, due >>> to the zero copy send issue. Now I've been told (but I have not >>> investigated) that OpenSSL can solve this. It would probably be a good >>> use >>> of time to look into that further. >>> >> >> Unfortunately, the way our application is written, we don't have the >> ability to enable SSL for producers and disable it for consumers. Changing >> the application to support something like that is doable, but I will have >> to make sure that this is one of the reasons for the slowness. >> >> >> Thanks for all your inputs. I was trying to avoid coming up with a >> reproducer which closely matches our application usage scenario (given our >> tight delivery timelines for the product), but from what you say and a >> reply Tom Crayford in this thread, I am starting to feel that not everyone >> is affected to the extent we are. I will see if I can narrow to this down >> to something more concrete. >> >> -Jaikiran >> >> >> >> >> >>> That said, switching the message format to the newer option (KIP-31 I >>> believe?) will result in the brokers not needing to recompress message >>> batches that are produced. This should result in a significant reduction >>> in >>> CPU usage, which may offset the cost of SSL. We haven't had a chance to >>> fully investigate this, however, as changing that config depends on the >>> clients being updated to support the new format. >>> >>> -Todd >>> >>> On Sunday, September 4, 2016, Jaikiran Pai <jai.forums2...@gmail.com> >>> wrote: >>> >>> We are using 0.10.0.1 of Kafka and (Java) client libraries. We recently >>>> decided to start using SSL for Kafka communication between broker and >>>> clients. Right now, we have a pretty basic setup with just 1 broker with >>>> SSL keystore setup and the Java client(s) communicate using the >>>> Producer/Consumer APIs against this single broker. There's no client >>>> auth >>>> (intentionally) right now. We also have plain text enabled for the >>>> initial >>>> testing. >>>> >>>> What we have noticed is that the consumer/producer performance when SSL >>>> is >>>> enabled is noticeably poor when compared to plain text. I understand >>>> that >>>> there are expected to be performance impacts when SSL is enabled but the >>>> order of magnitude is too high and in fact it shows up in a very >>>> noticeable >>>> fashion in our product. I do have the numbers, but I haven't been able >>>> to >>>> narrow it down yet (because there's nothing that stands out, except that >>>> it's slow). Our application code is exactly the same between non-SSL and >>>> SSL usage. >>>> >>>> Furthermore, I'm aware of this specific JIRA in Kafka >>>> https://issues.apache.org/jira/browse/KAFKA-2561 which acknowledges a >>>> similar issue. So what I would like to know is, in context of Kafka >>>> 0.10.x >>>> releases and Java 8 support, are there any timelines that the dev team >>>> is >>>> looking for in terms of improving this performance issue (which I >>>> believe >>>> requires usage of OpenSSL or other non-JDK implementations of >>>> SSLEngine)? >>>> We would like to go for GA of our product in the next couple of months >>>> and >>>> in order to do that, we do plan to have Kafka over SSL working with >>>> reasonably good performance, but the current performance isn't >>>> promising. >>>> Expecting this to be fixed in the next couple of months and have it >>>> available in 0.10.x is probably too much to expect, but if we know the >>>> plans around this, we should be able to come up with a plan of our own >>>> for >>>> our product. >>>> >>>> >>>> -Jaikiran >>>> >>>> >>> >> >
