On Tuesday 06 September 2016 09:42 AM, Jaikiran Pai wrote:
Hi Todd,
Note that this relies on the fix in KAFKA-4050 and switching the PRNG to
SHA1PRNG.
Thanks, I hadn't noticed that JIRA. I'll make sure we use it in our
setup. However, right now, (intentionally) we have single broker
within the setup and based on the JIRA details, I believe this one
won't affect us for now.
I read that JIRA again and saw the stacktrace. I might be wrong when I
said this probably doesn't affect us given that the code path appears to
the same for other SSL interactions. I'm going to introduce this patch
(or in fact the latest 0.10 branch) in our setup and see how it goes.
-Jaikiran
Right now, we're specifically avoiding moving consume traffic to SSL,
due
to the zero copy send issue. Now I've been told (but I have not
investigated) that OpenSSL can solve this. It would probably be a
good use
of time to look into that further.
Unfortunately, the way our application is written, we don't have the
ability to enable SSL for producers and disable it for consumers.
Changing the application to support something like that is doable, but
I will have to make sure that this is one of the reasons for the
slowness.
Thanks for all your inputs. I was trying to avoid coming up with a
reproducer which closely matches our application usage scenario (given
our tight delivery timelines for the product), but from what you say
and a reply Tom Crayford in this thread, I am starting to feel that
not everyone is affected to the extent we are. I will see if I can
narrow to this down to something more concrete.
-Jaikiran
That said, switching the message format to the newer option (KIP-31 I
believe?) will result in the brokers not needing to recompress message
batches that are produced. This should result in a significant
reduction in
CPU usage, which may offset the cost of SSL. We haven't had a chance to
fully investigate this, however, as changing that config depends on the
clients being updated to support the new format.
-Todd
On Sunday, September 4, 2016, Jaikiran Pai <jai.forums2...@gmail.com>
wrote:
We are using 0.10.0.1 of Kafka and (Java) client libraries. We recently
decided to start using SSL for Kafka communication between broker and
clients. Right now, we have a pretty basic setup with just 1 broker
with
SSL keystore setup and the Java client(s) communicate using the
Producer/Consumer APIs against this single broker. There's no client
auth
(intentionally) right now. We also have plain text enabled for the
initial
testing.
What we have noticed is that the consumer/producer performance when
SSL is
enabled is noticeably poor when compared to plain text. I understand
that
there are expected to be performance impacts when SSL is enabled but
the
order of magnitude is too high and in fact it shows up in a very
noticeable
fashion in our product. I do have the numbers, but I haven't been
able to
narrow it down yet (because there's nothing that stands out, except
that
it's slow). Our application code is exactly the same between non-SSL
and
SSL usage.
Furthermore, I'm aware of this specific JIRA in Kafka
https://issues.apache.org/jira/browse/KAFKA-2561 which acknowledges a
similar issue. So what I would like to know is, in context of Kafka
0.10.x
releases and Java 8 support, are there any timelines that the dev
team is
looking for in terms of improving this performance issue (which I
believe
requires usage of OpenSSL or other non-JDK implementations of
SSLEngine)?
We would like to go for GA of our product in the next couple of
months and
in order to do that, we do plan to have Kafka over SSL working with
reasonably good performance, but the current performance isn't
promising.
Expecting this to be fixed in the next couple of months and have it
available in 0.10.x is probably too much to expect, but if we know the
plans around this, we should be able to come up with a plan of our
own for
our product.
-Jaikiran
