So as the next step I see to increase the partition of the 2nd topic - do I
increase the instances of the consumer from that or keep it at 7?
Anything else (besides researching those libs)?
Are there any good tools for load testing kafka?
On Sun, Dec 20, 2020 at 7:23 PM Haruki Okada wrote:
> It
Hi Ning and all,
We got a crude way to solve this issue. Below are the high level steps:
Read the message from Replicator's internal topic for storing offsets.
[connect-offsets]
This topic stores the offsets for all topics which is getting replicated in
key:value pair . For e.g.
Key : ["replica
It depends on how you manually commit offsets.
Auto-commit does commits offsets in async manner basically, so as long as
you do manual-commit in the same way, there should be no much difference.
And, generally offset-commit mode doesn't make much difference in
performance regardless manual/auto o
Thank you so much Marina and Haruka.
Marina's response:
- When you say " if you are sure there is no room for perf optimization of
the processing itself :" - do you mean code level optimizations? Can you
please explain?
- On the second topic you say " I'd say at least 40" - is this based on 12
mil
Hi All,
I have a question about the inMemoryKeyValue store. I was under the
assumption that in-memory stores would not serialize the objects but when I
looked into the implementation I see InMemoryKeyValueStore uses a
NavigableMap of bytes which indicates the user objects have to be
serialized and
Thanks John & Blake. Will try to recreate the issue and see what's going on.
Regards,
Navneeth
On Thu, Dec 17, 2020 at 6:27 PM Blake Miller wrote:
> Navneeth,
>
> You may need this JVM option:
>
> -XX:-OmitStackTraceInFastThrow
>
> as some stack frame information can be optimized away by defau
Hello there,
One of our input topics does not have so much traffic.
Divided by the number of partitions, and given the default 'max.poll.records'
setting (being 1000 if I understand the doc correctly), it could happen
that fetching 1000 records at once, the event timestamps between the first
and l
