A different approach would be to integrate the Apache DataSketches
(https://datasketches.apache.org/) which have mathematical proofs behind them.
Using a DataSketch you can capture unique members for any given time period in
a very small data object and be able to aggregate them (even though
I had a similar problem before. I could find no way for the producer to
determine the smallest maximum buffer size of the servers.
We had an issue where we had very large items that we wanted to send through
kafka, basically to tunnel through a firewall.
We utilized an open source project
