Hi Team,
We have a requirement where we need to decrypt/encrypt any data coming
in/going out to Samza processor (Kafka used as the system stream). It is
achievable with the different serde for input/output stream however for
Rocksdb key value store backed up by kafka changelog it becomes
challenging.
With the current state of the system
<https://github.com/apache/samza/blob/master/samza-core/src/main/scala/org/apache/samza/serializers/SerdeManager.scala#L40>
the only way to achieve encryption/decryption for the changelog stream is
to make the key value store have a serde which also does
encryption/decryption. But the problem is if we try to do a iteration of
the rocksdb store in window task, we will be unnecessarily
encrypting/decrypting the data(which is present locally in the system)
which will have impact on the performance. So is there any other way to use
a different serde for key value store but before sending the data to kafka
changelog use a different serde(doing encryption)?
I believe we don't need to do the encryption if we can securely connect to
Kafka with 0.10 release, however Samza is using the old consumer, so we
cannot leverage the feature. Is there any timeline regarding the new kafka
client integration(Samza-855
<https://issues.apache.org/jira/browse/SAMZA-855>)?
Also unrelated to this topic when could we expect the next Samza release?
Thank you for the support.
Regards,
Nick
