Hey guys,
Apologies if this question has been asked before but is there a way to enable
(kafka) log compaction on the key-value store changelog from the samza
properties file?
I’m experimenting with the state management features of Samza 0.8 and I’ve
created a very simple single node test setup to deploy a super simple stream
processor that uses the RocksDb state store and I’ve set the following store
related config properties:
stores.my-store.factory=org.apache.samza.storage.kv.RocksDbKeyValueStorageEngineFactory
stores.my-store.changelog=kafka.my-store-changelog
stores.my-store.key.serde=string
stores.my-store.msg.serde=string
When I deployed the configuration I noticed the state store changelog topic did
not have compaction enabled. I was also experimenting with checkpointing and
noticed that the kafka checkpoint topic did have the log cleaner policy set to
compaction:
deploy/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --describe
Topic:__samza_checkpoint_ver_1_for_my-test_1 PartitionCount:1
ReplicationFactor:1 Configs:segment.bytes=26214400,cleanup.policy=compact
Topic: __samza_checkpoint_ver_1_for_my-test_1 Partition: 0 Leader:
0 Replicas: 0 Isr: 0
Topic:my-store-changelog PartitionCount:1 ReplicationFactor:1
Configs:
Topic: my-store-changelog Partition: 0 Leader: 0
Replicas: 0 Isr: 0
Is there a way to enable log compaction for the state store changelog through
samzas configuration? If not, is it safe to manually enable it using
kafka-topics.sh --alter —topic my-store-changelog —config
cleanup.policy=compact?
Many thanks,
-Mart
