Re: ChangeLog Question for TTL rocksDB stores

Hi David, My understanding is that Samza changelog could still be logging those entries deleted by RocksDB TTL. You could refer to SAMZA-677 for more information. Thanks, -Tao On Wed, Feb 3, 2016 at 5:46 PM, David Garcia wrote: > Boris, thank you for the

Re: ChangeLog Question for TTL rocksDB stores

As Jacob mentioned there is not direct relationship between the rocksdb tts (internal to rocksdb) and changelog (done by Samza). The problem may arise if the store is restored from the changelog, since the log will have the expired entries, and they will be entered with the NEW date (and as Yi

Re: ChangeLog Question for TTL rocksDB stores

Boris, thank you for the clarification. But just to make sure I understand, is it correct to say that entries deleted by the TTL-policy in rocksDB will NOT be logged in the change-log? My job processes a lot of data and saves a large portion of it to RocksDB (for reference later…but subject to a

Re: ChangeLog Question for TTL rocksDB stores

Ok, that makes sense. I had assumed that the changelog was supported because the docs mention that TTL is enforced upon ³compaction² (I had assumed compaction of the DB changelog). Which topic does the TTL policy listen for the compaction of (since compaction policies of topics can differ)?

Re: ChangeLog Question for TTL rocksDB stores

Hi, David, The "compaction" referred to together w/ TTL is referring to RocksDb's compaction, not the Kafka-based changelog topic. Currently, TTL is not applied to Kafka-based changelog topic. SAMZA-677 is opened for this. -Yi On Thu, Jan 28, 2016 at 11:36 AM, David Garcia