Guozhang,
Thanks for the suggestion, however, I'm afraid cardinality of keys
will grow indefinitely and AFAIU keys are permanent with log
compaction. Any chance keys could also be removed during compaction?
Thanks,
2015-03-02 5:27 GMT+03:00 Guozhang Wang wangg...@gmail.com:
From your
Guozhang,
I agree, but upon restart the application still needs to init
KV-storage. And even though values are empty, keys will generate
traffic (delaying app startup time).
Besides, the idea of keeping needless data in kafka forever, even keys
only, sounds rather unsettling.
I guess we could
Currently Kafka log compaction does not support removing keys, but as long
as you also have log cleaning done at the app level #.keys will not
increase indefinitely.
On Mon, Mar 2, 2015 at 2:08 AM, Ivan Balashov ibalas...@gmail.com wrote:
Guozhang,
Thanks for the suggestion, however, I'm
Wouldn't it be rather simple to add a retention time on deleted items ie
keys with null value for topics that are compacted?
The retention time would then be set to some large time to allow all
consumers to understand that a previous k/v is being deleted.
2015-03-02 17:30 GMT+01:00 Ivan
Svante,
Not sure if I understand your suggestion correctly, but I do think
that enabling retention for deleted values would make a useful
addition to the compact policy. Otherwise some data is bound to be
hanging around not used.
Guozhang, could this potentially deserve a feature request?
Ivan,
I think log.cleaner.delete.retention.ms does just that?
The amount of time to retain delete tombstone markers for log compacted
topics. This setting also gives a bound on the time in which a consumer must
complete a read if they begin from offset 0 to ensure that they get a valid
James,
Indeed, does exactly what is needed.
Thanks for noticing!
2015-03-02 22:34 GMT+03:00 James Cheng jch...@tivo.com:
Ivan,
I think log.cleaner.delete.retention.ms does just that?
The amount of time to retain delete tombstone markers for log compacted
topics. This setting also gives
This would be a good feature to add to log Cleaner.
Thanks,
Mayuresh
On Mon, Mar 2, 2015 at 8:57 AM, Ivan Balashov ibalas...@gmail.com wrote:
Svante,
Not sure if I understand your suggestion correctly, but I do think
that enabling retention for deleted values would make a useful
addition
Ivan,
From your description it seems Kafka stores source of truth of the data
and the k-v store is constructed via consuming from Kafka, right? In that
case time/size-based data retention policy is usually not preferred as it
may delete data out of expectation while people are querying the k-v
2015-03-01 18:41 GMT+03:00 Jay Kreps jay.kr...@gmail.com:
They are mutually exclusive. Can you expand on the motivation/use for
combining them?
Thanks, Jay
Let's say we need to build key-value storage semantically connected to
the data that also stored in kafka.
Once the particular pieces of
Hi,
Do I understand correctly that compaction and deletion are currently
mutually exclusive?
Is it possible to compact recent segments and delete older ones,
according to general deletion policies?
Thanks,
2014-11-30 15:10 GMT+03:00 Manikumar Reddy ku...@nmsworks.co.in:
Log cleaner does not
They are mutually exclusive. Can you expand on the motivation/use for
combining them?
-Jay
On Sunday, March 1, 2015, Ivan Balashov ibalas...@gmail.com wrote:
Hi,
Do I understand correctly that compaction and deletion are currently
mutually exclusive?
Is it possible to compact recent
I think currently you can issue delete markers (tombstones) for the keys.
That will delete the data associated with the respective keys during
compaction. But the keys still will exist in the log.
Thanks,
Mayuresh
On Sun, Mar 1, 2015 at 8:07 AM, Ivan Balashov ibalas...@gmail.com wrote:
Does the log cleaner in 0.8.2 support topics with compressed messages? IIRC
that wasn't supported in 0.8.1.1.
On 29 November 2014 at 17:23, Jun Rao jun...@gmail.com wrote:
Yes, log cleaner is in 0.8.2. You just need to set the retention policy of
a topic to compact.
Thanks,
Jun
On Thu,
Log cleaner does not support topics with compressed messages.
https://issues.apache.org/jira/browse/KAFKA-1374
On Sun, Nov 30, 2014 at 5:33 PM, Mathias Söderberg
mathias.soederb...@gmail.com wrote:
Does the log cleaner in 0.8.2 support topics with compressed messages? IIRC
that wasn't
Yes, log cleaner is in 0.8.2. You just need to set the retention policy of
a topic to compact.
Thanks,
Jun
On Thu, Nov 27, 2014 at 5:20 AM, Khandygo, Evgeny (EXT)
evgeny.khandygo@siemens.com wrote:
I’m wondering if you could tell me whether log cleaner implemented in
0.8.2 because it
I’m wondering if you could tell me whether log cleaner implemented in 0.8.2
because it seems like it didn’t.
Thanks
John
17 matches
Mail list logo
