[
https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14162336#comment-14162336
]
Joel Koshy commented on KAFKA-1499:
-----------------------------------
Talked to Jay offline and here is a summary of what we discussed: the main
motivations for this feature (currently) are:
# Log compaction
# Ensuring messages on the broker are compressed if a bunch of producers happen
to send messages uncompressed - say if all producers in an organization
happened to pick up a bad config over time
# Ensuring messages on the broker are compressed with a specific compression
type - perhaps if downstream consumers want only that compression type
For the first use-case, we can potentially get around it as described above by
picking any compression type - i.e., if we are writing out a batch of messages
that contains various compression types we can just pick one of those types.
This is not as neat as having an explicit target compression type but it seems
reasonable.
The second and third use-cases are likely only marginally useful.
So we have a couple of options:
* Do nothing - given that we have (what seems to be) a reasonable approach for
dealing log compaction. i.e., we can table this and revisit if we have a very
compelling use case for it.
* Add the compression.type config as a server config and topic-override config.
** In order to address the concern of forgetting to set this (or
misconfiguration) there are two approaches:
*** Make it an optional configuration as mentioned further above
*** Have it default to compression.type "producer" - which means use whatever
compression type the producer used.
> Broker-side compression configuration
> -------------------------------------
>
> Key: KAFKA-1499
> URL: https://issues.apache.org/jira/browse/KAFKA-1499
> Project: Kafka
> Issue Type: New Feature
> Reporter: Joel Koshy
> Assignee: Manikumar Reddy
> Labels: newbie++
> Fix For: 0.8.2
>
> Attachments: KAFKA-1499.patch, KAFKA-1499.patch,
> KAFKA-1499_2014-08-15_14:20:27.patch, KAFKA-1499_2014-08-21_21:44:27.patch,
> KAFKA-1499_2014-09-21_15:57:23.patch, KAFKA-1499_2014-09-23_14:45:38.patch,
> KAFKA-1499_2014-09-24_14:20:33.patch, KAFKA-1499_2014-09-24_14:24:54.patch,
> KAFKA-1499_2014-09-25_11:05:57.patch
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> A given topic can have messages in mixed compression codecs. i.e., it can
> also have a mix of uncompressed/compressed messages.
> It will be useful to support a broker-side configuration to recompress
> messages to a specific compression codec. i.e., all messages (for all
> topics) on the broker will be compressed to this codec. We could have
> per-topic overrides as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
