[
https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154336#comment-14154336
]
Jay Kreps commented on KAFKA-1499:
----------------------------------
Yeah I totally agree.
I agree that some heuristic that worked batch-by-batch might be okay, I hadn't
thought of that. Actually though I think the main motivation for this feature
was to fix the compaction issue, so if that is an okay fix just doing that
would be an alternative.
I also agree that NoCompressionCodec should be the default and unless people
know about the change they will surely be confused by this switch. However I
claim this is a temporary confusion based on the fact that previously Kafka
compression worked one way and now it will work a new way. Plus they will in
any case have this confusion if they turn on the feature. For any new user the
configuration docs will all be updated and in the process of learning how to
turn on compression they will learn how it works. I think we could help this
with good release notes (when doing an upgrade people always read that to
ensure it is in-place compatible).
I guess in the end what I am arguing is that we should make a choice. Either a
single compression codec per topic is better and it should work that way or
else having the producer specify compression is better and it should work that
way. Giving the user the choice seems nice but it actually just adds complexity
since now we will always have to document and explain both and tell people
about the configuration knob to choose and then advise them on how to best make
the choice (and then debug when they get lost in all this). If we think the
right choice is very situation specific (in situation x, chose
broker.compression.enabled=true, in situation y chose false) then okay maybe we
need a config, but then let's figure out what the situations you want one
versus the other. If it isn't situation specific we should just choose one and
implement and document that.
> Broker-side compression configuration
> -------------------------------------
>
> Key: KAFKA-1499
> URL: https://issues.apache.org/jira/browse/KAFKA-1499
> Project: Kafka
> Issue Type: New Feature
> Reporter: Joel Koshy
> Assignee: Manikumar Reddy
> Labels: newbie++
> Fix For: 0.8.2
>
> Attachments: KAFKA-1499.patch, KAFKA-1499.patch,
> KAFKA-1499_2014-08-15_14:20:27.patch, KAFKA-1499_2014-08-21_21:44:27.patch,
> KAFKA-1499_2014-09-21_15:57:23.patch, KAFKA-1499_2014-09-23_14:45:38.patch,
> KAFKA-1499_2014-09-24_14:20:33.patch, KAFKA-1499_2014-09-24_14:24:54.patch,
> KAFKA-1499_2014-09-25_11:05:57.patch
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> A given topic can have messages in mixed compression codecs. i.e., it can
> also have a mix of uncompressed/compressed messages.
> It will be useful to support a broker-side configuration to recompress
> messages to a specific compression codec. i.e., all messages (for all
> topics) on the broker will be compressed to this codec. We could have
> per-topic overrides as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
