Hey David, I think that behavior is meant to prevent an issue on the Kafka 0.8 Brokers. Samza 10.1 allows compression on log compacted topics, but you'll need to make sure you're using Kafka 0.9 or higher on the Brokers.
-Jake On Fri, Aug 5, 2016 at 10:57 PM, David Yu <david...@optimizely.com> wrote: > I guess this might be the problem: > > 2016-08-06 05:23:23,622 [main ] WARN o.a.s.s.kafka.KafkaSystemFactory$ - > System name 'kafka' is being used as a changelog. Disabling compression > since Kafka does not support compression for log compacted topics. > > > Is this a 0.8.x.x Kafka producer limitation? > > On Fri, Aug 5, 2016 at 4:25 PM Jacob Maes <jacob.m...@gmail.com> wrote: > > > Hey David, > > > > If you check your container logs Kafka should print the producer config > > > > > 2016-08-05 19:18:30.134 [main] ProducerConfig [INFO] ProducerConfig > > values: > > > compression.type = gzip > > > > ... > > > > If you see the correct compression type in that config, then the Kafka > > producer definitely has compression enabled. > > > > > > If it's still not compressing after confirming that setting then it > > means snappy wasn't able to compress your messages. I talked to our > > Kafka team and they said that's not uncommon. You might want to try > > gzip or some other compression. > > > > > > On Fri, Aug 5, 2016 at 12:10 PM, David Yu <david...@optimizely.com> > wrote: > > > > > I'm reporting back my observations after enabling compression. > > > > > > Looks like compression is not doing anything. I'm still seeing > > > "compression-rate-avg=1.0" and the same "record-size-avg" from JMX > > > "kafka.producer" metrics. > > > > > > I did set the following: > > > systems.kafka.producer.compression.type=snappy > > > > > > Am I missing anything? > > > > > > Thanks, > > > David > > > > > > On Wed, Aug 3, 2016 at 1:48 PM David Yu <david...@optimizely.com> > wrote: > > > > > > > Great. Thx. > > > > > > > > On Wed, Aug 3, 2016 at 1:42 PM Jacob Maes <jacob.m...@gmail.com> > > wrote: > > > > > > > >> Hey David, > > > >> > > > >> what gets written to the changelog topic > > > >> > > > >> The changelog gets the same value as the store, which is the > > serialized > > > >> form of the key and value. The serdes for the store are configured > > with > > > >> the > > > >> properties: > > > >> stores.store-name.key.serde > > > >> stores.store-name.msg.serde > > > >> > > > >> If I want to compress the changelog topic, do I enable that from the > > > >> > producer? > > > >> > > > >> Yes. When you specify the changelog for your store, you specify it > in > > > >> terms > > > >> of a SystemStream (typically a Kafka topic). In the part of the > config > > > >> where you define the Kafka system, you can pass any Kafka producer > > > config > > > >> <http://kafka.apache.org/documentation.html#newproducerconfigs>. So > > to > > > >> configure compression you should configure the following property. > > > >> systems.system-name.producer.compression.type > > > >> > > > >> Hope this helps. > > > >> -Jake > > > >> > > > >> > > > >> > > > >> On Wed, Aug 3, 2016 at 11:16 AM, David Yu <david...@optimizely.com> > > > >> wrote: > > > >> > > > >> > I'm trying to understand what gets written to the changelog topic. > > Is > > > it > > > >> > just the serialized value of the particular state store entry? If > I > > > >> want to > > > >> > compress the changelog topic, do I enable that from the producer? > > > >> > > > > >> > The reason I'm asking is that, we are seeing producer throughput > > > issues > > > >> and > > > >> > suspected that writing to changelog takes up most of the network > > > >> bandwidth. > > > >> > > > > >> > Thanks, > > > >> > David > > > >> > > > > >> > > > > > > > > > >
