[jira] [Commented] (KAFKA-4846) Use KafkaProducer without allocating a new ProducerRecord for each message sent
[
https://issues.apache.org/jira/browse/KAFKA-4846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897564#comment-15897564
]
Matt Sicker commented on KAFKA-4846:
I know I'd love this feature personally for distributed trace logging a la
Zipkin. I don't wish to contribute overhead to instrumented services, and
increasing memory pressure is one of those ways to get some pushback from
certain mechanical sympathy type people.
We spent a large epic in Log4j 2.6 making the main execution paths
garbage-free, and since then, we've added GC-free support to more and more
components. Of course, it doesn't make sense to add this sort of feature to
every library, but seeing as how I find that the KafkaAppender has the best
potential for networked logging other than the FlumeAppender (I'm not sure how
performance compares here), it'd be great to support this use case in Log4j as
such.
I did find an [alternative Kafka
client|https://github.com/blackberry/Krackle/], but it appears to be limited to
the 0.8 message format as well as not entirely what we're looking for.
> Use KafkaProducer without allocating a new ProducerRecord for each message
> sent
> ---
>
> Key: KAFKA-4846
> URL: https://issues.apache.org/jira/browse/KAFKA-4846
> Project: Kafka
> Issue Type: Improvement
> Components: producer
>Affects Versions: 0.10.2.0
>Reporter: Mikael Ståldal
>
> The KafkaProducer API requires you to allocate a new ProducerRecord for each
> record sent. This is unfortunate since some application wants to reduce
> object allocations to minimize GC work. This would be useful for Log4j to
> allow [garbage free
> logging|https://logging.apache.org/log4j/2.x/manual/garbagefree.html] in its
> [Kafka
> appender|https://logging.apache.org/log4j/2.x/manual/appenders.html#KafkaAppender].
> This could be solved by adding a new method with unrolled arguments to
> KafkaProducer, like this:
> {code}
> Future send(String topic, Integer partition, Long timestamp,
> K key, V value);
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KAFKA-4802) Support direct ByteBuffer serializers/deserializers in clients
[ https://issues.apache.org/jira/browse/KAFKA-4802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894657#comment-15894657 ] Matt Sicker commented on KAFKA-4802: We also found a related issue in the inability to reuse ProducerRecord objects. > Support direct ByteBuffer serializers/deserializers in clients > -- > > Key: KAFKA-4802 > URL: https://issues.apache.org/jira/browse/KAFKA-4802 > Project: Kafka > Issue Type: New Feature > Components: clients >Reporter: Matt Sicker > > RecordAccumulator and Fetcher are already written to take advantage of a pool > of ByteBuffers, but Serializer and Deserializer require you to return a byte > array. If I have a key or value format that is better handled directly via > ByteBuffer, the added conversion to a byte array introduces unnecessary > garbage. > An example use case would be in enhancing the KafkaAppender in Log4j 2 to > support garbage free logging (or minimal garbage; I haven't really looked at > the entire code path). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KAFKA-4802) Support direct ByteBuffer serializers/deserializers in clients
Matt Sicker created KAFKA-4802: -- Summary: Support direct ByteBuffer serializers/deserializers in clients Key: KAFKA-4802 URL: https://issues.apache.org/jira/browse/KAFKA-4802 Project: Kafka Issue Type: New Feature Components: clients Reporter: Matt Sicker RecordAccumulator and Fetcher are already written to take advantage of a pool of ByteBuffers, but Serializer and Deserializer require you to return a byte array. If I have a key or value format that is better handled directly via ByteBuffer, the added conversion to a byte array introduces unnecessary garbage. An example use case would be in enhancing the KafkaAppender in Log4j 2 to support garbage free logging (or minimal garbage; I haven't really looked at the entire code path). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
