Re: Dealing with large messages

Storing large blobs in S3 or HDFS and placing URIs in Kafka is the most common solution I've seen in use. On Tue, Oct 6, 2015 at 8:32 AM, Joel Koshy wrote: > The best practice I think is to just put large objects in a blob store > and have messages embed references to those

Re: Dealing with large messages

The best practice I think is to just put large objects in a blob store and have messages embed references to those blobs. Interestingly we ended up having to implement large-message-support at LinkedIn but for various reasons were forced to put messages inline (i.e., against the above

Re: Dealing with large messages

Thanks for the replies! I was rather hoping not to have to implement a side channel solution. :/ If we have to do this, we may use an HBase table with a TTL the same as our topic so the large objects are "gc'ed"... thoughts? On Tue, Oct 6, 2015 at 8:45 AM, Gwen Shapira

Dealing with large messages

Fellow Kafkaers, We have a pretty heavyweight legacy event logging system for batch processing. We're now sending the events into Kafka now for realtime analytics. But we have some pretty large messages (> 40 MB). I'm wondering if any of you have use cases where you have to send large messages

Re: Dealing with large messages

In addition to the config changes mentioned in that post, you may also have to change producer config if you are using the new producer. Specifically, *max.request.size* and *request.timeout.ms * have to be increased to allow the producer to send large messages. On 6

Re: Dealing with large messages

Here’s an article that Gwen wrote earlier this year on handling large messages in Kafka. http://ingest.tips/2015/01/21/handling-large-messages-kafka/ -James > On Oct 5, 2015, at 11:20 AM, Pradeep Gollakota wrote: > > Fellow Kafkaers, > > We have a pretty heavyweight