Thanks for the reply.  Actually my Lambda consumers are consuming batched 
messages from a Kinesis queue in AWS, process them, and send results to Kafka.  
Even with 'reserve concurrency' AWS will frequently stop and 
re-initiate/-invoke the function for different batches - resulting in 
recreation of the producers.  Hope there is a solution for this - otherwise I 
could not use Lambda or Spot-Instances in AWS with Kafka. 
    Am Donnerstag, 15. August 2019, 09:28:17 MESZ hat Jörn Franke 
<jornfra...@gmail.com> Folgendes geschrieben:  
 
 Even if it is not a memory leak it is not a good practice. You can put the 
messages on SQS and have a lambda function listening to the SQS queue with 
reserve concurrency to put it on Kafka

> Am 15.08.2019 um 08:52 schrieb Tianning Zhang 
> <tianningzh...@yahoo.de.invalid>:
> 
> Dear all, 
> 
> I am using Amazon AWS Lambda functions to produce messages to a Kafka 
> cluster. As I can not control how frequently a Lambda function is 
> initiated/invoked and I can not share object between invocations - I have to 
> create a new Kafka producer for each invocation and clean it up after the 
> invocation finishes. Each producer also set to the same "client.id".
> I notice that after deploying the lambda functions the heap size at the 
> brokers increases quickly - which finally resulted GC problems and problems 
> at the brokers. It is very likely that this increase is connected to the 
> Lambda producers.
> I know that it is recommended to reuse single producer instance for message 
> production. But in this case (with AWS Lambda) this is not possible.
> My question is that if it is possible that high number of producer 
> creation/cleanup can lead to memory leaks at the brokers?
> I am using Kafka cluster with 5 brokers - version 1.0.1. Kafka client lib 
> tested with versions 0.11.0.03, 1.0.1 and 2.3.0.
> Thanks in advance
> Tianning Zhang
> 
> T: +49(30)509691-8301M: +49 172 7095686E: tianning.zh...@awin.com
> 
> Eichhornstraße 310785 Berlin
> www.awin.com
>   

Reply via email to