ChrisSamo632 commented on pull request #4822: URL: https://github.com/apache/nifi/pull/4822#issuecomment-834187070
> @ChrisSamo632 It seems to me that the checkpoint at shutdown does not work. I have a few uncheckpointed messages that get processed again and again when I restart the processor. > > I don't really understand how the graceful shutdown works in KCL, but it seems to me we have no chance to exit cleanly (I am always seeing ZOMBIE state in the record processor's shutdown callback). > Could you please double check it? > > ... @turcsanyip I'd noticed ZOMBIE logs a few times and thought it seemed a "normal" thing for KCL (from what I could tell, but note I'm hardly a Kinesis/KCL expert). Taking a look around on the internet, there are many threads about people seeing this kind of behaviour in many different versions of the KCL, for example: * https://github.com/awslabs/amazon-kinesis-client/issues/108 * https://github.com/awslabs/amazon-kinesis-client/issues/677 * https://github.com/amazon-archives/amazon-kinesis-connectors/issues/10 A common thing between these seems to be that the KCL settings likely need tweaking depending upon one's setup. With the ability to configure most KCL settings via Dynamic Properties on the processor, it's largely open for users to figure this out I guess depending upon their setup. That said, a common theme seems to be the need to increase the `failoverTimeMillis` and/or `gracefulShutdownMillis` KCL settings, I've therefore added those to the `ConsumeKinesisStream` processor's properties to encourage users to investigate/tweak as required. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
