PanTheMan opened a new pull request #1289: SAMZA-2317: ProcessJob does not call CoordinatorStreamStore.close() URL: https://github.com/apache/samza/pull/1289 Symptom: Users when deploying a job in dev will see an error message from the kafka consumer finalizer complaining about "kafka consumer allocated and not closed" This error message will appear multiple times in their job and doesn't actually affect the job's performance or correctness. This leads to users falsely believing that any job failures is because of this message. Cause: All Kafka consumers have a finalize method in them. This means when the garbage collector determines that there's no more references to the consumer, it will try to close the consumer and find that the consumer wasn't closed properly yet. This happens because in the ProcessJobFactory, a CoordinatorStreamStore is initialized. It is then wrapped around a NamespaceAwareCoordinatorStreamStore. Later on when the ProcessJob is done, a close() is called on the NamespaceAwareCoordinatorStreamStore, however this doesn't close the original CoordinatorStreamStore (As there could be multiple NamespaceAware stores pointing to one CoordinatorStreamStore). So, from the garbage collector's perspective, it will see that there are no more references to the CoordinatorStreamStore, leading to the finalize method reporting the above error. Changes: ProcessJob will now have an additional parameter which will be the CoordinatorStreamStore. Then a close is called on the store when the job is done running. That way there is still a reference to the store after the ProcessJob starts. Also updated the TestProcessJob file to test that the store is closed properly. This way of fixing this error was chosen as any other solution would involved changing how the coordinatorStreamStore operates which could lead to many unintended side-effects Tests: To reproduce this error, deploy any job running the latest version of Samza and look in the .out log file. Then run the same job with this fix included and that error message shouldn't appear anymore
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
