alnzng opened a new pull request #1259: SAMZA-2444: JobModel save in CoordinatorStreamStore resulting flush for each message URL: https://github.com/apache/samza/pull/1259 ## Symptom When Samza's job creates lots of tasks/partitions, it can take over a long time for AM to save the metadata in a run which may cause timeout exception. We observed if the Samza's job has over 37k tasks/partitions, it takes over 10 min for AM to save it in a run. ## Cause JobModelManager uses CoordinatorStreamStore.put() to save job metadata information which does flush for each message, and the flush operation is heavy especially when the remote server suffering the performance issues. ## Changes - [ ] Use CoordinatorStreamStore's `putAll()` instead of `put()` to avoid flushing on each message - [ ] Add a new `deleteAll()` method in CoordinatorStreamStore, and replace `delete()` with it when updating job metadata informaiton ## Tests - [ ] Add unit tests for newly added methods - [ ] All unit tests and integration tests are passed ## API Changes - [ ] Add new `deleteAll()` function in `MetadataStore` interface and provide default implmentation ## Upgrade Instructions None ## Usage Instructions None
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
