lakshmi-manasa-g opened a new pull request, #1599: URL: https://github.com/apache/samza/pull/1599
Feature: Elasticity (SAMZA-2687) for a Samza job allows job to have more tasks than the number of input SystemStreamPartition(SSP). Thus, a job can scale up beyond its input partition count without needing the repartition the input stream. This current PR is to update the last processed offsets maintained by the OffsetManager during processing stage of the container correctly when elasticity is enabled. Changes: 1. Modify OffsetManager.udpate to correctly identify the ssp even with keyBucket info is not present in the given parameter ssp 2. Modify SystemConsumers to provide the ssp,offset to Chooser after all the registrations are done. This is because, chooser is given an ssp without keyBucket but a container could be processing multiple keyBuckets within the same ssp. So after registration for all is complete, the smallest offset is given to the chooser. Tests: TBD API changes: no public api change Upgrade instructions: none Usage instructions: None. Backwards compatible: yes. does not affect the existing flow -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
