lakshmi-manasa-g commented on a change in pull request #1580:
URL: https://github.com/apache/samza/pull/1580#discussion_r799921343
##########
File path:
samza-core/src/main/scala/org/apache/samza/system/SystemConsumers.scala
##########
@@ -172,7 +172,10 @@ class SystemConsumers (
def start {
for ((systemStreamPartition, offset) <- sspToRegisteredOffsets.asScala) {
val consumer = consumers(systemStreamPartition.getSystem)
- consumer.register(systemStreamPartition, offset)
+ // If elasticity is enabled then the RunLoop gives SSP with keybucket
+ // but the actual systemConsumer which consumes from the input does not
know about KeyBucket.
+ // hence, use an SSP without KeyBucket
+ consumer.register(new
SystemStreamPartition(systemStreamPartition.getSystemStream,
systemStreamPartition.getPartition), offset)
Review comment:
working on this. trying to find a way that will work well for the next
planned pr of existing container metrics reporting .
will update soon.
##########
File path: samza-core/src/main/java/org/apache/samza/container/RunLoop.java
##########
@@ -239,8 +258,29 @@ private void runTasks(IncomingMessageEnvelope envelope) {
if (!shutdownNow) {
if (envelope != null) {
PendingEnvelope pendingEnvelope = new PendingEnvelope(envelope);
- for (AsyncTaskWorker worker :
sspToTaskWorkerMapping.get(envelope.getSystemStreamPartition())) {
- worker.state.insertEnvelope(pendingEnvelope);
+ SystemStreamPartition sspOfEnvelope = null;
+ // when elasticity is enabled
+ // the tasks actually consume a keyBucket of the ssp.
+ // hence use the SSP with keybucket to find the worker(s) for the
envelope
+ if (elasticityFactor <= 1) {
+ sspOfEnvelope = envelope.getSystemStreamPartition();
+ } else {
+ sspOfEnvelope = envelope.getSystemStreamPartition(elasticityFactor);
+ log.trace("elasticity enabled. using the ssp of the envelope as {}",
sspOfEnvelope);
+ }
Review comment:
done.
But can not delete the envelope.getSSP() method as it is used by many samza
components which have no idea or access to elasticityFactor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
