[
https://issues.apache.org/jira/browse/SAMZA-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088114#comment-16088114
]
ASF GitHub Bot commented on SAMZA-1282:
---------------------------------------
GitHub user shanthoosh opened a pull request:
https://github.com/apache/samza/pull/244
SAMZA-1282: Spinning up more containers than number of tasks.
Changes
* Stop streamProcessor in onNewJobModelAvailable eventHandler(instead of
onNewJobModelConfirmed
eventHandler) when it's not part of the group and prevent it from joining
the barrier.
* When numContainerIds > numTaskModels, generate JobModel by choosing
lexicographically
least `x` containerIds(where x = numTaskModels).
* Added unit and integration tests in appropriate classes to verify the
expected behavior.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/shanthoosh/samza more_processor_than_tasks
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/244.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #244
----
commit 2e1d372e18fbb8077f5d02907ef4e68bbbdfc6a8
Author: Shanthoosh Venkataraman <[email protected]>
Date: 2017-07-12T23:19:28Z
SAMZA-1282: Spinning up more containers than number of tasks.
Changes
* Stop streamProcessor in onNewJobModelAvailable eventHandler(instead of
onNewJobModelConfirmed
eventHandler) when it's not part of the group and prevent it from joining
the barrier.
* When numContainerIds > numTaskModels, generate JobModel by choosing
lexicographically
least `x` containerIds(where x = numTaskModels).
* Added unit and integration tests in appropriate classes to verify the
expected behavior.
----
> Spinning up more containers than the number of tasks kills leader
> -----------------------------------------------------------------
>
> Key: SAMZA-1282
> URL: https://issues.apache.org/jira/browse/SAMZA-1282
> Project: Samza
> Issue Type: Bug
> Components: container
> Affects Versions: 0.13.0
> Reporter: Bharath Kumarasubramanian
> Assignee: Shanthoosh Venkataraman
> Fix For: 0.13.1
>
>
> When a user tries to spin up more containers than the max partitions or
> tasks, the leader process gets killed.
> We throw an exception in the TaskNameGrouper for the above scenario and that
> needs to be handled gracefully by the leader and kill the newly spun
> containers as opposed bailing out.
> Here is the stack trace
> {code}
> 2017-05-10 15:13:24.526 [debounce-thread-0] ScheduleAfterDebounceTime
> [ERROR] OnProcessorChange threw an exception.
> java.lang.IllegalArgumentException: number of containers 2 is bigger than
> number of tasks 1
> at
> org.apache.samza.container.grouper.task.GroupByContainerIds.group(GroupByContainerIds.java:68)
> at
> org.apache.samza.coordinator.JobModelManager$.readJobModel(JobModelManager.scala:258)
> at
> org.apache.samza.coordinator.JobModelManager.readJobModel(JobModelManager.scala)
> at
> org.apache.samza.zk.ZkJobCoordinator.generateNewJobModel(ZkJobCoordinator.java:212)
> at
> org.apache.samza.zk.ZkJobCoordinator.doOnProcessorChange(ZkJobCoordinator.java:125)
> at
> org.apache.samza.zk.ZkJobCoordinator.lambda$onProcessorChange$1(ZkJobCoordinator.java:120)
> at
> org.apache.samza.zk.ScheduleAfterDebounceTime.lambda$scheduleAfterDebounceTime$0(ScheduleAfterDebounceTime.java:89)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
