chetanmeh commented on a change in pull request #2795: enable concurrent
activation processing
URL:
https://github.com/apache/incubator-openwhisk/pull/2795#discussion_r225774615
##########
File path: core/invoker/src/main/scala/whisk/core/invoker/InvokerReactive.scala
##########
@@ -101,15 +102,16 @@ class InvokerReactive(
private val topic = s"invoker${instance.toInt}"
private val maximumContainers = (poolConfig.userMemory /
MemoryLimit.minMemory).toInt
private val msgProvider = SpiLoader.get[MessagingProvider]
- private val consumer = msgProvider.getConsumer(
- config,
- topic,
- topic,
- maximumContainers,
- maxPollInterval = TimeLimit.MAX_DURATION + 1.minute)
+
+ //number of peeked messages - increasing the concurrentPeekFactor improves
concurrent usage, but adds risk for message loss in case of crash
+ private val maxPeek =
+ math.max(maximumContainers, (maximumContainers * limitsConfig.max *
poolConfig.concurrentPeekFactor).toInt)
Review comment:
Let me try again
```scala
val maximumContainers = (poolConfig.userMemory / MemoryLimit.minMemory).toInt
```
Here
* `MemoryLimit.minMemory` = 128 MB
* `poolConfig.userMemory` = x MB
So `maximumContainers` = x/128
Now
```scala
val maxPeek =
math.max(maximumContainers, (maximumContainers * limitsConfig.max *
poolConfig.concurrentPeekFactor).toInt)
```
Where earlier the defauls were
* `limitsConfig.max` = 500
* `poolConfig.concurrentPeekFactor` = 0.5
So
```
maxPeek = max(x/128, x/128 * 500 * 0.5)
maxPeek = max(128x, ~ 2x)
```
So if `poolConfig.userMemory` is 32 GB = 32768 MB then `maxPeek` = 65536.
Which is a high number. Considering a worst case of 1 MB per activation it
results in loading 65 GB of data. So may be we introduce an explicit config for
`maxPeek`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
