[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-337231124 rebased yet again to master to resolve conflicts. PG1 2150 ? Could we please finally merge once CI approves? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-334764041 2 comments in favor of merging on dev list and no objections. Good to go? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-333544268 rebased to Oct 2 master. pg4 836 ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-332344816 pg1 2101 ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-332524505 pg1 2102 ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-332524292 Now simplified as suggested by @markusthoemmes. With the state for the id assignment externalized to redis, there is no need to involve the controller at all. Each invoker can self-assign ids on startup using redis for coordination. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-332344816 pg1 2101 ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker
id assignment
URL:
https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-332273637
update to address review comments from @rabbah and @markusthoemmes.
pg1 2100 ?
@markusthoemmes, is moving the blocking Redis calls into the handler really
enough, or do I need to be finer-grained wrapping in Future { blocking {
blocking_redis_call...} }
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-331531524 pg1 2093 ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-331531524 pg1 2092 ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-331525184 Rebased again to resolve conflicts with master. Assuming CI agrees, I think this PR could be ready for final review & merge. Additional comments/feedback from anyone? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-330938601 pg1 2083 ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-330938601 pg1 2083 ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-330924558 For this PR, I would prefer to keep Redis as the persistent store. I think the Redis client API is a better match for the needed function than Zookeeper's. It might not be the final solution, but it allows us to make forward progress. Note that the persistent store is not dynamically used if the deployment provides suggested invokerIds (current default behavior). I've done some updates and squashed down to a single commit again. Since the approach has changed quite a bit during the review process, I'm inling the updated commit comment here to make it easier to see what this PR is actually doing now. First stage of adding support for dynamic invoker id assignment Add a handshake between the invoker and controller during invoker startup to allow the controller an opportunity to dynamically assign the invokerId (the topic to which the invoker listens to accept work). The invoker provides an identifying name for itself and optionally a suggested invokerId to the controller via the invokerReg topic of the messaging provider. The controller responds with an assigned invokerID using a topic derived from the provided invoker name. After receiving its invokerId, the rest of invoker startup proceeds as before. To support different deployment scenarios, both the name and the suggested invokerId may be provided externally to the invoker. This supports two deployment modes: 1. A completely static assignment of invokerIds to invokers when the deployment specifies the suggested invokerId. This is the default option for this PR and matches the prior behavior of the system. 2. If the deployment provides an invoker name, but not a suggested invokerId, then we get a dynamically assigned but stable invokerIds. For example, the invoker name could be the IP address or hostname of the invoker's host or the nodeName of the kube worker node. A third option, in which the deployment provides neither the name nor the suggested invokerId is possible, but is currently trapped as a configuration error. This is treated as configuration error to avoid an explosion of invokerIds (topics) in the presence of frequent invoker instance restarts. Option 2 depends on persisting the state for the dynamic invoker id assignment protocol to support controller restarts and multiple controllers. For the purposes of this PR, we use the Redis instance stated for the apigateway as the persistent store since the Redis API is a good match for the needed operations. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-330855363 Doesn't have to be UUIDs. Just needs to be any stable unique alphanumeric String that can be used to distinguish invokers. UUIDs were convenient but I don't really care if we use them or change to just be a String that might be a UUID if the deployment hasn't specified anything else. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-330647740 rebased with master to resolve conflict in LoadBalancer.scala. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-330631211 pg1 2078 ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-330613666 pg1 2078 ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-330613666 pg1 2077 ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-330608467 pg 2076 ? investigating. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-330572212 pg1 2076 ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-330347712 updated to persist the state for dynamic invoker id assignment in a Redis instance. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-329778750 Those are good points. I hadn't thought carefully about a total controller restart. Maybe I should rework to use an external store. This is definitely cold-path code so we can afford to hit a database if we don't have a UUID=>invokerID mapping. Then the in-memory structure is just a cache of the system of record. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-329591647 For multiple controllers, the current code is assuming that every controller will see the messages on the invokerReg topic in the same order. My understanding of kafka is superficial, but I believe that is what it provides. Assuming it does, then the code in the Registrar agent should make exactly the same set of state updates in each controller process. I've done a few dozen runs with multiple controllers and it seems to work, but that isn't proof (could be dumb luck still). An alternative implementation could forgo relying on kafka and use Akka distributed data to maintain a shared counter and mapping data structure. If the invoker dies, but restarts with the same invokerUUID it had before, it will get the same invokerId assigned to it. The main difference from what you need to do today is that although your deployment tooling still needs to assign UUIDs, they no longer have to be dense integers. For example the invokerUUID can be the Kube nodeName or the IP address of the node (if you only deploy 1 invoker instance per IP address) or some other arbitrary alpha-numeric string. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment
dgrove-oss commented on issue #2689: first stage of support for dynamic invoker id assignment URL: https://github.com/apache/incubator-openwhisk/pull/2689#issuecomment-328573174 I've squashed to a single commit and updated the commit comment to reflect the final design. @rabbah I think this is ready for review and could have the WIP label removed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
