Re: [one-users] econe-run-instances idempotent?
Hi Parag, I have uploaded a first version of the client token support. You can apply the commit included in the ticket. It will be included in the next release Cheers On 16 October 2014 11:39, Daniel Molina dmol...@opennebula.org wrote: On 15 October 2014 18:57, Parag Mhashilkar pa...@fnal.gov wrote: Hi Daniel Such a feature would be really useful when provisioning VMs and avoiding leaks. To be useful, this also requires support for client token in ec2-describe-instances equivalent in open nebula http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-DescribeInstances.html I have opened a ticker in our dev portal, and we will try to implement it. AWS uses client token to achieve idempotency in few other calls too, but I will leave that upto you on if and where you want to support such a functionality. The easiest way to implement this is only for VMs as I explained before. If we want to support this for any call we would have to generate a client token pool and it will be trickier. Cheers Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== On Oct 15, 2014, at 11:44 AM, Daniel Molina wrote: I think tthat could be implemented in the econe server. 1. Include an EC2_CLIENT_TOKEN in the vm.allocate method containing the ClientToken param provided in the CreateInstance command. 2. When a new CreateInstance is requested and contains a ClientToken, the vmpool is retrieved to check if any vm contains that token On 15 October 2014 18:34, Parag Mhashilkar pa...@fnal.gov wrote: That won't help. If the middle ware like HTcondor or run-instance commands starts putting anything in the user data, users are deprived of the functionality of user data. AWS achieves idempotency with --client-token option, which to me seems like a tagging the request from the client side. http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-RunInstances.html Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== On Oct 15, 2014, at 11:27 AM, Daniel Molina wrote: On 15 October 2014 18:18, Parag Mhashilkar pa...@fnal.gov wrote: Hi Daniel, Let me rephrase what we meant. As Steve mentioned in one of his emails, we use HTCondor to launch VMs. There is always a possibility that something can go wrong after a run-instance command is issued and before the ec2 server gets back to you with a valid instance id or an error message. Note that in this case the server may serve the request, but there is no way for the client to know the exact instance name that resulted from the request. Life is much easier if the client crashes after it gets back the instance name. One way for the client to know the exact status of the request even after a crash (and before it gets the instance id) is to assign a unique identifier of its own like a tag name or something. This way client will never lose track of the request and result in the leaked VMs. The way HTCondor is implementing this is by creating a new key pair and giving it a unique name and tracking the requests based on the key pair. But then we are hit by other limit ~300 that we observed. Ideally, we would like to use the --keypair option and use the existing pre registered key. But than we cannot track the results of the create-instance in case something goes wrong before the instance id is issued. And what about including a tag in the UserData instead of using the unique keypar? Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109
Re: [one-users] econe-run-instances idempotent?
On 15 October 2014 18:57, Parag Mhashilkar pa...@fnal.gov wrote: Hi Daniel Such a feature would be really useful when provisioning VMs and avoiding leaks. To be useful, this also requires support for client token in ec2-describe-instances equivalent in open nebula http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-DescribeInstances.html I have opened a ticker in our dev portal, and we will try to implement it. AWS uses client token to achieve idempotency in few other calls too, but I will leave that upto you on if and where you want to support such a functionality. The easiest way to implement this is only for VMs as I explained before. If we want to support this for any call we would have to generate a client token pool and it will be trickier. Cheers Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== On Oct 15, 2014, at 11:44 AM, Daniel Molina wrote: I think tthat could be implemented in the econe server. 1. Include an EC2_CLIENT_TOKEN in the vm.allocate method containing the ClientToken param provided in the CreateInstance command. 2. When a new CreateInstance is requested and contains a ClientToken, the vmpool is retrieved to check if any vm contains that token On 15 October 2014 18:34, Parag Mhashilkar pa...@fnal.gov wrote: That won't help. If the middle ware like HTcondor or run-instance commands starts putting anything in the user data, users are deprived of the functionality of user data. AWS achieves idempotency with --client-token option, which to me seems like a tagging the request from the client side. http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-RunInstances.html Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== On Oct 15, 2014, at 11:27 AM, Daniel Molina wrote: On 15 October 2014 18:18, Parag Mhashilkar pa...@fnal.gov wrote: Hi Daniel, Let me rephrase what we meant. As Steve mentioned in one of his emails, we use HTCondor to launch VMs. There is always a possibility that something can go wrong after a run-instance command is issued and before the ec2 server gets back to you with a valid instance id or an error message. Note that in this case the server may serve the request, but there is no way for the client to know the exact instance name that resulted from the request. Life is much easier if the client crashes after it gets back the instance name. One way for the client to know the exact status of the request even after a crash (and before it gets the instance id) is to assign a unique identifier of its own like a tag name or something. This way client will never lose track of the request and result in the leaked VMs. The way HTCondor is implementing this is by creating a new key pair and giving it a unique name and tracking the requests based on the key pair. But then we are hit by other limit ~300 that we observed. Ideally, we would like to use the --keypair option and use the existing pre registered key. But than we cannot track the results of the create-instance in case something goes wrong before the instance id is issued. And what about including a tag in the UserData instead of using the unique keypar? Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23,
Re: [one-users] econe-run-instances idempotent?
Hi, What do you mean with idempotent? As long as the client implements the ec2 API, it should work Cheers On 13 October 2014 20:33, Parag Mhashilkar pa...@fnal.gov wrote: Hi, We are using HTCondor to launch VMs in OpenNebula using ec2 interface and would like to know if the submit calls are idempotent. Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org -- -- Daniel Molina Project Engineer OpenNebula - Flexible Enterprise Cloud Made Simple www.OpenNebula.org | dmol...@opennebula.org | @OpenNebula ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] econe-run-instances idempotent?
If you call the same CreateInstances command more than once is there any way that it will create the instance twice or not. Steve Timm On Wed, 15 Oct 2014, Daniel Molina wrote: Hi, What do you mean with idempotent? As long as the client implements the ec2 API, it should work Cheers On 13 October 2014 20:33, Parag Mhashilkar pa...@fnal.gov wrote: Hi, We are using HTCondor to launch VMs in OpenNebula using ec2 interface and would like to know if the submit calls are idempotent. Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org -- -- Daniel Molina Project Engineer OpenNebula - Flexible Enterprise Cloud Made Simple www.OpenNebula.org | dmol...@opennebula.org | @OpenNebula -- Steven C. Timm, Ph.D (630) 840-8525 t...@fnal.gov http://home.fnal.gov/~timm/ Fermilab Scientific Computing Division, Scientific Computing Services Quad. Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] econe-run-instances idempotent?
On 15 October 2014 17:57, Steven Timm t...@fnal.gov wrote: If you call the same CreateInstances command more than once is there any way that it will create the instance twice or not. If you call the command twice it will create 2 instances. You can also provide de MaxCount param in the CreateInstance command to create more than one instance at once. Steve Timm On Wed, 15 Oct 2014, Daniel Molina wrote: Hi, What do you mean with idempotent? As long as the client implements the ec2 API, it should work Cheers On 13 October 2014 20:33, Parag Mhashilkar pa...@fnal.gov wrote: Hi, We are using HTCondor to launch VMs in OpenNebula using ec2 interface and would like to know if the submit calls are idempotent. Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org -- -- Daniel Molina Project Engineer OpenNebula - Flexible Enterprise Cloud Made Simple www.OpenNebula.org | dmol...@opennebula.org | @OpenNebula -- Steven C. Timm, Ph.D (630) 840-8525 t...@fnal.gov http://home.fnal.gov/~timm/ Fermilab Scientific Computing Division, Scientific Computing Services Quad. Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing -- -- Daniel Molina Project Engineer OpenNebula - Flexible Enterprise Cloud Made Simple www.OpenNebula.org | dmol...@opennebula.org | @OpenNebula ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] econe-run-instances idempotent?
Hi Daniel, Let me rephrase what we meant. As Steve mentioned in one of his emails, we use HTCondor to launch VMs. There is always a possibility that something can go wrong after a run-instance command is issued and before the ec2 server gets back to you with a valid instance id or an error message. Note that in this case the server may serve the request, but there is no way for the client to know the exact instance name that resulted from the request. Life is much easier if the client crashes after it gets back the instance name. One way for the client to know the exact status of the request even after a crash (and before it gets the instance id) is to assign a unique identifier of its own like a tag name or something. This way client will never lose track of the request and result in the leaked VMs. The way HTCondor is implementing this is by creating a new key pair and giving it a unique name and tracking the requests based on the key pair. But then we are hit by other limit ~300 that we observed. Ideally, we would like to use the --keypair option and use the existing pre registered key. But than we cannot track the results of the create-instance in case something goes wrong before the instance id is issued. Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== On Oct 15, 2014, at 11:02 AM, Daniel Molina wrote: On 15 October 2014 17:57, Steven Timm t...@fnal.gov wrote: If you call the same CreateInstances command more than once is there any way that it will create the instance twice or not. If you call the command twice it will create 2 instances. You can also provide de MaxCount param in the CreateInstance command to create more than one instance at once. Steve Timm On Wed, 15 Oct 2014, Daniel Molina wrote: Hi, What do you mean with idempotent? As long as the client implements the ec2 API, it should work Cheers On 13 October 2014 20:33, Parag Mhashilkar pa...@fnal.gov wrote: Hi, We are using HTCondor to launch VMs in OpenNebula using ec2 interface and would like to know if the submit calls are idempotent. Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org -- -- Daniel Molina Project Engineer OpenNebula - Flexible Enterprise Cloud Made Simple www.OpenNebula.org | dmol...@opennebula.org | @OpenNebula -- Steven C. Timm, Ph.D (630) 840-8525 t...@fnal.gov http://home.fnal.gov/~timm/ Fermilab Scientific Computing Division, Scientific Computing Services Quad. Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing -- -- Daniel Molina Project Engineer OpenNebula - Flexible Enterprise Cloud Made Simple www.OpenNebula.org | dmol...@opennebula.org | @OpenNebula smime.p7s Description: S/MIME cryptographic signature ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] econe-run-instances idempotent?
On 15 October 2014 18:18, Parag Mhashilkar pa...@fnal.gov wrote: Hi Daniel, Let me rephrase what we meant. As Steve mentioned in one of his emails, we use HTCondor to launch VMs. There is always a possibility that something can go wrong after a run-instance command is issued and before the ec2 server gets back to you with a valid instance id or an error message. Note that in this case the server may serve the request, but there is no way for the client to know the exact instance name that resulted from the request. Life is much easier if the client crashes after it gets back the instance name. One way for the client to know the exact status of the request even after a crash (and before it gets the instance id) is to assign a unique identifier of its own like a tag name or something. This way client will never lose track of the request and result in the leaked VMs. The way HTCondor is implementing this is by creating a new key pair and giving it a unique name and tracking the requests based on the key pair. But then we are hit by other limit ~300 that we observed. Ideally, we would like to use the --keypair option and use the existing pre registered key. But than we cannot track the results of the create-instance in case something goes wrong before the instance id is issued. And what about including a tag in the UserData instead of using the unique keypar? Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== On Oct 15, 2014, at 11:02 AM, Daniel Molina wrote: On 15 October 2014 17:57, Steven Timm t...@fnal.gov wrote: If you call the same CreateInstances command more than once is there any way that it will create the instance twice or not. If you call the command twice it will create 2 instances. You can also provide de MaxCount param in the CreateInstance command to create more than one instance at once. Steve Timm On Wed, 15 Oct 2014, Daniel Molina wrote: Hi, What do you mean with idempotent? As long as the client implements the ec2 API, it should work Cheers On 13 October 2014 20:33, Parag Mhashilkar pa...@fnal.gov wrote: Hi, We are using HTCondor to launch VMs in OpenNebula using ec2 interface and would like to know if the submit calls are idempotent. Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org -- -- Daniel Molina Project Engineer OpenNebula - Flexible Enterprise Cloud Made Simple www.OpenNebula.org | dmol...@opennebula.org | @OpenNebula -- Steven C. Timm, Ph.D (630) 840-8525 t...@fnal.gov http://home.fnal.gov/~timm/ Fermilab Scientific Computing Division, Scientific Computing Services Quad. Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing -- -- Daniel Molina Project Engineer OpenNebula - Flexible Enterprise Cloud Made Simple www.OpenNebula.org | dmol...@opennebula.org | @OpenNebula -- -- Daniel Molina Project Engineer OpenNebula - Flexible Enterprise Cloud Made Simple www.OpenNebula.org | dmol...@opennebula.org | @OpenNebula ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
Re: [one-users] econe-run-instances idempotent?
That won't help. If the middle ware like HTcondor or run-instance commands starts putting anything in the user data, users are deprived of the functionality of user data. AWS achieves idempotency with --client-token option, which to me seems like a tagging the request from the client side. http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-RunInstances.html Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== On Oct 15, 2014, at 11:27 AM, Daniel Molina wrote: On 15 October 2014 18:18, Parag Mhashilkar pa...@fnal.gov wrote: Hi Daniel, Let me rephrase what we meant. As Steve mentioned in one of his emails, we use HTCondor to launch VMs. There is always a possibility that something can go wrong after a run-instance command is issued and before the ec2 server gets back to you with a valid instance id or an error message. Note that in this case the server may serve the request, but there is no way for the client to know the exact instance name that resulted from the request. Life is much easier if the client crashes after it gets back the instance name. One way for the client to know the exact status of the request even after a crash (and before it gets the instance id) is to assign a unique identifier of its own like a tag name or something. This way client will never lose track of the request and result in the leaked VMs. The way HTCondor is implementing this is by creating a new key pair and giving it a unique name and tracking the requests based on the key pair. But then we are hit by other limit ~300 that we observed. Ideally, we would like to use the --keypair option and use the existing pre registered key. But than we cannot track the results of the create-instance in case something goes wrong before the instance id is issued. And what about including a tag in the UserData instead of using the unique keypar? Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== On Oct 15, 2014, at 11:02 AM, Daniel Molina wrote: On 15 October 2014 17:57, Steven Timm t...@fnal.gov wrote: If you call the same CreateInstances command more than once is there any way that it will create the instance twice or not. If you call the command twice it will create 2 instances. You can also provide de MaxCount param in the CreateInstance command to create more than one instance at once. Steve Timm On Wed, 15 Oct 2014, Daniel Molina wrote: Hi, What do you mean with idempotent? As long as the client implements the ec2 API, it should work Cheers On 13 October 2014 20:33, Parag Mhashilkar pa...@fnal.gov wrote: Hi, We are using HTCondor to launch VMs in OpenNebula using ec2 interface and would like to know if the submit calls are idempotent. Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org -- -- Daniel Molina Project Engineer OpenNebula - Flexible Enterprise Cloud Made Simple www.OpenNebula.org |
Re: [one-users] econe-run-instances idempotent?
I think tthat could be implemented in the econe server. 1. Include an EC2_CLIENT_TOKEN in the vm.allocate method containing the ClientToken param provided in the CreateInstance command. 2. When a new CreateInstance is requested and contains a ClientToken, the vmpool is retrieved to check if any vm contains that token On 15 October 2014 18:34, Parag Mhashilkar pa...@fnal.gov wrote: That won't help. If the middle ware like HTcondor or run-instance commands starts putting anything in the user data, users are deprived of the functionality of user data. AWS achieves idempotency with --client-token option, which to me seems like a tagging the request from the client side. http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-RunInstances.html Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== On Oct 15, 2014, at 11:27 AM, Daniel Molina wrote: On 15 October 2014 18:18, Parag Mhashilkar pa...@fnal.gov wrote: Hi Daniel, Let me rephrase what we meant. As Steve mentioned in one of his emails, we use HTCondor to launch VMs. There is always a possibility that something can go wrong after a run-instance command is issued and before the ec2 server gets back to you with a valid instance id or an error message. Note that in this case the server may serve the request, but there is no way for the client to know the exact instance name that resulted from the request. Life is much easier if the client crashes after it gets back the instance name. One way for the client to know the exact status of the request even after a crash (and before it gets the instance id) is to assign a unique identifier of its own like a tag name or something. This way client will never lose track of the request and result in the leaked VMs. The way HTCondor is implementing this is by creating a new key pair and giving it a unique name and tracking the requests based on the key pair. But then we are hit by other limit ~300 that we observed. Ideally, we would like to use the --keypair option and use the existing pre registered key. But than we cannot track the results of the create-instance in case something goes wrong before the instance id is issued. And what about including a tag in the UserData instead of using the unique keypar? Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== On Oct 15, 2014, at 11:02 AM, Daniel Molina wrote: On 15 October 2014 17:57, Steven Timm t...@fnal.gov wrote: If you call the same CreateInstances command more than once is there any way that it will create the instance twice or not. If you call the command twice it will create 2 instances. You can also provide de MaxCount param in the CreateInstance command to create more than one instance at once. Steve Timm On Wed, 15 Oct 2014, Daniel Molina wrote: Hi, What do you mean with idempotent? As long as the client implements the ec2 API, it should work Cheers On 13 October 2014 20:33, Parag Mhashilkar pa...@fnal.gov wrote: Hi, We are using HTCondor to launch VMs in OpenNebula using ec2 interface and would like to know if the submit calls are idempotent. Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson
Re: [one-users] econe-run-instances idempotent?
Hi Daniel Such a feature would be really useful when provisioning VMs and avoiding leaks. To be useful, this also requires support for client token in ec2-describe-instances equivalent in open nebula http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-DescribeInstances.html AWS uses client token to achieve idempotency in few other calls too, but I will leave that upto you on if and where you want to support such a functionality. Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== On Oct 15, 2014, at 11:44 AM, Daniel Molina wrote: I think tthat could be implemented in the econe server. 1. Include an EC2_CLIENT_TOKEN in the vm.allocate method containing the ClientToken param provided in the CreateInstance command. 2. When a new CreateInstance is requested and contains a ClientToken, the vmpool is retrieved to check if any vm contains that token On 15 October 2014 18:34, Parag Mhashilkar pa...@fnal.gov wrote: That won't help. If the middle ware like HTcondor or run-instance commands starts putting anything in the user data, users are deprived of the functionality of user data. AWS achieves idempotency with --client-token option, which to me seems like a tagging the request from the client side. http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-RunInstances.html Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== On Oct 15, 2014, at 11:27 AM, Daniel Molina wrote: On 15 October 2014 18:18, Parag Mhashilkar pa...@fnal.gov wrote: Hi Daniel, Let me rephrase what we meant. As Steve mentioned in one of his emails, we use HTCondor to launch VMs. There is always a possibility that something can go wrong after a run-instance command is issued and before the ec2 server gets back to you with a valid instance id or an error message. Note that in this case the server may serve the request, but there is no way for the client to know the exact instance name that resulted from the request. Life is much easier if the client crashes after it gets back the instance name. One way for the client to know the exact status of the request even after a crash (and before it gets the instance id) is to assign a unique identifier of its own like a tag name or something. This way client will never lose track of the request and result in the leaked VMs. The way HTCondor is implementing this is by creating a new key pair and giving it a unique name and tracking the requests based on the key pair. But then we are hit by other limit ~300 that we observed. Ideally, we would like to use the --keypair option and use the existing pre registered key. But than we cannot track the results of the create-instance in case something goes wrong before the instance id is issued. And what about including a tag in the UserData instead of using the unique keypar? Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== On Oct 15, 2014, at 11:02 AM, Daniel Molina wrote: On 15 October 2014 17:57, Steven Timm t...@fnal.gov wrote: If you call the same CreateInstances command more than once is there any way that it will create the instance twice or not. If you call the command twice it will create 2
[one-users] econe-run-instances idempotent?
Hi, We are using HTCondor to launch VMs in OpenNebula using ec2 interface and would like to know if the submit calls are idempotent. Thanks Regards +== | Parag Mhashilkar | Fermi National Accelerator Laboratory, MS 120 | Wilson Kirk Road, Batavia, IL - 60510 |-- | Phone: 1 (630) 840-6530 Fax: 1 (630) 840-3109 |-- | Wilson Hall, 806E (Nov 8, 2012 - To date) | Wilson Hall, 867E (Nov 17, 2010 - Nov 7, 2012) | Wilson Hall, 863E (Apr 24, 2007 - Nov 16, 2010) | Wilson Hall, 856E (Mar 21, 2005 - Apr 23, 2007) +== smime.p7s Description: S/MIME cryptographic signature ___ Users mailing list Users@lists.opennebula.org http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
