Hi Curtis, Thank you for updating the status and correction about the OpenStack Perl SDKs.
If DNS lookup is not reliable to find the private ip of an instance for a multi-host openstack environment, then do you have any suggestion to deal with this issue? At this point, I think at least we have an agreement that we do NOT want to use /etc/hosts anymore, right? If so, we can first try to use the database to get/update the private ip address with DNS lookup for a single-host openstack environment. After that, we can try to find a solution to manage the DNS lookup or other methods of finding the private ip of an instance in multi-host openstack environment. If you have any other suggestions, please let me know. Thanks, Young-Hyun On Mon, Jun 2, 2014 at 3:56 PM, Curtis <[email protected]> wrote: > On Mon, Jun 2, 2014 at 12:45 PM, YOUNG OH <[email protected]> wrote: > > Hi All, > > > > I would like to bring your attention again to discuss what OpenStack > module > > should include. I would like to discus two issues first. > > > > First, we know that using /etc/hosts to get the private ip of an instance > > can be problematic. Specifically, Cameron (at Cybera) and I had found > that > > there might duplicated or blank entries and rarely but OpenStack APIs > > cannot get the private ip (not populated) of an instance properly. For > the > > later case, Cameron added DNS Lookup function to get the correct private > IP > > (Cameron, correct me if I am wrong). This problem leads us to avoid using > > /etc/hosts file to get the private ip but I think we need to first > discuss > > what the best way is to handle the private ip: > > > > (a). Use the privateIPaddress field in computer table. > > Andy suggests that we can change the get_computer_private_ip to get > private > > ip from the database instead of using /etc/hosts (Andy, correct me if I > am > > wrong). I agree with his idea because this can avoid duplicated or blank > > entry problem of using /etc/hosts. > > > > (b). OpenStack (DNS lookup for finding unpopulated private ip) > > In our openstack module, DNS lookup or other methods (if any) can give > the > > correct the private ip of the instance in order to deal with the problem > > for the private ip not being populated in the hosts file. Cameron > modified > > the get_computer_private_ip in DataStructure.pm to do a DNS lookup > instead > > of using the hosts file. But I would like to suggest that if possible, it > > can be better to add the DNS lookup function to openstack module rather > > than changing the DataStructure.pm. That is, we can update the > > privateIPaddress field in computer table using DNS lookup instead of > > injecting the private ip to /etc/hosts in openstack module. > > We've just found out recently that the helpful DNS resolution that > openstack provides does not work well in a mutlihost openstack > environment, so can't be relied on. In multihost mode each compute > host only knows about the vms running on it locally. > > > > > I think, if possible, one of the best methods for dealing with the > private > > ip is that we can use the database to get the private ip of an instance > and > > using DNS lookup in openstack module to update the information. Any > > suggestions and corrections about my opinion would be very helpful to > solve > > this problem. > > > > Second, currently we have two versions (not exactly but for convenience); > > one uses OpenStack command line interfaces (CLIs) and one uses OpenStack > > Perl SDKs. As I understand, the Perl SDKs are wrapper APIs for the > > OpenStcak CLIs and required to install through CPAN. I think both > versions > > have pros and cons. So, I would like to ask your opinions whether it is > > better to include two different openstack versions to the next VCL > release, > > to find a way to merge two versions into one openstack module, OR to > create > > a higher level of openstack module so that it can choose one version > based > > on user's environment. Any suggestions would be also very helpful to > decide > > how we can manage openstack module. Thank you. > > FYI -- The perl openstack sdk is not a wrapper for the command line > utils. It uses the openstack api, not the command line tools. > > Thanks, > Curtis. > > > > > Best regards, > > Young-Hyun > > > > > > > > On Mon, Jun 2, 2014 at 2:22 PM, YOUNG OH <[email protected]> > wrote: > > > >> Hi All, > >> > >> I have tested my openstack module ( > >> https://issues.apache.org/jira/browse/VCL-590) on the OpenStack > Icehouse > >> (The ninth release of OpenStack, April 17, 2014) and I could > provisioning a > >> Linux image through it. > >> > >> The environment details are as follows: > >> > >> 1. Host Blade > >> :- CPU: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz > >> :- Memory: 49G > >> :- Disk: 134G > >> :- OS: RHEL 6.4 x64 with KVM hypervisor > >> 2. OpenStack Icehouse Components > >> :- Compute > >> :- Network (Nova-Network, not OpenStack Neutron) > >> :- Glance > >> 3. Provisioning Linux image > >> :- CentOS 6.3 x64 > >> 4. VCL 2.3 > >> > >> In this test, I've just tested the provisioning an image but I would > like > >> to post the detail information to jira once I finish other test. In > >> addition, I will try to add a function of selecting OpenStack flavors > based > >> on the size of an image. The current openstack module sets flavors for > each > >> image because the previous OpenStack releases (Essex as I know) didn't > >> support the size information of images but now it supports. I will also > >> post to jira issue when I add the function to my module. Thank you. > >> > >> Best regards, > >> Young-Hyun > >> > >> > >> > >> On Mon, Mar 31, 2014 at 12:55 PM, Cameron Mann <[email protected]> > >> wrote: > >> > >>> > > >>> > I haven't looked at Try::Tiny or any newer exception handling > features > >>> of > >>> > Perl, but I'd prefer to keep the number of external modules to a > >>> minimum if > >>> > possible. If there are new features in Perl which would be > beneficial, > >>> I'd > >>> > rather make the newer version of Perl a requirement and use the > built in > >>> > features. Otherwise, we're locked into using an external module, > >>> usually > >>> > from CPAN, which has no quality assurance. I regret adding some of > the > >>> > current modules such as InsideOut, rather then writing the code which > >>> could > >>> > accomplish the same objective. > >>> > Did you need to catch exceptions because one of the OpenStack modules > >>> > you're using may throw a die exception? When this happens, the > entire > >>> vcld > >>> > process for the reservation exits abruptly which causes problems > such as > >>> > the request state remaining in pending indefinitely. This happens > even > >>> if > >>> > you wrap the call in an eval block. I have worked around this > problem > >>> > before by locally overriding the die handler. Take a look at > >>> > vSphere_SDK.pm and find lines such as: > >>> > # Override the die handler > >>> > local $SIG{__DIE__} = sub{}; > >>> > The vSphere SDK is one example which may call die (via Carp). If > this > >>> > happens with die overridden, the eval block returns normally. If an > >>> > exception occurred, $EVAL_ERROR should be set. > >>> > >>> > >>> The OpenStack SDK throws exceptions when requests that > >>> delete/modify/create > >>> resources fail, no die exceptions though. There wasn't any specific > >>> feature > >>> that made me choose Try::Tiny, it was more my unfamiliarity with > exception > >>> handling in Perl which seemed to have a number of gotchas prior to > 5.14 so > >>> I figured it'd be better to let someone else's code worry about that. > >>> > >>> > >>> > http://blog.twoshortplanks.com/2011/06/06/unexceptional-exceptions-in-perl-5-14/ > >>> > >>> I see no problems. Any barriers or difficulty you encounter with the > core > >>> > code should be addressed by improving the core code. > >>> > Reworking the dependency on /etc/hosts is something I'd like to > improve > >>> > regardless. People often encounter problems with this if they > forget to > >>> > add entries to /etc/hosts for all their computers, including the > >>> management > >>> > node(s). I'd like to not read /etc/hosts on the management node at > all > >>> and > >>> > have the value in the computer table be authoritative. This should > >>> > facilitate cases where the provisioning module needs to do something > >>> > different regarding private IPs such as if they are dynamically > >>> allocated. > >>> > The provisioning module would retrieve the IP when the image gets > >>> loaded > >>> > by querying DNS or some other method and then update the database. > The > >>> OS > >>> > module would then use this value to configure the computer. > >>> > >>> > >>> Sounds good. I agree that avoiding the hosts file would be best, > >>> especially > >>> in a case like this where it needs to be continuously modified. Here's > our > >>> changed get_computer_private_ip_address, lines 45-55 are the new code: > >>> > >>> https://gist.github.com/cmann/9896513 > >>> > >>> Finally we're still working on putting together more comprehensive > >>> installation documentation for anyone who wants to try it out. We're > use > >>> automation for a number of the tasks and I haven't had a chance to run > >>> through it from scratch yet. For anyone feeling adventurous these are > the > >>> most basic instructions: > >>> > >>> 1. Apply the NAT patch > >>> (https://issues.apache.org/jira/browse/VCL-174tested on 2.3 and 2.3.1, > >>> make sure to read the Google doc) > >>> 2. Follow the instructions in the readme in the Github repo for the > >>> OpenStack module. > >>> > >>> Cameron > >>> > >>> > >>> On Fri, Mar 28, 2014 at 7:54 AM, Andy Kurth <[email protected]> > wrote: > >>> > >>> > > > >>> > > > >>> > > 2. We use the Try::Tiny module to handle exceptions since the > version > >>> of > >>> > > Perl VCL 2.3 uses doesn't support the saner try/catch mechanisms > >>> > available > >>> > > in newer releases, however this would add an additional dependency > to > >>> > VCL. > >>> > > Depending on the version of Perl VCL 2.4 is planning to use what > are > >>> > > people's thoughts on using Try::Tiny versus switching to Perl's > >>> internal > >>> > > mechanisms? > >>> > > > >>> > > >>> > I haven't looked at Try::Tiny or any newer exception handling > features > >>> of > >>> > Perl, but I'd prefer to keep the number of external modules to a > >>> minimum if > >>> > possible. If there are new features in Perl which would be > beneficial, > >>> I'd > >>> > rather make the newer version of Perl a requirement and use the > built in > >>> > features. Otherwise, we're locked into using an external module, > >>> usually > >>> > from CPAN, which has no quality assurance. I regret adding some of > the > >>> > current modules such as InsideOut, rather then writing the code which > >>> could > >>> > accomplish the same objective. > >>> > > >>> > Did you need to catch exceptions because one of the OpenStack modules > >>> > you're using may throw a die exception? When this happens, the > entire > >>> vcld > >>> > process for the reservation exits abruptly which causes problems > such as > >>> > the request state remaining in pending indefinitely. This happens > even > >>> if > >>> > you wrap the call in an eval block. I have worked around this > problem > >>> > before by locally overriding the die handler. Take a look at > >>> > vSphere_SDK.pm and find lines such as: > >>> > > >>> > # Override the die handler > >>> > local $SIG{__DIE__} = sub{}; > >>> > > >>> > The vSphere SDK is one example which may call die (via Carp). If > this > >>> > happens with die overridden, the eval block returns normally. If an > >>> > exception occurred, $EVAL_ERROR should be set. > >>> > > >>> > > >>> > > > >>> > > 3. I've saved the best for last, but first some background. The > >>> biggest > >>> > > change we've made is that we no longer rely on the /etc/hosts file > at > >>> all > >>> > > (though we haven't removed the code that keeps it up to date). We > had > >>> a > >>> > > number of issues with duplicate entries ending up in the hosts file > >>> and > >>> > > weren't able to pin down the exact cause. This led us to adding a > >>> table > >>> > to > >>> > > the database to track VCL computer to OpenStack instance mappings. > The > >>> > next > >>> > > issue we ran into was the private IP address not being populated in > >>> the > >>> > > hosts file. This happened very rarely and we're still unsure of the > >>> cause > >>> > > especially since there's an explicit check that the address is > >>> populated > >>> > > with a value before we write the hosts file. I'd assume that the > Nova > >>> API > >>> > > occasionally returns a response where the addresses component is > >>> > structured > >>> > > differently than we expect, though I was never able to directly > >>> observe > >>> > > this behavior. Since OpenStack automatically creates DNS entries > for > >>> new > >>> > > instances we decided the best course of action was to bypass the > hosts > >>> > file > >>> > > entirely. In the end we modified get_computer_private_ip_address in > >>> > > DataStructure.pm to do a DNS lookup rather than looking in the > hosts > >>> > file. > >>> > > > >>> > > The question for the community is are changes to core VCL code > >>> > acceptable? > >>> > > Or should we find a solution that keeps everything contained to the > >>> > > provisioning module? If changes to core code are acceptable we'd > also > >>> > have > >>> > > to decide on a method of detecting when to do a DNS lookup. Should > it > >>> > > depend on the provisioning module in use? A configuration option? > >>> > > >>> > > >>> > I see no problems. Any barriers or difficulty you encounter with the > >>> core > >>> > code should be addressed by improving the core code. > >>> > > >>> > Reworking the dependency on /etc/hosts is something I'd like to > improve > >>> > regardless. People often encounter problems with this if they > forget to > >>> > add entries to /etc/hosts for all their computers, including the > >>> management > >>> > node(s). I'd like to not read /etc/hosts on the management node at > all > >>> and > >>> > have the value in the computer table be authoritative. This should > >>> > facilitate cases where the provisioning module needs to do something > >>> > different regarding private IPs such as if they are dynamically > >>> allocated. > >>> > The provisioning module would retrieve the IP when the image gets > >>> loaded > >>> > by querying DNS or some other method and then update the database. > The > >>> OS > >>> > module would then use this value to configure the computer. > >>> > > >>> > One comment regarding adding tables... great! Add all the tables you > >>> think > >>> > are beneficial. I have seen some try to work around the current > >>> database > >>> > schema rather than adding to it. There's no reason for this. If > there > >>> are > >>> > configuration options which apply only to OpenStack, add something > like > >>> an > >>> > openstackconfig table. If the vmprofile table doesn't align, add > >>> something > >>> > like an openstackprofile table. If there are general options such as > >>> how > >>> > to handle DNS or private IP resolution, we could add a more general > >>> table > >>> > or possibly extend the managementnode or computer table. > >>> > > >>> > Thanks, > >>> > Andy > >>> > > >>> > >> > >> > > > > -- > Twitter: @serverascode > Blog: serverascode.com >
