On Tue, Jun 3, 2014 at 11:53 AM, YOUNG OH <[email protected]> wrote: > Hi Curtis, > > Thank you for updating the status and correction about the OpenStack Perl > SDKs. > > If DNS lookup is not reliable to find the private ip of an instance for a > multi-host openstack environment, then do you have any suggestion to deal > with this issue? > > At this point, I think at least we have an agreement that we do NOT want to > use /etc/hosts anymore, right? > If so, we can first try to use the database to get/update the private ip > address with DNS lookup for a single-host openstack environment. After > that, we can try to find a solution to manage the DNS lookup or other > methods of finding the private ip of an instance in multi-host openstack > environment. If you have any other suggestions, please let me know. >
I'm not too sure what to do. I'm hoping the Apache VCL core devs chime in. As far as I can tell, Apache VCL needs DNS resolution to work, so simply adding the IP into the database isn't going to be enough. For the current Apache VCL + OpenStack multihost cloud installation I'm working on I'm just going to add a python script that queries OpenStack and generates /etc/hosts every 20 or 30 seconds or something like that. Thanks, Curtis. > Thanks, > Young-Hyun > > > > On Mon, Jun 2, 2014 at 3:56 PM, Curtis <[email protected]> wrote: > >> On Mon, Jun 2, 2014 at 12:45 PM, YOUNG OH <[email protected]> wrote: >> > Hi All, >> > >> > I would like to bring your attention again to discuss what OpenStack >> module >> > should include. I would like to discus two issues first. >> > >> > First, we know that using /etc/hosts to get the private ip of an instance >> > can be problematic. Specifically, Cameron (at Cybera) and I had found >> that >> > there might duplicated or blank entries and rarely but OpenStack APIs >> > cannot get the private ip (not populated) of an instance properly. For >> the >> > later case, Cameron added DNS Lookup function to get the correct private >> IP >> > (Cameron, correct me if I am wrong). This problem leads us to avoid using >> > /etc/hosts file to get the private ip but I think we need to first >> discuss >> > what the best way is to handle the private ip: >> > >> > (a). Use the privateIPaddress field in computer table. >> > Andy suggests that we can change the get_computer_private_ip to get >> private >> > ip from the database instead of using /etc/hosts (Andy, correct me if I >> am >> > wrong). I agree with his idea because this can avoid duplicated or blank >> > entry problem of using /etc/hosts. >> > >> > (b). OpenStack (DNS lookup for finding unpopulated private ip) >> > In our openstack module, DNS lookup or other methods (if any) can give >> the >> > correct the private ip of the instance in order to deal with the problem >> > for the private ip not being populated in the hosts file. Cameron >> modified >> > the get_computer_private_ip in DataStructure.pm to do a DNS lookup >> instead >> > of using the hosts file. But I would like to suggest that if possible, it >> > can be better to add the DNS lookup function to openstack module rather >> > than changing the DataStructure.pm. That is, we can update the >> > privateIPaddress field in computer table using DNS lookup instead of >> > injecting the private ip to /etc/hosts in openstack module. >> >> We've just found out recently that the helpful DNS resolution that >> openstack provides does not work well in a mutlihost openstack >> environment, so can't be relied on. In multihost mode each compute >> host only knows about the vms running on it locally. >> >> > >> > I think, if possible, one of the best methods for dealing with the >> private >> > ip is that we can use the database to get the private ip of an instance >> and >> > using DNS lookup in openstack module to update the information. Any >> > suggestions and corrections about my opinion would be very helpful to >> solve >> > this problem. >> > >> > Second, currently we have two versions (not exactly but for convenience); >> > one uses OpenStack command line interfaces (CLIs) and one uses OpenStack >> > Perl SDKs. As I understand, the Perl SDKs are wrapper APIs for the >> > OpenStcak CLIs and required to install through CPAN. I think both >> versions >> > have pros and cons. So, I would like to ask your opinions whether it is >> > better to include two different openstack versions to the next VCL >> release, >> > to find a way to merge two versions into one openstack module, OR to >> create >> > a higher level of openstack module so that it can choose one version >> based >> > on user's environment. Any suggestions would be also very helpful to >> decide >> > how we can manage openstack module. Thank you. >> >> FYI -- The perl openstack sdk is not a wrapper for the command line >> utils. It uses the openstack api, not the command line tools. >> >> Thanks, >> Curtis. >> >> > >> > Best regards, >> > Young-Hyun >> > >> > >> > >> > On Mon, Jun 2, 2014 at 2:22 PM, YOUNG OH <[email protected]> >> wrote: >> > >> >> Hi All, >> >> >> >> I have tested my openstack module ( >> >> https://issues.apache.org/jira/browse/VCL-590) on the OpenStack >> Icehouse >> >> (The ninth release of OpenStack, April 17, 2014) and I could >> provisioning a >> >> Linux image through it. >> >> >> >> The environment details are as follows: >> >> >> >> 1. Host Blade >> >> :- CPU: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz >> >> :- Memory: 49G >> >> :- Disk: 134G >> >> :- OS: RHEL 6.4 x64 with KVM hypervisor >> >> 2. OpenStack Icehouse Components >> >> :- Compute >> >> :- Network (Nova-Network, not OpenStack Neutron) >> >> :- Glance >> >> 3. Provisioning Linux image >> >> :- CentOS 6.3 x64 >> >> 4. VCL 2.3 >> >> >> >> In this test, I've just tested the provisioning an image but I would >> like >> >> to post the detail information to jira once I finish other test. In >> >> addition, I will try to add a function of selecting OpenStack flavors >> based >> >> on the size of an image. The current openstack module sets flavors for >> each >> >> image because the previous OpenStack releases (Essex as I know) didn't >> >> support the size information of images but now it supports. I will also >> >> post to jira issue when I add the function to my module. Thank you. >> >> >> >> Best regards, >> >> Young-Hyun >> >> >> >> >> >> >> >> On Mon, Mar 31, 2014 at 12:55 PM, Cameron Mann <[email protected]> >> >> wrote: >> >> >> >>> > >> >>> > I haven't looked at Try::Tiny or any newer exception handling >> features >> >>> of >> >>> > Perl, but I'd prefer to keep the number of external modules to a >> >>> minimum if >> >>> > possible. If there are new features in Perl which would be >> beneficial, >> >>> I'd >> >>> > rather make the newer version of Perl a requirement and use the >> built in >> >>> > features. Otherwise, we're locked into using an external module, >> >>> usually >> >>> > from CPAN, which has no quality assurance. I regret adding some of >> the >> >>> > current modules such as InsideOut, rather then writing the code which >> >>> could >> >>> > accomplish the same objective. >> >>> > Did you need to catch exceptions because one of the OpenStack modules >> >>> > you're using may throw a die exception? When this happens, the >> entire >> >>> vcld >> >>> > process for the reservation exits abruptly which causes problems >> such as >> >>> > the request state remaining in pending indefinitely. This happens >> even >> >>> if >> >>> > you wrap the call in an eval block. I have worked around this >> problem >> >>> > before by locally overriding the die handler. Take a look at >> >>> > vSphere_SDK.pm and find lines such as: >> >>> > # Override the die handler >> >>> > local $SIG{__DIE__} = sub{}; >> >>> > The vSphere SDK is one example which may call die (via Carp). If >> this >> >>> > happens with die overridden, the eval block returns normally. If an >> >>> > exception occurred, $EVAL_ERROR should be set. >> >>> >> >>> >> >>> The OpenStack SDK throws exceptions when requests that >> >>> delete/modify/create >> >>> resources fail, no die exceptions though. There wasn't any specific >> >>> feature >> >>> that made me choose Try::Tiny, it was more my unfamiliarity with >> exception >> >>> handling in Perl which seemed to have a number of gotchas prior to >> 5.14 so >> >>> I figured it'd be better to let someone else's code worry about that. >> >>> >> >>> >> >>> >> http://blog.twoshortplanks.com/2011/06/06/unexceptional-exceptions-in-perl-5-14/ >> >>> >> >>> I see no problems. Any barriers or difficulty you encounter with the >> core >> >>> > code should be addressed by improving the core code. >> >>> > Reworking the dependency on /etc/hosts is something I'd like to >> improve >> >>> > regardless. People often encounter problems with this if they >> forget to >> >>> > add entries to /etc/hosts for all their computers, including the >> >>> management >> >>> > node(s). I'd like to not read /etc/hosts on the management node at >> all >> >>> and >> >>> > have the value in the computer table be authoritative. This should >> >>> > facilitate cases where the provisioning module needs to do something >> >>> > different regarding private IPs such as if they are dynamically >> >>> allocated. >> >>> > The provisioning module would retrieve the IP when the image gets >> >>> loaded >> >>> > by querying DNS or some other method and then update the database. >> The >> >>> OS >> >>> > module would then use this value to configure the computer. >> >>> >> >>> >> >>> Sounds good. I agree that avoiding the hosts file would be best, >> >>> especially >> >>> in a case like this where it needs to be continuously modified. Here's >> our >> >>> changed get_computer_private_ip_address, lines 45-55 are the new code: >> >>> >> >>> https://gist.github.com/cmann/9896513 >> >>> >> >>> Finally we're still working on putting together more comprehensive >> >>> installation documentation for anyone who wants to try it out. We're >> use >> >>> automation for a number of the tasks and I haven't had a chance to run >> >>> through it from scratch yet. For anyone feeling adventurous these are >> the >> >>> most basic instructions: >> >>> >> >>> 1. Apply the NAT patch >> >>> (https://issues.apache.org/jira/browse/VCL-174tested on 2.3 and 2.3.1, >> >>> make sure to read the Google doc) >> >>> 2. Follow the instructions in the readme in the Github repo for the >> >>> OpenStack module. >> >>> >> >>> Cameron >> >>> >> >>> >> >>> On Fri, Mar 28, 2014 at 7:54 AM, Andy Kurth <[email protected]> >> wrote: >> >>> >> >>> > > >> >>> > > >> >>> > > 2. We use the Try::Tiny module to handle exceptions since the >> version >> >>> of >> >>> > > Perl VCL 2.3 uses doesn't support the saner try/catch mechanisms >> >>> > available >> >>> > > in newer releases, however this would add an additional dependency >> to >> >>> > VCL. >> >>> > > Depending on the version of Perl VCL 2.4 is planning to use what >> are >> >>> > > people's thoughts on using Try::Tiny versus switching to Perl's >> >>> internal >> >>> > > mechanisms? >> >>> > > >> >>> > >> >>> > I haven't looked at Try::Tiny or any newer exception handling >> features >> >>> of >> >>> > Perl, but I'd prefer to keep the number of external modules to a >> >>> minimum if >> >>> > possible. If there are new features in Perl which would be >> beneficial, >> >>> I'd >> >>> > rather make the newer version of Perl a requirement and use the >> built in >> >>> > features. Otherwise, we're locked into using an external module, >> >>> usually >> >>> > from CPAN, which has no quality assurance. I regret adding some of >> the >> >>> > current modules such as InsideOut, rather then writing the code which >> >>> could >> >>> > accomplish the same objective. >> >>> > >> >>> > Did you need to catch exceptions because one of the OpenStack modules >> >>> > you're using may throw a die exception? When this happens, the >> entire >> >>> vcld >> >>> > process for the reservation exits abruptly which causes problems >> such as >> >>> > the request state remaining in pending indefinitely. This happens >> even >> >>> if >> >>> > you wrap the call in an eval block. I have worked around this >> problem >> >>> > before by locally overriding the die handler. Take a look at >> >>> > vSphere_SDK.pm and find lines such as: >> >>> > >> >>> > # Override the die handler >> >>> > local $SIG{__DIE__} = sub{}; >> >>> > >> >>> > The vSphere SDK is one example which may call die (via Carp). If >> this >> >>> > happens with die overridden, the eval block returns normally. If an >> >>> > exception occurred, $EVAL_ERROR should be set. >> >>> > >> >>> > >> >>> > > >> >>> > > 3. I've saved the best for last, but first some background. The >> >>> biggest >> >>> > > change we've made is that we no longer rely on the /etc/hosts file >> at >> >>> all >> >>> > > (though we haven't removed the code that keeps it up to date). We >> had >> >>> a >> >>> > > number of issues with duplicate entries ending up in the hosts file >> >>> and >> >>> > > weren't able to pin down the exact cause. This led us to adding a >> >>> table >> >>> > to >> >>> > > the database to track VCL computer to OpenStack instance mappings. >> The >> >>> > next >> >>> > > issue we ran into was the private IP address not being populated in >> >>> the >> >>> > > hosts file. This happened very rarely and we're still unsure of the >> >>> cause >> >>> > > especially since there's an explicit check that the address is >> >>> populated >> >>> > > with a value before we write the hosts file. I'd assume that the >> Nova >> >>> API >> >>> > > occasionally returns a response where the addresses component is >> >>> > structured >> >>> > > differently than we expect, though I was never able to directly >> >>> observe >> >>> > > this behavior. Since OpenStack automatically creates DNS entries >> for >> >>> new >> >>> > > instances we decided the best course of action was to bypass the >> hosts >> >>> > file >> >>> > > entirely. In the end we modified get_computer_private_ip_address in >> >>> > > DataStructure.pm to do a DNS lookup rather than looking in the >> hosts >> >>> > file. >> >>> > > >> >>> > > The question for the community is are changes to core VCL code >> >>> > acceptable? >> >>> > > Or should we find a solution that keeps everything contained to the >> >>> > > provisioning module? If changes to core code are acceptable we'd >> also >> >>> > have >> >>> > > to decide on a method of detecting when to do a DNS lookup. Should >> it >> >>> > > depend on the provisioning module in use? A configuration option? >> >>> > >> >>> > >> >>> > I see no problems. Any barriers or difficulty you encounter with the >> >>> core >> >>> > code should be addressed by improving the core code. >> >>> > >> >>> > Reworking the dependency on /etc/hosts is something I'd like to >> improve >> >>> > regardless. People often encounter problems with this if they >> forget to >> >>> > add entries to /etc/hosts for all their computers, including the >> >>> management >> >>> > node(s). I'd like to not read /etc/hosts on the management node at >> all >> >>> and >> >>> > have the value in the computer table be authoritative. This should >> >>> > facilitate cases where the provisioning module needs to do something >> >>> > different regarding private IPs such as if they are dynamically >> >>> allocated. >> >>> > The provisioning module would retrieve the IP when the image gets >> >>> loaded >> >>> > by querying DNS or some other method and then update the database. >> The >> >>> OS >> >>> > module would then use this value to configure the computer. >> >>> > >> >>> > One comment regarding adding tables... great! Add all the tables you >> >>> think >> >>> > are beneficial. I have seen some try to work around the current >> >>> database >> >>> > schema rather than adding to it. There's no reason for this. If >> there >> >>> are >> >>> > configuration options which apply only to OpenStack, add something >> like >> >>> an >> >>> > openstackconfig table. If the vmprofile table doesn't align, add >> >>> something >> >>> > like an openstackprofile table. If there are general options such as >> >>> how >> >>> > to handle DNS or private IP resolution, we could add a more general >> >>> table >> >>> > or possibly extend the managementnode or computer table. >> >>> > >> >>> > Thanks, >> >>> > Andy >> >>> > >> >>> >> >> >> >> >> >> >> >> -- >> Twitter: @serverascode >> Blog: serverascode.com >> -- Twitter: @serverascode Blog: serverascode.com
