Would using the private IP instead of hostnames resolve it? It would take a bit of work, but could be done.
Aaron On Tue, Jun 3, 2014 at 3:22 PM, Curtis <[email protected]> wrote: > On Tue, Jun 3, 2014 at 11:53 AM, YOUNG OH <[email protected]> wrote: >> Hi Curtis, >> >> Thank you for updating the status and correction about the OpenStack Perl >> SDKs. >> >> If DNS lookup is not reliable to find the private ip of an instance for a >> multi-host openstack environment, then do you have any suggestion to deal >> with this issue? >> >> At this point, I think at least we have an agreement that we do NOT want to >> use /etc/hosts anymore, right? >> If so, we can first try to use the database to get/update the private ip >> address with DNS lookup for a single-host openstack environment. After >> that, we can try to find a solution to manage the DNS lookup or other >> methods of finding the private ip of an instance in multi-host openstack >> environment. If you have any other suggestions, please let me know. >> > > I'm not too sure what to do. I'm hoping the Apache VCL core devs chime in. > > As far as I can tell, Apache VCL needs DNS resolution to work, so > simply adding the IP into the database isn't going to be enough. > > For the current Apache VCL + OpenStack multihost cloud installation > I'm working on I'm just going to add a python script that queries > OpenStack and generates /etc/hosts every 20 or 30 seconds or something > like that. > > Thanks, > Curtis. > >> Thanks, >> Young-Hyun >> >> >> >> On Mon, Jun 2, 2014 at 3:56 PM, Curtis <[email protected]> wrote: >> >>> On Mon, Jun 2, 2014 at 12:45 PM, YOUNG OH <[email protected]> wrote: >>> > Hi All, >>> > >>> > I would like to bring your attention again to discuss what OpenStack >>> module >>> > should include. I would like to discus two issues first. >>> > >>> > First, we know that using /etc/hosts to get the private ip of an instance >>> > can be problematic. Specifically, Cameron (at Cybera) and I had found >>> that >>> > there might duplicated or blank entries and rarely but OpenStack APIs >>> > cannot get the private ip (not populated) of an instance properly. For >>> the >>> > later case, Cameron added DNS Lookup function to get the correct private >>> IP >>> > (Cameron, correct me if I am wrong). This problem leads us to avoid using >>> > /etc/hosts file to get the private ip but I think we need to first >>> discuss >>> > what the best way is to handle the private ip: >>> > >>> > (a). Use the privateIPaddress field in computer table. >>> > Andy suggests that we can change the get_computer_private_ip to get >>> private >>> > ip from the database instead of using /etc/hosts (Andy, correct me if I >>> am >>> > wrong). I agree with his idea because this can avoid duplicated or blank >>> > entry problem of using /etc/hosts. >>> > >>> > (b). OpenStack (DNS lookup for finding unpopulated private ip) >>> > In our openstack module, DNS lookup or other methods (if any) can give >>> the >>> > correct the private ip of the instance in order to deal with the problem >>> > for the private ip not being populated in the hosts file. Cameron >>> modified >>> > the get_computer_private_ip in DataStructure.pm to do a DNS lookup >>> instead >>> > of using the hosts file. But I would like to suggest that if possible, it >>> > can be better to add the DNS lookup function to openstack module rather >>> > than changing the DataStructure.pm. That is, we can update the >>> > privateIPaddress field in computer table using DNS lookup instead of >>> > injecting the private ip to /etc/hosts in openstack module. >>> >>> We've just found out recently that the helpful DNS resolution that >>> openstack provides does not work well in a mutlihost openstack >>> environment, so can't be relied on. In multihost mode each compute >>> host only knows about the vms running on it locally. >>> >>> > >>> > I think, if possible, one of the best methods for dealing with the >>> private >>> > ip is that we can use the database to get the private ip of an instance >>> and >>> > using DNS lookup in openstack module to update the information. Any >>> > suggestions and corrections about my opinion would be very helpful to >>> solve >>> > this problem. >>> > >>> > Second, currently we have two versions (not exactly but for convenience); >>> > one uses OpenStack command line interfaces (CLIs) and one uses OpenStack >>> > Perl SDKs. As I understand, the Perl SDKs are wrapper APIs for the >>> > OpenStcak CLIs and required to install through CPAN. I think both >>> versions >>> > have pros and cons. So, I would like to ask your opinions whether it is >>> > better to include two different openstack versions to the next VCL >>> release, >>> > to find a way to merge two versions into one openstack module, OR to >>> create >>> > a higher level of openstack module so that it can choose one version >>> based >>> > on user's environment. Any suggestions would be also very helpful to >>> decide >>> > how we can manage openstack module. Thank you. >>> >>> FYI -- The perl openstack sdk is not a wrapper for the command line >>> utils. It uses the openstack api, not the command line tools. >>> >>> Thanks, >>> Curtis. >>> >>> > >>> > Best regards, >>> > Young-Hyun >>> > >>> > >>> > >>> > On Mon, Jun 2, 2014 at 2:22 PM, YOUNG OH <[email protected]> >>> wrote: >>> > >>> >> Hi All, >>> >> >>> >> I have tested my openstack module ( >>> >> https://issues.apache.org/jira/browse/VCL-590) on the OpenStack >>> Icehouse >>> >> (The ninth release of OpenStack, April 17, 2014) and I could >>> provisioning a >>> >> Linux image through it. >>> >> >>> >> The environment details are as follows: >>> >> >>> >> 1. Host Blade >>> >> :- CPU: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz >>> >> :- Memory: 49G >>> >> :- Disk: 134G >>> >> :- OS: RHEL 6.4 x64 with KVM hypervisor >>> >> 2. OpenStack Icehouse Components >>> >> :- Compute >>> >> :- Network (Nova-Network, not OpenStack Neutron) >>> >> :- Glance >>> >> 3. Provisioning Linux image >>> >> :- CentOS 6.3 x64 >>> >> 4. VCL 2.3 >>> >> >>> >> In this test, I've just tested the provisioning an image but I would >>> like >>> >> to post the detail information to jira once I finish other test. In >>> >> addition, I will try to add a function of selecting OpenStack flavors >>> based >>> >> on the size of an image. The current openstack module sets flavors for >>> each >>> >> image because the previous OpenStack releases (Essex as I know) didn't >>> >> support the size information of images but now it supports. I will also >>> >> post to jira issue when I add the function to my module. Thank you. >>> >> >>> >> Best regards, >>> >> Young-Hyun >>> >> >>> >> >>> >> >>> >> On Mon, Mar 31, 2014 at 12:55 PM, Cameron Mann <[email protected]> >>> >> wrote: >>> >> >>> >>> > >>> >>> > I haven't looked at Try::Tiny or any newer exception handling >>> features >>> >>> of >>> >>> > Perl, but I'd prefer to keep the number of external modules to a >>> >>> minimum if >>> >>> > possible. If there are new features in Perl which would be >>> beneficial, >>> >>> I'd >>> >>> > rather make the newer version of Perl a requirement and use the >>> built in >>> >>> > features. Otherwise, we're locked into using an external module, >>> >>> usually >>> >>> > from CPAN, which has no quality assurance. I regret adding some of >>> the >>> >>> > current modules such as InsideOut, rather then writing the code which >>> >>> could >>> >>> > accomplish the same objective. >>> >>> > Did you need to catch exceptions because one of the OpenStack modules >>> >>> > you're using may throw a die exception? When this happens, the >>> entire >>> >>> vcld >>> >>> > process for the reservation exits abruptly which causes problems >>> such as >>> >>> > the request state remaining in pending indefinitely. This happens >>> even >>> >>> if >>> >>> > you wrap the call in an eval block. I have worked around this >>> problem >>> >>> > before by locally overriding the die handler. Take a look at >>> >>> > vSphere_SDK.pm and find lines such as: >>> >>> > # Override the die handler >>> >>> > local $SIG{__DIE__} = sub{}; >>> >>> > The vSphere SDK is one example which may call die (via Carp). If >>> this >>> >>> > happens with die overridden, the eval block returns normally. If an >>> >>> > exception occurred, $EVAL_ERROR should be set. >>> >>> >>> >>> >>> >>> The OpenStack SDK throws exceptions when requests that >>> >>> delete/modify/create >>> >>> resources fail, no die exceptions though. There wasn't any specific >>> >>> feature >>> >>> that made me choose Try::Tiny, it was more my unfamiliarity with >>> exception >>> >>> handling in Perl which seemed to have a number of gotchas prior to >>> 5.14 so >>> >>> I figured it'd be better to let someone else's code worry about that. >>> >>> >>> >>> >>> >>> >>> http://blog.twoshortplanks.com/2011/06/06/unexceptional-exceptions-in-perl-5-14/ >>> >>> >>> >>> I see no problems. Any barriers or difficulty you encounter with the >>> core >>> >>> > code should be addressed by improving the core code. >>> >>> > Reworking the dependency on /etc/hosts is something I'd like to >>> improve >>> >>> > regardless. People often encounter problems with this if they >>> forget to >>> >>> > add entries to /etc/hosts for all their computers, including the >>> >>> management >>> >>> > node(s). I'd like to not read /etc/hosts on the management node at >>> all >>> >>> and >>> >>> > have the value in the computer table be authoritative. This should >>> >>> > facilitate cases where the provisioning module needs to do something >>> >>> > different regarding private IPs such as if they are dynamically >>> >>> allocated. >>> >>> > The provisioning module would retrieve the IP when the image gets >>> >>> loaded >>> >>> > by querying DNS or some other method and then update the database. >>> The >>> >>> OS >>> >>> > module would then use this value to configure the computer. >>> >>> >>> >>> >>> >>> Sounds good. I agree that avoiding the hosts file would be best, >>> >>> especially >>> >>> in a case like this where it needs to be continuously modified. Here's >>> our >>> >>> changed get_computer_private_ip_address, lines 45-55 are the new code: >>> >>> >>> >>> https://gist.github.com/cmann/9896513 >>> >>> >>> >>> Finally we're still working on putting together more comprehensive >>> >>> installation documentation for anyone who wants to try it out. We're >>> use >>> >>> automation for a number of the tasks and I haven't had a chance to run >>> >>> through it from scratch yet. For anyone feeling adventurous these are >>> the >>> >>> most basic instructions: >>> >>> >>> >>> 1. Apply the NAT patch >>> >>> (https://issues.apache.org/jira/browse/VCL-174tested on 2.3 and 2.3.1, >>> >>> make sure to read the Google doc) >>> >>> 2. Follow the instructions in the readme in the Github repo for the >>> >>> OpenStack module. >>> >>> >>> >>> Cameron >>> >>> >>> >>> >>> >>> On Fri, Mar 28, 2014 at 7:54 AM, Andy Kurth <[email protected]> >>> wrote: >>> >>> >>> >>> > > >>> >>> > > >>> >>> > > 2. We use the Try::Tiny module to handle exceptions since the >>> version >>> >>> of >>> >>> > > Perl VCL 2.3 uses doesn't support the saner try/catch mechanisms >>> >>> > available >>> >>> > > in newer releases, however this would add an additional dependency >>> to >>> >>> > VCL. >>> >>> > > Depending on the version of Perl VCL 2.4 is planning to use what >>> are >>> >>> > > people's thoughts on using Try::Tiny versus switching to Perl's >>> >>> internal >>> >>> > > mechanisms? >>> >>> > > >>> >>> > >>> >>> > I haven't looked at Try::Tiny or any newer exception handling >>> features >>> >>> of >>> >>> > Perl, but I'd prefer to keep the number of external modules to a >>> >>> minimum if >>> >>> > possible. If there are new features in Perl which would be >>> beneficial, >>> >>> I'd >>> >>> > rather make the newer version of Perl a requirement and use the >>> built in >>> >>> > features. Otherwise, we're locked into using an external module, >>> >>> usually >>> >>> > from CPAN, which has no quality assurance. I regret adding some of >>> the >>> >>> > current modules such as InsideOut, rather then writing the code which >>> >>> could >>> >>> > accomplish the same objective. >>> >>> > >>> >>> > Did you need to catch exceptions because one of the OpenStack modules >>> >>> > you're using may throw a die exception? When this happens, the >>> entire >>> >>> vcld >>> >>> > process for the reservation exits abruptly which causes problems >>> such as >>> >>> > the request state remaining in pending indefinitely. This happens >>> even >>> >>> if >>> >>> > you wrap the call in an eval block. I have worked around this >>> problem >>> >>> > before by locally overriding the die handler. Take a look at >>> >>> > vSphere_SDK.pm and find lines such as: >>> >>> > >>> >>> > # Override the die handler >>> >>> > local $SIG{__DIE__} = sub{}; >>> >>> > >>> >>> > The vSphere SDK is one example which may call die (via Carp). If >>> this >>> >>> > happens with die overridden, the eval block returns normally. If an >>> >>> > exception occurred, $EVAL_ERROR should be set. >>> >>> > >>> >>> > >>> >>> > > >>> >>> > > 3. I've saved the best for last, but first some background. The >>> >>> biggest >>> >>> > > change we've made is that we no longer rely on the /etc/hosts file >>> at >>> >>> all >>> >>> > > (though we haven't removed the code that keeps it up to date). We >>> had >>> >>> a >>> >>> > > number of issues with duplicate entries ending up in the hosts file >>> >>> and >>> >>> > > weren't able to pin down the exact cause. This led us to adding a >>> >>> table >>> >>> > to >>> >>> > > the database to track VCL computer to OpenStack instance mappings. >>> The >>> >>> > next >>> >>> > > issue we ran into was the private IP address not being populated in >>> >>> the >>> >>> > > hosts file. This happened very rarely and we're still unsure of the >>> >>> cause >>> >>> > > especially since there's an explicit check that the address is >>> >>> populated >>> >>> > > with a value before we write the hosts file. I'd assume that the >>> Nova >>> >>> API >>> >>> > > occasionally returns a response where the addresses component is >>> >>> > structured >>> >>> > > differently than we expect, though I was never able to directly >>> >>> observe >>> >>> > > this behavior. Since OpenStack automatically creates DNS entries >>> for >>> >>> new >>> >>> > > instances we decided the best course of action was to bypass the >>> hosts >>> >>> > file >>> >>> > > entirely. In the end we modified get_computer_private_ip_address in >>> >>> > > DataStructure.pm to do a DNS lookup rather than looking in the >>> hosts >>> >>> > file. >>> >>> > > >>> >>> > > The question for the community is are changes to core VCL code >>> >>> > acceptable? >>> >>> > > Or should we find a solution that keeps everything contained to the >>> >>> > > provisioning module? If changes to core code are acceptable we'd >>> also >>> >>> > have >>> >>> > > to decide on a method of detecting when to do a DNS lookup. Should >>> it >>> >>> > > depend on the provisioning module in use? A configuration option? >>> >>> > >>> >>> > >>> >>> > I see no problems. Any barriers or difficulty you encounter with the >>> >>> core >>> >>> > code should be addressed by improving the core code. >>> >>> > >>> >>> > Reworking the dependency on /etc/hosts is something I'd like to >>> improve >>> >>> > regardless. People often encounter problems with this if they >>> forget to >>> >>> > add entries to /etc/hosts for all their computers, including the >>> >>> management >>> >>> > node(s). I'd like to not read /etc/hosts on the management node at >>> all >>> >>> and >>> >>> > have the value in the computer table be authoritative. This should >>> >>> > facilitate cases where the provisioning module needs to do something >>> >>> > different regarding private IPs such as if they are dynamically >>> >>> allocated. >>> >>> > The provisioning module would retrieve the IP when the image gets >>> >>> loaded >>> >>> > by querying DNS or some other method and then update the database. >>> The >>> >>> OS >>> >>> > module would then use this value to configure the computer. >>> >>> > >>> >>> > One comment regarding adding tables... great! Add all the tables you >>> >>> think >>> >>> > are beneficial. I have seen some try to work around the current >>> >>> database >>> >>> > schema rather than adding to it. There's no reason for this. If >>> there >>> >>> are >>> >>> > configuration options which apply only to OpenStack, add something >>> like >>> >>> an >>> >>> > openstackconfig table. If the vmprofile table doesn't align, add >>> >>> something >>> >>> > like an openstackprofile table. If there are general options such as >>> >>> how >>> >>> > to handle DNS or private IP resolution, we could add a more general >>> >>> table >>> >>> > or possibly extend the managementnode or computer table. >>> >>> > >>> >>> > Thanks, >>> >>> > Andy >>> >>> > >>> >>> >>> >> >>> >> >>> >>> >>> >>> -- >>> Twitter: @serverascode >>> Blog: serverascode.com >>> > > > > -- > Twitter: @serverascode > Blog: serverascode.com -- Aaron Peeler Program Manager Virtual Computing Lab NC State University All electronic mail messages in connection with State business which are sent to or received by this account are subject to the NC Public Records Law and may be disclosed to third parties.
